Presentation
MDD9 - Developing a Heuristic Checklist for the Evaluation of Rapid Self-Diagnostic Test Kits
SessionPoster Session 1
DescriptionThe focus of this research is to understand the unique human factors design considerations for rapid self-diagnostic test kits (RSDT kits). Human factors practitioners are involved in the summative testing phases of medical devices and kits of this nature. However, in this research we seek to target a method in which diagnostic kits can be evaluated at the earliest stages of development to identify usability issues and human factors considerations in an inexpensive manner without the need to test with users, which is often less feasible during early stages of design. A heuristic checklist—developed specifically for use for self-diagnostic test kits—can provide practitioners with a quick and inexpensive assessment tool to evaluate the usability of RDST kits and detect possible usability issues during early-stage development and throughout the iterative design cycle.
At-home medical self-testing kits have been used to detect a vast array of conditions, such as infectious diseases, allergies, deficiencies, pregnancy, substance abuse, and other health issues (Roma, 2022). These kits require taking a small sample of a bodily fluid—primarily blood, urine, saliva, or nasal secretions (MedlinePlus, 2022). While home-based self-testing is not a novel concept, recent years have seen the proliferation of at-home medical self-testing. In fact, the global home diagnostics market is projected to expand at a compound annual growth rate of 4.4% from 2021 to 2031, with a projected market value of $8.2 billion by the end of 2031 (Transparency Market Research, 2021). Self-testing promises a number of benefits, including privacy and convenience, affordability, patient autonomy, earlier diagnosis and fast turn-around times, and simplicity and ease of use.
There is an inherent value in taking a closer look at the category of self-testing as to determine unique design elements and usability considerations. For example, during the Covid-19 pandemic, the FDA guidelines included usability recommendations for RDSTs which included the need for external controls, easy-to-follow reference instructions with reading comprehension below a seventh-grade level, and simple smart phone applications (Drain et al., 2023). Despite these recommendations, Drain et al. (2023) noted that the rapid Emergency Use Authorization (EUA) process did not consider how products were labeled, designed, and utilized in the field under various operating conditions, and that, moreover, the limited assessments of usability factors for self-testing have impaired the utility of RDSTs overall. It is reasonable to conclude that self-test kits involve specialized usability considerations. For example, the level of training and education is likely different for the average user of a RDST compared to users of traditional medical devices. For traditional lab tests there is an assumption that the lab technicians have a certain level of knowledge of lab-related terminology and techniques, but the skill and knowledge base of RDST kits users is likely to be far more varied (Farmer et al., 2022).
The development of a heuristic checklist is an appropriate method to investigate these unique considerations, which can in turn be utilized as an evaluation tool. This method requires that a small set of evaluators examine the interface and judge its compliance with recognized usability principles (the “heuristics”), and assess the severity of any violations (Nielsen, 1992 & Zhang et al., 2003). Heuristic evaluation is not meant to completely replace user testing, as each method provides its own value. Traditional heuristics such as Nielsen’s 10 and Shneiderman’s Golden Eight are often cited because they can apply generally to a broad range of products and interfaces. However, these more general heuristics may be less effective for assessing unique features of certain domains, which in turn neglects the evaluation of important design elements and features (Quiñones et al., 2018).
As a result, a number of heuristics (and checklists) have been outlined for a wide array of domains because the classic heuristics were not sufficient to effectively evaluate the domain in question (Hermawati & Lawson, 2016). Prominent categories within the health care domain that have attracted the use and/or development of heuristics include medical devices, health information systems (Yeratziotis, Pottas, & Van Greunen, 2012; Carvalho, Borycki, & Kushniruk, 2009), clinical decision support (Miller et al., 2018; Taft et al., 2016; Leverenz, 2019), and digital health (Nasr et al., 2023; Aabel & Abeywarna, 2018; Reolon et al., 2016; Aldekhyyel et al., 2021; Baumel & Muench, 2016; Galavi & Khajouei, 2023; Khowaja & Al-Thani, 2020). The most prevalent set of usability heuristics in the healthcare domain are Zhang et al.’s (2003) usability heuristics to evaluate patient safety of medical devices. Zhang and his colleagues formalized a set of fourteen heuristics with semantic tags, names, and descriptions for each, which were based Nielsen’s 10 heuristics, Shneiderman’s eight golden rules, and their own considerations.
Despite the proliferation of heuristic sets and checklists designed for the health care domain, there has been a noticeable lack of heuristics designed for or applied to self-diagnostic test kits. However, Nielsen’s 10 heuristics and even Zhang’s heuristics for medical devices are insufficient to effectively assess self-diagnostic test kit usability and are susceptible to failure to capture design considerations associated with various elements such as unboxing and set up, instructions, physical comfort, hygiene, temporal demands, proper disposal of materials, and accessibility.
This research to develop and validate a heuristic checklist for RSDT kits follows Quiñones et al.’s (2018) formalized methodology. This eight-stage methodology includes reviewing the literature for the domain, identifying common usability issues within the domain, selecting and matching features to heuristic items, refinement, and validation through expert judgments, heuristic evaluation, and/or user testing.
This presentation will further discuss the literature review, including the published usability studies on RSDT kits and the associated use errors identified by researchers. It will also outline the heuristic sets that informed the development of this new checklist, such as Zhang’s Heuristics for Medical Device Evaluation, Ravden and Johnson’s (1989) HCI Checklist Heuristics, and others. The presentation will also cover the task analyses conducted across five kit types meant to reasonably represent the market, including a covid-19 test, HIV test, blood typing test, glucometer, and cholesterol meter. The task analyses identified common themes and use phases, including set up, sample collection, sample processing, and reading and interpretation of results. Data collection to formalize the checklist is still ongoing, though we expect to formally specify approximately ten to sixteen heuristics with an estimated 90-150 checklist items. This new checklist for RSDT kits is expected to identify more usability issues and domain specific issues than Zhang’s heuristics for medical device evaluation and will provide more granular opportunities to detect potential usability issues.
At-home medical self-testing kits have been used to detect a vast array of conditions, such as infectious diseases, allergies, deficiencies, pregnancy, substance abuse, and other health issues (Roma, 2022). These kits require taking a small sample of a bodily fluid—primarily blood, urine, saliva, or nasal secretions (MedlinePlus, 2022). While home-based self-testing is not a novel concept, recent years have seen the proliferation of at-home medical self-testing. In fact, the global home diagnostics market is projected to expand at a compound annual growth rate of 4.4% from 2021 to 2031, with a projected market value of $8.2 billion by the end of 2031 (Transparency Market Research, 2021). Self-testing promises a number of benefits, including privacy and convenience, affordability, patient autonomy, earlier diagnosis and fast turn-around times, and simplicity and ease of use.
There is an inherent value in taking a closer look at the category of self-testing as to determine unique design elements and usability considerations. For example, during the Covid-19 pandemic, the FDA guidelines included usability recommendations for RDSTs which included the need for external controls, easy-to-follow reference instructions with reading comprehension below a seventh-grade level, and simple smart phone applications (Drain et al., 2023). Despite these recommendations, Drain et al. (2023) noted that the rapid Emergency Use Authorization (EUA) process did not consider how products were labeled, designed, and utilized in the field under various operating conditions, and that, moreover, the limited assessments of usability factors for self-testing have impaired the utility of RDSTs overall. It is reasonable to conclude that self-test kits involve specialized usability considerations. For example, the level of training and education is likely different for the average user of a RDST compared to users of traditional medical devices. For traditional lab tests there is an assumption that the lab technicians have a certain level of knowledge of lab-related terminology and techniques, but the skill and knowledge base of RDST kits users is likely to be far more varied (Farmer et al., 2022).
The development of a heuristic checklist is an appropriate method to investigate these unique considerations, which can in turn be utilized as an evaluation tool. This method requires that a small set of evaluators examine the interface and judge its compliance with recognized usability principles (the “heuristics”), and assess the severity of any violations (Nielsen, 1992 & Zhang et al., 2003). Heuristic evaluation is not meant to completely replace user testing, as each method provides its own value. Traditional heuristics such as Nielsen’s 10 and Shneiderman’s Golden Eight are often cited because they can apply generally to a broad range of products and interfaces. However, these more general heuristics may be less effective for assessing unique features of certain domains, which in turn neglects the evaluation of important design elements and features (Quiñones et al., 2018).
As a result, a number of heuristics (and checklists) have been outlined for a wide array of domains because the classic heuristics were not sufficient to effectively evaluate the domain in question (Hermawati & Lawson, 2016). Prominent categories within the health care domain that have attracted the use and/or development of heuristics include medical devices, health information systems (Yeratziotis, Pottas, & Van Greunen, 2012; Carvalho, Borycki, & Kushniruk, 2009), clinical decision support (Miller et al., 2018; Taft et al., 2016; Leverenz, 2019), and digital health (Nasr et al., 2023; Aabel & Abeywarna, 2018; Reolon et al., 2016; Aldekhyyel et al., 2021; Baumel & Muench, 2016; Galavi & Khajouei, 2023; Khowaja & Al-Thani, 2020). The most prevalent set of usability heuristics in the healthcare domain are Zhang et al.’s (2003) usability heuristics to evaluate patient safety of medical devices. Zhang and his colleagues formalized a set of fourteen heuristics with semantic tags, names, and descriptions for each, which were based Nielsen’s 10 heuristics, Shneiderman’s eight golden rules, and their own considerations.
Despite the proliferation of heuristic sets and checklists designed for the health care domain, there has been a noticeable lack of heuristics designed for or applied to self-diagnostic test kits. However, Nielsen’s 10 heuristics and even Zhang’s heuristics for medical devices are insufficient to effectively assess self-diagnostic test kit usability and are susceptible to failure to capture design considerations associated with various elements such as unboxing and set up, instructions, physical comfort, hygiene, temporal demands, proper disposal of materials, and accessibility.
This research to develop and validate a heuristic checklist for RSDT kits follows Quiñones et al.’s (2018) formalized methodology. This eight-stage methodology includes reviewing the literature for the domain, identifying common usability issues within the domain, selecting and matching features to heuristic items, refinement, and validation through expert judgments, heuristic evaluation, and/or user testing.
This presentation will further discuss the literature review, including the published usability studies on RSDT kits and the associated use errors identified by researchers. It will also outline the heuristic sets that informed the development of this new checklist, such as Zhang’s Heuristics for Medical Device Evaluation, Ravden and Johnson’s (1989) HCI Checklist Heuristics, and others. The presentation will also cover the task analyses conducted across five kit types meant to reasonably represent the market, including a covid-19 test, HIV test, blood typing test, glucometer, and cholesterol meter. The task analyses identified common themes and use phases, including set up, sample collection, sample processing, and reading and interpretation of results. Data collection to formalize the checklist is still ongoing, though we expect to formally specify approximately ten to sixteen heuristics with an estimated 90-150 checklist items. This new checklist for RSDT kits is expected to identify more usability issues and domain specific issues than Zhang’s heuristics for medical device evaluation and will provide more granular opportunities to detect potential usability issues.
Event Type
Poster Presentation
TimeMonday, March 314:45pm - 6:15pm EDT
LocationFrontenac Foyer
Digital Health (DH)
Simulation and Education (SE)
Hospital Environments (HE)
Medical and Drug Delivery Devices (MDD)
Patient Safety and Research Initiatives (PS)