Presentation
MDD14 - Ethical Challenges of Using Artificial Intelligence in Usability Testing Practices for Medical Solutions Design
SessionPoster Session 1
DescriptionEarly detection and prevention of patient harm in healthcare is an international policy priority, as it is a leading cause of morbidity and mortality internationally, with one in 20 patients subjected to preventable harm. Human factors engineering (HFE) has become a practice used in the medical design process to successfully reduce preventable medical errors that result in injury and death.
Usability testing throughout all stages of development is critical to verifying that a given design is safe, efficient, and easy to use for diverse user groups, use cases, and use environments. The United States Food and Drug Administration (FDA) emphasizes the importance of HFE throughout the product development lifecycle to minimize errors and adverse events and to gain a deeper understanding of patients and their conditions. Depending on the classification of a medical device, the FDA mandates a pre-market review, including a human factors assessment based on their recognized standards, before the product's launch.
However, user testing is often a bottleneck in the development cycle, as it requires a lot of time and resources to run sessions with enough participants and analyze large amounts of data. Moreover, research has raised concerns about the adequacy of traditional usability testing methodologies and their capacity to accurately represent the diversity of user populations. Molich et al. found variations of usability results of the same product tested between different organizations– questioning the assumption that current usability testing practices are uniform.
This finding raises concerns as to whether current usability testing practices are equitable, inclusive, and accessible for medical device and service development. If a product/service is poorly designed, interventions can become more accessible to, adopted more frequently by, adhered to more closely by, or more effective to socioeconomically advantaged groups with greater access to resources–thereby increasing health disparity between patient groups. This healthcare disparity is often linked to social conditions such as level of education, occupational status, and income; residential segregation; environmental barriers; stigmatization; and discrimination. Healthcare disparities are associated with poor health outcomes and premature death of vulnerable groups, as well as increased healthcare costs. Therefore, inadequate usability testing practices of medical devices and services can exacerbate the health gaps in marginalized communities.
Efforts have been made to improve equitable usability testing practices through expert discussions and equity assessment checklists. Artificial intelligence (AI) also presents an opportunity to revolutionize usability testing by improving efficiency and providing less biased outcomes. Many companies involved in user testing are now integrating AI into their evaluation workflows to streamline the process, including automating the generation of test scripts, analyzing data from testing sessions, and even using AI-driven agents to simulate human behaviour. Despite these promising advancements, there remains a lack of formal research on how AI-supported usability tools account for diverse user groups, particularly marginalized communities, and whether AI improves current testing practices or inadvertently perpetuates existing biases and inequalities.
The primary objective of the proposed presentation is to outline potential opportunities and challenges AI poses on equitable medical solutions development when integrated into usability testing practices. This presentation is based on the results of a narrative literature review, which aimed to evaluate the greater role of AI in usability testing and its impact on automating evaluator tasks and equity. This study reviewed 46 papers and grey literature pertaining to seven AI-supported usability testing tools on the market. This presentation will solely focus on the ethical implications of their conclusions. The AI-supported usability tools included in this review were evaluated on a zero-to-five and zero-to-three scale respectively, to determine their consideration of equity in their intervention. From the review of the 46 articles, 23 studies did not address the inclusion of diverse user groups (Level 0 equity consideration), while 22 studies acknowledged equity considerations without incorporating them into their methodologies (Levels 1-3 equity consideration). Only three studies specifically explored the application of AI-supported usability tools with marginalized communities. From the grey literature of seven AI-informed usability testing platforms, it was difficult to find any explicit discussion on their AI-supported usability tools' impact on the equity, inclusivity, and diversity of usability testing practices. Only one platform incorporated equity considerations in some of its features. From these conclusions, we found that future research efforts should develop standardized and equitable usability testing guidelines for traditional and AI-informed methods. Collaboration among AI, UX, and ethics experts is essential to ensure that research in this area keeps pace with technological advancements, ultimately supporting equitable usability practices. Moreover, AI-supported usability tools should be well-integrated into UX evaluator workflows, and address the concerns and preferences that UX evaluators have upon implementation. Lastly, the ethical implications of on-the-market AI-informed usability tools should be easily accessible to ensure they are used appropriately and consistently generate accurate results. Overall, this presentation aims to initiate discussions with HFE practitioners to evaluate the ethical implications of AI in their practices and, more generally, consider how their traditional practices impact equitable medical solutions development.
Usability testing throughout all stages of development is critical to verifying that a given design is safe, efficient, and easy to use for diverse user groups, use cases, and use environments. The United States Food and Drug Administration (FDA) emphasizes the importance of HFE throughout the product development lifecycle to minimize errors and adverse events and to gain a deeper understanding of patients and their conditions. Depending on the classification of a medical device, the FDA mandates a pre-market review, including a human factors assessment based on their recognized standards, before the product's launch.
However, user testing is often a bottleneck in the development cycle, as it requires a lot of time and resources to run sessions with enough participants and analyze large amounts of data. Moreover, research has raised concerns about the adequacy of traditional usability testing methodologies and their capacity to accurately represent the diversity of user populations. Molich et al. found variations of usability results of the same product tested between different organizations– questioning the assumption that current usability testing practices are uniform.
This finding raises concerns as to whether current usability testing practices are equitable, inclusive, and accessible for medical device and service development. If a product/service is poorly designed, interventions can become more accessible to, adopted more frequently by, adhered to more closely by, or more effective to socioeconomically advantaged groups with greater access to resources–thereby increasing health disparity between patient groups. This healthcare disparity is often linked to social conditions such as level of education, occupational status, and income; residential segregation; environmental barriers; stigmatization; and discrimination. Healthcare disparities are associated with poor health outcomes and premature death of vulnerable groups, as well as increased healthcare costs. Therefore, inadequate usability testing practices of medical devices and services can exacerbate the health gaps in marginalized communities.
Efforts have been made to improve equitable usability testing practices through expert discussions and equity assessment checklists. Artificial intelligence (AI) also presents an opportunity to revolutionize usability testing by improving efficiency and providing less biased outcomes. Many companies involved in user testing are now integrating AI into their evaluation workflows to streamline the process, including automating the generation of test scripts, analyzing data from testing sessions, and even using AI-driven agents to simulate human behaviour. Despite these promising advancements, there remains a lack of formal research on how AI-supported usability tools account for diverse user groups, particularly marginalized communities, and whether AI improves current testing practices or inadvertently perpetuates existing biases and inequalities.
The primary objective of the proposed presentation is to outline potential opportunities and challenges AI poses on equitable medical solutions development when integrated into usability testing practices. This presentation is based on the results of a narrative literature review, which aimed to evaluate the greater role of AI in usability testing and its impact on automating evaluator tasks and equity. This study reviewed 46 papers and grey literature pertaining to seven AI-supported usability testing tools on the market. This presentation will solely focus on the ethical implications of their conclusions. The AI-supported usability tools included in this review were evaluated on a zero-to-five and zero-to-three scale respectively, to determine their consideration of equity in their intervention. From the review of the 46 articles, 23 studies did not address the inclusion of diverse user groups (Level 0 equity consideration), while 22 studies acknowledged equity considerations without incorporating them into their methodologies (Levels 1-3 equity consideration). Only three studies specifically explored the application of AI-supported usability tools with marginalized communities. From the grey literature of seven AI-informed usability testing platforms, it was difficult to find any explicit discussion on their AI-supported usability tools' impact on the equity, inclusivity, and diversity of usability testing practices. Only one platform incorporated equity considerations in some of its features. From these conclusions, we found that future research efforts should develop standardized and equitable usability testing guidelines for traditional and AI-informed methods. Collaboration among AI, UX, and ethics experts is essential to ensure that research in this area keeps pace with technological advancements, ultimately supporting equitable usability practices. Moreover, AI-supported usability tools should be well-integrated into UX evaluator workflows, and address the concerns and preferences that UX evaluators have upon implementation. Lastly, the ethical implications of on-the-market AI-informed usability tools should be easily accessible to ensure they are used appropriately and consistently generate accurate results. Overall, this presentation aims to initiate discussions with HFE practitioners to evaluate the ethical implications of AI in their practices and, more generally, consider how their traditional practices impact equitable medical solutions development.
Authors
Event Type
Poster Presentation
TimeMonday, March 314:45pm - 6:15pm EDT
LocationFrontenac Foyer
Digital Health (DH)
Simulation and Education (SE)
Hospital Environments (HE)
Medical and Drug Delivery Devices (MDD)
Patient Safety and Research Initiatives (PS)