Presentation
MDD13 - Equity Implications of Regulation on Usability Evaluations of Medical Devices
SessionPoster Session 1
DescriptionMedical device regulation has played a significant role in improving the safety and effectiveness of medical devices through mandating the rigorous application of human factors and usability engineering. While the application of usability engineering to medical device design emphasizes the need to understand the users, uses, and contexts, in practice, what is foregrounded in usability evaluations may be determined by existing societal inequity. Yet, there is still little discussion of equity in this application of usability engineering in the design of medical devices. At the same time, the FDA is committed to advancing health equity in medical devices: one of the strategic priorities of the CDRH is to “bridge the health technology divide.” Our aim in this presentation is to call attention to the role of regulation in (dis)incentivizing equitable usability evaluations and how we might continue to advance equity in usability evaluations of medical devices.
This presentation draws on a subset of data collected from four workshops. The goal of these workshops was to conduct a failure modes and effects analysis (FMEA) of various usability evaluation methods to identify ways in which the evaluation may reproduce societal structures that work to oppress. Each workshop was conducted over two hours and included a total of 13 human factors practitioners and designers at a consultancy in Toronto, Canada. Practitioners at this consultancy have expertise in conducting usability evaluations of regulated medical devices for clients globally, having extensive knowledge of FDA human factors guidance and international standards for usability engineering (e.g. ISO 62366). Data was collected during the workshops by recording notes in a FMEA worksheet and audio was recorded and transcribed. The FMEA worksheet was further refined by inductively coding for failure modes, potential effects, and potential causes. We focus here on findings that highlight some of the potential equity implications of regulation on the practice of usability testing of medical devices.
Seven potential failure modes were identified for which regulation was attributed as a potential cause: (1) The realities of developing medical devices requires developers to obtain pre-market regulatory approval. The goals of usability evaluation in this context may be aligned with the priorities participants insofar as they align with the priorities of the FDA; (2) The characteristics that the FDA is interested in may influence the characteristics that are recruited for in the evaluation; (3) Regulatory bodies may only require testing participants who speak the dominant language; (4) Manufacturers may be incentivized to limit the number of distinct user groups to test to reduce total number of participants; (5) Manufacturers may choose sample sizes of 15 by default because of regulatory guidance; (6) Participants may be selected based on who might be a “good” participant, as manufacturers may be incentivized not to find problems to report in regulatory submissions; (7) Manufacturers may only report demographics that the regulatory body is interested in, potentially precluding reporting demographics like race.
Based on these failure modes, we summarize two ways in which regulation may produce unintended consequences. Regulation may act as a disincentive to find usability problems when intending to submit validation study results. In this mode, concerns about obtaining regulatory approval may be prioritized over ensuring usability for a diversity of user groups. This may be especially true if regulation emphasizes validation studies over formative studies. Secondly, regulation may act as a standard to meet, with little incentive to exceed. In this mode, manufacturers may deem that, by meeting the requirements for regulatory approval, they have achieved sufficient usability across all the users, uses, and contexts relevant to their medical device. We propose that to achieve equity, manufacturers will likely need to exceed the standard.
As key takeaways we offer two questions as provocations for practitioners and regulators. (1) As a practitioner, am I designing my usability studies to uncover usability problems for the most marginalized of my intended users? (2) As a regulator, how can regulatory strategy be reoriented towards advancing equity in medical device design?
This presentation draws on a subset of data collected from four workshops. The goal of these workshops was to conduct a failure modes and effects analysis (FMEA) of various usability evaluation methods to identify ways in which the evaluation may reproduce societal structures that work to oppress. Each workshop was conducted over two hours and included a total of 13 human factors practitioners and designers at a consultancy in Toronto, Canada. Practitioners at this consultancy have expertise in conducting usability evaluations of regulated medical devices for clients globally, having extensive knowledge of FDA human factors guidance and international standards for usability engineering (e.g. ISO 62366). Data was collected during the workshops by recording notes in a FMEA worksheet and audio was recorded and transcribed. The FMEA worksheet was further refined by inductively coding for failure modes, potential effects, and potential causes. We focus here on findings that highlight some of the potential equity implications of regulation on the practice of usability testing of medical devices.
Seven potential failure modes were identified for which regulation was attributed as a potential cause: (1) The realities of developing medical devices requires developers to obtain pre-market regulatory approval. The goals of usability evaluation in this context may be aligned with the priorities participants insofar as they align with the priorities of the FDA; (2) The characteristics that the FDA is interested in may influence the characteristics that are recruited for in the evaluation; (3) Regulatory bodies may only require testing participants who speak the dominant language; (4) Manufacturers may be incentivized to limit the number of distinct user groups to test to reduce total number of participants; (5) Manufacturers may choose sample sizes of 15 by default because of regulatory guidance; (6) Participants may be selected based on who might be a “good” participant, as manufacturers may be incentivized not to find problems to report in regulatory submissions; (7) Manufacturers may only report demographics that the regulatory body is interested in, potentially precluding reporting demographics like race.
Based on these failure modes, we summarize two ways in which regulation may produce unintended consequences. Regulation may act as a disincentive to find usability problems when intending to submit validation study results. In this mode, concerns about obtaining regulatory approval may be prioritized over ensuring usability for a diversity of user groups. This may be especially true if regulation emphasizes validation studies over formative studies. Secondly, regulation may act as a standard to meet, with little incentive to exceed. In this mode, manufacturers may deem that, by meeting the requirements for regulatory approval, they have achieved sufficient usability across all the users, uses, and contexts relevant to their medical device. We propose that to achieve equity, manufacturers will likely need to exceed the standard.
As key takeaways we offer two questions as provocations for practitioners and regulators. (1) As a practitioner, am I designing my usability studies to uncover usability problems for the most marginalized of my intended users? (2) As a regulator, how can regulatory strategy be reoriented towards advancing equity in medical device design?
Event Type
Poster Presentation
TimeMonday, March 314:45pm - 6:15pm EDT
LocationFrontenac Foyer
Digital Health (DH)
Simulation and Education (SE)
Hospital Environments (HE)
Medical and Drug Delivery Devices (MDD)
Patient Safety and Research Initiatives (PS)