Presentation
Enhanced Expert Review: A Method for Identifying Usability Issues and Evaluating Mitigations
DescriptionBackground
AAMI/IEC TIR62366-2:2016 defines an expert review as a process where a usability expert inspects a medical device to identify its strengths and weaknesses. However, there are no recommendations on how to structure an expert review to effectively evaluate the usability of a medical device. Despite the lack of guidelines, IEC 62366-2 states in Annex E.7 that expert reviews can be a cost-saving alternative to conducting usability testing in simulated use conditions. While a single usability specialist siting in a room listing out strengths and weaknesses is less costly than conducting a usability test, it may not be the most effective method for identifying a plurality of usability issues and use-related risks.
Since IEC 62366-2 claims that expert reviews can be used in a variety of different usability activities (e.g., formative and summative evaluations), Healthcare Human Factors (HHF) has developed a method that shares the same purported advantages of an expert review (e.g., time and resource efficiency), while producing more comprehensive results than the lonely usability specialist. The novel method that HHF developed is called Enhanced Expert Review (EER).
Outline of the Method
There is a common misconception that empirical (e.g., usability testing) and analytical (e.g., expert review) usability evaluation methods must be mutually exclusive. With EER, the experts participate in an individual session that comprises of the following three phases:
1. Completion of realistic tasks within representative workflows using a high-fidelity or production-equivalent device
a. A facilitator provides clinically representative prompts to get the expert to conduct realistic tasks with the subject device while thinking aloud.
b. The facilitator records any observed use issues and asks a series of debrief questions to identify the root causes.
c. If the expert omits a task/sub-task or does not execute the task/sub-task as expected, the facilitator will guide them through the expected workflow to identify any use issues.
2. Perception-Cognition-Action (PCA) interview
a. The facilitator asks specific questions about how the end-user might perceive, understand, and execute each task, to identify any additional use-related risks.
3. Direct assessment of mitigations
a. The facilitator describes any known use issues associated with each mitigation under evaluation (i.e., existing, new, or modified mitigations) and asks open-ended questions to directly assess those mitigations.
IEC 62366-2 states that the expert must be a human factors engineering or usability specialist. However, the Food and Drug Administration’s (FDA) Human Factors Guidance (2016) suggests that expert reviews can be conducted by clinical or human factors experts. Thus, the EER method includes both subject matter experts (SMEs) and usability experts to provide a broader range of perspectives when conducting tasks and assessing mitigations. Furthermore, EER recommends that at least two usability experts and at least one SME per intended user group participate in the expert review. This method also calls for experts with varying experience levels with the subject device, from completely naïve to highly experienced.
Applying the Method
HHF has used EER to evaluate critical care patient monitors in both formative and summative evaluations. EER was used in a summative evaluation because the scope was limited to minor changes to the user interface that did not involve any hazard-related use scenarios associated with serious harm and was not used for validation testing for pre-market submissions to the FDA. Although EER does not align with the FDA’s guidance for conducting validation testing, it does align with its guidance for conducting preliminary analyses and evaluations.
EER was successful in identifying a range of usability issues and helped inform the use-related risk management documentation as well as the design of the user interface. EER also provided an effective analysis on the implemented mitigations, most notably identifying deficiencies where further mitigation may be necessary. When compared to usability testing under simulated use conditions EER required less time for data collection and analysis.
Future Research
1. Confirm that EER provides more reliable and comprehensive results than expert reviews with only usability specialists or other standalone analytical methods (e.g., heuristic analysis).
2. Determine the effectiveness of EER in different contexts (e.g., stages in the design cycle, types of medical devices, types of known use issues).
Key Takeaways
1. Expert reviews are poorly defined in AAMI/IEC TIR62366-2:2016, as the standard does not provide any guidelines or recommendations on how to conduct an expert review.
2. EER combines empirical and analytical methods, including (1) a usability test, (2) a PCA interview, and (3) a direct assessment of mitigations.
3. Include SMEs and usability experts with varying levels of subject device experience in expert reviews.
4. HHF has used EER to evaluate medical devices and been successful in identifying a variety of unique usability issues.
5. Further research is needed to determine the effectiveness of EER in different contexts.
AAMI/IEC TIR62366-2:2016 defines an expert review as a process where a usability expert inspects a medical device to identify its strengths and weaknesses. However, there are no recommendations on how to structure an expert review to effectively evaluate the usability of a medical device. Despite the lack of guidelines, IEC 62366-2 states in Annex E.7 that expert reviews can be a cost-saving alternative to conducting usability testing in simulated use conditions. While a single usability specialist siting in a room listing out strengths and weaknesses is less costly than conducting a usability test, it may not be the most effective method for identifying a plurality of usability issues and use-related risks.
Since IEC 62366-2 claims that expert reviews can be used in a variety of different usability activities (e.g., formative and summative evaluations), Healthcare Human Factors (HHF) has developed a method that shares the same purported advantages of an expert review (e.g., time and resource efficiency), while producing more comprehensive results than the lonely usability specialist. The novel method that HHF developed is called Enhanced Expert Review (EER).
Outline of the Method
There is a common misconception that empirical (e.g., usability testing) and analytical (e.g., expert review) usability evaluation methods must be mutually exclusive. With EER, the experts participate in an individual session that comprises of the following three phases:
1. Completion of realistic tasks within representative workflows using a high-fidelity or production-equivalent device
a. A facilitator provides clinically representative prompts to get the expert to conduct realistic tasks with the subject device while thinking aloud.
b. The facilitator records any observed use issues and asks a series of debrief questions to identify the root causes.
c. If the expert omits a task/sub-task or does not execute the task/sub-task as expected, the facilitator will guide them through the expected workflow to identify any use issues.
2. Perception-Cognition-Action (PCA) interview
a. The facilitator asks specific questions about how the end-user might perceive, understand, and execute each task, to identify any additional use-related risks.
3. Direct assessment of mitigations
a. The facilitator describes any known use issues associated with each mitigation under evaluation (i.e., existing, new, or modified mitigations) and asks open-ended questions to directly assess those mitigations.
IEC 62366-2 states that the expert must be a human factors engineering or usability specialist. However, the Food and Drug Administration’s (FDA) Human Factors Guidance (2016) suggests that expert reviews can be conducted by clinical or human factors experts. Thus, the EER method includes both subject matter experts (SMEs) and usability experts to provide a broader range of perspectives when conducting tasks and assessing mitigations. Furthermore, EER recommends that at least two usability experts and at least one SME per intended user group participate in the expert review. This method also calls for experts with varying experience levels with the subject device, from completely naïve to highly experienced.
Applying the Method
HHF has used EER to evaluate critical care patient monitors in both formative and summative evaluations. EER was used in a summative evaluation because the scope was limited to minor changes to the user interface that did not involve any hazard-related use scenarios associated with serious harm and was not used for validation testing for pre-market submissions to the FDA. Although EER does not align with the FDA’s guidance for conducting validation testing, it does align with its guidance for conducting preliminary analyses and evaluations.
EER was successful in identifying a range of usability issues and helped inform the use-related risk management documentation as well as the design of the user interface. EER also provided an effective analysis on the implemented mitigations, most notably identifying deficiencies where further mitigation may be necessary. When compared to usability testing under simulated use conditions EER required less time for data collection and analysis.
Future Research
1. Confirm that EER provides more reliable and comprehensive results than expert reviews with only usability specialists or other standalone analytical methods (e.g., heuristic analysis).
2. Determine the effectiveness of EER in different contexts (e.g., stages in the design cycle, types of medical devices, types of known use issues).
Key Takeaways
1. Expert reviews are poorly defined in AAMI/IEC TIR62366-2:2016, as the standard does not provide any guidelines or recommendations on how to conduct an expert review.
2. EER combines empirical and analytical methods, including (1) a usability test, (2) a PCA interview, and (3) a direct assessment of mitigations.
3. Include SMEs and usability experts with varying levels of subject device experience in expert reviews.
4. HHF has used EER to evaluate medical devices and been successful in identifying a variety of unique usability issues.
5. Further research is needed to determine the effectiveness of EER in different contexts.
Authors
Event Type
Oral Presentations
TimeTuesday, April 19:30am - 10:00am EDT
LocationHarbour A/B
Medical and Drug Delivery Devices (MDD)