Presentation
MDD28 - The Dilemma of Task Prompt Wording in HF Validation: History, Challenges, and Strategies for Success
SessionPoster Session 1
DescriptionDesigning effective task prompts for usability testing is a critical yet challenging aspect of medical device evaluation. Regulatory guidance and industry best practices emphasize the importance of prompts that reflect real-world scenarios without leading participants toward specific actions. However, the balance between avoiding leading language and maintaining clarity can be difficult to achieve. This presentation will explore the history, challenges, and best practices for task prompt phrasing, highlighting the implications for medical device safety and usability.
Three case studies underscore the importance of this issue.
The first case study involves a Patient-Controlled Analgesia (PCA) pump. The initial prompt, “Please demonstrate how a patient would request a dose using this PCA pump,” was too leading, directly guiding participants to a specific feature. A second version removed the feature name but still told participants what to do. The final version, "Your patient reports pain following surgery. Please show how you would manage their pain using the PCA pump," required participants to make contextual decisions, leading to a more realistic and nuanced assessment of the device's usability.
In the second case study, a usability study was conducted on an autoinjector intended for emergency use, such as during an anaphylactic reaction. The initial prompt, "Demonstrate how to administer the medication using the autoinjector," was rejected for being too leading, as it explicitly instructed participants to use the device. A subsequent version, "Your patient is experiencing a severe allergic reaction and requires immediate intervention. Show us what you would do," resulted in participants hesitating or searching for alternative methods, such as calling for emergency services, before considering the autoinjector. The final refined prompt, "You are presented with a situation where an individual shows signs of a severe allergic reaction. Please proceed as you would in real life to address their condition," provided enough context without directing participants to the device. This improved prompt allowed evaluators to observe natural decision-making processes and identify potential confusion related to device discoverability and use sequence.
The third case study focused on a diagnostic imaging system used for CT scans. The initial task prompt, "Demonstrate how to set up and perform a CT scan for this patient," was too directive, guiding participants toward specific system features and bypassing natural decision-making processes. This prompt failed to reveal challenges users might encounter when navigating the system or interpreting its options.
A second version of the prompt, "You have a patient who presents with abdominal pain, and the physician has ordered a CT scan of the abdomen. Please demonstrate how you would prepare the system and the patient for the scan, and then initiate the imaging process." introduced a more scenario-based approach. However, it left some participants uncertain about the steps needed to position the patient, select the correct imaging protocol, or verify settings, creating inconsistencies in task performance.
The final version provided a more realistic clinical context: "A physician has ordered a CT scan of the abdomen for a patient. Show how you would proceed". This version encouraged participants to think through the task comprehensively, revealing usability issues such as unclear patient positioning guidance, difficulty identifying the appropriate imaging protocol, and challenges with system feedback during the scan initiation process.
The key takeaway is that while non-leading, scenario-based prompts are essential for obtaining authentic user behavior, they can also introduce ambiguity. This often results in unintended actions or test artifacts, complicating the analysis of whether observed issues are related to user error or device design. The message is clear: task prompts must be crafted to strike a balance between realism and clarity. Furthermore, aligning with FDA expectations and discussing the potential for test artifacts early in the process are crucial steps for success.
In addition to strategies for creating effective task prompts, we will address the limitations of this approach. These include difficulties in isolating specific device features, increased ambiguity in participant responses, and the risk of generating test artifacts. The presentation will emphasize the importance of thoroughly documenting and discussing these artifacts in final reports to ensure a comprehensive evaluation.
The overarching takeaway for attendees is that thoughtful task prompt design can make or break a usability study. By understanding and addressing the inherent challenges, human factors professionals can enhance the accuracy and reliability of testing outcomes, leading to better device designs and improved patient safety.
Three case studies underscore the importance of this issue.
The first case study involves a Patient-Controlled Analgesia (PCA) pump. The initial prompt, “Please demonstrate how a patient would request a dose using this PCA pump,” was too leading, directly guiding participants to a specific feature. A second version removed the feature name but still told participants what to do. The final version, "Your patient reports pain following surgery. Please show how you would manage their pain using the PCA pump," required participants to make contextual decisions, leading to a more realistic and nuanced assessment of the device's usability.
In the second case study, a usability study was conducted on an autoinjector intended for emergency use, such as during an anaphylactic reaction. The initial prompt, "Demonstrate how to administer the medication using the autoinjector," was rejected for being too leading, as it explicitly instructed participants to use the device. A subsequent version, "Your patient is experiencing a severe allergic reaction and requires immediate intervention. Show us what you would do," resulted in participants hesitating or searching for alternative methods, such as calling for emergency services, before considering the autoinjector. The final refined prompt, "You are presented with a situation where an individual shows signs of a severe allergic reaction. Please proceed as you would in real life to address their condition," provided enough context without directing participants to the device. This improved prompt allowed evaluators to observe natural decision-making processes and identify potential confusion related to device discoverability and use sequence.
The third case study focused on a diagnostic imaging system used for CT scans. The initial task prompt, "Demonstrate how to set up and perform a CT scan for this patient," was too directive, guiding participants toward specific system features and bypassing natural decision-making processes. This prompt failed to reveal challenges users might encounter when navigating the system or interpreting its options.
A second version of the prompt, "You have a patient who presents with abdominal pain, and the physician has ordered a CT scan of the abdomen. Please demonstrate how you would prepare the system and the patient for the scan, and then initiate the imaging process." introduced a more scenario-based approach. However, it left some participants uncertain about the steps needed to position the patient, select the correct imaging protocol, or verify settings, creating inconsistencies in task performance.
The final version provided a more realistic clinical context: "A physician has ordered a CT scan of the abdomen for a patient. Show how you would proceed". This version encouraged participants to think through the task comprehensively, revealing usability issues such as unclear patient positioning guidance, difficulty identifying the appropriate imaging protocol, and challenges with system feedback during the scan initiation process.
The key takeaway is that while non-leading, scenario-based prompts are essential for obtaining authentic user behavior, they can also introduce ambiguity. This often results in unintended actions or test artifacts, complicating the analysis of whether observed issues are related to user error or device design. The message is clear: task prompts must be crafted to strike a balance between realism and clarity. Furthermore, aligning with FDA expectations and discussing the potential for test artifacts early in the process are crucial steps for success.
In addition to strategies for creating effective task prompts, we will address the limitations of this approach. These include difficulties in isolating specific device features, increased ambiguity in participant responses, and the risk of generating test artifacts. The presentation will emphasize the importance of thoroughly documenting and discussing these artifacts in final reports to ensure a comprehensive evaluation.
The overarching takeaway for attendees is that thoughtful task prompt design can make or break a usability study. By understanding and addressing the inherent challenges, human factors professionals can enhance the accuracy and reliability of testing outcomes, leading to better device designs and improved patient safety.
Event Type
Poster Presentation
TimeMonday, March 314:45pm - 6:15pm EDT
LocationFrontenac Foyer
Digital Health (DH)
Simulation and Education (SE)
Hospital Environments (HE)
Medical and Drug Delivery Devices (MDD)
Patient Safety and Research Initiatives (PS)