Presentation
MDD4 - “This Would Never Happen in My Hospital” - The Importance of Adjusting Your Simulated Use Environment Based on Participant Feedback
SessionPoster Session 1
DescriptionIn this presentation, we will discuss the importance of reducing study artifacts during validation studies by utilizing participant feedback and performance-related insights gleaned during formative studies. While it is crucial during formative studies to refine a product and its user interface to improve usability and product safety, researchers should also take these opportunities to critically assess a study’s simulated use environment. The insights gained about a study’s environment can have a considerable impact on the success of later validation studies by identifying potential study artifacts. Study artifacts are a significant factor in a successful validation study that all researchers must consider, however, they can fall to the wayside during formative studies. Human factors practitioners can aid in eliminating study artifacts in a validation study by continually refining the simulated use environment based on participant feedback in early rounds of research. If a participant’s root cause for a use issue was due to a study artifact, for example, “this is not real life” or “not using the product like this in my workplace,” this is the first indication that the simulated environment’s realism can be improved to appropriately assess the product in validation.
Albeit often smaller sample sizes, early formative studies can reveal important indications of study artifact, which may be amplified by the higher sample sizes required for validation studies. We will supplement this presentation with a case study of a recent collaboration between the contributors of this poster describing the adjustments made to the simulated use environment between studies, and the impact they had on effectively decreasing the incidence of study artifacts. This case study will then lead to a framework of considerations that can be employed as a research program approaches validation testing.
Key discussion points from the case study, and elements of the framework include:
• Scenario realism
o Create scenarios that will assess the necessary critical tasks but are also realistic to what the participant may encounter or experience
• Room positioning
o Home setups (living room, sink, phone etc.)
o Hospital setups (positioning of one or more patient beds with bedside monitors and tables for bedside prep)
• Ambiance
o Hospital/clinic (patient monitor vitals, participants and moderator standing and moving around the bedside as if working on a unit, moderators wearing scrubs)
• Assessing alarm states
o Deciding whether it’s appropriate to assess in simulated use or KTAs by understanding if it’s possible to covertly trigger alarm states
o Positioning of the stimuli in the simulated use environment
• Technical setup
o Considerations if pilots and fieldwork are not taking place in the same facility.
Ultimately, making study design and simulated use setup considerations to decrease the incidence of study artifacts increases the validation study’s acceptability during an FDA submission by producing more complete and accurate usability data.
Albeit often smaller sample sizes, early formative studies can reveal important indications of study artifact, which may be amplified by the higher sample sizes required for validation studies. We will supplement this presentation with a case study of a recent collaboration between the contributors of this poster describing the adjustments made to the simulated use environment between studies, and the impact they had on effectively decreasing the incidence of study artifacts. This case study will then lead to a framework of considerations that can be employed as a research program approaches validation testing.
Key discussion points from the case study, and elements of the framework include:
• Scenario realism
o Create scenarios that will assess the necessary critical tasks but are also realistic to what the participant may encounter or experience
• Room positioning
o Home setups (living room, sink, phone etc.)
o Hospital setups (positioning of one or more patient beds with bedside monitors and tables for bedside prep)
• Ambiance
o Hospital/clinic (patient monitor vitals, participants and moderator standing and moving around the bedside as if working on a unit, moderators wearing scrubs)
• Assessing alarm states
o Deciding whether it’s appropriate to assess in simulated use or KTAs by understanding if it’s possible to covertly trigger alarm states
o Positioning of the stimuli in the simulated use environment
• Technical setup
o Considerations if pilots and fieldwork are not taking place in the same facility.
Ultimately, making study design and simulated use setup considerations to decrease the incidence of study artifacts increases the validation study’s acceptability during an FDA submission by producing more complete and accurate usability data.
Event Type
Poster Presentation
TimeMonday, March 314:45pm - 6:15pm EDT
LocationFrontenac Foyer
Digital Health (DH)
Simulation and Education (SE)
Hospital Environments (HE)
Medical and Drug Delivery Devices (MDD)
Patient Safety and Research Initiatives (PS)