Presentation
MDD11 - Emerging Technologies in Usability Evaluations: Exploring Advancements in Data Collection Methods and Their Impact
SessionPoster Session 1
DescriptionTraditionally, Human Factors (HF) study methods applied to medical device development have relied largely on observed objective data – ‘did they perform the step?’ and subjective data expressed by the participant – ‘how did you find performing that step?’. As Human Factors settles into its traditional methods and approaches, it is a great time to start experimenting with additional opportunities to collect reliable and insightful data.
A project has been executed to first assess different technologies that support traditional HF methods, and then use some of the emerging technologies to conduct a HF usability evaluation. The project aimed to advance our knowledge of alternative ways to collect objective and quantitative data to compliment the traditional HF interviews and observation data, which can now be transferred to industry to advance our knowledge as a community.
In this instance, facial expression, heart rate, sweat production, and eye movement data were collected in conjunction with the traditional approaches. In with the old and in with the new!
The evaluation used high-fidelity physical and digital prototypes and labelling to support simulated use scenarios and reasonable knowledge-based assessments. The study design incorporated traditional structure and data collection methods alongside the use of new technology and quantitative data collection methods.
This presentation will tell the story of one experience investigating opportunities presented with new technologies, where data was collected using eye tracking, electrocardiogram (for heart rate), electrodermal activity sensors (‘EDA’) and facial expression analysis software, (‘FEA’) applied to a traditional formative study for an injection device.
These technologies were chosen based on research conducted as a preliminary stage to the study-phase of the project, however it was also taken into account how widespread these technologies were in other applications such as consumer research and psychology.
The project aimed to utilize several new technologies to determine whether these technologies complimented the current human factors study data collection methods. As the project was exploring the methods, as well as reporting on the test data there was an analysis and report on the technologies themselves. The project explored several hypotheses:
• Study logistics hypotheses
o Impact on participant’s comfort and performance:
The addition of sensors will have no negative impact on participant’s comfort
The addition of sensors will have no negative impact on the participant’s ability to perform the tasks in a natural way
The use of the EDA will have no impact on the task ‘wash hands’
o Impact on time required for device setup and analysis:
Setting up the research equipment and sensors will account for no more than 17% of the session length
Post-session analysis of the data will take no more than 2 hours per participant
• Study outcome hypotheses
o Performance data scoring: The additional data collected will assist in identifying use difficulties
o Participant’s preference: The additional data will give quantifiable preference data that aligns with participant verbal preference data
o Quantifiable data: The additional data collected will give quantifiable data in alignment with participant performance, verbal response and subjective feedback
Extensive analysis of the data was conducted, to determine conclusions per hypothesis and look toward possible use of these technologies in future HF research. There was a mix of outcomes for hypotheses, some proven false in this instance and some proven to be true. For example, the addition of sensors to the session had no negative impact on participants’ comfort or ability to perform tasks in a natural way during the session, except the ‘wash hands’ task. It is inconclusive whether the additional data collected assisted in identifying use issues during post-session analysis which participants may not have verbally announced during the session. In some tasks, the additional data collected gave quantifiable data that were in alignment with the performance of participants, their verbal response and subjective feedback.
Learnings from this research could support the opportunity for greater robustness of data collection in a future attempt to investigate these hypotheses (or others).
A project has been executed to first assess different technologies that support traditional HF methods, and then use some of the emerging technologies to conduct a HF usability evaluation. The project aimed to advance our knowledge of alternative ways to collect objective and quantitative data to compliment the traditional HF interviews and observation data, which can now be transferred to industry to advance our knowledge as a community.
In this instance, facial expression, heart rate, sweat production, and eye movement data were collected in conjunction with the traditional approaches. In with the old and in with the new!
The evaluation used high-fidelity physical and digital prototypes and labelling to support simulated use scenarios and reasonable knowledge-based assessments. The study design incorporated traditional structure and data collection methods alongside the use of new technology and quantitative data collection methods.
This presentation will tell the story of one experience investigating opportunities presented with new technologies, where data was collected using eye tracking, electrocardiogram (for heart rate), electrodermal activity sensors (‘EDA’) and facial expression analysis software, (‘FEA’) applied to a traditional formative study for an injection device.
These technologies were chosen based on research conducted as a preliminary stage to the study-phase of the project, however it was also taken into account how widespread these technologies were in other applications such as consumer research and psychology.
The project aimed to utilize several new technologies to determine whether these technologies complimented the current human factors study data collection methods. As the project was exploring the methods, as well as reporting on the test data there was an analysis and report on the technologies themselves. The project explored several hypotheses:
• Study logistics hypotheses
o Impact on participant’s comfort and performance:
The addition of sensors will have no negative impact on participant’s comfort
The addition of sensors will have no negative impact on the participant’s ability to perform the tasks in a natural way
The use of the EDA will have no impact on the task ‘wash hands’
o Impact on time required for device setup and analysis:
Setting up the research equipment and sensors will account for no more than 17% of the session length
Post-session analysis of the data will take no more than 2 hours per participant
• Study outcome hypotheses
o Performance data scoring: The additional data collected will assist in identifying use difficulties
o Participant’s preference: The additional data will give quantifiable preference data that aligns with participant verbal preference data
o Quantifiable data: The additional data collected will give quantifiable data in alignment with participant performance, verbal response and subjective feedback
Extensive analysis of the data was conducted, to determine conclusions per hypothesis and look toward possible use of these technologies in future HF research. There was a mix of outcomes for hypotheses, some proven false in this instance and some proven to be true. For example, the addition of sensors to the session had no negative impact on participants’ comfort or ability to perform tasks in a natural way during the session, except the ‘wash hands’ task. It is inconclusive whether the additional data collected assisted in identifying use issues during post-session analysis which participants may not have verbally announced during the session. In some tasks, the additional data collected gave quantifiable data that were in alignment with the performance of participants, their verbal response and subjective feedback.
Learnings from this research could support the opportunity for greater robustness of data collection in a future attempt to investigate these hypotheses (or others).
Event Type
Poster Presentation
TimeMonday, March 314:45pm - 6:15pm EDT
LocationFrontenac Foyer
Digital Health (DH)
Simulation and Education (SE)
Hospital Environments (HE)
Medical and Drug Delivery Devices (MDD)
Patient Safety and Research Initiatives (PS)