Presentation
How Many Users Do We Really Need? Analyzing the Impact of Key Characteristics on Usability Test Performance and Sample Size Selection of HF Validation for Combination Products
DescriptionSample sizes for human factors (HF) validation studies of combination products has been a major pain point in the medical device human factors industry over the past several years. Specifically, sample sizes have grown from 15 participants per user group (e.g., adult patients) to 15 participants per sub-group within a user group (e.g., trained injection-naïve adult patients, untrained injection-naïve adult patients, trained injection-experienced adult patients, untrained injection-experienced adult patients; a total of 60 participants per user group). These large sample sizes increase the burden of HF validation testing, even when many combination products are platform devices with well understood use-risk profiles.
Furthermore, the rationale and benefit of increased sample sizes stratified by various characteristics (e.g., training, experience), and the difference in performance between these sub-groups, have not been deeply examined or understood. Additionally, the reference cited in FDA’s 2016 Human Factors Guidance regarding including 15 participants per user group (Faulkner, 2003) is based on a study of telecommunications software and is not specific to the stimuli examined in HF validation studies of combination products.
Our objective is to analyze data from recent HF validation studies of a variety of injection products with the following goals:
• understanding how the variables of experience and training impact performance
• identifying when sample sizes reach a point of diminishing returns; in other words, does including 60 participants per user group allow us to gain more robust insights, or do unique findings plateau at a smaller sample size?
To perform this analysis, we started by conducting a literature review on sample size estimation to understand previous work, including the work by Faulkner cited within FDA guidance. We then shifted to standardize and process unstructured study performance data collected on five recent HF validation studies of injection devices to enable performance trend comparisons across the studies, a dataset that included over 470 participants and 1500+ use errors. From this, we developed a performance model to understand the impact of the variables of training and experience on performance and use error distribution. This fed into final sample size estimation models, which explored both Frequentist and Bayesian statistical frameworks for informing user group sample sizes estimation and sample size estimation tool development.
Our work has found that certain variables do have a significant impact on performance and that different approaches to sample sizes estimation consistently uncover most use errors found with increased sample sizes.
Furthermore, the rationale and benefit of increased sample sizes stratified by various characteristics (e.g., training, experience), and the difference in performance between these sub-groups, have not been deeply examined or understood. Additionally, the reference cited in FDA’s 2016 Human Factors Guidance regarding including 15 participants per user group (Faulkner, 2003) is based on a study of telecommunications software and is not specific to the stimuli examined in HF validation studies of combination products.
Our objective is to analyze data from recent HF validation studies of a variety of injection products with the following goals:
• understanding how the variables of experience and training impact performance
• identifying when sample sizes reach a point of diminishing returns; in other words, does including 60 participants per user group allow us to gain more robust insights, or do unique findings plateau at a smaller sample size?
To perform this analysis, we started by conducting a literature review on sample size estimation to understand previous work, including the work by Faulkner cited within FDA guidance. We then shifted to standardize and process unstructured study performance data collected on five recent HF validation studies of injection devices to enable performance trend comparisons across the studies, a dataset that included over 470 participants and 1500+ use errors. From this, we developed a performance model to understand the impact of the variables of training and experience on performance and use error distribution. This fed into final sample size estimation models, which explored both Frequentist and Bayesian statistical frameworks for informing user group sample sizes estimation and sample size estimation tool development.
Our work has found that certain variables do have a significant impact on performance and that different approaches to sample sizes estimation consistently uncover most use errors found with increased sample sizes.
Event Type
Oral Presentations
TimeMonday, March 3111:00am - 11:30am EDT
LocationHarbour A/B
Medical and Drug Delivery Devices (MDD)