Presentation
Iterative Insights: A Mixed-Design Strategy for Comprehensive Usability Testing in Health Technologies
DescriptionEffective health communication systems are essential for promoting health literacy and improving patient outcomes. The rapid evolution of health technologies necessitates rigorous testing methodologies to evaluate their effectiveness. However, have you ever embarked on a research journey only to find yourself overwhelmed by an array of stimuli or (health) technology options, making it nearly impossible to draw meaningful comparisons between subjects? This common dilemma can obscure the path to valuable usability testing insights, leaving researchers grappling with an unwieldy dataset that lacks clarity and potentially also hides alternative options and stimuli. To navigate this challenge, it is essential to implement a systematic approach to narrowing usability testing research choices. A focused and systematic selection during exploratory phase process not only enhances the validity of our empirical comparisons but also streamlines the research, allowing for a more nuanced understanding of user preferences and performance though a more manageable process. This study proposes a two-phase methodology for assessing health technologies to serve as communication systems, utilizing both within-subjects and between-subjects designs to optimize insights while addressing resource constraints typical in feasibility studies.
In the first phase, a larger online study employs a within-subjects approach, in which participants are exposed to a diverse range of stimuli presented in randomized and counterbalanced order to attenuate the worry about the order of exposure affecting reference and the slew of confounds when looking at the aggregate data. This design presents several notable benefits. Firstly, within-subjects designs reduce participant variability, as each participant serves as their own control. It enables researchers to account for individual differences in responses, thus isolating the effects of specific stimuli more effectively while internal validity is not dependent on random assignment. Additionally, from a theoretical perspective (e.g., Dual Processing Theories, Bounded Rationality, Nudge Theory), it is often more insightful in usability testing to envision a health technology user responding to two competing products rather than considering two users in different markets making choices about distinct technologies.
Following the completion of the first phase, findings will be analyzed to narrow down the most effective communication materials based on participants’ engagement, comprehension, and overall impact. The second phase will focus on a narrowed selection of these options, enabling a more targeted evaluation and strategic usability testing. This phase will employ a between-subjects design, where different groups of participants are exposed to distinct sets of stimuli. The advantages of a between-subjects design include the reduction of potential carryover effects—where the experience of one stimulus influences the response to another—allowing for clearer comparisons between materials. Furthermore, often participants are less likely to experience fatigue or boredom, which can occur from the exposure of multiple conditions, allowing the investigation of a deeper understanding from this narrowed selection of choices. This structure enables researchers to gather unbiased assessments of each communication strategy, facilitating a focused analysis of user preferences and effectiveness.
In addition, at the end of the between-subjects portion of usability testing, researchers can have the option to add a direct comparison within-subjects, as needed, if questions remain around competing alternatives. This within-subjects usability testing will allow participants to declare their preferences and satisfaction while simultaneously comparing performance metrics such as comprehension, retention, error rates, task success rates, time on tasks, and navigation path, among others. This approach capitalizes on the strengths of both methodologies: the robustness of within-subjects analysis and the clarity of between-subjects comparisons.
To assess usability effectively, metrics such as task completion time, accuracy of responses, and subjective ratings of user experience can be collected. By employing this dual methodological design, the proposed approach aims to provide a comprehensive evaluation of health communication materials, capturing both qualitative and quantitative data on user preferences, perceived usefulness, perceived ease of use and performance. The proposed methodology also considers the practical constraints often faced in health technology research. By integrating both within-subjects and between-subjects designs, researchers can optimize resource allocation while still obtaining meaningful insights leveraging an iterative mindset on the initial within-subjects phase. This is particularly important in the context of feasibility studies, where limited funding and time may restrict the scope of research. By strategically utilizing a two-phase approach, the proposed method can support researchers to deliver robust and actionable findings that can inform the development of effective health communication systems. Additionally, the first stage of the within-subjects online study can be broken down into multiple waves, based on the number of tests required, in line with an iterative approach to usability testing.
Hence, this study seeks to bridge the best practices of experimental psychology and statistics with the practical needs of the fields of human factors and ergonomics. The first phase focuses on a larger sample size to explore a diverse range of stimuli through a within-subjects approach, reducing variability and enhancing statistical power. The second phase narrows the focus to systematically selected materials, employing primarily a between-subjects design to facilitate clear comparisons and reduce carryover effects. By integrating rigorous experimental designs commonly employed in psychological research—such as the within-subjects and between-subjects approaches—this study aims to enhance the validity and reliability of findings in health communication systems. Moreover, to circumvent a potential limitation of the two-stage method, we note that participants should be recruited from an assortment of online platforms to ensure a diverse sample that accurately reflects the target population. This measure aims to avoid risks of prematurely eliminating initial options and rendering them unavailable for the second, between-subjects phase. Human factors and ergonomics emphasize user-centered design and the optimization of systems for human use; thus, the application of psychology principles can inform the development of more effective communication materials, and usability testing of technologies tailored to user needs. Furthermore, leveraging statistical techniques from experimental psychology can help in the precise analysis of user interactions with health technologies, ensuring that insights gleaned from the research translate into actionable recommendations for practitioners. By aligning the scientific rigor of experimental psychology methodologies with the practical applications in human factors and ergonomics, this research aspires to foster interdisciplinary collaboration that enhances and expands both theoretical understanding and real-world applications in health communication.
In the first phase, a larger online study employs a within-subjects approach, in which participants are exposed to a diverse range of stimuli presented in randomized and counterbalanced order to attenuate the worry about the order of exposure affecting reference and the slew of confounds when looking at the aggregate data. This design presents several notable benefits. Firstly, within-subjects designs reduce participant variability, as each participant serves as their own control. It enables researchers to account for individual differences in responses, thus isolating the effects of specific stimuli more effectively while internal validity is not dependent on random assignment. Additionally, from a theoretical perspective (e.g., Dual Processing Theories, Bounded Rationality, Nudge Theory), it is often more insightful in usability testing to envision a health technology user responding to two competing products rather than considering two users in different markets making choices about distinct technologies.
Following the completion of the first phase, findings will be analyzed to narrow down the most effective communication materials based on participants’ engagement, comprehension, and overall impact. The second phase will focus on a narrowed selection of these options, enabling a more targeted evaluation and strategic usability testing. This phase will employ a between-subjects design, where different groups of participants are exposed to distinct sets of stimuli. The advantages of a between-subjects design include the reduction of potential carryover effects—where the experience of one stimulus influences the response to another—allowing for clearer comparisons between materials. Furthermore, often participants are less likely to experience fatigue or boredom, which can occur from the exposure of multiple conditions, allowing the investigation of a deeper understanding from this narrowed selection of choices. This structure enables researchers to gather unbiased assessments of each communication strategy, facilitating a focused analysis of user preferences and effectiveness.
In addition, at the end of the between-subjects portion of usability testing, researchers can have the option to add a direct comparison within-subjects, as needed, if questions remain around competing alternatives. This within-subjects usability testing will allow participants to declare their preferences and satisfaction while simultaneously comparing performance metrics such as comprehension, retention, error rates, task success rates, time on tasks, and navigation path, among others. This approach capitalizes on the strengths of both methodologies: the robustness of within-subjects analysis and the clarity of between-subjects comparisons.
To assess usability effectively, metrics such as task completion time, accuracy of responses, and subjective ratings of user experience can be collected. By employing this dual methodological design, the proposed approach aims to provide a comprehensive evaluation of health communication materials, capturing both qualitative and quantitative data on user preferences, perceived usefulness, perceived ease of use and performance. The proposed methodology also considers the practical constraints often faced in health technology research. By integrating both within-subjects and between-subjects designs, researchers can optimize resource allocation while still obtaining meaningful insights leveraging an iterative mindset on the initial within-subjects phase. This is particularly important in the context of feasibility studies, where limited funding and time may restrict the scope of research. By strategically utilizing a two-phase approach, the proposed method can support researchers to deliver robust and actionable findings that can inform the development of effective health communication systems. Additionally, the first stage of the within-subjects online study can be broken down into multiple waves, based on the number of tests required, in line with an iterative approach to usability testing.
Hence, this study seeks to bridge the best practices of experimental psychology and statistics with the practical needs of the fields of human factors and ergonomics. The first phase focuses on a larger sample size to explore a diverse range of stimuli through a within-subjects approach, reducing variability and enhancing statistical power. The second phase narrows the focus to systematically selected materials, employing primarily a between-subjects design to facilitate clear comparisons and reduce carryover effects. By integrating rigorous experimental designs commonly employed in psychological research—such as the within-subjects and between-subjects approaches—this study aims to enhance the validity and reliability of findings in health communication systems. Moreover, to circumvent a potential limitation of the two-stage method, we note that participants should be recruited from an assortment of online platforms to ensure a diverse sample that accurately reflects the target population. This measure aims to avoid risks of prematurely eliminating initial options and rendering them unavailable for the second, between-subjects phase. Human factors and ergonomics emphasize user-centered design and the optimization of systems for human use; thus, the application of psychology principles can inform the development of more effective communication materials, and usability testing of technologies tailored to user needs. Furthermore, leveraging statistical techniques from experimental psychology can help in the precise analysis of user interactions with health technologies, ensuring that insights gleaned from the research translate into actionable recommendations for practitioners. By aligning the scientific rigor of experimental psychology methodologies with the practical applications in human factors and ergonomics, this research aspires to foster interdisciplinary collaboration that enhances and expands both theoretical understanding and real-world applications in health communication.
Event Type
Oral Presentations
TimeWednesday, April 211:00am - 11:30am EDT
LocationPier 2/3
Digital Health (DH)