Presentation
Knowledge Tasks in Human Factors Formative and Validation Studies – What to Assess and How to Assess It?
DescriptionThe primary activity in human factors formative and validation studies involves observing representative test participants use the medical device being tested. However, some use tasks are not observable, for various reasons. Particularly, knowledge tasks are required to be included in such studies to assess the effectiveness of information in the labeling that is used as strategies to reduce or minimize risks (i.e., risk control measures) associated with critical tasks on which use error could lead to harm.
However, a great deal of confusion exists in the industry regarding what elements of a medical device’s user interface should be assessed in knowledge tasks, how to formulate knowledge task questions, and how to assess the resulting data as part of a human factors formative or validation study.
In this presentation, the presenter will discuss the information and advice provided by the US Food and Drug Administration (FDA) regarding knowledge tasks in all their guidance documents since 2011 and the struggles the medical device industry has had applying the information and advice when conducting human factors studies. In addition, the presentation will include knowledge task guidance provided in the upcoming update of ANSI/AAMI HE75, which is expected to be published in early 2025 and was heavily reviewed by the FDA and the AAMI HF committee members in an effort to provide clarity.
The FDA first mentioned knowledge tasks in their 2011 Center for Devices and Radiological Health (CDRH) draft guidance document, “Applying Human Factors and Usability Engineering to Optimize Medical Device Design.” In the draft, little guidance was provided on how to structure or apply knowledge tasks in human factors formative and validation studies. More guidance was provided in the 2016 CDRH final guidance, “Applying Human Factors and Usability Engineering to Medical Devices.” In the final guidance, FDA specifically described knowledge tasks as being a way to assess test participants’ understanding of information, which was difficult to ascertain by observing their actions during handling tasks. They also specifically mentioned that the medical device user interface components involved in knowledge tasks were usually the labeling elements, e.g., the user manual, quick start guide, labeling on the device itself, and training.
The FDA’s Center for Drug Evaluation and Research (CDER) (in conjunction with its Center for Biologics Evaluation and Research [CBER] and Office of Combination Products [OCP]) first mentioned knowledge tasks in their 2016 draft guidance, “Human Factors Studies and Related Clinical Study Considerations in Combination Product Design and Development.” In the document, CDER/CBER/OCP specifically stated that the goal of knowledge tasks was to assess users’ understanding of information associated with critical tasks in a combination product’s labels or labeling. CDER and CBER’s 2018 draft guidance document, “Contents of a Complete Submission for Threshold Analyses and Human Factors Submissions to Drug and Biologic Applications,” also mentioned knowledge tasks. In the document, CDER and CBER stated that tasks requiring understanding of warnings, caution statements, or contraindications in the product labels or labeling would generally be considered critical knowledge tasks.
However, in recent years medical device and combination product manufacturers have gotten feedback from the FDA’s CDRH and CDER that it is not acceptable for study personnel to direct test participants to look at the labeling materials during a human factors formative and validation study because having such help is not representative of real-world use. A challenge then arises when a test participant does not look at the labeling and provides an incorrect response to a knowledge task question. What does that response indicate about the effectiveness of risk control measures in the labeling? What should the manufacturer do with that data? Should the test stop at that point or continue in some manner so that the effectiveness of the risk controls in the labeling will still be assessed? If the study continues, how should it continue?
To date, the Association for the Advancement of Medical Instrumentation (AAMI) has not described knowledge tasks in any of their standards or technical information reports (TIRs). However, the update of AAMI HE75, which was originally issued in 2009, will include this information (in the section on usability testing). As of the date of this proposal, the final Draft Committee Draft for Vote (CDV-4) is about to be issued for final review prior to its publication in early 2025. Multiple representatives from FDA’s CDRH and CDER have been actively involved in the discussions of this content and approved of the wording in this final draft.
In the presentation, the presenter, who is Co-chair of the AAMI Human Factors Engineering (HE) committee, will share with the audience how knowledge tasks will be explained and described in the updated version of AAMI HE75. Because the FDA was actively engaged in reviewing and providing input on the update of AAMI HE75, the way in which knowledge tasks are discussed in this standard could influence future FDA guidance documents that discuss this topic.
However, a great deal of confusion exists in the industry regarding what elements of a medical device’s user interface should be assessed in knowledge tasks, how to formulate knowledge task questions, and how to assess the resulting data as part of a human factors formative or validation study.
In this presentation, the presenter will discuss the information and advice provided by the US Food and Drug Administration (FDA) regarding knowledge tasks in all their guidance documents since 2011 and the struggles the medical device industry has had applying the information and advice when conducting human factors studies. In addition, the presentation will include knowledge task guidance provided in the upcoming update of ANSI/AAMI HE75, which is expected to be published in early 2025 and was heavily reviewed by the FDA and the AAMI HF committee members in an effort to provide clarity.
The FDA first mentioned knowledge tasks in their 2011 Center for Devices and Radiological Health (CDRH) draft guidance document, “Applying Human Factors and Usability Engineering to Optimize Medical Device Design.” In the draft, little guidance was provided on how to structure or apply knowledge tasks in human factors formative and validation studies. More guidance was provided in the 2016 CDRH final guidance, “Applying Human Factors and Usability Engineering to Medical Devices.” In the final guidance, FDA specifically described knowledge tasks as being a way to assess test participants’ understanding of information, which was difficult to ascertain by observing their actions during handling tasks. They also specifically mentioned that the medical device user interface components involved in knowledge tasks were usually the labeling elements, e.g., the user manual, quick start guide, labeling on the device itself, and training.
The FDA’s Center for Drug Evaluation and Research (CDER) (in conjunction with its Center for Biologics Evaluation and Research [CBER] and Office of Combination Products [OCP]) first mentioned knowledge tasks in their 2016 draft guidance, “Human Factors Studies and Related Clinical Study Considerations in Combination Product Design and Development.” In the document, CDER/CBER/OCP specifically stated that the goal of knowledge tasks was to assess users’ understanding of information associated with critical tasks in a combination product’s labels or labeling. CDER and CBER’s 2018 draft guidance document, “Contents of a Complete Submission for Threshold Analyses and Human Factors Submissions to Drug and Biologic Applications,” also mentioned knowledge tasks. In the document, CDER and CBER stated that tasks requiring understanding of warnings, caution statements, or contraindications in the product labels or labeling would generally be considered critical knowledge tasks.
However, in recent years medical device and combination product manufacturers have gotten feedback from the FDA’s CDRH and CDER that it is not acceptable for study personnel to direct test participants to look at the labeling materials during a human factors formative and validation study because having such help is not representative of real-world use. A challenge then arises when a test participant does not look at the labeling and provides an incorrect response to a knowledge task question. What does that response indicate about the effectiveness of risk control measures in the labeling? What should the manufacturer do with that data? Should the test stop at that point or continue in some manner so that the effectiveness of the risk controls in the labeling will still be assessed? If the study continues, how should it continue?
To date, the Association for the Advancement of Medical Instrumentation (AAMI) has not described knowledge tasks in any of their standards or technical information reports (TIRs). However, the update of AAMI HE75, which was originally issued in 2009, will include this information (in the section on usability testing). As of the date of this proposal, the final Draft Committee Draft for Vote (CDV-4) is about to be issued for final review prior to its publication in early 2025. Multiple representatives from FDA’s CDRH and CDER have been actively involved in the discussions of this content and approved of the wording in this final draft.
In the presentation, the presenter, who is Co-chair of the AAMI Human Factors Engineering (HE) committee, will share with the audience how knowledge tasks will be explained and described in the updated version of AAMI HE75. Because the FDA was actively engaged in reviewing and providing input on the update of AAMI HE75, the way in which knowledge tasks are discussed in this standard could influence future FDA guidance documents that discuss this topic.
Event Type
Oral Presentations
TimeWednesday, April 210:30am - 11:00am EDT
LocationHarbour A/B
Medical and Drug Delivery Devices (MDD)