Presentation
MDD26 - Toward Never Doing Known Use Problem Analysis Again
SessionPoster Session 2
DescriptionArtificial Intelligence is a hot topic in all fields, including Human Factors. Last year, we presented a poster describing a proof of concept for leveraging a custom-made AI model to work toward automating the known use problem assessments (KUPA) required for FDA submissions. As shared last year, our goal is to never have to manually do a KUPA again. The work involved in reviewing complaints data for relevance and use problems is lengthy (i.e., costly), and tedious. We successfully proved that a model trained on even relatively small amounts of MAUDE data for injection devices can identify other MAUDE use-related events with around 70% accuracy. However, more work was needed to evaluate if the same process works equally well with other device types and other sources of data, and to determine if a model will always require device-specific human-annotated training data in order to be effective.
This poster presentation seeks to expand that concept in those three areas: first, to apply the same process and model to complex medical devices; second to leverage internal complaints data (with the company’s permission) as an alternative data source; and third, so see how well data trained on one device can identify use-related issues for a related but different device of the same manufacturer.
The original proof of concept focused on injection devices such as pen injectors and autoinjectors. These devices are relatively simple and therefore have relatively few use-related errors (compared to something like a surgical robot). Additionally, because they are common lay-user devices, there are lots of data entries in MAUDE with standardized (by manufacturers) reporting formats. As of last year’s presentation it was an open question whether AI would work equally well with a more complex system.
The original proof of concept also only utilized data from MAUDE, so we wanted to evaluate how well it works with other data formats. Manufacturers all have internal mechanisms for tracking product complaints and adverse events. Many manufacturers have very conservative data collection practices, meaning that they have lots of internal data to sift through, but they don’t have reliable ways of flagging use-related events. This means that a Human Factors Engineer tasked with conducting a known use assessment or other type of post-market Human Factors review has to spend significant amounts of time poring over data or must develop alternative data-limiting strategies to reduce the burden to something manageable within the program’s budget and schedule. We propose that using AI as the primary strategy will significantly reduce this burden without loss of accuracy.
With this presentation we also will show how well the AI model works on other devices when trained on a single device type.
To accomplish these three goals, with our client’s permission, we will leverage data from an existing KUPA that we conducted for a complex medical device that included significant amounts of internal data review, and present our findings. Using this method we will also be able to speak to cost and accuracy tradeoffs, as we will be able to compare data review hours and findings from the original KUPA to one that leverages AI for data review. The data and presentation will be anonymized but the findings will still help others evaluate whether to apply the same processes to their own work.
This poster presentation seeks to expand that concept in those three areas: first, to apply the same process and model to complex medical devices; second to leverage internal complaints data (with the company’s permission) as an alternative data source; and third, so see how well data trained on one device can identify use-related issues for a related but different device of the same manufacturer.
The original proof of concept focused on injection devices such as pen injectors and autoinjectors. These devices are relatively simple and therefore have relatively few use-related errors (compared to something like a surgical robot). Additionally, because they are common lay-user devices, there are lots of data entries in MAUDE with standardized (by manufacturers) reporting formats. As of last year’s presentation it was an open question whether AI would work equally well with a more complex system.
The original proof of concept also only utilized data from MAUDE, so we wanted to evaluate how well it works with other data formats. Manufacturers all have internal mechanisms for tracking product complaints and adverse events. Many manufacturers have very conservative data collection practices, meaning that they have lots of internal data to sift through, but they don’t have reliable ways of flagging use-related events. This means that a Human Factors Engineer tasked with conducting a known use assessment or other type of post-market Human Factors review has to spend significant amounts of time poring over data or must develop alternative data-limiting strategies to reduce the burden to something manageable within the program’s budget and schedule. We propose that using AI as the primary strategy will significantly reduce this burden without loss of accuracy.
With this presentation we also will show how well the AI model works on other devices when trained on a single device type.
To accomplish these three goals, with our client’s permission, we will leverage data from an existing KUPA that we conducted for a complex medical device that included significant amounts of internal data review, and present our findings. Using this method we will also be able to speak to cost and accuracy tradeoffs, as we will be able to compare data review hours and findings from the original KUPA to one that leverages AI for data review. The data and presentation will be anonymized but the findings will still help others evaluate whether to apply the same processes to their own work.
Event Type
Poster Presentation
TimeTuesday, April 14:45pm - 6:15pm EDT
LocationFrontenac Foyer