Presentation
DH17 - Survey Development for Measuring Human Performance of AI Based Clinical Decision Support Implementations
SessionPoster Session 1
DescriptionArtificial intelligence (AI) has been touted as a tool to improve sepsis outcomes with hundreds of sepsis machine learning models published. However, clinically implemented models in the electronic health record (EHR) have yielded mixed results. Even the same sepsis prediction model developed by the same vendor has led to widely different results. To explain such differences, it is necessary to understand how implementation of the intervention leads to an expected mechanism of impact to produce the intended outcomes across different contexts. The intended mechanism by which prediction models improve outcomes is through enhancing human performance, surfacing previously undetected signals leading to better outcomes. However, real time evaluation of these measures is challenging as clinicians are busy working when exposed to these AI models. In this work, we aimed to develop a survey tool and implementation mechanism to feasibly evaluate human performance measures in the context of a real-world AI-based sepsis tool.
Following the survey scale design process described by AMEE (An International Association for Medical Education) Guide No. 87, we first conducted a literature review to identify measurement constructs. We then conducted interviews with users of the sepsis AI system to learn how they interacted with the AI tool and understand how they describe the constructs of interest. Specifically, we contacted providers and nurses who saw the AI sepsis alert in production, contacted them within 24 hours, and performed interviews within 72 hours of their seeing the AI tool. After synthesizing these learnings, we developed survey items. We then conducted expert validation of the survey instrument. We used content validity index (CVI) to evaluated how well a the items represent each human performance being measured. CVI of >=0.8 is generally recommended.
From the literature review, we identified the following as important applicable human performance constructs for AI in healthcare: AI tool impact on clinician situational awareness, explainability, trust, automation bias, human computer agreement, workload, and relationship with staff. Based on interviews, we added to this list AI tool impact on clinician differential diagnosis and treatment plan, team communication, tool satisfaction and clinician alert response. Combining these two sources, we developed a 10-item survey tool for expert validation.
Six clinical informatics experts with experience in AI and clinical decision support design, implementation, and evaluation were recruited from the Pediatric CDS Collaborative. Each expert rated each survey item assessing how clear and relevant the items were with respect to the construct of interest. Of the 10 survey items, content ratings for “human computer agreement”, “workload” and “relationship between patients and staff” items had content validity index (CVI) <0.8. Expert ratings on clarity of survey questions on “workload” and “relationship between patients and staff” had CVI <0.8. All other questions had CVI > 0.8.
Experts mentioned that the two question on “human computer agreement” were very similar and suggested that we merge them into one question. They were also skeptical of using a single point question for workload. Specifically, they were concerned that survey respondents would not separate workload with respect to the AI tool (i.e. interpretation of AI recommendations and associated decision-making) from workload of overall patient care which is impacted by several factors unrelated to the AI tool (e.g. patient complexity, communication needs, etc.). They were also not sure how respondents would interpret the question. Finally, experts recommended that with slight wording adjustments, this survey tool could be generalized for use for all CDS interruptive alerts irrespective of it being AI based or not as many human performance measures would be applicable to other CDS.
After completing updates to the survey instrument based on expert feedback and cognitive interviews with prospective respondents, we anticipate we will have an evidence-based survey capable of capturing human performance measures in near real time (within 1 day of an event). This approach will enable measurement of human performance measures as a mechanism of impact for AI and similar CDS implementations in healthcare.
Following the survey scale design process described by AMEE (An International Association for Medical Education) Guide No. 87, we first conducted a literature review to identify measurement constructs. We then conducted interviews with users of the sepsis AI system to learn how they interacted with the AI tool and understand how they describe the constructs of interest. Specifically, we contacted providers and nurses who saw the AI sepsis alert in production, contacted them within 24 hours, and performed interviews within 72 hours of their seeing the AI tool. After synthesizing these learnings, we developed survey items. We then conducted expert validation of the survey instrument. We used content validity index (CVI) to evaluated how well a the items represent each human performance being measured. CVI of >=0.8 is generally recommended.
From the literature review, we identified the following as important applicable human performance constructs for AI in healthcare: AI tool impact on clinician situational awareness, explainability, trust, automation bias, human computer agreement, workload, and relationship with staff. Based on interviews, we added to this list AI tool impact on clinician differential diagnosis and treatment plan, team communication, tool satisfaction and clinician alert response. Combining these two sources, we developed a 10-item survey tool for expert validation.
Six clinical informatics experts with experience in AI and clinical decision support design, implementation, and evaluation were recruited from the Pediatric CDS Collaborative. Each expert rated each survey item assessing how clear and relevant the items were with respect to the construct of interest. Of the 10 survey items, content ratings for “human computer agreement”, “workload” and “relationship between patients and staff” items had content validity index (CVI) <0.8. Expert ratings on clarity of survey questions on “workload” and “relationship between patients and staff” had CVI <0.8. All other questions had CVI > 0.8.
Experts mentioned that the two question on “human computer agreement” were very similar and suggested that we merge them into one question. They were also skeptical of using a single point question for workload. Specifically, they were concerned that survey respondents would not separate workload with respect to the AI tool (i.e. interpretation of AI recommendations and associated decision-making) from workload of overall patient care which is impacted by several factors unrelated to the AI tool (e.g. patient complexity, communication needs, etc.). They were also not sure how respondents would interpret the question. Finally, experts recommended that with slight wording adjustments, this survey tool could be generalized for use for all CDS interruptive alerts irrespective of it being AI based or not as many human performance measures would be applicable to other CDS.
After completing updates to the survey instrument based on expert feedback and cognitive interviews with prospective respondents, we anticipate we will have an evidence-based survey capable of capturing human performance measures in near real time (within 1 day of an event). This approach will enable measurement of human performance measures as a mechanism of impact for AI and similar CDS implementations in healthcare.
Event Type
Poster Presentation
TimeMonday, March 314:45pm - 6:15pm EDT
LocationFrontenac Foyer
Digital Health (DH)
Simulation and Education (SE)
Hospital Environments (HE)
Medical and Drug Delivery Devices (MDD)
Patient Safety and Research Initiatives (PS)