Presentation
Competency Assessment of Physicians Using Full-Scale Simulation of Acute Crisis Management Scenarios: The IMPACTS Study
DescriptionINTRODUCTION.
The public expects healthcare professionals to competently manage acute critical events; yet deficiencies are common during actual events leading to ‘failure to rescue’ and poorer outcomes. To successfully manage acute care events, both technical (e.g., correct diagnosis & therapy) and non-technical (e.g., teamwork/communication) skills are crucial. We know too little about the ‘fast’ dynamic decision strategies needed for effective management of acute care events. Understanding and addressing why or how some clinicians manage such critical events well, while others do not, is vital to improved outcomes.
Beyond use in training, simulation is an ideal tool to probe clinicians’ and teams’ critical event management. Our prior multisite study [1,2] assessed 263 consenting board-certified anesthesiologists (BCAs) ON two of four 20-minute standardized critical intraoperative anesthesia scenarios AS PART OF existing simulation-based maintenance of certification courses. Independent expert clinicians were trained to reliably rate the resulting scenario videos. Almost 1/3 of encounters received “poor” holistic technical and non-technical performance ratings. While we identified broad patterns of performance deficiencies (e.g., failure to escalate care) [2], that study was not designed to ascertain how or why this happened.
We therefore designed a study to better understand the critical aspects of ‘successful’ decision-making and the fixed and the mutable clinician attributes that predict optimal vs. sub-optimal event management. This study was designed to: 1) develop and test a unified cognitive model and taxonomy of the decision-making strategies clinicians use during critical event management [3]; 2) Evaluate the factors affecting physicians’ critical event performance; 3) Develop and delineate methods to measure and understand decision making strategies; and 4) Evaluate the relationship between simulation-based performance assessment and existing metrics of physician competence (participants’ primary board certification exam scores). In this abstract, we describe the design and conduct of this complex study and present some of our early results.
METHODS.
Using the prior study as a foundation [1,2], the Simulation Assessment Research Group (SARG, www.vumc.org/sarg), with funding from the Agency for Healthcare Research and Quality (AHRQ), conducted a multicenter study of physician performance during high-fidelity simulated acute crisis events. A large team of clinicians, human factors experts, and educators developed four 15-minute standardized scenarios with associated performance expectations (‘clinical performance elements’ or CPEs). The team was assisted by an independent multidisciplinary group of clinicians who provided content feedback and ranked CPEs using a modified Delphi technique. To enhance standardization, each scenario had a comprehensive set of scripts, rules, and delivery expectations. Each scenario was pilot tested at each study site to assure consistent and scorable delivery prior to use. The team also developed and pilot tested a comprehensive eSurvey to elicit study participants’ individual characteristics, clinical training, current clinical practice attributes, prior exposure to simulation and to crisis event management, and a validated hazardous attitudes questionnaire [4].
Study data collection commenced in 2021 after a nearly one-year delay due to COVID. We ultimately conducted 46 study days at four diverse sites with 102 anesthesiologists of varying experience consisting of 62 BCAs, 26 senior trainees (≥24 months of training and not board-certified) and 16 junior (≥6 and <18 months of training).
Before the 7-hour study day, each participant signed an eConsent and completed the 7-module eSurvey. On the study day, 2-4 participants received a comprehensive orientation to the study, simulation, and the local simulated environment. They then individually provided care in all four scenarios, presented in predetermined random order.
The four scenarios were:
• Abdominal sepsis presenting initially as shortness of breath (SOB) in the post-anesthesia care unit (PACU) using a patient manikin.
• Altered mental status (AMS) presenting initially as delayed emergency in the PACU using a live standardized patient.
•. Urosepsis presenting as hypotension (Hypo) in the PACU using a patient manikin.
• Thyrotoxicosis and acute cardiac syndrome (ACS) presenting initially as anxiety in the pre-operative holding area with a standardized patient.
Each scenario had at least two additional actors portraying clinician team members according to strict behavioral scripts. After each scenario, we conducted 40-minute cognitive interviews (CIs) to delineate participants’ thought processes and decision-making strategies. These innovative methods have already been published and will not be discussed further here [5]. At the end of the study day (i.e., after all 4 scenarios and 4 CIs), participants completed two eSurvey evaluations. All participants received appropriate compensation.
As was the case in our prior study [1], all encounters were video recorded and were scored subsequently using a custom video review platform (ViDARA) by at least one of six independent blinded and trained raters using several performance measures – the scenario-specific time-stamped CPEs, behaviorally anchored rating scales (BARS) on a 1-9 scale, and 1-9 holistic scores for technical, non-technical, and overall performance. Raters were taught to first decide whether the observed performance fell in the Poor (1-3), Middle (4-6), or Excellent (7-9) bin and then to assign the final score within that bin (e.g., 1 – low, 2 – medium, or 3 – high Poor). ~30% of all performances were subjected to re-review to assess intra- and inter-rater reliability.
Each participant’s national board scores were also obtained and linked to their other performance measures. We used a principal components analysis to statistically summarize into 3 components the multiple eSurvey variables related to participants’ clinical, simulation, and crisis management experience. Multivariate regression models were developed to examine the distribution of performance measures.
RESULTS.
To date performance on 3 of the scenarios have been fully rated. Overall, intra- and inter-rater reliability coefficients were >0.8 for CPEs and >0.5 for other measures. Overall, senior residents received the highest performance scores followed closely by BCAs with junior residents’ scores being much lower. Senior residents and BCAs were more likely than junior residents to perform specific CPEs. There were no differences in performance between academic and community practice-based BCAs. Initial multivariate models suggest that the two most significant predictors of performance are time from completion of training (in either direction) and prior exposure to crisis events. The transcripts of the cognitive interviews have been coded and meaningful information about participants’ decision-making approaches is being extracted. Additional results will be provided.
IMPLICATIONS.
These findings appear to validate our model of simulation-based performance by showing poorer performances in junior vs. senior residents. Moreover, we show that practitioners who are further away from their original training and certification have a tendency for lower performance scores supporting the value of maintenance of certification programs. Upon completion, we expect this study to yield guidelines on decision-making in critical event management and information to support policymaking about the content, execution, and timing of simulation-based training and assessment.
REFERENCES.
1. Weinger MB, Banerjee A, et al: Simulation-based assessment of the management of critical events by board-certified anesthesiologists. Anesthesiology 127(3): 475-89, Sep 2017. PMID: 28671903
2. Banerjee A, Burden A, et al: Key performance gaps of practicing anesthesiologists: How they contribute to hazards in anesthesiology and proposals for addressing them. Int Anesthesiol Clin 2019. 58(1):13-20. PMID: 31800410
3. Reale C, Salwei ME, et al: Decision-making during high-risk events: A systematic literature review. J Cogn Engin Dec Mak 2023 Jun; 17(2):188-212. PMID: 37823061
4. Rask JP, Duran H-T, DeClercq J, et al: The prevalence, patterns and predictors of hazardous attitudes among anesthesiologists: An incompletely studied area of clinical decision safety. Brit J Anaesth 2023 Nov; 131(5):e157-e160. PMID: 37741719
5. Militello LG, Salwei ME, Reale C, et al: Adapting cognitive task analysis methods for use in a large sample simulation study of high-risk healthcare events. J Cogn Engin Decis Making 2023 Dec;17(4):315-331. PMID: 37941803
Supported by the Agency for Healthcare Research and Quality (AHRQ, Rockville, MD, USA).
The public expects healthcare professionals to competently manage acute critical events; yet deficiencies are common during actual events leading to ‘failure to rescue’ and poorer outcomes. To successfully manage acute care events, both technical (e.g., correct diagnosis & therapy) and non-technical (e.g., teamwork/communication) skills are crucial. We know too little about the ‘fast’ dynamic decision strategies needed for effective management of acute care events. Understanding and addressing why or how some clinicians manage such critical events well, while others do not, is vital to improved outcomes.
Beyond use in training, simulation is an ideal tool to probe clinicians’ and teams’ critical event management. Our prior multisite study [1,2] assessed 263 consenting board-certified anesthesiologists (BCAs) ON two of four 20-minute standardized critical intraoperative anesthesia scenarios AS PART OF existing simulation-based maintenance of certification courses. Independent expert clinicians were trained to reliably rate the resulting scenario videos. Almost 1/3 of encounters received “poor” holistic technical and non-technical performance ratings. While we identified broad patterns of performance deficiencies (e.g., failure to escalate care) [2], that study was not designed to ascertain how or why this happened.
We therefore designed a study to better understand the critical aspects of ‘successful’ decision-making and the fixed and the mutable clinician attributes that predict optimal vs. sub-optimal event management. This study was designed to: 1) develop and test a unified cognitive model and taxonomy of the decision-making strategies clinicians use during critical event management [3]; 2) Evaluate the factors affecting physicians’ critical event performance; 3) Develop and delineate methods to measure and understand decision making strategies; and 4) Evaluate the relationship between simulation-based performance assessment and existing metrics of physician competence (participants’ primary board certification exam scores). In this abstract, we describe the design and conduct of this complex study and present some of our early results.
METHODS.
Using the prior study as a foundation [1,2], the Simulation Assessment Research Group (SARG, www.vumc.org/sarg), with funding from the Agency for Healthcare Research and Quality (AHRQ), conducted a multicenter study of physician performance during high-fidelity simulated acute crisis events. A large team of clinicians, human factors experts, and educators developed four 15-minute standardized scenarios with associated performance expectations (‘clinical performance elements’ or CPEs). The team was assisted by an independent multidisciplinary group of clinicians who provided content feedback and ranked CPEs using a modified Delphi technique. To enhance standardization, each scenario had a comprehensive set of scripts, rules, and delivery expectations. Each scenario was pilot tested at each study site to assure consistent and scorable delivery prior to use. The team also developed and pilot tested a comprehensive eSurvey to elicit study participants’ individual characteristics, clinical training, current clinical practice attributes, prior exposure to simulation and to crisis event management, and a validated hazardous attitudes questionnaire [4].
Study data collection commenced in 2021 after a nearly one-year delay due to COVID. We ultimately conducted 46 study days at four diverse sites with 102 anesthesiologists of varying experience consisting of 62 BCAs, 26 senior trainees (≥24 months of training and not board-certified) and 16 junior (≥6 and <18 months of training).
Before the 7-hour study day, each participant signed an eConsent and completed the 7-module eSurvey. On the study day, 2-4 participants received a comprehensive orientation to the study, simulation, and the local simulated environment. They then individually provided care in all four scenarios, presented in predetermined random order.
The four scenarios were:
• Abdominal sepsis presenting initially as shortness of breath (SOB) in the post-anesthesia care unit (PACU) using a patient manikin.
• Altered mental status (AMS) presenting initially as delayed emergency in the PACU using a live standardized patient.
•. Urosepsis presenting as hypotension (Hypo) in the PACU using a patient manikin.
• Thyrotoxicosis and acute cardiac syndrome (ACS) presenting initially as anxiety in the pre-operative holding area with a standardized patient.
Each scenario had at least two additional actors portraying clinician team members according to strict behavioral scripts. After each scenario, we conducted 40-minute cognitive interviews (CIs) to delineate participants’ thought processes and decision-making strategies. These innovative methods have already been published and will not be discussed further here [5]. At the end of the study day (i.e., after all 4 scenarios and 4 CIs), participants completed two eSurvey evaluations. All participants received appropriate compensation.
As was the case in our prior study [1], all encounters were video recorded and were scored subsequently using a custom video review platform (ViDARA) by at least one of six independent blinded and trained raters using several performance measures – the scenario-specific time-stamped CPEs, behaviorally anchored rating scales (BARS) on a 1-9 scale, and 1-9 holistic scores for technical, non-technical, and overall performance. Raters were taught to first decide whether the observed performance fell in the Poor (1-3), Middle (4-6), or Excellent (7-9) bin and then to assign the final score within that bin (e.g., 1 – low, 2 – medium, or 3 – high Poor). ~30% of all performances were subjected to re-review to assess intra- and inter-rater reliability.
Each participant’s national board scores were also obtained and linked to their other performance measures. We used a principal components analysis to statistically summarize into 3 components the multiple eSurvey variables related to participants’ clinical, simulation, and crisis management experience. Multivariate regression models were developed to examine the distribution of performance measures.
RESULTS.
To date performance on 3 of the scenarios have been fully rated. Overall, intra- and inter-rater reliability coefficients were >0.8 for CPEs and >0.5 for other measures. Overall, senior residents received the highest performance scores followed closely by BCAs with junior residents’ scores being much lower. Senior residents and BCAs were more likely than junior residents to perform specific CPEs. There were no differences in performance between academic and community practice-based BCAs. Initial multivariate models suggest that the two most significant predictors of performance are time from completion of training (in either direction) and prior exposure to crisis events. The transcripts of the cognitive interviews have been coded and meaningful information about participants’ decision-making approaches is being extracted. Additional results will be provided.
IMPLICATIONS.
These findings appear to validate our model of simulation-based performance by showing poorer performances in junior vs. senior residents. Moreover, we show that practitioners who are further away from their original training and certification have a tendency for lower performance scores supporting the value of maintenance of certification programs. Upon completion, we expect this study to yield guidelines on decision-making in critical event management and information to support policymaking about the content, execution, and timing of simulation-based training and assessment.
REFERENCES.
1. Weinger MB, Banerjee A, et al: Simulation-based assessment of the management of critical events by board-certified anesthesiologists. Anesthesiology 127(3): 475-89, Sep 2017. PMID: 28671903
2. Banerjee A, Burden A, et al: Key performance gaps of practicing anesthesiologists: How they contribute to hazards in anesthesiology and proposals for addressing them. Int Anesthesiol Clin 2019. 58(1):13-20. PMID: 31800410
3. Reale C, Salwei ME, et al: Decision-making during high-risk events: A systematic literature review. J Cogn Engin Dec Mak 2023 Jun; 17(2):188-212. PMID: 37823061
4. Rask JP, Duran H-T, DeClercq J, et al: The prevalence, patterns and predictors of hazardous attitudes among anesthesiologists: An incompletely studied area of clinical decision safety. Brit J Anaesth 2023 Nov; 131(5):e157-e160. PMID: 37741719
5. Militello LG, Salwei ME, Reale C, et al: Adapting cognitive task analysis methods for use in a large sample simulation study of high-risk healthcare events. J Cogn Engin Decis Making 2023 Dec;17(4):315-331. PMID: 37941803
Supported by the Agency for Healthcare Research and Quality (AHRQ, Rockville, MD, USA).
Event Type
Oral Presentations
TimeMonday, March 311:52pm - 2:15pm EDT
LocationQueens Quay
Patient Safety and Research Initiatives (PS)