Presentation
Usability Evaluation of OptiBrain, a Clinical Decision Support System for Traumatic Brain Injury
DescriptionContext: Since their first implementation in the 1980s, Clinical Decision Support Systems (CDSS) have significantly evolved, having a profound impact on the healthcare field. These systems assist clinicians in making complex decisions by compiling clinical data from different sources and presenting an inference model to support clinical reasoning. Their use is often associated with improved clinician performance and better clinical outcomes for patients. However, despite their potential in the medical sector, adoption of CDSS remains limited in practice due to intercompatibility issues, change in medical practices and poor usability. It is therefore essential to adopt a user-centered perspective in the design of CDSS and evaluate its usability to integrate the CDSS more harmoniously into clinicians' workflows.
Recent work at CHU Sainte-Justine hospital (Montreal, QC) developed OptiBrain, a CDDS to manage pediatric severe traumatic brain injury. Using live data from the electronic health record and bedside monitors, it computes the adherence to clinical guidelines of 14 indicators i.e., to what extent was the patient within the recommended range for each indicator, the patient's cerebral condition, and the cerebral autoregulation status including optimal cerebral perfusion pressure. A graphical user interface was designed for OptiBrain through iterative cycles involving clinicians and human factors specialists. It consists of 5 main information areas (see the full manuscript for detailed figures): main navigation bar, system-based navigation bar, patient records, adherence to guidelines, and real-time data. Before OptiBrain can make its way to the patient’s bedside, it requires an usability evaluation with realistic patient data. This will ensure fixing usability problems found in tests before its deployment and improve clinicians’ acceptance of OptiBrain within the intensive care unit.
Objective: In this work, we conducted an usability evaluation of OptiBrain, using realistic patient data. The evaluation aimed to ensure the proper implementation of a new system within the intensive care unit at CHU Sainte-Justine, as no similar initiative has been undertaken thus far.
Methodology: 8 participants took part in the study: 4 intensive care physicians with an average clinical experience of 17 years, and 4 fellows with an average clinical experience of 3 years. Only 2 participants had prior exposure to OptiBrain.
Clinical case. We used a realistic case of a 13 year old patient following a bike accident. The case was presented in two phases: phase 1 was the most acute phase as the patient has just been taken in charge and experienced several intracranial pressure peaks; phase 2 was on the second day after the patient had a craniectomy surgery and was more stable.
Material. We implemented the OptiBrain mockup as an interactive prototype using Figma and populated its data using realistic patient’s information. Test was conducted in-person with 6 participants and remotely with 2 participants. In both cases participants interacted with the prototype from a laptop.
Procedure. Before the test, we briefed participants on OptiBrain and let them familiarize with the interface. Then we presented the clinical case: phase 1, followed by phase 2. We asked participants to verbalize their clinical evaluation of the patient’s condition using OptiBrain. Participants’ verbalizations allowed us to analyze the clinicians cognitive processes. We analyzed the transcript using three main coding categories: clinical reasoning, usability interactions, and user suggestions. The session lasted 1 hour.
Results: Clinical reasoning. Clinicians first scanned the interface and reviewed key parameters like ICP and CPAP to support their diagnoses. They employed non-analytical processes to explain the shape of curves, while using analytical reasoning, or clinical scripting, for case management strategies. They formed hypotheses based on their knowledge and experience, as seen when one clinician questioned the accuracy of arterial pressures. OptiBrain facilitated the transition between non-analytical and analytical approaches, allowing practitioners to generate and refine hypotheses, while simultaneously investigating off-target parameters to better understand their origins.
Interaction. Clinicians appreciated the inclusion of key parameters like ICP, MAP, and ETCO2. They also understood the neurological parameters presented and actively explored other systems, demonstrating engagement with the interface. However, there were notable usability issues. Navigation difficulties were reported, especially in accessing key parameters and understanding how to change the time window of the data displayed. The adherence banner cluttered the interface during the acute phase, as clinicians had to scroll down to see the parameters of interest. Clinicians also reported some parameters were missing like EVD and SaO2. Additionally, the optimal CPP, a new concept for users, required further clarification to clinicians less familiar with neurological diagnosis.
Suggestions. The interviews allowed us to gather several relevant suggestions from clinicians, particularly regarding the representation of data over time. Many found it challenging to understand the connection between hour-based navigation and the real-time data displayed at the bottom of the screen. Additionally, clinicians emphasized the importance of having more information available on the graphs used to analyze real-time indicators and the need for clearer axis labeling. Significant work will be needed to ensure a more efficient interaction between time management and patient progress.Our results also revealed clinicians’ interest in reducing visual overload and rethinking the prioritization of certain information. Most participants believed that real-time data should take precedence over adherence percentages, particularly for binary indicators such as nutrition or sedation. It will be important to reassess this hierarchy to ensure a better analysis and quality of the indicators provided by the system.
Lastly, clinicians reported their need for more transparency regarding the algorithm developed. For example, while clinicians appreciated receiving the optimal CPP results, they were somewhat hesitant to use it because they did not understand how the results were generated. Therefore, providing clear explanations and allowing clinicians to interact with the interface to review automated data will be necessary. This will contribute to increased trust in the system.
Limitations. It is important to note that during the tests, the system was not connected to the hospital's electronic health record in real-time. Indeed, the scenarios were built using a previous patient record, with some associated data missing. Furthermore, it was difficult for clinicians to provide a comprehensive and confident diagnosis, as the OptiBrain interface was limited to the neurological system and did not account for the interrelation with other physiological systems.
Outcomes: Following the study's recommendations, a second phase of bedside testing will be necessary to ensure the relevance of the proposed interface and the changes made accordingly. In parallel with this evaluation, the application was programmed and designed to be easily integrated into the existing systems at the CHU Sainte-Justine Hospital. These recommendations will be incorporated into the application to ensure better adoption of the system within the intensive care unit.
Recent work at CHU Sainte-Justine hospital (Montreal, QC) developed OptiBrain, a CDDS to manage pediatric severe traumatic brain injury. Using live data from the electronic health record and bedside monitors, it computes the adherence to clinical guidelines of 14 indicators i.e., to what extent was the patient within the recommended range for each indicator, the patient's cerebral condition, and the cerebral autoregulation status including optimal cerebral perfusion pressure. A graphical user interface was designed for OptiBrain through iterative cycles involving clinicians and human factors specialists. It consists of 5 main information areas (see the full manuscript for detailed figures): main navigation bar, system-based navigation bar, patient records, adherence to guidelines, and real-time data. Before OptiBrain can make its way to the patient’s bedside, it requires an usability evaluation with realistic patient data. This will ensure fixing usability problems found in tests before its deployment and improve clinicians’ acceptance of OptiBrain within the intensive care unit.
Objective: In this work, we conducted an usability evaluation of OptiBrain, using realistic patient data. The evaluation aimed to ensure the proper implementation of a new system within the intensive care unit at CHU Sainte-Justine, as no similar initiative has been undertaken thus far.
Methodology: 8 participants took part in the study: 4 intensive care physicians with an average clinical experience of 17 years, and 4 fellows with an average clinical experience of 3 years. Only 2 participants had prior exposure to OptiBrain.
Clinical case. We used a realistic case of a 13 year old patient following a bike accident. The case was presented in two phases: phase 1 was the most acute phase as the patient has just been taken in charge and experienced several intracranial pressure peaks; phase 2 was on the second day after the patient had a craniectomy surgery and was more stable.
Material. We implemented the OptiBrain mockup as an interactive prototype using Figma and populated its data using realistic patient’s information. Test was conducted in-person with 6 participants and remotely with 2 participants. In both cases participants interacted with the prototype from a laptop.
Procedure. Before the test, we briefed participants on OptiBrain and let them familiarize with the interface. Then we presented the clinical case: phase 1, followed by phase 2. We asked participants to verbalize their clinical evaluation of the patient’s condition using OptiBrain. Participants’ verbalizations allowed us to analyze the clinicians cognitive processes. We analyzed the transcript using three main coding categories: clinical reasoning, usability interactions, and user suggestions. The session lasted 1 hour.
Results: Clinical reasoning. Clinicians first scanned the interface and reviewed key parameters like ICP and CPAP to support their diagnoses. They employed non-analytical processes to explain the shape of curves, while using analytical reasoning, or clinical scripting, for case management strategies. They formed hypotheses based on their knowledge and experience, as seen when one clinician questioned the accuracy of arterial pressures. OptiBrain facilitated the transition between non-analytical and analytical approaches, allowing practitioners to generate and refine hypotheses, while simultaneously investigating off-target parameters to better understand their origins.
Interaction. Clinicians appreciated the inclusion of key parameters like ICP, MAP, and ETCO2. They also understood the neurological parameters presented and actively explored other systems, demonstrating engagement with the interface. However, there were notable usability issues. Navigation difficulties were reported, especially in accessing key parameters and understanding how to change the time window of the data displayed. The adherence banner cluttered the interface during the acute phase, as clinicians had to scroll down to see the parameters of interest. Clinicians also reported some parameters were missing like EVD and SaO2. Additionally, the optimal CPP, a new concept for users, required further clarification to clinicians less familiar with neurological diagnosis.
Suggestions. The interviews allowed us to gather several relevant suggestions from clinicians, particularly regarding the representation of data over time. Many found it challenging to understand the connection between hour-based navigation and the real-time data displayed at the bottom of the screen. Additionally, clinicians emphasized the importance of having more information available on the graphs used to analyze real-time indicators and the need for clearer axis labeling. Significant work will be needed to ensure a more efficient interaction between time management and patient progress.Our results also revealed clinicians’ interest in reducing visual overload and rethinking the prioritization of certain information. Most participants believed that real-time data should take precedence over adherence percentages, particularly for binary indicators such as nutrition or sedation. It will be important to reassess this hierarchy to ensure a better analysis and quality of the indicators provided by the system.
Lastly, clinicians reported their need for more transparency regarding the algorithm developed. For example, while clinicians appreciated receiving the optimal CPP results, they were somewhat hesitant to use it because they did not understand how the results were generated. Therefore, providing clear explanations and allowing clinicians to interact with the interface to review automated data will be necessary. This will contribute to increased trust in the system.
Limitations. It is important to note that during the tests, the system was not connected to the hospital's electronic health record in real-time. Indeed, the scenarios were built using a previous patient record, with some associated data missing. Furthermore, it was difficult for clinicians to provide a comprehensive and confident diagnosis, as the OptiBrain interface was limited to the neurological system and did not account for the interrelation with other physiological systems.
Outcomes: Following the study's recommendations, a second phase of bedside testing will be necessary to ensure the relevance of the proposed interface and the changes made accordingly. In parallel with this evaluation, the application was programmed and designed to be easily integrated into the existing systems at the CHU Sainte-Justine Hospital. These recommendations will be incorporated into the application to ensure better adoption of the system within the intensive care unit.
Authors
Event Type
Oral Presentations
TimeWednesday, April 211:30am - 12:00pm EDT
LocationPier 2/3
Digital Health (DH)