Presentation
DH5 - Building Trust in Clinical Decision Support Systems: Lessons Learnt From a Pilot Implementation of CDS in an Australian Rural Hospital
SessionPoster Session 2
DescriptionBackground:
Trust is a critical factor in the successful adoption of Clinical Decision Support (CDS) systems. Offering real-time, evidence-based recommendations, CDS systems have the potential to greatly enhance clinical decision-making and improve patient care. However, they are often underutilised by clinicians in practice. Building and sustaining trust in CDS is essential, as clinicians are unlikely to rely on information from these systems if they question their accuracy, reliability, or intent. Without trust, clinicians are less likely to consistently engage with CDS, limiting its potential to improve care quality and outcomes. Despite its critical role, the development and evolution of trust in CDS over time has rarely been explored.
This presentation will share findings from a case study exploring the pilot implementation of a CDS system in the Emergency Department (ED) and Patient Flow departments of a rural hospital in New South Wales, Australia. The CDS was deployed as two distinct applications: a mobile application used by ED clinicians, and a dashboard intended for patient flow nurses. Both systems were discontinued either before or after the completion of the pilot. The mobile application, designed to enhance doctors' access to patient information, test results, and critical alerts, was used in the ED for a period of 3-6 months. In contrast, patient flow nurses used a dashboard designed to remotely monitor COVID-19 patients for a period of 2 weeks.
Following pilot cessation, we interviewed 11 clinicians, managers and vendor staff involved in implementing or using the CDS, exploring participants’ experiences with the system, the pilot implementation approach and key reasons for the CDS not continuing post-pilot. Interviews were transcribed and thematically analysed. This presentation will focus specifically on factors that influenced clinicians' trust in CDS and how trust, or lack thereof, affected willingness to engage with the technology between different clinical settings and users. Our findings offer valuable insights into how early user experiences, technical performance, and organisational support collectively shaped clinicians’ trust in the CDS, impacting its adoption and eventual discontinuation. By examining these aspects, we aim to highlight critical considerations and recommendations that can be employed in future CDS implementations in healthcare settings.
Key findings:
For many ED users, early setbacks post-implementation, including system crashes, delays, and poor integration with the Electronic Medical Record (EMR) caused frustration and cast doubt on the systems’ utility. Although some technical issues were resolved later, the initial negative experiences resulted in a persistent distrust of the system among several clinicians, who did not re-adopt the technology. In contrast, several clinicians who early adopters of the system in the ED, remained engaged with the system as they were optimistic about the potential benefit it could deliver. As they were actively involved in the CDS’ design, they had established trusting relationships with the CDS vendor and saw value in providing feedback to refine the CDS, motivated by the prospect of improving its clinical utility. However, this group’s trust was conditional on the organisation and vendors’ ability to respond to their feedback and support their continued engagement. When these clinicians encountered persistent organisational constraints, such as resource limitations that stalled system improvement, their trust waned, underscoring the need for responsive support systems in sustaining trust. These clinicians however remained positive about the value of the CDS and described being willing to readopt the system if technical issues were resolved.
Patient flow nurses were less involved in CDS design, and this contributed to a more immediate and pronounced distrust in the system. Clinicians in this setting also appeared to be less forgiving of initial technical challenges as the CDS was intended to support the remote monitoring of COVID-19 patients in the initial stages of the pandemic, an urgent and high-stress period. Given their high workload and critical nature of their tasks, they expected the CDS to function effectively without requiring additional input. For these clinicians, trust was contingent on the system’s ability to perform reliably from the outset. Although nurses perceived a need for technology to support enhanced patient monitoring, the CDS was perceived to be poorly adapted to their specific needs and workflows. For instance, the system introduced new processes that lacked clear guidance, such as escalation protocols, while its overly sensitive alerting system contributed to a lack of trust in alerts and consequently, alert fatigue. Unlike ED clinicians, who had a mixed response to the system, most patient flow nurses rejected the CDS outright, demonstrating how different technical features, organisational contexts and user expectations can influence trust. Participants further expressed concern that this erosion of trust might carry over to future CDS implementations, potentially hindering nurses’ willingness to engage with new system implementations.
Overall, our findings underscore that trust in CDS systems is built not only on the technical functionality of systems but also on the organisation and vendors’ intentions and responsiveness to user concerns. Early technical failures and delayed resolutions are particularly detrimental to trust, as they can create persistent scepticism that overshadows later improvements. Engaging users in CDS design can help to mitigate the impact of these issues by fostering trust between clinicians and vendors, while also encouraging a sense of ownership of the system among users. To maintain trust with these users, strong organisational support and timely resolution of feedback is required. However, different approaches may be required to support users who do not have the capacity to be engaged in co-design processes and may be less likely to engage further if the system fails to meet their expectations on first use.
Presentation summary:
This presentation will highlight the key challenges involved in building clinicians’ trust in CDS systems, drawing on insights from our case study. We will describe lessons learnt and offer recommendations on how to enhance trust in CDS during and beyond implementation. The presentation will emphasise the importance of co-design and rapid issue resolution, tailored support structures, and a deep understanding of the unique requirements of different clinical settings and user groups. These insights are intended to aid healthcare organisations and developers in designing and implementing CDS systems that align more closely with clinicians’ needs, thereby enhancing adoption and long-term success.
Trust is a critical factor in the successful adoption of Clinical Decision Support (CDS) systems. Offering real-time, evidence-based recommendations, CDS systems have the potential to greatly enhance clinical decision-making and improve patient care. However, they are often underutilised by clinicians in practice. Building and sustaining trust in CDS is essential, as clinicians are unlikely to rely on information from these systems if they question their accuracy, reliability, or intent. Without trust, clinicians are less likely to consistently engage with CDS, limiting its potential to improve care quality and outcomes. Despite its critical role, the development and evolution of trust in CDS over time has rarely been explored.
This presentation will share findings from a case study exploring the pilot implementation of a CDS system in the Emergency Department (ED) and Patient Flow departments of a rural hospital in New South Wales, Australia. The CDS was deployed as two distinct applications: a mobile application used by ED clinicians, and a dashboard intended for patient flow nurses. Both systems were discontinued either before or after the completion of the pilot. The mobile application, designed to enhance doctors' access to patient information, test results, and critical alerts, was used in the ED for a period of 3-6 months. In contrast, patient flow nurses used a dashboard designed to remotely monitor COVID-19 patients for a period of 2 weeks.
Following pilot cessation, we interviewed 11 clinicians, managers and vendor staff involved in implementing or using the CDS, exploring participants’ experiences with the system, the pilot implementation approach and key reasons for the CDS not continuing post-pilot. Interviews were transcribed and thematically analysed. This presentation will focus specifically on factors that influenced clinicians' trust in CDS and how trust, or lack thereof, affected willingness to engage with the technology between different clinical settings and users. Our findings offer valuable insights into how early user experiences, technical performance, and organisational support collectively shaped clinicians’ trust in the CDS, impacting its adoption and eventual discontinuation. By examining these aspects, we aim to highlight critical considerations and recommendations that can be employed in future CDS implementations in healthcare settings.
Key findings:
For many ED users, early setbacks post-implementation, including system crashes, delays, and poor integration with the Electronic Medical Record (EMR) caused frustration and cast doubt on the systems’ utility. Although some technical issues were resolved later, the initial negative experiences resulted in a persistent distrust of the system among several clinicians, who did not re-adopt the technology. In contrast, several clinicians who early adopters of the system in the ED, remained engaged with the system as they were optimistic about the potential benefit it could deliver. As they were actively involved in the CDS’ design, they had established trusting relationships with the CDS vendor and saw value in providing feedback to refine the CDS, motivated by the prospect of improving its clinical utility. However, this group’s trust was conditional on the organisation and vendors’ ability to respond to their feedback and support their continued engagement. When these clinicians encountered persistent organisational constraints, such as resource limitations that stalled system improvement, their trust waned, underscoring the need for responsive support systems in sustaining trust. These clinicians however remained positive about the value of the CDS and described being willing to readopt the system if technical issues were resolved.
Patient flow nurses were less involved in CDS design, and this contributed to a more immediate and pronounced distrust in the system. Clinicians in this setting also appeared to be less forgiving of initial technical challenges as the CDS was intended to support the remote monitoring of COVID-19 patients in the initial stages of the pandemic, an urgent and high-stress period. Given their high workload and critical nature of their tasks, they expected the CDS to function effectively without requiring additional input. For these clinicians, trust was contingent on the system’s ability to perform reliably from the outset. Although nurses perceived a need for technology to support enhanced patient monitoring, the CDS was perceived to be poorly adapted to their specific needs and workflows. For instance, the system introduced new processes that lacked clear guidance, such as escalation protocols, while its overly sensitive alerting system contributed to a lack of trust in alerts and consequently, alert fatigue. Unlike ED clinicians, who had a mixed response to the system, most patient flow nurses rejected the CDS outright, demonstrating how different technical features, organisational contexts and user expectations can influence trust. Participants further expressed concern that this erosion of trust might carry over to future CDS implementations, potentially hindering nurses’ willingness to engage with new system implementations.
Overall, our findings underscore that trust in CDS systems is built not only on the technical functionality of systems but also on the organisation and vendors’ intentions and responsiveness to user concerns. Early technical failures and delayed resolutions are particularly detrimental to trust, as they can create persistent scepticism that overshadows later improvements. Engaging users in CDS design can help to mitigate the impact of these issues by fostering trust between clinicians and vendors, while also encouraging a sense of ownership of the system among users. To maintain trust with these users, strong organisational support and timely resolution of feedback is required. However, different approaches may be required to support users who do not have the capacity to be engaged in co-design processes and may be less likely to engage further if the system fails to meet their expectations on first use.
Presentation summary:
This presentation will highlight the key challenges involved in building clinicians’ trust in CDS systems, drawing on insights from our case study. We will describe lessons learnt and offer recommendations on how to enhance trust in CDS during and beyond implementation. The presentation will emphasise the importance of co-design and rapid issue resolution, tailored support structures, and a deep understanding of the unique requirements of different clinical settings and user groups. These insights are intended to aid healthcare organisations and developers in designing and implementing CDS systems that align more closely with clinicians’ needs, thereby enhancing adoption and long-term success.
Event Type
Poster Presentation
TimeTuesday, April 14:45pm - 6:15pm EDT
LocationFrontenac Foyer