Alejandro Huete-Garcia; Sara Rodriguez-Lopez

doi:10.4266/acc.2024.00759

Articles

Page Path: HOME > Acute Crit Care > Volume 39(4); 2024 > Article

Original Article Rapid response system Development and implementation of an artificial intelligence–enhanced care model to improve patient safety in hospital wards in Spain: Alejandro Huete-Garcia^1,2, Sara Rodriguez-Lopez³; Acute and Critical Care 2024;39(4):488-498.
DOI: https://doi.org/10.4266/acc.2024.00759
Published online: November 18, 2024

¹Servicio de Medicina Intensiva, H.U. Torrejón, Servicio de Salud de Madrid, Madrid, Spain

²Servicio de Urgencias Médicas de Madrid-SUMMA112, Servicio de Salud de Madrid, Madrid, Spain

³Servicio de Cardiologia, H.U. Puerta de Hierro-Majadahonda, Servicio de Salud de Madrid, Madrid, Spain

Corresponding author: Alejandro Huete-García Servicio de Urgencias Médicas de Madrid-SUMMA112, Servicio de Salud de Madrid, Calle Antracita 2, Madrid 28001, Spain Tel: +34-61-889-5195 E-mail: alejandro.huete@salud.madrid.org

• Received: June 19, 2024 • Revised: July 24, 2024 • Accepted: August 15, 2024

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

290 Views
22 Download

prev next

Full Article

Download PDF

Abstract
INTRODUCTION
MATERIALS AND METHODS
RESULTS
DISCUSSION
KEY MESSAGES
NOTES
REFERENCES

Abstract

Background
Early detection of critical events in hospitalized patients improves clinical outcomes and reduces mortality rates. Traditional early warning score systems, such as the National Early Warning Score 2 (NEWS2), effectively identify at-risk patients. Integrating artificial intelligence (AI) could enhance the predictive accuracy and operational efficiency of such systems. The study describes the development and implementation of an AI-enhanced early warning system based on a modified NEWS2 scale with laboratory parameters (mNEWS2-Lab) and evaluates its ability to improve patient safety in hospital wards.
Methods
For this retrospective cohort study of 3,790 adults admitted to hospital wards, data were collected before and after implementing the mNEWS2-Lab protocol with and without AI enhancement. The study used a multivariate prediction model with statistical analyses such as Fisher's chi-square test, relative risk (RR), RR reduction, and various AI models (logistic regression, decision trees, neural networks). The economic cost of the intervention was also analyzed.
Results
The mNEWS2-Lab reduced critical events from 6.15% to 2.15% (RR, 0.35; P<0.001), representing a 65% risk reduction. AI integration further reduced events to 1.59% (RR, 0.26; P<0.001) indicating a 10% additional risk reduction and enhancing early warning accuracy by 15%. The intervention was cost-effective, resulting in substantial savings by reducing critical events in hospitalized patients.
Conclusions
The mNEWS2-Lab scale, particularly when integrated with AI models, is a powerful and cost-effective tool for the early detection and prevention of critical events in hospitalized patients.
Keywords: artificial intelligence; early warning score; hospital costs; hospital mortality; patient safety

INTRODUCTION

Many hospital deaths are potentially predictable and preventable. Eighty percent of patients admitted to hospitals show basic vital sign abnormalities 6–24 hours before cardiac arrest, and 50% received suboptimal care before intensive care unit (ICU) admission [1,2]. Moreover, 41% of ICU admissions were potentially avoidable [3]. Unplanned ICU admissions, in-hospital cardiac arrests, and unexpected patient deaths are critical events (CEs) in hospital care [4].

Early detection of CEs in patients is crucial for improving clinical outcomes and reducing hospital mortality rates [5,6]. Improving patient safety involves implementing proactive models throughout the hospital based on an early warning score (EWS) system. These systems are intended to detect critical or potentially critical patients early to enable early and appropriate care to meet their needs [7,8]. Some examples of EWSs are: National Early Warning Score 2 (NEWS2) [9], Critical Care Outreach Team [3,6], and ICU without walls [10,11]. EWS systems have been proven effective, but integrating artificial intelligence (AI) could take this methodology a step further by providing faster and more accurate analyses [12]. This study describes the development and implementation of a single-hospital EWS, a modified NEWS2 scale that includes critical clinical laboratory blood test results (modified NEWS2 scale with clinical laboratory parameters [mNEWS2-Lab]) and demonstrates how it reduced CEs and how AI enhanced those results [13,14].

We designed and developed an in-hospital system for surveillance, assessment, and early warning of severity based on the mNEWS2-Lab scale. The process begins with nursing surveillance and detection, where routine observations are conducted, and staff and family concerns are noted using the mNEWS2-Lab scale (Figure 1). Upon CE detection, the responsible physician and on-duty physician are notified for assessment. Activation of the response depends on the mNEWS2-Lab score: for scores between 1 and 6, an evaluation is performed within 15 minutes; for scores of 7 or more, an immediate evaluation is conducted, and the ICU team is called. The responsible physician and the intensivist then carry out a joint evaluation, provide initial treatment, and determine the best therapeutic place for the patient. The system emphasizes early warning through continuous surveillance, thorough assessment, and an early severity alert system to ensure timely and appropriate care for critical patients.

The mNEWS2-Lab score triggers an alert in the nursing comment section of the electronic record. The nurse does not have to calculate anything because the electronic record program itself integrates the automatically collected vital signs with the results of the patient’s latest analysis, or with only the vital signs if no recent analysis is available. To ensure agile, clear, and effective exchanges of patient information among healthcare professionals, the transmission of information must use the ISOBAR (Identify, Situation, Observation, Backround, Agreed plan, Read back) structured communication tool.

The objectives of this initiative are to positively influence the clinical course and prognosis of patients by standardizing the evaluation and care response for disease, to develop and implement the mNEWS2-Lab early warning system to enhance the detection and care of critical patients in hospital wards, and to develop and validate an AI model that integrates with the mNEWS2-Lab system to further improve the accuracy and timeliness of early warnings for critical patients.

MATERIALS AND METHODS

Because this study did not involve any direct interventions with patients or experimental procedures with human subjects, the Research Ethics Committee of the involved institution declined to issue a formal report. The research focused on the retrospective analysis of anonymized clinical data previously collected in electronic records, which did not require obtaining informed consent from patients. Therefore, an exemption from IRB/IACUC approval was granted for this study.

This retrospective cohort study evaluates the implementation of the mNEWS2-Lab protocol. The study included adult patients admitted to hospital wards, with data collected pre-implementation (April 3–26, 2018) and post-implementation (December 1, 2018–December 1, 2021). Following the transfer of the lead researcher to a new institution, the protocol was re-evaluated and enhanced with AI in December 2023. This enhancement included integration into the electronic health record as an alert system. The AI-enhanced system was implemented on December 15, 2023, and data on its effectiveness were collected from December 15, 2023, to April 15, 2024.

Inclusion and Exclusion Criteria

The inclusion criteria were adult patients (older than 18 years) admitted to hospital wards who did not meet the exclusion criteria. The exclusion criteria were admission to a psychiatric ward, admission to an obstetric ward, admission to an ICU, admission to the emergency department, and admission to home hospitalization. Figure 2 illustrates the patient selection and analysis process, providing a visual representation of the criteria and steps followed for inclusion in this study.

Care model design

The care model design consisted of a scale design, an impact study, and a validity and efficiency study. The key variables were vital signs, critical laboratory parameters, and CEs. The statistical analyses used Fisher's chi-square test or Student t-test and logistic regression models.

Scale design

The cohort study analyzed the following variables. (1) Vital signs: oxygen saturation as measured by pulse oximetry (SpO₂), respiratory rate, blood pressure, heart rate, temperature. (2) Laboratory parameters: elevated white blood cell count, C-reactive protein, procalcitonin, troponin T, pH, partial pressure of carbon dioxide (pCO₂), lactate. (3) Other variables: level of consciousness, need for oxygen therapy, CEs, and the intervention group. For this study, Fisher's chi-square test was used, and relative risk (RR) and RR reduction were calculated. The impact study calculated sensitivity, specificity, positive predictive value, and negative predictive value, along with receiver operating characteristic (ROC) curves and economic cost analyses. The validity study maintained a confidence level of 95% for all cases.

Methodology for the AI Model

AI model development used several software tools and libraries, all of which are open-access to ensure transparency and reproducibility. (1) Python: the primary programming language used for developing the AI model, Python is widely used in the data science community due to its readability and extensive library support [15]. (2) Pandas: used for data manipulation and analysis, Pandas provides the data structures and functions needed to clean and process data efficiently [16]. (3) NumPy: a fundamental package for scientific computing with Python, NumPy supports large, multi-dimensional arrays and matrices, along with a collection of mathematical functions that operate on those arrays [17]. (4) Scikit-learn: this library was used to implement various machine learning algorithms, including logistic regression, decision trees, and ensemble models. Scikit-learn also provides tools for model evaluation and validation [18]. (5) TensorFlow/Keras: to develop and train the neural network models, TensorFlow and its high-level application programming interface Keras were used. These libraries are particularly good at handling complex neural network architectures and large datasets [19]. (6) SMOTE (synthetic minority over-sampling technique): implemented via the "imbalanced-learn" library to address class imbalance in the dataset by generating synthetic samples for the minority class [20].

Data preprocessing includes (1) data cleaning, which involves the removal of duplicate records and handling of missing values through mean or median imputation techniques; (2) normalization, where variables are scaled to ensure all features are within the same range using min-max normalization to enhance the performance of machine learning models; and (3) class balancing, which addresses class imbalance using SMOTE to prevent the model from being biased toward the majority class. The dataset was split into training (70%) and testing (30%) sets to train and validate the model. The training process involved the following steps. (1) Logistic regression: used as a baseline model to predict the probability of binary outcomes based on one or more predictors [20]. (2) Decision trees: these models split data into branches based on input features, facilitating the interpretation and visualization of decisions [21]. (3) Neural networks: deep learning models were used to capture complex non-linear relationships among variables. TensorFlow and Keras were used to build and train these networks [22]. (4) Ensemble models: several models were combined (stacked) to improve overall performance by leveraging the strengths of each algorithm [23].

The models were evaluated using the following metrics. Precision: the proportion of all positive predictions that was correct. Sensitivity (recall): the model's ability to correctly identify CEs. Specificity: the model's ability to correctly identify non-CEs. Area under the ROC curve (AUROC): The model's ability to distinguish among classes.

The trained AI models were deployed using open-source frameworks and tools to ensure accessibility. (1) Flask: a lightweight web framework for deploying machine learning models as web services. It enabled the integration of the AI model into existing hospital information systems [24]. (2) Docker: used to containerize an application, Docker ensures that a model can run consistently across different environments by packaging the code and dependencies together [25]. (3) GitHub: the codebase, including preprocessing scripts, model training code, and deployment scripts, is available on GitHub under an open-access license, promoting transparency and enabling other researchers to replicate and build upon this work [26].

RESULTS

Patient Demographics

This study analyzed data from 3,790 patients with an average age of 65±21 years. Among them, 56.3% were female, and 60.1% were surgical patients (n=2,277). During the study period, there were 94 CEs, accounting for 3.7% of the cohort (Table 1).

Design of the MNEWS2-Lab Scale

The appearance of CEs was significantly associated with vital signs and critical laboratory values (Table 2). The multivariate prediction model identified several key variables: SpO₂, respiratory rate, need for oxygen therapy, blood pressure, heart rate, level of consciousness, and temperature. These variables emerged as strong predictors of CE risk, with high Exp(β) values indicating their substantial contributions (Table 3). Figure 3 represents the mNEWS2-Lab scale as it was developed from the logistic regression models.

Validity of the MNEWS2-Lab Scale

For high-risk patients (mNEWS2-Lab score of 7 or more), the sensitivity of the mNEWS2-Lab score was 89.7%, and the specificity was 97.9%. The scale showed a high negative predictive value (99.2%) but only a moderate positive predictive value (75.5%). The AUROC for the mNEWS2-Lab scale was 0.938, indicating excellent discriminative ability.

Results of the AI Model

The performance of the various AI models tested with the mNEWS2-Lab system is summarized in Table 4. The AI simulations indicated that integrating an AI model with the mNEWS2-Lab system would significantly improve early CE detection, resulting in a 15% increase in the precision of early warnings and an additional 10% reduction in the incidence of CEs post-implementation.

Impact and Outcomes of mNEWS2-Lab System Implementation and AI Enhancement

The implementation of the mNEWS2-Lab system led to a significant reduction in CEs among hospitalized patients (Table 5). Initially, the incidence of CEs decreased from 6.15% pre-implementation to 2.15% post-implementation, which corresponds to an RR of 0.35 or a 65% risk reduction (P<0.001). With the introduction of the AI-enhanced mNEWS2-Lab system, the incidence of CEs was further reduced to 1.59%, with an RR of 0.26 indicating a 74% risk reduction (P<0.001). These results confirm the projections from the AI-simulated models, which predicted a 10% additional reduction in the incidence of CEs (P<0.001) and a 15% increase in the precision of early warnings. Overall, this intervention achieved a 74% reduction in the incidence of CEs in hospitalized patients.

Table 6 illustrates the distribution of patients according to the mNEWS2-Lab score. After implementing the mNEWS2-Lab system, there was a significant increase in patients classified as "none," which rose further with AI integration, indicating better identification of non-risk patients. The "very low" risk group remained stable initially but decreased with AI, which improved risk stratification and reduced false positives. The "low" risk group decreased and stabilized, showing better discrimination. The "intermediate" risk group decreased and remained stable, but the “high-risk” group decreased significantly, showing enhanced management and reduced CEs. Significant differences were noted across all risk levels, particularly in the high-risk group of patients with scores of 7 or more. Overall, the system improved patient risk stratification and management, reducing CEs.

Cost Analysis

The intervention was cost-effective. The initial economic cost in 2019 was 60.00 Euros (€), and that increased to €8,000 in 2023 with the integration of the AI models developed in Python, including the salaries of the system engineers and data scientists. Although the specific costs for system integration and personnel were managed by the institution, the overall expenditure was offset by the substantial reduction in CEs, demonstrating the economic and clinical value of the enhanced mNEWS2-Lab system.

As detailed in Table 7, we calculated the costs associated with CEs across the different phases of the study. According to data provided by the Servicio Madrileño de Salud, the average daily cost for general wards is €986.00, and that for the ICU is €1933.00. Based on observations from our institution, each CE typically results in an average of 5 additional days in the ICU and 10 additional days in the general ward, reflecting the severity of CEs and the required recovery period.

During the pre-mNEWS2-Lab phase (1 month), 22 CEs cost €19,525 each, totaling €429,550, or €5,146,600 annually. In the post-mNEWS2-Lab phase (3 years), 65 CEs cost €1,269,125, or €423,042 annually, saving €4,723,558 compared with pre-implementation. The post-AI phase (4 months) had 7 CEs costing €136,675, or €410,025 annually, yielding savings of €4,736,575 compared with pre-implementation. Thus, despite the higher initial costs, the mNEWS2-Lab system and its AI enhancement offer significant financial benefits by reducing ICU admissions and hospital stays, making the initial investment cost-effective.

DISCUSSION

This cohort study, encompassing 3,790 patients with an average age of 64.46 years, provides insightful findings on the effectiveness of the mNEWS2-Lab scale and the integration of AI models in predicting and reducing CEs in patients hospitalized in general wards. During the pre-implementation period, vital signs were collected every 8 hours from patients admitted over a 1-month span, resulting in a cohort of 358 patients. This sample size was adequate for developing the mNEWS2-Lab scale because vital signs are crucial for this process. The post-implementation phase lasted 3 years and included 3,117 patients, allowing for a comprehensive evaluation of the mNEWS2-Lab system. The AI-enhanced mNEWS2-Lab system was implemented on December 15, 2023, and its effectiveness was monitored for 4 months, resulting in a smaller cohort than in the post-implementation phase, but the available data provide valuable initial insights into the system’s performance. The differences in patient numbers across these phases are attributed to the varying durations of data collection. These variations in cohort size are inherent to the study design and do not compromise the validity of the findings.

The implementation of the mNEWS2-Lab system revealed a significant association between the occurrence of CEs and various vital signs and critical laboratory values. The multivariate prediction model identified several key variables as strong predictors of CE risk: SpO₂, respiratory rate, need for oxygen therapy, blood pressure, heart rate, level of consciousness, and temperature. For instance, the SpO₂ value had an Exp(β) of 22.328, suggesting that decreased oxygen saturation considerably increases the risk of CEs [1,4,7]. This aligns with existing literature that emphasizes the importance of oxygen saturation and respiratory rate as critical indicators of patient deterioration [1,2,7].

The impact of the mNEWS2-Lab system on patient outcomes was profound. Before the implementation, the incidence of CEs was 6.31%, which decreased significantly to 2.1% post-implementation. This 66% reduction underscores the system’s effectiveness in enabling early detection and timely intervention for at-risk patients. Furthermore, the integration of the AI-enhanced mNEWS2-Lab system reduced the incidence of CEs to 1.59%, highlighting an additional improvement in patient outcomes. The stratification of patients into different risk categories further enabled targeted monitoring and intervention, enhancing overall patient management [2,3,5].

The validity metrics of the mNEWS2-Lab scale are impressive. With a sensitivity of 89.7%, the scale effectively identified a high proportion of patients at risk for CEs. Its specificity of 97.9% indicated a low rate of false positives, which is crucial in preventing unnecessary alarms and interventions. The positive predictive value of 75.5% and negative predictive value of 99.2% further validate the scale's reliability, confirming its utility in a clinical setting [1,3,4]. These metrics are essential for ensuring that the mNEWS2-Lab scale can be confidently used to guide clinical decision-making and resource allocation [2,5].

The economic analysis highlights the cost-effectiveness of the intervention. The initial economic cost of the intervention in 2019 was €60.00, which increased to €8,000.00 in 2023 with the integration of AI. Despite the higher cost, the significant reduction in CEs and the associated decrease in critical care costs demonstrate the financial viability of the mNEWS2-Lab system [6,8]. The annualized cost of CEs dropped from €5,146,600 in the pre-implementation phase to €423,042 in the post-implementation phase over 3 years, and it further decreased to €410,025 in the first 4 months following the AI enhancement. These reductions led to substantial annual savings of approximately €4.7 million compared with the pre-implementation period, with reduced ICU admissions and shorter hospital stays offsetting the initial implementation costs and making it an economically sound investment for healthcare facilities [6,8].

The integration of AI models with the mNEWS2-Lab system has added a new dimension to patient monitoring and prediction. Among the AI models, the neural networks had an AUROC of 0.92, indicating predictive performance superior to that of the traditional logistic regression (AUROC, 0.85) and decision trees (AUROC, 0.88). The AI-enhanced system improved the precision of early warnings by 15% and further reduced the incidence of CEs by 10% [9,12-14]. These findings suggest that AI integration can significantly enhance the predictive accuracy and operational efficiency of early warning systems [9,10,13].

AI models’ ability to continuously learn and adapt based on new data makes them a dynamic and robust tool for predicting patient deterioration. The combination of AI and the mNEWS2-Lab system facilitated more precise and timely interventions, thereby improving patient outcomes and optimizing resource use. The implementation of such integrated systems in clinical practice could revolutionize patient care by offering more personalized and accurate health monitoring [10,12,13]. These projected improvements were confirmed during this study, which has validated the effectiveness of the AI-enhanced system in a real-world clinical setting.

The AI system improved CE detection by using advanced algorithms to analyze large volumes of patient data in real time, identifying patterns and correlations that traditional methods might miss. This capability allows for the detection of subtle changes in vital signs before CEs occur. The AI’s ability to continuously learn from new data further enhances its predictive accuracy. By integrating AI with the mNEWS2-Lab system, we achieved more accurate early warnings and timely interventions, thus significantly improving patient safety and clinical outcomes.

This study’s findings have substantial implications for clinical practice. The significant reduction in CE incidence and the high predictive validity of the mNEWS2-Lab scale support its adoption as a standard early warning tool for hospitalized patients. The scale's integration with AI models further enhanced its utility, offering a more sophisticated and reliable method for the early detection of patient deterioration [1,2,4]. Healthcare facilities could benefit from implementing the mNEWS2-Lab system, particularly when integrated with AI models. This approach not only improves patient safety and outcomes but also enhances clinical workflow efficiency by providing timely alerts and reducing unnecessary interventions. Training healthcare professionals to effectively use these tools and interpret the results will be crucial for maximizing their potential benefits [2,5,7].

Despite the robust evidence supporting the efficacy of the mNEWS2-Lab system, several limitations should be acknowledged. This study was conducted in a single healthcare setting, potentially limiting the generalizability of the findings. Future research should aim to validate these results across diverse clinical environments and patient populations to ensure broad applicability. Additionally, long-term studies are needed to assess the sustained effects of the mNEWS2-Lab system and AI integration on patient outcomes and healthcare costs [3,6,8]. Additionally, the post-implementation period for the AI-enhanced mNEWS2-Lab protocol ran from December 2023 to April 2024, so the outcomes of this third phase should be closely monitored and evaluated. Another limitation is the potential for variability in the implementation and adherence to the mNEWS2-Lab protocols across different healthcare settings. Standardized training to ensure consistent application of the system will be critical for achieving the desired outcomes [2,5]. Future research should also explore the integration of other predictive variables and advanced AI techniques to further enhance the system's accuracy and predictive power. Investigating how real-time data integration and continuous monitoring affect patient outcomes could provide valuable insights for the future development of early warning systems [9,10,12].

In conclusion, the mNEWS2-Lab scale, particularly when integrated with AI models, is a powerful tool for the early detection and prevention of critical health events in hospitalized patients. Its implementation can lead to significant improvements in patient safety, clinical outcomes, and healthcare efficiency. Embracing these advances in early warning systems holds promise for transforming patient care and enhancing the overall quality of healthcare services [1,3,7]. This study has demonstrated that the mNEWS2-Lab scale is highly effective in predicting and reducing CEs in hospitalized patients. The significant reduction in CE incidence from 6.15% pre-implementation to 2.15% post-implementation underscores the scale's ability to enhance early detection and timely intervention for at-risk patients.

Integrating AI models with the mNEWS2-Lab system further improved its predictive accuracy and operational efficiency. The AI-enhanced system reduced the incidence of CEs by an additional 10% and increased the early warning precision by 15%. These findings highlight the potential of AI technologies to augment traditional early warning systems, leading to better patient outcomes and optimized resource utilization. The mNEWS2-Lab scale demonstrated high sensitivity (89.7%) and specificity (97.9%), along with a strong positive predictive value (75.5%) and negative predictive value (99.2%), confirming its reliability as a clinical tool. Its implementation proved to be cost-effective, with an initial economic cost of €60.00 in 2019 that increased to €8,000 in 2023 with the integration of AI. Despite the higher cost, the intervention resulted in substantial savings of approximately €4.7 million by reducing CEs in hospitalized patients, demonstrating the economic and clinical value of the enhanced mNEWS2-Lab system.

This study's findings support the adoption of the mNEWS2-Lab system as a standard early warning tool in clinical practice, particularly when integrated with AI models. This approach not only enhances patient safety and clinical outcomes but also improves workflow efficiency by providing timely and accurate alerts. In conclusion, the mNEWS2-Lab scale, especially when enhanced with AI technologies, is a powerful and cost-effective tool for the early detection and prevention of critical health events in hospitalized patients. Its implementation has the potential to transform patient care, improve clinical outcomes, and enhance the overall quality of healthcare services. Future research should focus on validating these results across diverse healthcare settings and exploring additional predictive variables and advanced AI techniques to further enhance system accuracy.

KEY MESSAGES

▪ Improved Detection: Integrating artificial intelligence (AI) with Modified National Early Warning Score 2 with Laboratory Parameters significantly enhances the early detection of critical health events, reducing the incidence by an additional 10%.

▪ AI improves the precision of early warnings by 15%, leading to more timely and accurate interventions.

▪ Despite higher initial costs, the intervention results in substantial savings by reducing critical events and their associated healthcare costs.

NOTES

CONFLICT OF INTEREST

No potential conflict of interest relevant to this article was reported.

FUNDING

None.

ACKNOWLEDGMENTS

None.

AUTHOR CONTRIBUTIONS

Conceptualization: AHG. Data curation: all authors. Formal analysis: all authors. Methodology: all authors. Project administration: AHG. Visualization: all authors. Writing – original draft: all authors. Writing – review & editing: all authors. All authors read and agreed to the published version of the manuscript.

Figure 1.

In-hospital system for surveillance, assessment, and early warning of severity. mNEWS2-Lab: modified National Early Warning Score 2 with Laboratory Parameters; ICU: intensive care unit.

Figure 2.

Flowchart of the patient selection process. AI: artificial intelligence; mNEWS2-Lab: modified National Early Warning Score 2 with Laboratory Parameters.

Figure 3.

In-hospital system for surveillance, assessment, and early warning of severity using a modified National Early Warning Score 2 (NEWS2) scale that includes critical clinical laboratory blood test results (mNEWS2-Lab). SpO₂: oxygen saturation as measured by pulse oximetry; SPB: systolic blood pressure; pCO₂: partial pressure of carbon dioxide; CRP: C-reactive protein; ICU: intensive care unit.

Table 1.

Distribution of the study population

Variable	Total	Pre-mNEWS2-Lab	Post-mNEWS2-Lab	Post-AI enhanced mNEWS2-Lab
Patients	3,790	358	3,117	315
Total clinical records	45,780	3,120	42,660	3,916
Critical events	94	22	65	7

mNEWS2-Lab: modified National Early Warning Score 2 with Laboratory Parameters; AI: artificial intelligence.

Table 2.

Univariate prediction model for the risk of critical events in hospitalized patients

Variable	β	Standard error	Wald	P-value	Exp(β)
SpO₂	3.050	0.202	227.545	<0.001	21.126
Respiratory rate	2.271	0.161	198.680	<0.001	9.691
Oxygen therapy	1.970	0.124	252.206	<0.001	7.169
Blood pressure	1.287	0.212	36.937	<0.001	3.622
Heart rate	2.991	0.510	34.399	<0.001	19.896
Level of consciousness	1.337	0.107	156.039	<0.001	3.807
Temperature	1.635	0.269	37.003	<0.001	5.131
Laboratory	1.417	0.418	11.499	<0.001	4.124

SpO₂: oxygen saturation as measured by pulse oximetry.

Table 3.

Multivariate prediction model for the risk of critical events in hospitalized patients

Variable	β	Standard error	Wald	P-value	Exp(β)	95% CI
SpO₂	3.106	0.339	84.004	<0.001	22.328	11.492–43.381
Respiratory rate	2.228	0.242	84.585	<0.001	9.277	5.771–14.914
Oxygen therapy	1.819	0.146	155.228	<0.001	6.167	4.632–8.211
Blood pressure	2.346	0.365	41.412	<0.001	10.444	5.111–21.338
Heart rate	3.818	0.886	18.550	<0.001	45.499	8.008–258.529
Level of consciousness	1.288	0.171	56.492	<0.001	3.625	2.591–5.072

SpO₂: oxygen saturation as measured by pulse oximetry.

Table 4.

AI models performance

Model	AUROC	Precision (PPV)	Sensitivity	Specificity
Logistic regression	0.85	0.82	0.78	0.87
Decision trees	0.88	0.85	0.81	0.89
Neural networks	0.92	0.90	0.88	0.91
Ensemble models	Variable results depending on the combination of models and simulated data

AI: artificial intelligence; AUROC: area under the receiver operating characteristic curve; PPV: positive predictive value.

Table 5.

Critical events incidence and risk reduction analysis

Study phase	Patient	Critical events	Critical events incidence (%)	Relative risk vs. pre-implementation	Relative risk reduction (%)
Pre-implementation mNEWS2-Lab system	358	22	6.15	-	-
Post-implementation mNEWS2-Lab system	3,117	67	2.15	0.35	65^a)
Post-implementation AI enhanced mNEWS2-Lab system	315	5	1.59	0.26	74^a)

mNEWS2-Lab: modified National Early Warning Score 2 with Laboratory Parameters; AI: artificial intelligence.

^a)Statistical significance: P<0.001.

Table 6.

Patient distribution according to mNEWS2-Lab score

Risk level	MNEWS2-Lab score	Pre-MNEWS2-Lab	Post-MNEWS2-Lab	Post-AI enhanced mNEWS2-Lab
None	0	52 (14.5)	970 (31.1)^a)	125 (39.7)^a)
Very low	1–2	138 (38.6)	1199 (38.5)^a)	88 (28.0)^a)
Low	3–4	112 (31.3)	654 (21.0)^a)	67(21.3)^a)
Intermediate	5–6 or 3 in one item	31 (8.7)	196 (6.3)^a)	23 (7.3)^a)
High	7 or more	25 (7.0)	98 (3.2)^a)	12 (3.5)^a)

Values are presented as number (%).

mNEWS2-Lab: modified National Early Warning Score 2 with Laboratory Parameters; AI: artificial intelligence.

^a)Statistical significance, P<0.001.

Table 7.

Cost analysis of the mNEWS2-Lab system

Phase	Number of CEs	Total cost per CE (€)^a)	Total cost (€)	Annualized cost (€)^b)	Annual savings (€)^c)
Pre-MNEWS2-Lab (1 mo)	22	19,525.00	429,550.00	5,146,600.00	-
Post-MNEWS2-Lab (3 yr)	65	19,525.00	1,269,125.00	423,042.00	4,723,558.00
Post-AI enhanced mNEWS2-Lab (4 mo)	7	19,525.00	136,675.00	410,025.00	4,736,575.00

mNEWS2-Lab: modified National Early Warning Score 2 with Laboratory Parameters; CE: critical event; AI: artificial intelligence.

^a)Total cost per critical event: calculated as (5 days in ICU × €1933) + (10 days in general ward × €986) = €9,665.00 + €9,860.00 = €19,525;

^b)Annualized cost: for pre-MNEWS2-Lab, the 1-month cost is annualized by multiplying by 12. For post-MNEWS2-Lab, the total cost over 3 years is divided by 3. For post-AI enhancement MNEWS2-Lab, the 4-month cost is annualized by multiplying by 3.

^c)Annual savings: calculated by subtracting the annualized cost of each phase from the annualized cost of the pre-implementation phase.

REFERENCES

1. Sandroni C, Ferro G, Santangelo S, Tortora F, Mistura L, Cavallaro F, et al. In-hospital cardiac arrest: survival depends mainly on the effectiveness of the emergency response. Resuscitation 2004;62:291-7.Article PubMed
2. Schein RM, Hazday N, Pena M, Ruben BH, Sprung CL. Clinical antecedents to in-hospital cardiopulmonary arrest. Chest 1990;98:1388-92.Article PubMed
3. Goldhill DR, Worthington L, Mulcahy A, Tarling M, Sumner A. The patient-at-risk team: identifying and managing seriously ill ward patients. Anaesthesia 1999;54:853-60.Article PubMed PDF
4. Kazaure HS, Roman SA, Sosa JA. Cardiac arrest among surgical patients: an analysis of incidence, patient characteristics, and outcomes in ACS-NSQIP. JAMA Surg 2013;148:14-21.Article PubMed
5. Hillman KM, Bristow PJ, Chey T, Daffurn K, Jacques T, Norman SL, et al. Duration of life-threatening antecedents prior to intensive care admission. Intensive Care Med 2002;28:1629-34.Article PubMed PDF
6. Buist MD, Moore GE, Bernard SA, Waxman BP, Anderson JN, Nguyen TV. Effects of a medical emergency team on reduction of incidence of and mortality from unexpected cardiac arrests in hospital: preliminary study. BMJ 2002;324:387-90.Article PubMed PMC
7. Nguyen OK, Makam AN, Clark C, Zhang S, Xie B, Velasco F, et al. Vital signs are still vital: instability on discharge and the risk of post-discharge adverse outcomes. J Gen Intern Med 2017;32:42-8.Article PubMed PDF
8. McQuillan P, Pilkington S, Allan A, Taylor B, Short A, Morgan G, et al. Confidential inquiry into quality of care before admission to intensive care. BMJ 1998;316:1853-8.Article PubMed PMC
9. Royal College of Physicians. National Early Warning Score (NEWS) 2: standardising the assessment of acute-illness severity in the NHS [Internet]. Royal College of Physicians; 2020 [cited 2024 May 25]. Available from: https://www.rcp.ac.uk/media/umzn4ntq/news2_additional-guidance-002-_0.pdf
10. Abella A, Enciso V, Torrejón I, Hermosa C, Mozo T, Molina R, et al. Effect upon mortality of the extension to holidays and weekends of the “ICU without walls” project: a before-after study. Med Intensiva 2016;40:273-9.Article PubMed
11. Gordo F, Abella A. Intensive care unit without walls: seeking patient safety by improving the efficiency of the system. Med Intensiva 2014;38:438-43.Article PubMed
12. Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol 2017;2:230-43.Article PubMed PMC
13. Kostakis I, Smith GB, Prytherch D, Meredith P, Price C, Chauhan A, Portsmouth Academic ConsortIum For Investigating COVID-19 (PACIFIC-19). The performance of the National Early Warning Score and National Early Warning Score 2 in hospitalised patients infected by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Resuscitation 2021;159:150-7.Article PubMed
14. Morley J, Machado CC, Burr C, Cowls J, Joshi I, Taddeo M, et al. The ethics of AI in health care: a mapping review. Soc Sci Med 2020;260:113172. Article PubMed
15. Pandas documentation [Internet]. Pandas; 2024 [cited 2024 Aug 20]. Available from: https://pandas.pydata.org
16. NumPy documentation [Internet]. NumPy; 2024 [cited 2024 Aug 20]. Available from: https://numpy.org
17. Scikit-learn documentation [Internet]. Scikit-learn; 2024 [cited 2024 Aug 20]. Available from: https://scikit-learn.org/stable/
18. TensorFlow documentation [Internet]. TensorFlow; 2024 [cited 2024 Aug 20]. Available from: https://www.tensorflow.org
19. Imbalanced-learn documentation [Internet]. Imbalanced-learn; 2024 [cited 2024 Aug 20]. Available from: https://imbalanced-learn.org
20. Scikit-learn. Logistic regression [Internet]. Scikit-learn; 2024 [cited 2024 Aug 20]. Available from: https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
21. Scikit-learn. Decision trees [Internet]. Scikit-learn; 2024 [cited 2024 Aug 20]. Available from: https://scikit-learn.org/stable/modules/tree.html
22. Keras. Keras: deep learning for humans [Internet]. Keras; 2024 [cited 2024 Aug 20]. Available from: https://keras.io
23. Scikit-learn. Ensemble models [Internet]. Scikit-learn; 2024 [cited 2024 Aug 20]. Available from: https://scikit-learn.org/stable/modules/ensemble.html
24. Flask documentation [Internet]. Flask; 2024 [cited 2024 Aug 20]. Available from: https://flask.palletsprojects.com
25. Docker documentation [Internet]. Docker Inc.; 2024 [cited 2024 Aug 20]. Available from: https://www.docker.com
26. GitHub repository [Internet]. GitHub; 2024 [cited 2024 Aug 20]. Available from: https://github.com

Figure & Data

References

Citations

Citations to this article as recorded by

PubReader
ePub Link

Cite

CITE: export Copy Format; Close

Download Citation

Download a citation file in RIS format that can be imported by all major citation management software, including EndNote, ProCite, RefWorks, and Reference Manager.

Format:

RIS — For EndNote, ProCite, RefWorks, and most other reference management software
BibTeX — For JabRef, BibDesk, and other BibTeX-specific software

Include:

Citation for the content below
Citation and abstract for the content below

Development and implementation of an artificial intelligence–enhanced care model to improve patient safety in hospital wards in Spain

Acute Crit Care. 2024;39(4):488-498. Published online November 18, 2024

DOI: https://doi.org/10.4266/acc.2024.00759

XML Download

Figure

Development and implementation of an artificial intelligence–enhanced care model to improve patient safety in hospital wards in Spain

Figure 1. In-hospital system for surveillance, assessment, and early warning of severity. mNEWS2-Lab: modified National Early Warning Score 2 with Laboratory Parameters; ICU: intensive care unit.

Figure 2. Flowchart of the patient selection process. AI: artificial intelligence; mNEWS2-Lab: modified National Early Warning Score 2 with Laboratory Parameters.

Figure 3. In-hospital system for surveillance, assessment, and early warning of severity using a modified National Early Warning Score 2 (NEWS2) scale that includes critical clinical laboratory blood test results (mNEWS2-Lab). SpO2: oxygen saturation as measured by pulse oximetry; SPB: systolic blood pressure; pCO2: partial pressure of carbon dioxide; CRP: C-reactive protein; ICU: intensive care unit.

Figure 1.

Figure 2.

Figure 3.

Development and implementation of an artificial intelligence–enhanced care model to improve patient safety in hospital wards in Spain

Variable	Total	Pre-mNEWS2-Lab	Post-mNEWS2-Lab	Post-AI enhanced mNEWS2-Lab
Patients	3,790	358	3,117	315
Total clinical records	45,780	3,120	42,660	3,916
Critical events	94	22	65	7

Variable	β	Standard error	Wald	P-value	Exp(β)
SpO₂	3.050	0.202	227.545	<0.001	21.126
Respiratory rate	2.271	0.161	198.680	<0.001	9.691
Oxygen therapy	1.970	0.124	252.206	<0.001	7.169
Blood pressure	1.287	0.212	36.937	<0.001	3.622
Heart rate	2.991	0.510	34.399	<0.001	19.896
Level of consciousness	1.337	0.107	156.039	<0.001	3.807
Temperature	1.635	0.269	37.003	<0.001	5.131
Laboratory	1.417	0.418	11.499	<0.001	4.124

Variable	β	Standard error	Wald	P-value	Exp(β)	95% CI
SpO₂	3.106	0.339	84.004	<0.001	22.328	11.492–43.381
Respiratory rate	2.228	0.242	84.585	<0.001	9.277	5.771–14.914
Oxygen therapy	1.819	0.146	155.228	<0.001	6.167	4.632–8.211
Blood pressure	2.346	0.365	41.412	<0.001	10.444	5.111–21.338
Heart rate	3.818	0.886	18.550	<0.001	45.499	8.008–258.529
Level of consciousness	1.288	0.171	56.492	<0.001	3.625	2.591–5.072

Model	AUROC	Precision (PPV)	Sensitivity	Specificity
Logistic regression	0.85	0.82	0.78	0.87
Decision trees	0.88	0.85	0.81	0.89
Neural networks	0.92	0.90	0.88	0.91
Ensemble models	Variable results depending on the combination of models and simulated data

Study phase	Patient	Critical events	Critical events incidence (%)	Relative risk vs. pre-implementation	Relative risk reduction (%)
Pre-implementation mNEWS2-Lab system	358	22	6.15	-	-
Post-implementation mNEWS2-Lab system	3,117	67	2.15	0.35	65^a)
Post-implementation AI enhanced mNEWS2-Lab system	315	5	1.59	0.26	74^a)

Risk level	MNEWS2-Lab score	Pre-MNEWS2-Lab	Post-MNEWS2-Lab	Post-AI enhanced mNEWS2-Lab
None	0	52 (14.5)	970 (31.1)^a)	125 (39.7)^a)
Very low	1–2	138 (38.6)	1199 (38.5)^a)	88 (28.0)^a)
Low	3–4	112 (31.3)	654 (21.0)^a)	67(21.3)^a)
Intermediate	5–6 or 3 in one item	31 (8.7)	196 (6.3)^a)	23 (7.3)^a)
High	7 or more	25 (7.0)	98 (3.2)^a)	12 (3.5)^a)

Phase	Number of CEs	Total cost per CE (€)^a)	Total cost (€)	Annualized cost (€)^b)	Annual savings (€)^c)
Pre-MNEWS2-Lab (1 mo)	22	19,525.00	429,550.00	5,146,600.00	-
Post-MNEWS2-Lab (3 yr)	65	19,525.00	1,269,125.00	423,042.00	4,723,558.00
Post-AI enhanced mNEWS2-Lab (4 mo)	7	19,525.00	136,675.00	410,025.00	4,736,575.00

Table 1. Distribution of the study population

mNEWS2-Lab: modified National Early Warning Score 2 with Laboratory Parameters; AI: artificial intelligence.

Table 2. Univariate prediction model for the risk of critical events in hospitalized patients

SpO₂: oxygen saturation as measured by pulse oximetry.

Table 3. Multivariate prediction model for the risk of critical events in hospitalized patients

SpO₂: oxygen saturation as measured by pulse oximetry.

Table 4. AI models performance

AI: artificial intelligence; AUROC: area under the receiver operating characteristic curve; PPV: positive predictive value.

Table 5. Critical events incidence and risk reduction analysis

mNEWS2-Lab: modified National Early Warning Score 2 with Laboratory Parameters; AI: artificial intelligence.

Statistical significance: P<0.001.

Table 6. Patient distribution according to mNEWS2-Lab score

Values are presented as number (%).

mNEWS2-Lab: modified National Early Warning Score 2 with Laboratory Parameters; AI: artificial intelligence.

Statistical significance, P<0.001.

Table 7. Cost analysis of the mNEWS2-Lab system

mNEWS2-Lab: modified National Early Warning Score 2 with Laboratory Parameters; CE: critical event; AI: artificial intelligence.

Total cost per critical event: calculated as (5 days in ICU × €1933) + (10 days in general ward × €986) = €9,665.00 + €9,860.00 = €19,525;

Annualized cost: for pre-MNEWS2-Lab, the 1-month cost is annualized by multiplying by 12. For post-MNEWS2-Lab, the total cost over 3 years is divided by 3. For post-AI enhancement MNEWS2-Lab, the 4-month cost is annualized by multiplying by 3.

Annual savings: calculated by subtracting the annualized cost of each phase from the annualized cost of the pre-implementation phase.