Prediction of hospital-acquired influenza using machine learning algorithms: a comparative study

This was a retrospective, observational, single-centre study using EMR data. The dataset was obtained from the Yonsei University Health System, a tertiary teaching hospital in Seoul, South Korea. The study was conducted in 2022 and encompassed the influenza seasons spanning from 2011 to 2012 to 2019–2020, covering the months from October to April of the subsequent year. The exclusion of March and April 2020 from the 2019–2020 season was justified by the onset of the COVID-19 pandemic in March 2020.

The sample consisted of patients aged 19 years and older, who had stayed in the general adult wards for more than four days. Patients solely diagnosed with influenza and showing a positive polymerase chain reaction (PCR) test within four days of admission were excluded because of their classification as cases of CAI infections. Patients who had undergone surgery during admission were also excluded. In total, 189,321 patients were included in the study, comprising 117 HAI patients and 182,204 non-HAI patients (Fig. 1). Patients with negative PCR results were typically categorized as non-HAI cases. However, given that these individuals underwent testing because they exhibited symptoms and considering the inherent non-100% accuracy of the test, it is possible that some of them could indeed be HAI cases. To mitigate this uncertainty, patients were excluded from the analysis to prevent any potentially skewed impact on the training of the predictive model.

The outcome variable was the presence of HAI. HAI patients were defined as those with a positive result from an influenza A or B PCR test conducted more than four days after admission. Patients who did not undergo PCR were categorized as non-HAI.

The predictor variables were chosen based on an extensive literature review, considering the factors influencing influenza. General characteristics included sex [23], age [1, 3, 8, 23‐30], body mass index (BMI) [11], pregnancy status [3, 11], smoking history (past or present) [31], immunosuppression status [1‐4, 8, 23, 32], and corticosteroid use [33] (Appendix Table A.1). Comorbidities were ascribed if patients had received diagnoses of diabetes [2, 8, 9], obesity [11], heart disease [2, 8, 11, 23], liver disease [9, 11], renal disease [2, 8, 11, 32], hematologic disease [3, 11, 23], malignancy [1, 4, 9], organ transplantation [1], asthma [11], or chronic obstructive pulmonary disease (COPD) [8, 9, 11, 32] before the index date. This study applied the method of means and changes from previous values [34] to transform vital signs, including body temperature (BT), heart rate (HR), respiration rate (RR), systolic blood pressure (SBP), and diastolic blood pressure (DBP) [35].

Laboratory results [23] and haematological inflammatory parameters, specifically the neutrophil-to-lymphocyte ratio (NLR), platelet-to-neutrophil ratio (PNR), and platelet-to-lymphocyte ratio (PLR) [36], were included. The radiological results consisted of selected chest X-ray findings [23]. Patient rooms and units were included as factors because the type of hospital room [24, 37] and sharing a room or unit with an influenza patient [9, 38] are risk factors for HAI infection.

The observation period for each patient spanned four days before the index date, considering the incubation period of influenza [39]. The index date corresponded to the PCR test date [32, 35], except for patients who did not undergo PCR testing, for whom the index date was established on the fifth day after admission.

Among these were no laboratory results for 108,590 patients, while 205 had missing smoking or BMI information, and 11 had no diagnostic information. Finally, 108,806 patients were excluded (Fig. 1). This resulted in the remaining 73,859 patients, of whom 111 exhibited HAI. In cases where certain laboratory test results were missing, the following approach was adopted despite the presence of other results. Due to an absence rate of 80.8% among the patients, the direct bilirubin variable was removed. For other laboratory results, the missing rates were less than 5%, including calcium (4.6%), total bilirubin (3.7%), alanine transaminase (ALT; 2%), albumin (1.1%), aspartate transaminase (AST; 0.9%), blood urea nitrogen (BUN; 0.4%), creatinine (0.3%), and CO2 (0.02%). Consequently, imputation was employed to address missing data for laboratory test variables. The absence of laboratory test results indicated that the attending physician did not consider the test necessary for the patient; therefore, missing laboratory test results were not considered abnormal [40]. Continuous laboratory variables were imputed using the median values within the normal range.

Of the 73,859 patients included in this study, only 111 (0.15%) were diagnosed with HAI, which resulted in an unbalanced dataset. Imbalanced classes are common in real-world healthcare data and can diminish the predictive efficacy of models [41]. To address this issue, a synthetic minority oversampling technique (SMOTE) was employed, which involves generating new and reasonably accurate data based on existing minority cases [41]. SMOTE generates data by computing the Euclidean distance between any two randomly selected k-nearest neighbours (KNN) from two minority samples and creating new data points along the line connecting them [41].

Feature selection is a prevalent technique in forecasting, pattern recognition, and classification modelling, designed to reduce the dimensionality and complexity of datasets by eliminating irrelevant and redundant features [42]. Various methods, including Information GainRatio Attribute Evaluation (GA), Forward Elimination, Backward Elimination, and One Rule Attribute Evaluation (ORAE), have been proposed for selecting pertinent features in predictive modelling [43]. In this study, we employed RFECV (Recursive Feature Elimination with Cross-Validation), a form of Backward Elimination, utilising a random forest classifier as the estimator with accuracy as the scoring metric, for feature selection. As a result, the following 36 variables were retained, encompassing features such as age, sex, BMI, malignancy, BT, HR, RR, SBP, DBP, red blood cell (RBC), haemoglobin (Hb), white blood cell (WBC), platelet, haematocrit, RDW, delta neutrophil index (DNI), neutrophil, lymphocyte, NLR, PNR, PLR, sodium, potassium, chloride (Cl), CO2, calcium, albumin, total bilirubin, BUN, creatinine, ALT, AST, normal chest X-ray, abnormal chest X-ray, multi-occupancy room, and double room (variables marked with an asterisk in Appendix Table A.1).

After processing the raw data, 53 variables were categorized into seven groups (see Appendix Table A.1). Descriptive and univariate analyses were performed to determine the characteristics and factors associated with HAI. Chi-square and t-tests were used to analysed categorical and continuous variables, respectively.

To develop prediction models for HAI, a combination of ML classification methods, including Random Forest (RF), Extreme Gradient Boosting (XGB), Artificial Neural Networks (ANN), and Logistic Regression (LR), was employed with the selected 36 variables. LR, widely utilized for predicting patient outcomes, such as mortality or disease onset, was juxtaposed with ML methods in healthcare data analysis studies [16]. RF is an ensemble model of decision trees that amalgamates multiple weak classifier models into a robust model that outperforms individual components [44]. Decision-tree algorithms can be sensitive to minor cases in datasets; however, RF mitigates this by aggregating the outcomes of various decision trees [45]. Despite their longer training times, straightforward ensemble models exhibit noteworthy performance [44, 46]. XGB builds on the gradient boosting model, known for its reliability but has a prolonged training period. XGB considerably reduces this training duration, rendering it one of the most advanced supervised ML algorithms and faster than other ensemble classifiers [44]. ANNs possess significant predictive capability among classification algorithms and are extensively employed. The transparency and interpretability of models hold significance within healthcare [16] to explicate the rationale underlying outcomes. Despite their limitations in interpretability, ANNs have demonstrated robust predictive properties.

Five-fold grid search cross-validation (GSCV) was performed on the training set. GSCV identifies the optimal combination of hyperparameters that enhances model performance while preventing overfitting [44]. The optimized hyperparameters for each ML model examined in this study were as follows. The RF model featured a maximum depth of 20 m, a minimum of two sample splits, and 100 n estimators. The XGB model had a maximum depth of 5, a learning rate of 0.2, a subsample of 0.75, and 10 n estimators. The ANN model comprised 50 and 100 activation-rectified linear units, a hidden layer size of 50, a learning rate of 0.005, and an Adam solver.

It is imperative that the models not be trained or evaluated using the same dataset to ascertain their accuracy [47]. In this study, 80% of the dataset was randomly assigned to the training set, and the remaining 20% was assigned to the test set. No variables showed significant differences between the training and test sets (see Appendix Table A.2). The assessment of the discriminatory ability of a classification model involves metrics such as accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) [48]. In this study, particular emphasis was placed on the AUC and the number of false negatives (FN). AUC, the most commonly used metric for evaluating prediction models and FN count, is crucial in healthcare as it signifies untreated patients potentially spreading the virus and is deemed paramount. In addition, SHAP (SHapley Additive exPlanations) was employed to assess feature importance by utilizing Shapley values [49]. This methodology considers contributions across all possible combinations for fair attribution, accommodating feature interactions and enabling a more accurate evaluation of individual feature importance [49]. SHAP is versatile, applicable to diverse machine learning models, including regression, classification, and ensemble models. Visualized through a dot plot, the results depict Shapley values for each feature, offering an intuitive understanding of their impact on model predictions. Positive values indicate contributions that increase predictions, while negative values suggest contributions that decrease predictions. This analysis provides clear insights into the most influential features, contributing valuable information for a quantitative interpretation of the model’s feature importance [49].

Data analysis was performed using SQL Server Management Studio v18.10 (Microsoft, Seattle, US) and Python 3.5. SQL was used to integrate, preprocess, and transform the data. Python was used for the univariate analyses and ML.

This study was approved by the Yonsei University Health System Institutional Review Board (IRB No. 4-2021-1252) and Data Review Board (DRB No. 2,021,300,331). After obtaining approval, the data were extracted and anonymized by authorized personnel from the hospital’s records management department before being sent to the researcher.