Results

Study population

In total, 40 critical care registries were approached to participate. 21 registries expressed an initial interest in participation, of which 9 did not proceed. 12 registries representing 13 countries from five continents agreed to participate and completed the analysis. These are described in Supplementary Table 2. The original study populations for development, customisation and validation of the SMS-ICU score are presented in Supplementary Table 3 for comparison.11,14

In-hospital mortality analysis

Primary in-hospital mortality analysis

The study population for the in-hospital mortality analysis is described in Table 1. In total, 447,350 admissions were included from 512 ICUs in nine countries. Hospital mortality ranged from 8.1% to 21.2%. The median missingness rate for SMS-ICU variables was 0.1% (IQR 0–0.2%) (Table 1). Case-mix for each registry is described in Supplementary Table 4a–g.

Registry location | Aus & NZ | Brazil | Finland | India | Nepal | Netherlands | Uruguay
Number of ICUs included | 194 | 207 | 25 | 11 | 1 | 68 | 6
Total ICU admissions during study period | 197 713 | 180 268 | 16 460 | 7610 | 822 | 70 080 | 2240
Number of admissions included | 183 444 | 176 866 | 13 573 | 6615 | 621 | 64 163 | 2068
Number of exclusions | Total patients excluded | 14 269 | 3402 | 2887 | 995 | 201 | 5917 | 172
Age not over 18 years | 1592 | 1300 | 998 | 322 | 24 | 414 | 22
Readmission to ICU within same hospital encounter | 8245 | 0 | 1472 | 213 | 40 | 3163 | 0
Interhospital transfer | 3685 | 2102 | 416 | 77 | 34 | 2340 | 150
Missing outcome | 747 | 0 | 1 | 383 | 103 | 0 | 0
Median patient age (IQR) | 66 (52–76) | 65 (47–78) | 65 (53–73) | 60 (48–70) | 68 (51–79) | 66 (55–74) | 64 (49–74)
Median systolic blood pressure (IQR) | 97 (88–108) | 121 (105–140) | 90 (82–102) | 120 (110–140) | 130 (112–151) | 91 (80–105) | 120 (100–142)
Number of male patients (% of total) | 111 525 (56.4) | 85 743 (48.5) | 8578 (63.2) | 4064 (61.4) | 345 (55.6) | 42 839 (61.1) | 1225 (59.2)
Number of patients admitted after emergency surgery (% of total) | 28 933 (15.8) | 12 602 (7.1) | 2446 (18.0) | 87 (1.3) | 15 (2.4) | 7369 (11.5) | 224 (10.8)
Number of patients admitted after elective surgery (% of total) | 69 482 (37.9) | 31 899 (18.0) | 3552 (26.2) | 1019 (15.4) | 92 (14.8) | 22 615 (35.2) | 423 (20.4)
Number of patients receiving infused vasopressors/inotropes (% of total) | 70 527 (38.4) | 25 473 (14.4) | 8232 (60.7) | 1294 (19.6) | 67 (10.8) | 31 223 (48.7) | 461 (22.3)
Number of patients receiving NIV/CPAP (% of total) | 18 873 (10.3) | 20 583 (11.6) | 1050 (7.1) | 564 (10.4) | 30 (4.8) | NAb | NA
Number of patients receiving invasive mechanical ventilation (% of total) | 55 687 (30.4) | 12 602 (7.1) | 6100 (44.9) | 1457 (22.0) | 57 (9.2) | NA | 718 (34.7)
Number of patients receiving RRT (% of total) | 7043 (3.8) | 4889 (2.8) | 774 (4.7) | 283 (4.3) | 7 (1.1) | 2172 (3.4) | 65 (3.1)
Total number of in-hospital deaths (% of total) | 14 864 (8.1) | 22 382 (12.7) | 1636 (12.1) | 1022 (15.4) | 75 (12.1) | 8384 (13.1) | 439 (21.2)
Prevalence of missing data for study variables | Blood pressure (% of total) | 0 (0) | 2549 (1.4) | 210 (1.6) | 10 (0.15) | 0 (0) | 482 (0.8) | 84 (4.1)
Metastasis/haematological malignancy (% of total) | 0 (0) | 81 (0.05) | 692 (5.9) | 0 (0) | 0 (0) | 0 (0) | 3 (0.1)
Vasopressor/inotrope use (% of total) | 11 747 (6.4) | 204 (0.1) | 0 (0) | 10 (0.2) | 0 (0) | 0 (0) | 2 (0.1)
Respiratory support (% of total) | 0 (0) | 204 (0.1) | 1544 (11.4) | 10 (0.2) | 0 (0) | 0 (0) | 2 (0.1)
Renal replacement therapy (% of total) | 13 037 (7.1) | 204 (0.1) | 0 (0) | 10 (0.2) | 0 (0) | 36 195 (56.4)c | 2 (0.1)
Admission type (% of total) | 0 (0) | 0 (0) | 29 (0.2) | 0 (0) | 0 (0) | 201 (0.3) | 0 (0)

Discrimination of SMS-ICU, assessed using ROC curves, is shown for each registry in Fig. 1. AUROC met the pre-specified threshold of 0.7 in six of seven registries (Table 2). The pooled AUROC was 0.76 (95% CI 0.72–0.80).

Fig. 1: ROC curves (solid blue) presented by registry for the primary (in-hospital mortality) analysis. Sensitivity is plotted on the y axis, against (1 – specificity) on the x axis.

 | Aus & NZ | Brazil | Finland | India | Nepal | Netherlands | Uruguay
AUROC (95% CI)a | 0.79 (0.79–0.8) | 0.8 (0.8–0.81) | 0.78 (0.77–0.79) | 0.66 (0.64–0.68) | 0.75 (0.69–0.8) | 0.76 (0.76–0.77) | 0.79 (0.76–0.81)
Calibration slopeb | 0.9225 | 1.0095 | 0.9397 | 0.5536 | 1.045 | 0.8159 | 0.8445
Calibration interceptc | −1.2509 | −0.2486 | −1.0363 | −0.5829 | −0.0511 | −1.1667 | 0.0026
O/E ratio (95% CI)d | 0.42 (0.41–0.43) | 0.83 (0.82–0.84) | 0.5 (0.47–0.52) | 1.13 (1.07–1.18) | 0.91 (0.74–1.12) | 0.49 (0.48–0.51) | 1.12 (1.05–1.21)
Brier scoree | 0.0873 | 0.0916 | 0.113 | 0.125 | 0.0921 | 0.1282 | 0.136

Calibration varied between registries (Fig. 2, Table 2), with O/E ratios ranging from 0.42 to 1.13 (Table 2).

Fig. 2: Calibration curves for primary (in-hospital mortality) analysis. Observed probability (y-axis) is plotted against predicted probability (x-axis). Logistic calibration curves (solid black) and nonparametric smoothed (dotted) calibration curves are plotted.

Overall model fit is represented by Brier scores ranging from 0.087 to 0.136 (Table 2).

Sub-group in-hospital mortality analysis

Results from the pre-specified subgroup analyses are shown in Supplementary Table 5. For all subgroups, pooled AUROC point estimates were approximately 0.70 or higher (Fig. 3). The worst discrimination was observed in the invasively ventilated subgroup (AUROC 0.70 [0.65–0.74]). Score performance in each subgroup was either comparable to or marginally worse than in the total study population.

Fig. 3: A forest plot showing the pooled AUROC (95% CI) for each subgroup of the primary (in-hospital mortality) analysis. Pooled AUROCs were obtained using restricted maximum likelihood estimation for each subgroup. The area shaded orange is above the pre-specified threshold AUROC of 0.7, and the area shaded grey is below this threshold.

Sensitivity analyses

A complete case analysis and an analysis excluding all inward and outward interhospital transfers both showed comparable results to the primary analysis (Supplementary Table 6). Both sensitivity analyses yielded pooled AUROCs of 0.76 (0.72–0.80).

ICU mortality analysis

Main ICU mortality analysis

The study population for the ICU mortality analysis is described in Table 3. In total, 465,790 admissions were included from 546 ICUs in 13 countries. ICU mortality ranged from 5.3% to 56.8%. The median missingness rate for SMS-ICU variables was 0.1% (IQR 0–1.0%) (Table 3). Case-mix for each registry is described in Supplementary Table 7a-l.

Registry location | Aus & NZ | Bangladesh | Brazil | Ethiopia | Finland | Ghana | India | Malaysia | Nepal | Netherlands | South Africa | Uruguay
Number of ICUs included | 194 | 1 | 207 | 2 | 25 | 1 | 22 | 4 | 12 | 68 | 4 | 6
Total ICU admissions during study period | 197 713 | 1334 | 180 268 | 564 | 16 460 | 209 | 15 002 | 2363 | 7187 | 70 080 | 1285 | 2240
Number of admissions included | 183 944 | 1221 | 176 866 | 480 | 13 573 | 163 | 13 690 | 2106 | 6326 | 64 163 | 1190 | 2068
Number of exclusions | Total patients excluded | 13 768 | 113 | 3402 | 84 | 2887 | 46 | 1312 | 257 | 861 | 5917 | 95 | 172
Age not over 18 years | 1582 | 101 | 1300 | 48 | 998 | 31 | 513 | 95 | 210 | 414 | 52 | 22
Readmission to ICU within same hospital encounter | 8245 | 11 | 0 | 15 | 1472 | 13 | 401 | 144 | 344 | 3163 | 18 | 0
Interhospital transfer | 3685 | 1 | 2102 | 21 | 416 | 2 | 281 | 18 | 307 | 2340 | 25 | 150
Missing outcome | 247 | 0 | 0 | 0 | 26 | 0 | 117 | 0 | 0 | 0 | 0 | 0
Median patient age (IQR) | 66 (52–76) | 40 (28–60) | 65 (47–78) | 40 (28–55) | 65 (53–73) | 47 (30–61) | 60 (47–70) | 60 (42–69) | 59 (42–73) | 66 (55–74) | 41 (30–56) | 64 (49–74)
Median systolic blood pressure (IQR) | 97 (88–108) | 110 (90–130) | 121 (105–140) | 116 (104–130) | 90 (82–102) | 123 (102–141) | 129 (110–140) | 129 (114–146) | 122 (110–140) | 91 (80-105 | 125 (110–143) | 120 (100–142)
Number of male patients (% of total) | 103 444 (56.2) | 622 (50.9) | 85 743 (48.5) | 256 (53.3) | 8632 (63.6) | 73 (44.8) | 8123 (59.3) | 1175 (55.8) | 3572 (56.5) | 42 839 (61.1) | 758 (63.7) | 1225 (59.2)
Number of patients admitted after emergency surgery (% of total) | 29 006 (15.8) | 316 (25.9) | 12 602 (7.1) | 95 (19.8) | 2441 (18.0) | 43 (26.4) | 744 (5.4) | 248 (11.8) | 436 (6.9) | 7369 (11.5) | 225 (18.9) | 224 (10.8)
Number of patients admitted after elective surgery (% of total) | 69 651 (37.9) | 239 (19.6) | 31 899 (18.0) | 80 (16.7) | 3548 (26.2) | 69 (42.3) | 2743 (20.0) | 368 (17.5) | 929 (14.7) | 22 615 (35.2) | 611 (51.3) | 423 (20.4)
Number of patients receiving infused vasopressors/inotropes (% of total) | 70 665 (38.4) | 300 (24.6) | 25 473 (14.4) | 105 (21.9) | 8231 (60.8) | 51 (31.3) | 2621 (19.1) | 925 (43.9) | 1047 (16.6) | 31 223 (48.7) | 282 (23.7) | 461 (22.3)
Number of patients receiving NIV/CPAP (% of total) | 18 926 (10.3) | 5 (0.41) | 20 583 (11.6) | 12 (2.5) | 1050 (7.1) | 1 (0.6) | 1105 (8.1) | 279 (13.2) | 288 (4.6) | NAa | 101 (8.5) | NA
Number of patients receiving invasive mechanical ventilation (% of total) | 55 797 (30.3) | 368 (30.1) | 4889 (2.8) | 331 (69.0) | 6101 (45.0) | 116 (71.2) | 3177 (23.2) | 1174 (55.7) | 1321 (20.9) | NA | 837 (70.3) | 718 (34.7)
Number of patients receiving RRT (% of total) | 7069 (3.8) | 6 (0.5) | 14 297 (8.1) | 38 (7.9) | 638 (4.7) | 3 (1.8) | 468 (3.4) | 163 (7.7) | 188 (3.0) | 2172 (3.4) | 12 (1.0) | 65 (3.1)
Total number of ICU deaths (% of total) | 9810 (5.3) | 694 (56.8) | 12 602 (7.1) | 160 (33.3) | 1008 (7.4) | 50 (30.7) | 1263 (9.2) | 297 (14.1) | 815 (12.9) | 8384 (13.1) | 143 (12.0) | 355 (17.2)
Prevalence of missing data for study variables | Blood pressure (% of total) | 0 (0) | 1 (0.1) | 2549 (1.4) | 5 (1.0) | 187 (1.4) | 3 (1.8) | 66 (0.5) | 0 (0) | 11 (0.2) | 482 (0.8) | 18 (1.5) | 84 (4.1)
Metastasis/haematological malignancy (% of total) | 0 (0) | 0 (0) | 81 (0.0) | 0 (0) | 669 (4.9) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 3 (0.1)
Vasopressor/inotrope use (% of total) | 11 795 (6.4) | 1 (0.1) | 204 (0.1) | 5 (1.0) | 0 (0) | 3 (1.8) | 66 (0.5) | 0 (0) | 11 (0.2) | 0 (0) | 18 (1.5) | 2 (0.1)
Respiratory support (% of total) | 0 (0) | 1 (0.1) | 204 (0.1) | 5 (1.0) | 1528 (11.3) | 3 (1.8) | 66 (0.5) | 0 (0) | 11 (0.2) | 0 (0) | 18 (1.5) | 2 (0.1)
Renal replacement therapy (% of total) | 13 084 (7.1) | 1 (0.1) | 204 (0.1) | 5 (1.0) | 0 (0) | 3 (1.8) | 66 (0.5) | 0 (0) | 11 (0.2) | 36 195 (56.4)b | 18 (1.5) | 2 (0.1)
Admission type (% of total) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 29 (0.2) | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 201 (0.3) | 0 (0) | 0 (0)

ROC curves are shown in Figure 4. AUROCs varied from 0.63 to 0.82, with the pre-specified threshold of 0.70 met in eight of 12 registries (Table 4). The pooled AUROC was 0.73 (0.69–0.77).

 | Aus & NZ | Bangladesh | Brazil | Ethiopia | Finland | Ghana | India | Malaysia | Nepal | Netherlands | South Africa | Uruguay
AUROC (95% CI)a | 0.82 (0.81–0.82) | 0.67 (0.64–0.7) | 0.82 (0.81–0.82) | 0.66 (0.61–0.71) | 0.81 (0.79–0.82) | 0.70 (0.61–0.78) | 0.70 (0.68–0.72) | 0.71 (0.68–0.74) | 0.68 (0.66–0.7) | 0.79 (0.78–0.79) | 0.63 (0.58–0.67) | 0.80 (0.78–0.83)
Calibration slopeb | 1.0116 | 0.542 | 1.0473 | 0.5702 | 1.0508 | 0.7143 | 0.7067 | 0.6278 | 0.6131 | 0.9225 | 0.4626 | 0.8986
Calibration interceptc | −1.691 | 1.8638 | −0.8062 | 0.7306 | −1.6543 | 0.8963 | −0.8969 | −0.9466 | −0.6197 | −1.5156 | −0.8046 | −0.2594
O/E ratio (95% CI)d | 0.28 (0.27–0.29) | 6.18 (6.02–6.35) | 0.53 (0.52–0.54) | 3.19 (2.96–3.45) | 0.31 (0.28–0.33) | 2.76 (2.39–3.19) | 0.71 (0.67–0.75) | 0.66 (0.59–0.74) | 1.03 (0.97–1.09) | 0.36 (0.35–0.37) | 1.34 (1.18–1.53) | 0.91 (0.83–0.99)
Brier scoree | 0.0775 | 0.4571 | 0.0703 | 0.2573 | 0.1019 | 0.2289 | 0.0822 | 0.127 | 0.107 | 0.1185 | 0.1055 | 0.1158

There was extensive variation in calibration between registries (Figure 5, Table 4). O/E ratios ranged from 0.28 to 6.18 (Table 4) .

Fig. 4: ROC curves (solid blue) presented by registry for the ICU mortality analysis. Sensitivity is plotted on the y axis, against (1 – specificity) on the x axis.

Fig. 5: Calibration curves for ICU mortality analysis. Observed probability (y-axis) is plotted against predicted probability (x-axis). Logistic calibration curves (solid black) and nonparametric smoothed (dotted) calibration curves are plotted.

Overall model fit is quantified by Brier scores ranging from 0.070 to 0.457 (Table 4).

Sub-group ICU mortality analysis

Supplementary Table 8 shows results from the pre-specified subgroup analyses, with pooled AUROCs presented in Fig. 6. The worst discrimination was observed in the invasively ventilated subgroup (AUROC 0.68 [0.66–0.71]). All other subgroups had pooled AUROC point estimates above 0.70. As in the primary analysis, score performance was typically comparable or marginally worse than in the total study population.

Fig. 6: A forest plot showing the pooled AUROC (95% CI) for each subgroup of the ICU mortality analysis. Pooled AUROCs were obtained using restricted maximum likelihood estimation for each subgroup. The area shaded orange is above the pre-specified threshold AUROC of 0.7, and the area shaded grey is below this threshold.

Sensitivity analyses

The complete case analysis and exclusion of all interhospital transfers yielded comparable results to those described above (Supplementary Table 9). The pooled AUROCs were 0.72 (0.67–0.76) and 0.73 (0.69–0.77), respectively.