INTRODUCTION

Measuring the quality of healthcare is a critical prerequisite for improvements to be achieved [1-3]. Among the three types of healthcare quality measures, outcome measures are of great intrinsic interest because they reflect all of the aspects of the services provided, unlike structure or process indicators [4]. Mortality is one of the most commonly used outcome measures because the data is readily available and very useful, but robust risk adjustment methodologies are required if mortality is used as a quality measure. Disease-specific risk adjustment models for mortality prediction have been developed and are available for various diseases such as cardiovascular diseases and cancers. Although disease-specific risk adjustment is useful for quality improvement in certain clinical areas, generic risk-adjustment methodology that encompasses all clinical conditions and reflects the overall quality of a hospital is more useful for hospital-wide quality improvement than disease-specific risk adjustment is.

Several generic risk-adjustment measures such as the MedisGroups and Disease Staging have been available for decades [5-7]. However, these generic measures require extensive efforts to collect detailed clinical data, thus are not widely used. In 1999, the hospital standardized mortality ratio (HSMR) was developed and is a generic risk-adjustment measure that can be calculated from administrative data [8]. After its introduction in the UK, the HSMR has been widely used in many countries including the US, Canada, and the Netherlands [9].

In Korea, disease-specific risk-adjusted mortality has been evaluated nationally for various clinical conditions such acute myocardial infarction and major cancers since 2005. However, to the best of our knowledge, hospital-wide overall mortality has never been assessed using adequate risk-adjustment methodology. Thus, we sought to measure and compare hospital inpatient mortality among Korean hospitals using HSMR methodology. Specifically, the aim of this study was to assess variations in hospital HSMRs overall and to compare hospital HSMRs across hospital characteristics and disease groups.

METHODS

Study Population

All inpatients who were discharged from any tertiary or general hospital in Korea with more than 700 beds between January 1 and December 31 in 2008 were eligible for this study. We chose 700-bed hospitals as the cutoff point for inclusion to increase the accuracy of the measured mortality rates using an adequate sample size. Sixty-six hospitals fit this description, and more than half of all Koreans admitted to a hospital have visited one of these hospitals.

Using the inpatient claims data from the National Health Insurance and Medical Aid from 2008, inpatients with conditions accounting for 80% of all inpatient deaths at these hospitals were selected. Our final study population included 703 571 patients, which includes 27 718 (3.94% of all patients) in-hospital deaths nested in 66 hospitals (Table 1). We excluded deaths due to respiratory failure, cardiac arrest, and ventricular fibrillation from the analysis because these deaths cannot be attributed to poor quality health care rather than disease characteristics. We also excluded patients who were transferred in and/or out of the hospital, those who were more than 100 years old, those who were less than 28 days old (neonates), and those who had a length of stay of less than one day or more than 365 days [9-16]. Patients admitted for palliative therapy were also excluded because their mortality correlates poorly with the quality of care received at the hospital. Patients who had inpatient claims within 7 days before or after the index admission were defined as transferred patients. We excluded transferred patients because they tended to be sicker than non-transferred patients were. According to these exclusion criteria, 2884 patients were excluded from the study population (Supplemental Figure 1).

Variables

The patient outcome (alive and dead) and dates of death (if relevant), were obtained from the 2009 National Health Insurance registry. The patient characteristics required for calculating HSMRs were derived from the National Health Insurance and Medical Aid inpatient claims data. The claims data include patient demographic information such as age, gender, principal diagnosis, comorbidities, surgical procedures, and the whether admission was elective or emergency. Patient comorbidities were abstracted from secondary diagnoses codes. In addition, hospital characteristics such as academic status (academic vs. non-academic), ownership (public vs. private), bed size, and geographic location (metropolitan vs. non-metropolitan) were identified.

Hospital Standardized Mortality Ratio

The final outcome variable in this study was the HSMR, which represents hospital-wide inpatient mortality. The HSMR is the ratio of the actual number of in-hospital deaths to the expected number of in-hospital deaths, for the most responsible diagnosis (MRDx) accounting for 80% of inpatient mortality [16]. Patients were grouped according to their MRDx category using the HSMR methodology, and the Korean Classification of Diseases, 6th revision was used to map their principal diagnoses to MRDx. Among 258 MRDx of all inpatient, 29 MRDx accounted for 80% of all deaths in this study.

Using logistic regression, risk-adjusted mortality prediction models were built for each MRDx. Then, using model coefficients, the number of expected deaths was determined based on the sum of the probabilities of in-hospital death for patients from each hospital. The logistic regression models incorporated the factors affecting mortality, including age, gender, urgency of admission, income level (National Health Insurance vs. Medical Aid), and comorbidities. The Elixhauser comorbidity index was used to adjust for comorbidities [17-20]. The diagnosis related group (DRG) screening method embedded in the Elixhauser comorbidity index was used to differentiate comorbid conditions present on admission from complications that occurred during hospitalization. If the MRDx was a malignant neoplasm, variables reflecting cancer stage such as undergoing surgery, chemotherapy, and/or radiation were also included in the models. We also included principal diagnoses codes and disease-specific mortality risk factors for further risk adjustment. The predictive power of the model was assessed using the c-statistic and the fitness of the model was assessed using the Hosmer-Lemeshow (H-L) statistic [21]. To avoid over-fitting the model, the bootstrapping technique was applied to each model. In addition, 95% confidence intervals of the HSMRs were calculated using Byar’s approximation [22]. Further information about our risk-adjustment model can be found elsewhere [23].

Statistical Analyses

Variations in the hospital HSMRs across the included hospitals were assessed and their statistical significance was compared with the average HSMR, which is 100. In addition, multiple regression models were used to measure associations with hospital characteristics. To evaluate variations in hospital HSMR across different clinical conditions, we categorized hospitals into tertiles based on their overall HSMRs and the HSMRs of each MRDx group. Then, the HSMR distribution for each MRDx group across each tertile was analyzed within the tertile of the overall HSMR. Using the 6th edition of the Korean Classification of Diseases system, 29 MRDx were categorized into the following seven MRDx groups: particular infectious and parasitic diseases (A00-B99), neoplasms (C00-D48), and diseases of the circulatory system (I00-I99), respiratory system (J00-J99), digestive system (K00-K93), and genitourinary system (N00-N99) as well as injury, poisoning, and certain other consequences of external events (S00-T98). The institutional review board of Seoul National University College of Medicine (H-0910-015-296) approved this study. All data analyses were performed using SAS version 9.3 (SAS Inc., Cary, NC, USA).

RESULTS

Validity of Risk-adjusted Mortality Prediction Model

Table 2 shows the c-statistics and H-L statistics from the logistic regression models across the 29 MRDx. Except for the mortality prediction models of three MRDx, all of the models had c-statistics higher than 0.8. The c statistic was over 0.9 for five MRDx (17.2%) that included colon cancer, breast cancer, and pneumonia; between 0.9 and 0.8 in 21 MRDx (72.4%) that included acute cerebrovascular disease, acute myocardial infarction, and acute renal failure; and less than 0.8 in the remaining three MRDx (10.3%) that included poisoning, aspiration pneumonia, and congestive heart failure. However, in 14 out of 29 MRDx, the H-L statistics of these models were significant (p<0.05), which suggests poor model fitness.

Overall Variation Among Each Hospital’s Hospital Standardized Mortality Ratio

The HSMR widely varied across the hospitals included in this study (Figure 1). Of all of the hospitals, 25 (37.9%) had a significantly low HSMR and 17 hospitals (25.8%) had a significantly high HSMR. The remaining 24 hospitals (36.4%) showed an average HSMR. The highest hospital HSMR (189.0) was 4.1 times higher than the lowest HSMR was (46.0).

Variation Across Hospital Characteristics

The HSMR for different sub-groups of hospitals are presented in Table 3 and Supplemental Table 1. The academic status of the hospital and Medical Aid insurance status of the patients were significantly associated with HSMR; however, the other hospital characteristics such as hospital location, the number of doctors, having older age patients, having an emergency admission, having undergone surgical procedures, and having cancer patients were not related. Moreover, large variations were found among non-academic hospitals (vs. academic), public hospitals (vs. private hospitals), and hospitals with less than 800 beds (vs. those with 800 beds or more).

Variation Across Most Responsible Diagnosis Groups

To evaluate agreement between overall HSMR and disease-specific HSMR among hospitals, hospitals were categorized into tertiles according to overall HSMR and HSMR by MRDx groups. The agreement between overall HSMR and disease-specific HSMR was limited (Table 4). In four out of seven MRDx groups, the MRDx group-specific HSMR showed fair agreement (kappa, 0.21 to 0.40) with the overall HSMR regarding the strength of the agreement. The level of agreement between the overall HSMR and HSMR by MRDx was fair and was highest among those with neoplasms (kappa, 0.682), yet lowest among those with injury, poisoning, and other external causes (kappa, 0.156).

DISCUSSION

We found that the hospital HSMR among large hospitals in Korea varied greatly. The mortality of the highest HSMR was 4.1 times higher than that of the lowest HSMR. Out of the 66 hospitals, about two-thirds of them showed either significantly high or low mortality while only about one-third showed average mortality. The magnitude of variations in hospital HSMR was larger than those of previous studies were. The range of the HSMR in this study was larger (46 to 189) than that in the UK (53 to 137 in 1999), Canada (56 to 142 from 2004 to 2007), and the Netherlands (68 to 132 in 2009) [10,15,24,25]. We considered the HSMR of large hospitals in Korea only; therefore, the HSMR variation among Korean hospitals could be larger if all hospitals are compared. Hospitals with a high mortality rate should make efforts to reduce it, and at the national level, efforts should be made to decrease the large variation in the mortality rates of all hospitals.

Among hospital characteristics included in the analysis, the academic status of the hospital and Medical Aid insurance status of the patients were associated with hospital HSMR. This finding was consistent with those of previous studies. Teaching hospitals showed lower hospital mortality than non-teaching hospitals did [26-28]. In addition, Medical Aid patients generally have higher mortality rates than non-Medical Aid patients do [29,30]. However, hospital location, number of doctors, having older age patients, emergency admission, undergoing a surgical procedure, and having cancer patients were not associated with hospital HSMR. Nevertheless, hospital characteristics might differ in their association with hospital HSMR if small hospitals are added to the analysis. The hospital HSMR also showed a fair level of agreement in most MRDx groups; therefore, a hospital showing a low mortality for certain clinical conditions did not necessarily have low mortalities for other clinical conditions. For example, hospitals that had a low mortality for neoplasm tended to have a high mortality in other conditions such as digestive diseases. This finding indicates that there are disease-specific factors determining mortality in addition to generic factors and that disease-specific hospital mortality should be evaluated with overall hospital mortality. The limited agreement of hospital HSMR across the MRDx groups also suggests that disease group-specific mortality should be disclosed to facilitate targeted quality improvement effort in hospitals and informed choices among patients.

The c statistic and H-L statistic are generally used to evaluate the predictive power and robustness of risk-adjusted models. The predictive power of our mortality prediction models was high. For most models, the c statistic had excellent (0.8 to 0.9) or outstanding (≥0.9) discriminative value while only three models showed acceptable discriminative value (0.7 to 0.8). The prediction models of this study also had a better range of c statistics (0.706 to 0.938) than previous studies did. In the UK and the Netherlands, the range was 0.668 to 0.939 [31] and 0.677 to 0.957 [32], respectively. Approximately half of all mortality prediction models indicated poor fitness according to the H-L statistic. However, when mortality prediction models were rebuilt using a subset of the study population for each MRDx, the H-L statistics of the models were no longer significant except for three MRDx models. This finding suggests that the significant H-L statistics of these models might have resulted from our large sample size rather than from poor fitness.

This study has several limitations regarding use of claims data for risk adjustment. First, claims data may not be accurate enough for risk-adjusted mortality prediction because code creep in claims data has been known to be prevalent. Thus, to evaluate the accuracy of our claims data, we chose two MRDx (pneumonia and colon cancer), and then compared comorbidities obtained from the medical records of all inpatients with these diseases who were admitted at four randomly selected hospitals in our study. We found that only 6% of comorbid conditions that appeared in the claims data were not found in the medical records. Second, the DRG screening method embedded in the Elixhauser comorbidity index may not accurately differentiate comorbid conditions from complications that occurred during hospitalization. Thus, to evaluate the accuracy of the DRG screening method, we compared the results of the DRG screening with that of the medical records we reviewed using the inpatient episode to evaluate the accuracy of the claims data. We found that only 6% of comorbid conditions based on DRG screening were judged as complications according to the medical records. The rate of disagreement was 4.1% for pneumonia and 23.4% for colon cancer. However, to differentiate comorbid conditions from complications accurately, the diagnosis coding system for claims data should include flags to discern whether a disease developed before or after hospitalization [33]. Third, the predictive power of mortality prediction models using claims data may not be as robust as those made using clinical data are. Thus, to compare predictive power of the claim data based models and clinical data based models, c-statistics of the claim data based model and clinical based model (using the inpatient data) were calculated. The model based on clinical data included comprehensive risk factors such as vital signs (temperature, blood pressure, and respiration rate) and laboratory results (pH, hemoglobin, and creatinine). The c-statistic of the model based on clinical data for pneumonia was 0.005 higher than that of the model based on claims data (0.940 vs. 0.945). Because of the small sample size for this analysis, a model based on clinical data that incorporated all variables of the model based on claims data could not have been built.

This study also has several methodological limitations, which are common in studies using HSMR methodology. First, coding variations among hospitals can result in inappropriate risk adjustment [10,14,33]. Because the comorbidity index and emergency admission episodes are two important variables for risk adjustment in HSMR methodology, variations in coding practice for these variables may have a potentially large impact on the HSMR of individual hospitals. When we included the interaction term of the Elixhauser comorbidity index for each hospital and individual hospital in our mortality prediction models, the interaction terms were significant only in 20 hospitals out of the 66 total hospitals. In addition, the odds ratios were not large (0.713 to 1.382) across these hospitals with significant interaction terms. Hospitals that tend to treat emergency patients with mild symptoms may have a lower HSMR than those that tend to treat more severe emergency patients, regardless of their quality of care [14]. In this study, the emergency admission and number of transfer episodes with a 1-day length of stay had a low mortality of 2.19% compared to the overall mortality. Therefore, we included only patients who visited the emergency room if they were charged a fee for emergency medical service and had a length of stay longer than 1 day. By clearly those who visited the emergency room, we tried to minimize the potential impact of different case-mix of these emergency patients on the HSMR calculation of individual hospitals. Second, applying different mortality timeframes may result in discrepancies between the HSMR of individual hospitals, and previous studies have reported inconsistent findings regarding the consequences of applying different mortality timeframes on HSMR [34,35]. We used in-hospital mortality as the mortality timeframe because other countries have reported national HSMR using in-hospital mortality as the timeframe [9], and the c-statistics of the mortality prediction models used in calculating the in-hospital mortality timeframe were higher than those employing a different mortality timeframe were. Moreover, a patient’s cause of death post-discharge could have been unrelated to the patient’s reason for previous admission. Third, the exclusion criteria of the HSMR methodology might create a selection bias [33]. We excluded transferred patients, admissions with an extremely long length of stay, and patients older than 100 years old from the analysis. However, in our study, excluding these patients did not seem to cause a substantial selection bias because it required only approximately 0.4% of the study population (2884 out of 706 455) to be excluded from the analysis.

Notes

The authors have no conflicts of interest with the material presented in this paper.

SUPPLEMENTARY MATERIAL

Supplemental Figure 1.

Schematic representation of the study population selection an analysis process. LOS, length of stay; HSMR, hospital standardized mortality ratio.

Figure 1.

Caterpillar plot of the overall hospital standardized mortality ratio (HSMR) for large hospitals in 2008. Each hospital's HSMR is graphed as the upper and lower 95% confidence limits. ■17 Hospitals had a significantly high HSMR, ▲24 hospitals had a non-significant HSMR, ●25 hospitals had significantly low HSMR.

REFERENCES

• 1. Donabedian A. Evaluating the quality of medical care. Milbank Mem Fund Q 1966;44(3):166-206Article
• 2. Werner RM, Bradlow ET. Relationship between Medicare’s hospital compare performance measures and mortality rates. JAMA 2006;296(22):2694-2702ArticlePubMed
• 3. DesHarnais S, McMahon LF Jr, Wroblewski R. Measuring outcomes of hospital care using multiple risk-adjusted indexes. Health Serv Res 1991;26(4):425-445PubMedPMC
• 4. Mant J. Process versus outcome indicators in the assessment of quality of health care. Int J Qual Health Care 2001;13(6):475-480ArticlePubMed
• 5. Iezzoni LI, Shwartz M, Ash AS, Hughes JS, Daley J, Mackiernan YD. Severity measurement methods and judging hospital death rates for pneumonia. Med Care 1996;34(1):11-28ArticlePubMed
• 6. Iezzoni LI. Using risk-adjusted outcomes to assess clinical practice: an overview of issues pertaining to risk adjustment. Ann Thorac Surg 1994;58(6):1822-1826ArticlePubMed
• 7. Iezzoni LI, Ash AS, Coffman GA, Moskowitz MA. Predicting in-hospital mortality: a comparison of severity measurement approaches. Med Care 1992;30(4):347-359ArticlePubMed
• 8. DesHarnais SI, Chesney JD, Wroblewski RT, Fleming ST, McMahon LF Jr. The Risk-Adjusted Mortality Index: a new measure of hospital performance. Med Care 1988;26(12):1129-1148ArticlePubMed
• 9. Scott IA, Brand CA, Phelps GE, Barker AL, Cameron PA. Using hospital standardised mortality ratios to assess quality of care: proceed with extreme caution. Med J Aust 2011;194(12):645-648ArticlePubMedPDF
• 10. Jarman B, Gault S, Alves B, Hider A, Dolan S, Cook A, et al. Explaining differences in English hospital death rates using routinely collected data. BMJ 1999;318(7197):1515-1520ArticlePubMedPMC
• 11. Richardus JH, Graafmans WC, Verloove-Vanhorick SP, Mackenbach JP. The perinatal mortality rate as an indicator of quality of care in international comparisons. Med Care 1998;36(1):54-66ArticlePubMed
• 12. Gordon HS, Rosenthal GE. Impact of interhospital transfers on outcomes in an academic medical center. Implications for profiling hospital quality. Med Care 1996;34(4):295-309Article
• 13. Westert GP, Lagoe RJ, Keskimaki I, Leyland A, Murphy M. An international study of hospital readmissions and related utilization in Europe and the USA. Health Policy 2002;61(3):269-278
• 14. Mohammed MA, Deeks JJ, Girling A, Rudge G, Carmalt M, Stevens AJ, et al. Evidence of methodological bias in hospital standardised mortality ratios: retrospective database study of English hospitals. BMJ 2009;337: b780Article
• 15. Canadian Institute for Health Information. HSMR: a new approach for measuring hospital mortality trends in Canada [cited 2013 Sep 18]; Available from: https://secure.cihi.ca/estore/productSeries.htm?pc=PCC374
• 16. Canadian Institute for Health Information. Technical notes: hospital standardized mortality ratio (HSMR) [cited 2013 Sep 18]; Available from: http://www.cihi.ca/CIHI-ext-portal/pdf/internet/HSMR_TECH_NOTES_EN
• 17. Southern DA, Quan H, Ghali WA. Comparison of the Elixhauser and Charlson/Deyo methods of comorbidity measurement in administrative data. Med Care 2004;42(4):355-360ArticlePubMed
• 18. Elixhauser A, Steiner C, Harris DR, Coffey RM. Comorbidity measures for use with administrative data. Med Care 1998;36(1):8-27ArticlePubMed
• 19. Li P, Kim MM, Doshi JA. Comparison of the performance of the CMS Hierarchical Condition Category (CMS-HCC) risk adjuster with the Charlson and Elixhauser comorbidity measures in predicting mortality. BMC Health Serv Res 2010;10: 245ArticlePubMedPMC
• 20. Stukenborg GJ, Wagner DP, Connors AF Jr. Comparison of the performance of two comorbidity measures, with and without information from prior hospitalizations. Med Care 2001;39(7):727-739ArticlePubMed
• 21. Hosmer DW, Lemeshow S. Applied logistic regression. 2nd ed. New York: Wiley; 2000. p. 147-156
• 22. Wright J, Dugdale B, Hammond I, Jarman B, Neary M, Newton D, et al. Learning from death: a hospital mortality reduction programme. J R Soc Med 2006;99(6):303-8ArticlePubMedPMC
• 23. Lee EJ. Development of hospital standardized mortality ratio index using administrative data [dissertation]. Seoul: Seoul National University; 2011 (Korean)
• 24. Statistics Netherlands. HSMR 2012: methodological report [cited 2013 Sep 18]; Available from: http://www.cbs.nl/NR/rdonlyres/3F1FA6BD-1B97-4A37-B0A0-E323A732CA1E/0/20-13x3703.pdf
• 25. Pouw ME, Peelen LM, Lingsma HF, Pieter D, Steyerberg E, Kalkman CJ, et al. Hospital standardized mortality ratio: consequences of adjusting hospital mortality with indirect standardization. PLoS One 2013;8(4):e59160ArticlePubMedPMC
• 26. Ayanian JZ, Weissman JS. Teaching hospitals and quality of care: a review of the literature. Milbank Q 2002;80(3):569-593ArticlePubMedPMC
• 27. Allison JJ, Kiefe CI, Weissman NW, Person SD, Rousculp M, Canto JG, et al. Relationship of hospital teaching status with quality of care and mortality for Medicare patients with acute MI. JAMA 2000;284(10):1256-1262Article
• 28. Rosenthal GE, Harper DL, Quinn LM, Cooper GS. Severity-adjusted mortality and length of stay in teaching and nonteaching hospitals: results of a regional study. JAMA 1997;278(6):485-490ArticlePubMed
• 29. Ayanian JZ, Kohler BA, Abe T, Epstein AM. The relation between health insurance coverage and clinical outcomes among women with breast cancer. N Engl J Med 1993;329(5):326-331ArticlePubMed
• 30. Hasan O, Orav EJ, Hicks LS. Insurance status and hospital care for myocardial infarction, stroke, and pneumonia. J Hosp Med 2010;5(8):452-459ArticlePubMed
• 31. Dr Foster Intelligence. Understanding HSMRs: a toolkit on hospital standardized mortality ratios [cited 2014 Mar 5]; Available from: http://drfosterhealth.co.uk/docs/HSMR_Toolkit_Version_7.pdf
• 32. Jarman B, Pieter D, van der Veen AA, Kool RB, Aylin P, Bottle A, et al. The hospital standardised mortality ratio: a powerful tool for Dutch hospitals to assess their quality of care. Qual Saf Health Care 2010;19(1):9-13ArticlePubMedPMC
• 33. Bottle A, Jarman B, Aylin P. Strengths and weaknesses of hospital standardised mortality ratios. BMJ 2010;342: c7116ArticlePubMed
• 34. Rosenthal GE, Baker DW, Norris DG, Way LE, Harper DL, Snow RJ. Relationships between in-hospital and 30-day standardized hospital mortality: implications for profiling hospitals. Health Serv Res 2000;34(7):1449-1468PubMedPMC
• 35. Pouw ME, Peelen LM, Moons KG, Kalkman CJ, Lingsma HF. Including post-discharge mortality in calculation of hospital standardised mortality ratios: retrospective analysis of hospital episode statistics. BMJ 2013;347(7):f5913ArticlePubMedPMC

Figure & Data

References

Citations

Citations to this article as recorded by  

• Differences in trends in discharge location in a cohort of hospitalized patients with cancer and non-cancer diagnoses receiving specialist palliative care: A retrospective cohort study
  Michael Bonares, Kalli Stillos, Lise Huynh, Debbie Selby
  Palliative Medicine.2023; 37(8): 1241.     CrossRef
• Functional training and timed nutrition intervention in infectious medical patients
  M Holst, L N Søndergaard, M D Bendtsen, J Andreasen
  European Journal of Clinical Nutrition.2016; 70(9): 1039.     CrossRef