Skip Navigation
Skip to contents

JPMPH : Journal of Preventive Medicine and Public Health



Page Path
HOME > Search
1 "Intra-rater reliability"
Article category
Publication year
Original Article
Assessing Reliability of Medical Record Reviews for the Detection of Hospital Adverse Events
Minsu Ock, Sang-il Lee, Min-Woo Jo, Jin Yong Lee, Seon-Ha Kim
J Prev Med Public Health. 2015;48(5):239-248.   Published online September 11, 2015
  • 9,396 View
  • 112 Download
  • 12 Crossref
AbstractAbstract PDFSupplementary Material
The purpose of this study was to assess the inter-rater reliability and intra-rater reliability of medical record review for the detection of hospital adverse events.
We conducted two stages retrospective medical records review of a random sample of 96 patients from one acute-care general hospital. The first stage was an explicit patient record review by two nurses to detect the presence of 41 screening criteria (SC). The second stage was an implicit structured review by two physicians to identify the occurrence of adverse events from the positive cases on the SC. The inter-rater reliability of two nurses and that of two physicians were assessed. The intra-rater reliability was also evaluated by using test-retest method at approximately two weeks later.
In 84.2% of the patient medical records, the nurses agreed as to the necessity for the second stage review (kappa, 0.68; 95% confidence interval [CI], 0.54 to 0.83). In 93.0% of the patient medical records screened by nurses, the physicians agreed about the absence or presence of adverse events (kappa, 0.71; 95% CI, 0.44 to 0.97). When assessing intra-rater reliability, the kappa indices of two nurses were 0.54 (95% CI, 0.31 to 0.77) and 0.67 (95% CI, 0.47 to 0.87), whereas those of two physicians were 0.87 (95% CI, 0.62 to 1.00) and 0.37 (95% CI, -0.16 to 0.89).
In this study, the medical record review for detecting adverse events showed intermediate to good level of inter-rater and intra-rater reliability. Well organized training program for reviewers and clearly defining SC are required to get more reliable results in the hospital adverse event study.


Citations to this article as recorded by  
  • The Korea National Patient Safety Incidents Inquiry Survey: Feasibility of Medical Record Review for Detecting Adverse Events in Regional Public Hospitals
    Min Ji Kim, Hee Jung Seo, Hong Mo Koo, Minsu Ock, Jee-In Hwang, Sang-Il Lee
    Journal of Patient Safety.2022; 18(5): 389.     CrossRef
  • The Korea National Patient Safety Incidents Inquiry Survey: Characteristics of Adverse Events Identified Through Medical Records Review in Regional Public Hospitals
    Min Ji Kim, Hee Jung Seo, Hong Mo Koo, Minsu Ock, Jee-In Hwang, Sang-Il Lee
    Journal of Patient Safety.2022; 18(5): 382.     CrossRef
  • Feasibility of Capturing Adverse Events From Insurance Claims Data Using International Classification of Diseases, Tenth Revision, Codes Coupled to Present on Admission Indicators
    Juyoung Kim, Eun Young Choi, Won Lee, Hae Mi Oh, Jeehee Pyo, Minsu Ock, So Yoon Kim, Sang-il Lee
    Journal of Patient Safety.2022; 18(5): 404.     CrossRef
  • Quality of end-of-life communication in 2 high-risk ICU cohorts: a retrospective cohort study
    Tammy L. Pham, Allan Garland
    CMAJ Open.2021; 9(2): E570.     CrossRef
  • A Systematic Review of Methods for Medical Record Analysis to Detect Adverse Events in Hospitalized Patients
    Dorthe O. Klein, Roger J.M.W. Rennenberg, Richard P. Koopmans, Martin H. Prins
    Journal of Patient Safety.2021; 17(8): e1234.     CrossRef
  • Patient Safety Incidents Reported by the General Public in Korea: A Cross-Sectional Study
    Minsu Ock, Min-Woo Jo, Eun Young Choi, Sang-Il Lee
    Journal of Patient Safety.2020; 16(2): e90.     CrossRef
  • Perceptions of Hospital Health Information Managers Regarding Present on Admission Indicators in Korea: A Qualitative Study
    Jee-Hee Pyo, Eun-Young Choi, Hae-Mi Oh, Won Lee, Ju-Young Kim, Min-Su Ock, So-Yoon Kim, Sang-Il Lee
    Quality Improvement in Health Care.2020; 26(1): 23.     CrossRef
  • Variation between hospitals and reviewers in detection of adverse events identified through medical record review in Korea
    Sukyeong Kim, Ho Gyun Shin, A E Jeong Jo, Ari Min, Minsu Ock, Jee-In Hwang, Youngjin Jeong, Moon Sung Park, Jong Bouk Lee, Tae I K Chang, Eunhyang Song, Heungseon Kim, Sang-Il Lee
    International Journal for Quality in Health Care.2020; 32(8): 495.     CrossRef
  • The Harvard medical practice study trigger system performance in deceased patients
    Dorthe O. Klein, Roger J. M. W. Rennenberg, Richard P. Koopmans, Martin H. Prins
    BMC Health Services Research.2019;[Epub]     CrossRef
  • Optimising Hepatitis C care in an urban Aboriginal and Torres Strait Islander primary health care clinic
    Prabha Lakhan, Deborah Askew, Noel Hayman, Leigh‐anne Pokino, Cheryl Sendall, Paul J. Clark
    Australian and New Zealand Journal of Public Health.2019; 43(3): 228.     CrossRef
  • Patient and hospital characteristics that influence incidence of adverse events in acute public hospitals in Portugal: a retrospective cohort study
    Paulo Sousa, António Sousa Uva, Florentino Serranheira, Mafalda Sousa Uva, Carla Nunes
    International Journal for Quality in Health Care.2018; 30(2): 132.     CrossRef
  • Interpregnancy weight change and adverse pregnancy outcomes: a systematic review and meta-analysis
    Eugene Oteng-Ntim, Sofia Mononen, Olga Sawicki, Paul T Seed, Debra Bick, Lucilla Poston
    BMJ Open.2018; 8(6): e018778.     CrossRef

JPMPH : Journal of Preventive Medicine and Public Health