Editorial Views  |   August 2015
The Importance of Validation Studies in Perioperative Database Research
Author Affiliations & Notes
  • Mark D. Neuman, M.D., M.Sc.
    From the Department of Anesthesiology and Critical Care, and the Department of Internal Medicine, Division of Geriatric Medicine, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, Pennsylvania; and Leonard Davis Institute for Health Economics, the University of Pennsylvania, Philadelphia, Pennsylvania.
  • Corresponding article on page 253.
    Corresponding article on page 253.×
  • Accepted for publication March 6, 2015.
    Accepted for publication March 6, 2015.×
  • Address correspondence to Dr. Neuman:
Article Information
Editorial / Respiratory System / Sleep Medicine / Quality Improvement
Editorial Views   |   August 2015
The Importance of Validation Studies in Perioperative Database Research
Anesthesiology 8 2015, Vol.123, 243-245. doi:
Anesthesiology 8 2015, Vol.123, 243-245. doi:

“… [systematic] efforts to validate such [administrative] data for identifying specific diagnoses remain relatively uncommon [in the context of surgery and anesthesia].”

Image: ©Thinkstock.
Image: ©Thinkstock.
Image: ©Thinkstock.
OVER the past decade, administrative data have come to play a major role in research on perioperative care. Beyond providing a basis for studies focusing on the organization and delivery of care for patients undergoing surgery and anesthesia, insurance claims and other administrative records have been used to examine fundamental questions of clinical epidemiology and comparative effectiveness research. Yet, despite the growing number of investigators who seek clinical and policy insights from administrative data, efforts to validate such data for identifying specific diagnoses remain relatively uncommon although important examples do exist within1  and outside of2,3  the perioperative setting. In this issue of Anesthesiology, McIsaac et al.4  offer quantitative insights that can help readers and investigators make better sense of the opportunities and challenges involved in using administrative data for perioperative research.
McIsaac et al. examine the validity of administrative data for obstructive sleep apnea (OSA). OSA is a widespread public health problem that may increase the risk of adverse events after surgery,5,6  and to date, research using administrative data has played a major role in highlighting it as an important issue for perioperative medicine.7–9  Using a large clinical database from one major Canadian teaching hospital, McIsaac et al. identified 4,965 surgical patients who underwent a major surgical procedure between 2003 and 2012 and who also had undergone a polysomnogram at that same hospital before surgery. The authors linked these clinical data to national and provincial health administrative databases that included claims for inpatient, emergency department, and day-surgery care, as well as for physician services and durable medical equipment. They obtained a successful linkage for 4,353 patients, or 88% of their original cohort, and found 56% of linked patients to have a confirmed diagnosis of OSA based on polysomnogram results or documentation by a sleep medicine physician.
The authors used this linked dataset to study the accuracy of a range of case-ascertainment algorithms for obstructive OSA; these algorithms used codes from the 9th and 10th revisions of the International Classification of Diseases (ICD), including ICD codes that have been previously used to identify patients with sleep apnea in administrative data, as well as claims for polysomnography or positive airway pressure devices. Several of the algorithms that the authors assessed demonstrated high specificity (i.e., a high proportion of people who truly lacked OSA who were correctly identified by the algorithm); however, most of the methods assessed, and particularly those that relied exclusively on ICD codes, showed low sensitivity when tested against a standard of a diagnostic polysomnogram or documentation of OSA by a sleep medicine physician (i.e., a low proportion of people who truly had OSA who were correctly identified by the algorithm). Extrapolating their findings to a general surgical population, the authors estimate that only 40% of patients identified as having OSA based on an ICD-9 or ICD-10 diagnosis code in their administrative database could be expected to actually have the condition, whereas 10% of patients without such a code could still be expected to have OSA.
The study by McIsaac et al. has limitations of its own. Their use of data from Canada potentially limits the relevance of their work to studies that use administrative data sources from the United States because coding practices could potentially differ across countries due to variations in the organization and reimbursement of care. Its single-center design may also limit generalizability because coding practices may vary from institution to institution. Although they present findings for a range of ascertainment algorithms, these algorithms overlap incompletely with methods used in recent large database studies focusing on outcomes for surgical patients with and without OSA, thus limiting direct comparisons.
Yet this work offers important lessons for consumers of perioperative database research as well as for investigators–myself included–who use ICD codes to identify patients with specific health conditions. Essentially all empirical studies—regardless of the source of the data used—suffer from some degree of misclassification; in the present context, the low sensitivity of administrative claims for OSA creates the possibility that patients who actually have OSA might be incorrectly classified by common administrative data algorithms as lacking the disease. As McIsaac et al. note, such misclassification is often cited as being likely to bias study results toward the null hypothesis or a finding of no effect. In a study aiming to compare outcomes among patients with and without OSA, the failure to identify OSA in subjects who actually have the disease might make the two comparison groups appear more similar due to the presence in both groups of patients who actually have OSA. Such misclassification would work to diminish the differences in outcomes between groups that would be attributed to the presence or absence of OSA. Study results affected by such bias might be argued to be “conservative” because the true differences between groups would be expected to be larger than those that such a study would find.
Such an argument assumes that misclassification occurs independently of other important factors (i.e., is “nondifferential”). Yet if other factors that might impact outcomes also influence an individual’s likelihood of being classified as having the condition under study, such “differential” misclassification can also work to bias study findings away from the null hypothesis, potentially yielding spuriously large results.10 
The work by McIsaac et al. suggests that the misclassification of OSA in administrative data may indeed be related to other important factors; administrative diagnosis codes were more likely to correctly identify OSA in patients who also had other major perioperative risk factors, such as heart disease and diabetes, and who moderate or severe versus mild OSA. Such considerations create the potential that studies relying on administrative data to identify patients with OSA could potentially overstate the overall impact of this condition on postoperative outcomes if those patients classified in the study as having OSA tended to be sicker or to have more severe disease than patients with OSA overall.
To their credit, authors of studies that have used administrative data to examine the impact of OSA on postoperative outcomes have been frank about the limitations of such data sources for health research.9,11  Rather than invalidating these past studies, the work by McIsaac et al. provides an opportunity to add nuance to their interpretation. In particular, it offers new quantitative data that can help readers and researchers to better gauge the potential extent and nature of misclassification of OSA within administrative data. Future investigators might draw lessons from this work, as well as other investigations12,13  that have shown substantial variability in the coding of common diagnoses over time, as to what insights (regarding OSA or other conditions) might or might not be obtainable from administrative data sources. This investigation also offers useful information that can help practitioners and researcher more correctly interpret the results of past studies by newly quantifying and explicating limitations of administrative data sources in this context that have been suspected and debated, yet not fully described. Most importantly, this work offers a strong example of the importance of well-done validation studies in perioperative database research, not only for guiding the types of questions that we choose to investigate through the lens of retrospective data but also for making better sense of what insights such investigations might be able to convey.
Support was provided from institutional/departmental sources and from the National Institute on Aging, Bethesda, Maryland (grant no. K08AG043548).
Competing Interests
The author is not supported by, nor maintains any financial interest in, any commercial activity that may be associated with the topic of this article.
Lawson, EH, Louie, R, Zingmond, DS, Brook, RH, Hall, BL, Han, L, Rapp, M, Ko, CY A comparison of clinical registry versus administrative claims data for reporting of 30-day surgical complications.. Ann Surg. (2012). 256 973–81 [Article] [PubMed]
Carnahan, RM, Moores, KG Mini-Sentinel’s systematic reviews of validated methods for identifying health outcomes using administrative and claims data: Methods and lessons learned.. Pharmacoepidemiol Drug Saf. (2012). 21suppl 1 82–9 [Article] [PubMed]
Quan, H, Parsons, GA, Ghali, WA Validity of information on comorbidity derived rom ICD-9-CCM administrative data.. Med Care. (2002). 40 675–85 [Article] [PubMed]
McIsaac, DI, Gershon, A, Wijeysundera, D, Bryson, GL, Badner, N, van Walraven, C Identifying obstructive sleep apnea in administrative data: A study of diagnostic accuracy.. Anesthesiology. (2015). 123 253–63
Kaw, R, Chung, F, Pasupuleti, V, Mehta, J, Gay, PC, Hernandez, AV Meta-analysis of the association between obstructive sleep apnoea and postoperative outcome.. Br J Anaesth. (2012). 109 897–906 [Article] [PubMed]
Memtsoudis, SG, Besculides, MC, Mazumdar, M A rude awakening—The perioperative sleep apnea epidemic.. N Engl J Med. (2013). 368 2352–3 [Article] [PubMed]
Memtsoudis, S, Liu, SS, Ma, Y, Chiu, YL, Walz, JM, Gaber-Baylis, LK, Mazumdar, M Perioperative pulmonary outcomes in patients with sleep apnea after noncardiac surgery.. Anesth Analg. (2011). 112 113–21 [Article] [PubMed]
Spurr, KF, Graven, MA, Gilbert, RW Prevalence of unspecified sleep apnea and the use of continuous positive airway pressure in hospitalized patients, 2004 National Hospital Discharge Survey.. Sleep Breath. (2008). 12 229–34 [Article] [PubMed]
Mokhlesi, B, Hovda, MD, Vekhter, B, Arora, VM, Chung, F, Meltzer, DO Sleep-disordered breathing and postoperative outcomes after elective surgery: Analysis of the nationwide inpatient sample.. Chest. (2013). 144 903–14 [Article] [PubMed]
Rothman, KJ, Greenland, S, Lash, TL Rothman, KJ, Greenland, S, Lash, TL Validity in epidemiologic studies. Modern Epidemiology. (2008). 3rd edition Philadelphia Lippincott, Williams & Wilkins 128–47
Memtsoudis, SG In reply.. Anesthesiology. (2014). 120 246–7 [Article] [PubMed]
Lindenauer, PK, Lagu, T, Shieh, MS, Pekow, PS, Rothberg, MB Association of diagnostic coding with trends in hospitalizations and mortality of patients with pneumonia, 2003–2009.. JAMA. (2012). 307 1405–13 [Article] [PubMed]
Rhee, C, Murphy, MV, Li, L, Platt, R, Klompas, M Centers for Disease Control and Prevention Epicenters Program, Comparison of trends in sepsis incidence and coding using administrative claims versus objective clinical data.. Clin Infect Dis. (2015). 60 88–95 [Article] [PubMed]
Image: ©Thinkstock.
Image: ©Thinkstock.
Image: ©Thinkstock.