Inappropriate Survey Design Analysis of the Korean National Health and Nutrition Examination Survey May Produce Biased Results

Article information

Abstract

INTRODUCTION

The Korean National Health and Nutrition Examination Survey (KNHANES) is one of a series of health-related programs sponsored by the Korean Centers for Disease Control and Prevention [1]. The KNHANES has been used to assess the health and nutritional status of the non-institutionalized civilian population in Korea since 1998. Since the survey data became available for official public use in Korea, research papers using KNHANES data have become more common and have increased in numbers year by year [1]. Public access to the survey data has generated new momentum for research organizations, universities, and healthcare providers, including clinical medical researchers who had experienced difficulties in obtaining the raw data [2-4].

The KNHANES is a complex, stratified, multistage, probability-cluster survey of a representative sample of the non-institutionalized civilian population in Korea. The stratified multistage cluster sampling design creates non-independence among units, along with disproportionate sampling where some groups may be oversampled or a probability proportional to size has been applied [5,6]. This creates two primary challenges that must be addressed when analyzing complex survey data: 1) homogeneity that results from the non-simple random sample (i.e., non-independence) and 2) disproportionate sampling that results in unequal selection probabilities (e.g., oversampling or adjustment for non-response) [7,8]. Failure to address these issues results in incorrect estimation of standard errors, which often results in underestimation and an increased probability of a type I error [6,9].

Given the inherent nature of multiple complex survey designs, the KNHANES and the US NHANES provide specific guidelines for analysis and reporting of sample weights, so the stratification and clustering of the design must be incorporated into analyses to ensure appropriate estimates and standard errors. These guidelines warn that ignoring the complex design leads to biased estimates and overstated significance levels [10,11]. Regrettably, most of the published research articles using KNHANES data do not incorporate survey design analysis into their statistical analysis. Instead, most use regular standard statistical analysis, treating KNHANES data as if they were collected using a simple random sampling method. The objectives of this study were to: 1) explore the serious problem of misuse of statistical analyses using KNHANES data, 2) provide several examples of differing results when KNHANES data are analyzed using ordinary statistics and survey design statistics, and 3) recommend survey design statistics to calculate non-biased estimates for descriptive or analytical data analysis.

METHODS

RESULTS

During the period between 2007 and June 2012, a total of 256 relevant research papers were cited in PubMed. Table 1 lists the proportions that used ordinary statistical data analyses and design-based weighted statistical analyses. The number of relevant articles published increased steadily by year, and increased more rapidly in recent years. Throughout the 5.5 year time period, 19.8% of the articles used design-based data analyses.

Table 1.

Number (%) of research articles published in peerreviewed journals cited in PubMed using Korean National Health and Nutrition Examination Survey data during 2007 to 2012¹

Table 2 lists the differences in point estimates and standard errors of means between the un-weighted and weighted analyses by gender and age group. Not only did the means differ between the two analyses, but the standard errors in the weighted analyses also tended to be larger compared to those in the un-weighted analyses.

Table 2.

Comparison of geometric mean±standard error of blood metals between ordinary statistical data analysis and designbased analysis by gender, age groups in Korean adult population

ORs and 95% CIs for hypertension and prehypertension were calculated for log-transformed blood metal levels after adjusting for covariates (Table 3). The covariates for models 1 and 2 were gender, age, residence area, education level, smoking and drinking status, hemoglobin, and serum creatinine. In a logistic regression analysis using log-transformed blood cadmium as a continuous independent variable after covariate adjustment, the ORs and 95% CIs for hypertension with a doubling of blood cadmium were 1.240 (1.123 to 1.367) and 1.249 (1.117 to 1.398) in the ordinary statistical data analyses and design-based data analyses, respectively. However, a logistic regression analysis using log-transformed blood cadmium as a continuous independent variable after covariate adjustment revealed that the ORs and 95% CIs for pre-hypertension with a doubling of blood cadmium were 1.295 (1.178 to1.423) and 1.279 (1.147 to 1.426) in ordinary statistical data analyses and design-based analyses, respectively. While log-transformed blood lead did not exhibit any significant association with hypertension or prehypertension in logistic regression analyses regardless of type of analysis, log-transformed blood mercury exhibited a significant protective effect on prehypertension in un-weighted ordinary logistic regression analyses, while weighted design-based analyses did not reveal any significant associations between blood mercury and prehypertension.

Table 3.

Comparison of logistic regression analysis¹ of blood metals on hypertensive status in adults between ordinary statistical data analysis and design-based analysis during 2008 to 2010

Finally, Table 4 lists logistic regression analyses of blood metals on osteoporosis in menopausal women by ordinary statistical data analyses and design-based analyses. In the logistic regression analyses using log-transformed metals as a continuous independent variable after covariate adjustment, the ORs for osteoporosis with doubled blood lead were statistically significant between two types of analyses. The doubled blood mercury had a statistically significant OR (0.488: 0.320 to 0.743) in ordinary statistical data analyses, but no association was observed between blood mercury and osteoporosis in the design-based analysis. Similar findings were observed in logistic regression analyses using quartiles of blood mercury as an independent variable for osteoporosis. The OR (95% CI) of the highest quartile of blood mercury in ordinary statistical data analyses was 0.450 (0.251 to 0.808) with statistical significance; in design-based analyses, it was 0.575 (0.281 to 1.181) with no statistical significance.

Table 4.

Comparison of logistic regression analysis¹ of blood metals on osteoporosis in menopausal women between ordinary statistical data analysis and design-based analysis

DISCUSSION

Large-scale health surveys including the KNHANES and the US NHANES are increasingly being used to investigate associations between risk factors of diseases, nutritional status, and health-related outcomes [15-17]. Because these surveys record many variables for a large number of individuals, they enable many hypotheses to be addressed, some of which may not even be formulated until years after the surveys have been completed. Additionally, these surveys are frequently population-based, thereby avoiding the potential biases of hospital-based studies [3].

Generally, these large-scale health surveys use complex probability procedures, such as stratification, multiple stages of cluster selection, and oversampling to obtain a representative sample of the target population [5,18,19]. Because of these sampling procedures, analyses of these data using traditional statistical software (such as SPSS or SAS) that use ordinary and generalized least squares estimation techniques tend to result in an underestimated standard error, inappropriate confidence intervals, and misleading tests of significance [20]. Until a decade ago, many analysts of data collected in sample surveys felt comfortable using standard procedures in statistical software packages to analyze the data. These standard procedures were based on the assumption that the data being analyzed emerged from a simple random sample of the population of interest, where all sample respondents were randomly selected without replacement from the population, and all respondents had equal probability of being included in the sample [21]. During the last several decades, many survey data analysts have expressed their concerns that survey data were being analyzed using standard statistical procedures instead of specific procedures based on survey design [6,8].

In Korea, many research articles using KNHANES data have been published in peer-reviewed journals since the Korea Centers for Disease Control and Prevention released the KNHANES dataset to the research community in 2007, and the number of research articles using KNHANES data has increased remarkably over the last six years. The KNHANES provides a good source of data for studying the relationship between nutritional status, food habits, environmental factors, and health-related outcomes. Even though the proportion of weighted research articles has increased from 7.7% in 2007 to 26.9% in 2012, the majority of peer-reviewed research articles using KNHANES data cited in PubMed did not use appropriate analyses. Standard statistical analyses that do not use weighted multiple complex survey data generally 1) yield biased point estimates of population parameters, 2) underestimate the standard error for point estimates, 3) produce overly-narrow confidence intervals on population parameters, and 4) yield tests of significance that are overly likely to reject the null hypothesis because standard error or variability of statistics is generally underestimated, resulting in more frequent type I errors [3,6,20,22].

This study compared ordinary statistical data analyses without sampling weight and design-based (data analyzed using special survey design statistical procedures) GMs and their standard errors by gender and age group (Table 2). As anticipated, the GMs differed between the two groups, and standard errors of data based on weighted analyses were larger than those based on un-weighted data analyses. Clearly, ordinary statistical un-weighted GMs and standard errors are not the same as design-based GMs and standard errors, and ordinary un-weighted GMs and standard errors cannot be used as an alternative to weighted GMs in analyses of survey data [23].

The failure of data analysts to use appropriate statistical software procedures for unbiased analyses of survey data essentially negates the considerable money and effort needed to collect the data, and can result in scientific publications that present skewed pictures of populations of research interest [21]. Un-weighted analyses of complex multi-stage cluster sampling data tend to have a narrow confidence interval, which creates biased and skewed results and generates a type I error. This study illustrated that un-weighted data analyses generated incorrect conclusions, for example that blood mercury is associated with a protective effect on prehypertension (Table 3, model 2) in an adult population and is associated with having osteoporosis among menopausal women (Table 4); these results were due to the narrow confidence intervals.

The cross-sectional nature of most survey designs like KNHANES makes it difficult to draw causal associations between health-related outcomes and certain independent variables, including heavy metals such as blood mercury. The survey data can only provide statistical associations through conservative and appropriate statistical analysis and plausible scientific justification. In this study, blood mercury was a significant predictor for the prevention of osteoporosis among menopausal women using an un-weighted analysis. This finding reconfirmed a recent publication regarding this topic, which claimed that high blood mercury has a protective effect for osteoporosis among Korean menopausal women [24]. The authors analyzed 481 postmenopausal women who participated in KNHANES 2008 and underwent bone mineral density measurement. We reanalyzed the data on these subjects using the same analysis format, but were unable to recreate the results due to the total number of menopausal women who were eligible for their inclusion criteria (menopausal and measurement of bone mineral density). In 2008, KNHANES only measured the bone mineral density of the participants who were enrolled in the later part of the survey (about 50% of all participants). Therefore, we analyzed two years of data (2008 to 2009) including 567 postmenopausal women and obtained the same results of protective effect of mercury for osteoporosis using an ordinary statistical data analysis without sampling weight, but failed to obtain statistical significance in design-based statistical analysis. The KNHANES measured total blood mercury, which included mainly organic mercury with a small portion of inorganic mercury. Because most organic mercury comes from foods including various fishes, high levels of fish consumption may be beneficial from a nutritional standpoint, but may also increase blood mercury levels. Therefore, it is unclear whether high blood mercury has a protective effect on osteoporosis. It is possible that the blood mercury level is simply another unknown index of fish consumption.

Table 3, model 1, reveals no differences between the ordinary statistical data analysis and design-based analysis. This is the most desirable outcome for an un-weighted analysis of survey data. In the past, it was common to use ordinary statistical procedures due to the lack of availability of statistical software capable of more complex analyses. However, during the last two decades, suitable software has become available and is easily incorporated into ordinary statistical packages such as SAS, SPSS, and STATA. Therefore, analysts cannot claim the scarcity of software tools as a justification for using ordinary statistical packages, which treat the sample design as simple random sampling [6,8].

Two basic approaches are generally available for the analysis of survey design data: design- and model-based approaches [6,25]. Design-based analysis incorporates all the characteristics of multiple complex sampling designs (weighting, stratification, clustering, and post-stratification) and is the standard analysis method in the majority of national survey designs [26-28]. In contrast, model-based analysis does not use sample weighting, but instead uses a standard statistical method that considers weighting, stratification, and clustering in the modeling. As long as the model describes every possible observation to explain relationships accurately, model-based analysis can ignore the complexity of survey design, even for weighted correlated data. Unfortunately, theoretically derived models known to hold for all observations often do not exist for survey situations, unlike those derived from the physical sciences [6,8].

Over the past 10 years, many researchers in the health sciences have become interested in conducting secondary analyses using data from complex sample surveys. These analyses are descriptive, analytical, hypothesis-generating, or model-building. Sample survey statisticians are aware that specialized software should be used to analyze complex sample survey data, particularly when analyses are descriptive or analytical and the survey design includes clustering [20]. Until recently, some analysts in Korea thought that un-weighted ordinary statistical data analysis was the same as model-based analysis, and did not require sample weighting [29-31]. The lack of awareness about the necessity of using weighted analysis in survey design analysis resulted in biased estimates, generating model misspecification and type I errors in the majority of published articles using the KNHANES in Korea.

Given the inherent nature of multiple complex survey design, the NHANES and KNHANES provide specific guidelines for analysis and reporting of sample weights, and state that the stratification and clustering of the design must be incorporated into the analysis to obtain accurate estimates and standard errors. The guidelines warn that ignoring the complex design will lead to biased estimates and overstated significance levels [10,11]. Hahs-Vaughn et al. [32] reviewed methodological considerations in using complex survey data from the Head Start Family and Child Experiences Survey. They warned that failing to address survey design issues resulted in underestimated standard errors and biased parameters. To avoid methodological problems in survey design data analyses, they recommended reporting practices for authors, editorial board members, and editors to assist in ensuring transparency and greater ease in replication when using complex sample data. Their recommendations for authors were as follows: 1) specify the survey weight applied during the analyses to correct for disproportionate sampling; 2) if a design-based approach is followed, specify how variances have been estimated; 3) if the complex sample design is not addressed in the analyses, provide information to help the reader understand why that choice was appropriate and how it affects interpretation of the results.

The recommendations for authors should be implemented as part of the journal's manuscript submission guidelines. Their recommendations for journal editors were as follows: 1) require authors to report the survey weights and variance estimation method when their manuscript includes analyses of complex sample data; 2) providing resources to the editorial board, such as a manuscript or other primer [15] that provides information about complex survey data and can be a helpful tool when reviewing manuscripts that have analyzed this type of data. The analysis guidelines of data providers such as NHANES and KNHANES that incorporate a multiple complex sampling design must be followed; if not, any use of other analysis methods must be justified scientifically, as with model-based analysis, which uses standard statistical methods that consider survey design complexity, such as stratification, clustering, and non-response.

Although two basic approaches are generally available, it should be noted that there may be three possible statistical methods for handling the data of a multistage complex sampling survey design : the first is the un-weighted ordinary statistical method that takes into account neither complex sample survey design nor sample weights; the second is the weighted ordinary statistical method without consideration of the complex sample survey design; the last is incorporating sample weights and properly taking into consideration complex sample survey design. However, we were not able to identify the second method due to the lack of information on statistical methods provided by the reviewed studies. Instead, we classified both the first and second methods as ordinary statistical analysis, and the last one as design-based analysis. This is a limitation of the present study.

The emerging new era of data analysis using the KNHANES promises a bright future for health and nutrition-related scientific research in Korea, which will eventually increase our knowledge and contribute to a better understanding of human health. As the survey data accumulate annually, the richness of the data for more in-depth analysis becomes increasingly apparent. The demand for taking fuller advantage of such a rich data source is increasing, but it is vital to ensure sound scientific justification and awareness of the limitations of the survey design data. Thus, appropriate statistical analysis that incorporates the sample survey design needs to be mandatory.

Article information Continued