Originally posted on on
The results of a recent study, Complication Rates, Hospital Size, and Bias in the CMS Hospital-Acquired Condition Reduction Program that was published in The American Journal of Medical Quality, have been trumpeted in the media and blogosphere [i]. However, the authors, Koenig and colleagues, made several major design and methodological errors that rendered their research invalid for health policy and hospital administration applications. These flaws call into question the validity of the authors’ conclusion that hospital size affects a hospital’s chances of receiving a Hospital Acquired Condition Reduction Program (HAC-RP) penalty from the Centers for Medicare & Medicaid Services (CMS), especially for quality measures that rarely occur [ii].First, study design characteristics and statistical methods used to analyze the data were not adequately described in the paper which means that other researchers can’t replicate the research to confirm the results. For example, several data analysis assumptions were made that are not supported either by referring to the subject matter or statistical research literature or by providing adequate explanation of the authors’ rationale for making the assumptions. More importantly, while the authors stated a general research goal of querying whether surveillance bias and inadequate risk adjustment affected receiving penalties, the authors did not state specific hypotheses to be tested and did not present appropriate statistical evidence to support accepting or rejecting whether surveillance bias and inadequate risk adjustment, specifically hospital size and complication incidence, affected receiving penalties. Second, statistical methods were not appropriately applied. For example, the authors presented their results as likelihoods when the results were actually probabilities. [iii] Likelihood is calculated by being given a known number of total possible occurrences and a known number of successful occurrences and data is gathered by varying the probability of a successful occurrence. From these data, statisticians can determine which of two hypotheses are more reasonable given the observed data. On the other hand, probability is calculated by being given a known number of total occurrences and a specific probability of a successful occurrence and then data is gathered by varying the number of successful occurrences. From these data, statisticians can determine the statistical characteristics of an unknown distribution that describes the probability of observing different numbers of successes. In addition, the authors applied one pseudo R square statistic to statistically compare whether a single logistic regression model equally described two data sets. However, the authors didn’t state that they conducted logistic regression and, even if they had conducted logistic regression, a single pseudo R square statistic was insufficient to compare how well the model described the data. Third, the authors included an unnecessary simulation of the probability of a hospital receiving a HAC-RP penalty to the actual probability of a hospital receiving a HAC-RP penalty. The simulation was unnecessary because probability calculations could have been performed by using actual data that hospitals submit to CMS to test obtain the same results more directly. Finally, differences in simulated probability of being in the bottom quartile of performance were presented as results of the data analysis when, in fact, these differences are mathematical attributes of data when incidence of complications and number of cases are varied. These design and methodological errors mean that the results in this paper can’t be used to draw conclusions about whether or not CMS’ calculations to determine HAC-RP penalties contain inherent bias that cause certain hospitals to receive unwarranted penalties for poor quality care when, in fact, those hospitals may be delivering high quality care.
[ii] I would like to thank my fellow members of the American Statistical Association’s ASA Connect list serve for statistical input on this blog post.