That’s probably not the headline you were expecting if you’ve turned on the news in the last few days. The Congressional Budget Office (CBO) says raising the wage to $10.10 will cost 500,000 jobs! (We posted a full analysis of the CBO report yesterday). Thats certainly seems significant. But statistical significance is not a measure of magnitude, it’s a measure of certainty. How sure are we that we’ve got the correct number? In this case, the CBO is not very certain at all. In fact, by the standards normally used in social science research, they aren’t justified in rejecting the hypothesis that the minimum wage has no impact at all on employment.
But let’s back up a little to talk about how we can evaluate certainty. In most cases, when we test for statistical significance, we’re testing if our result is different from zero. The most intuitive way to test this is to look at a confidence interval. Social scientists usually report a 95% confidence interval, indicating a range that they would expect to observe 19 times out of 20 if we were enact a policy (in this case, raising the minimum wage). A wide confidence interval indicates a high range of uncertainty. If the confidence interval includes zero, then we say that result is not statistically significant. (1)
Now, based on the CBO’s own numbers, the confidence interval for the impact of raising the minimum wage on employment includes zero. So, why doesn’t the CBO report clearly say this? Well, instead of reporting the 95% confidence interval, they chose to report a 67% confidence interval. Now, I’ve never seen anyone report a 67% confidence interval. So I asked a few economists and an econometrician (someone who deals with advanced statistics in economics), and they’ve never seen a 67% confidence interval either. They only reason someone would choose such a low confidence interval is to make their result appear statistically significant.
Choosing a confidence interval after running the analysis in order report a significant result is a form of manipulating data. Chances are, even people who routinely deal with confidence intervals skipped the footnote in which the CBO noted that they were not reporting the standard 95% confidence interval, but instead a 67% confidence interval.
The CBO’s analysis showed, correctly, that they really have no idea what the impact of a minimum wage increase would be on employment. However, they then chose to report that analysis in a misleading fashion by intentionally choosing a confidence interval that allowed them to report a significant result. The correct thing to say would have been along the lines of, “we really have no idea, but we think there might be a slight negative impact.” Or, in slightly more technical terms, “we found a negative coefficient, but it wasn’t statistically significant.” It’s a shame the CBO didn’t choose to report the data in a more straightforward manner.
(1) I’m assuming that we’re testing for a statistically significant difference from zero. This is the most common, but one could also test if the result is significantly different from 1 or 2 or any value one cares to test. However, in the case of raising the minimum wage, the pressing question is if the employment impact is zero, or not.