Don’t believe everything you read in the Wall Street Journal, even if it comes out of Harvard…
Here’s an example of statistics gone wrong in a commentary featured in the November 7th Wall Street Journal. The authors, working at Harvard’s Program on Educational Policy and Governance took a look at schools in Philadelphia, namely to see who performed (improved) best – the kids in the schools managed by for-profit orgnizations, non-profit organizations, or the school district.
Four years into the study with nearly 400,000 test scores at their disposal – they claim the for-profits outperformed the others. But here’s the kicker: The differences in achievement were not statistically significant. In other words, you cannot say there are meaningful differences between kids in the for-profit schools and the others.
This kind of glossing over the facts (or findings) happens with alarming frequency. You might wonder, well, if it’s close to significant – why not say it’s significant. We adhere to rigorous statistical tests to weed out the very, very, very unlikely findings (due to chance) and the more likely and thus meaningful findings. And with exceeding large samples (400, 000 test scores is a massive sample) – the larger the sample, the more power of your statistical test. What this means in English, the larger the sample, the easier it is to get statistical significance. So let’s say we have an infinitely large sample, ANY difference we find between the schools here – it would be significant. So let’s say kids in school 1 were .05 days ahead of school 2 in reading – with an infinite sample this would be significant. Now that’s an exaggeration – but the point is – if the for-profit schools were doing a better job of educating kids – in this large sample – over 4 years of 400, 000 test scores – we should have had a statistically significant result. But that didn’t stop the researchers from staking their claim nor the Wall Street Journal from publishing the misleading information from “experts” in the field…
And no, the results have not been published in a peer-reviewed scholarly journal. Go figure –
This drives me nuts….