Bad Studies That Won’t Go Away: How Bad Science is like the Bratty Kid with the Slushy

I have no idea why this slurpy is sideways!

What happens to crappy studies?

Unless they ignite international brouhahas (e.g.. thousands of parents choosing not to vaccinate), not much according to an op-ed in the New York Times. It’s Science, But Not Necessarily Right. 

(Title in paper: Why Science Struggles To Correct Its Mistakes).

Turns out bad science is like the bratty kid with the push-over parents who keeps getting away with less than acceptable behavior.

Bad science gets rewarded in the short-term (i.e. published) then is ignored and thus ultimately excused much like the bad behavior of the kid in Target whose temper tantrum earns him a slushy. Mom or dad are too busy, too distracted, too unmotivated, too fill-in-the-blank to address the dreadful performance and thus it gets a pass and remains a regular feature of the kid’s behavioral repertoire.

Just as the other shoppers (in addition to neighbors, family members, Facebook friends) will dissect any incidents of poor parenting, academics are only too happy to trash bad studies at department meetings, conferences, maybe the blogosphere but they’re not gonna often lift a hand or research dollar to correct it.

They know it’s ridiculous, why bother proving it? If it’s a study of little merit, who’s gonna want to replicate it except for some beleaguered grad student who’s searching for a dissertation project.

Researchers are too busy with their own research gigs (for good reason), not to mention visions of tenure, to waste precious time and grant money trying to take down the competition even if the work appears deeply flawed. It’s no fun doing what’s right and dragging your child out of the mall mid-tantrum to make a point and there’s little glamour let well enough alone prestige in replicating already-published results.

Top journals want significant and original results. So do tenure committees.

Add to this dilemma the media’s eagerness to publicize studies with dramatic or entertainment value regardless of scientific merit and no wonder people don’t trust experts or anything they read or hear in the parenting sphere.

Now it’s not always clear that the results are flawed right away, especially if they are highly speculative and doctored (ahem, Andrew Wakefield’s MMR-autism results).   

Sometimes the results are just freak findings – the 1 in 100 chance of finding “statistically significant” results that truly are not significant.  Like the now debunked Mozart Effect based on the study showing a few minutes of classical music improves cognitive performance, work that launched the Baby Einstein empire but has ultimately not been successfully replicated despite many attempts.

More often small or limited results get blown out of proportion, especially in areas without much research. Like the incidence of fatal analphylatic reactions attributed to food.  Meredith Broussard blew the lid off that small, shoddy, greatly stretched set of data a few years back (see my earlier post).   

After reading the op-ed in the NY Times about shyness a couple days ago I searched high and low for updated figures on the percentage of introverts in the population. ( see Shyness: Evolutionary Tactic?).  Not much out there.  For years the accepted figure was 10 – 20% of the population.  Why? According to Susan Cain who wrote the op-ed and also a book on the topic, the estimate came from the creator of the Myers-Brigg personality test – and it was just a guess so not based on actual research.

Cain cites Jonathan Cheek’s figure (50% of adults) but in a quick check of his work (and a quick check in Google Scholar) I can’t find the relevant study nor any other estimates of introversion.  So who knows the origin of that number.  If you know, please let me know because now I’m intrigued and slightly irritated – if only expert sources would provide a full reference then my life would be complete.

So have any good or pressing examples of bad science on your mind?  The journals might not care but I do…rant away.