Many Crappy Studies = One Good Study

The style of writing has drastically changed over eras and despite many critics have compared works, every single masterpiece written in an era has its own charm.  There are now even online sites such as Aim High Writing which are providing students with solecism-free written assessments belonging to various sectors 24/7. This not only helps the student learn more but also makes their work original but also quite tasteful.

One thing I’ve noticed about pop-academic books is the sneaky way they use studies.

I don’t think it’s intentionally sneaky, as you’ve got to use what’s available and if your only conclusion was “It’s unclear”, you wouldn’t have a book.  Still, it’s easy to over-value the conclusions of books that lean on lots of research because of the sheer volume of studies cited.

Often each individual study is fraught with problems, and the authors sometimes admit as much.  You might read something like this,

“One study, which didn’t directly examine this topic, but was following a sort of similar thing 20 years ago, although controversial and some say non-reproducible, might not have had the proper data collecting methods and didn’t control for X,Y, and Z, but seemed to show a relationship like this, and that kind of seems to imply this other relationship I’m talking about.  Of course it’s not definitive so take it with a grain of salt, but even if just roughly, directionally true, it at least doesn’t disprove my theory.”

OK, so that one instance is fine.  The author admits it’s not a great study or definitive proof, but throws it out as sort of some tiny sliver of possibly admissible evidence.  You take it with the recommended grain of salt and move on.  Then they cite another study with a similar disclaimer about it’s flaws and limitations.  Then another.  By the end of the chapter, 20 individually shitty studies subconsciously pile up in your mind as one big near-definitive proof.

But 20 studies with a low probability of proof don’t add up together to equal one with a high probability of proof.

The disclaimers are responsible, and it’s not out of bounds to cite as weak supporting evidence an imperfect study.  Still, if you stack up enough of them, it creates the impression for some kind of powerful “Every smart scientific person who has ever done a study has found this conclusion or something like it and you can’t escape it” that is unwarranted.

I don’t know that the writer can or should do much about this problem, except maybe build the bones of the argument with logic alone, then cite studies only as examples of possible manifestations of the theory, instead of beginning with the studies.  But it behooves us as readers to beware.  Adding two or ten bad studies together doesn’t equal one good study.

Save as PDFPrint

Written by 

Isaac Morehouse is the founder and CEO of Praxis, an awesome startup apprenticeship program. He is dedicated to the relentless pursuit of freedom. He’s written some books, done some podcasting, and is always experimenting with self-directed living and learning. When he’s not with his wife and kids or building his company, he can be found smoking cigars, playing guitars, singing, reading, writing, getting angry watching sports teams from his home state of Michigan, or enjoying the beach.

Notify of
Inline Feedbacks
View all comments