Mark Twain famously said that there are “lies, damned lies, and statistics.”
Yet when you hear a statement presented in quantitative terms, like a doctor telling you that the test he’s just given you means there’s a 95% chance you have a fatal disease, or a journalist stating that heavy metal fans are ten times less likely to cheat on their lovers than fans of jazz music, then your brain tends to assume the number is true. After all, it’s a statistic! They conducted a rigorous, scientifically controlled survey “” right?
Even though doctors and journalists are supposed to be professionals, they often fail to understand either the data or its implications, and that’s assuming they’re actually doing the math correctly, which isn’t always the case. This was recently shown when a team of famous, influential economists were forced to admit that much of their conclusions were based on a spreadsheet typo.
If you’ve never taken the time to think about the details of a statistic, that’s okay “” you are not alone! I never gave it a second thought either, until I took statistics classes in college. But don’t worry, this isn’t going to turn into a “mathy” lecture; I’m going to describe real-world examples of how numbers are often unreliable.
There is a term called “statistical process control” (hence the title of this article), and it means that studies are supposed to be done with very specific protocols, which often aren’t followed in the popular press. For example, the claim about heavy metal and jazz was based on data from a web site called “Victoria Milan,” which is a dating site for those seeking affairs. The data was self reported, which meant that it just looked at people’s profiles.
There are a number of process controls that they missed. For example, Victoria Milan can hardly be considered an adequate sample of everyone who commits infidelity. Also, the study only noted the number of sign ups at the site, not the percent of fans of each genre who are on the site. If jazz is several times more popular than metal with the public at large, it would make sense why they would have more sign ups, even if fans of both genres had the same rate of cheating.
In short, the data doesn’t tell us much of anything beyond how many people on one particular website say they like certain genres of music “” but the press created a headline out of it as if it was the results of a legitimate study.
Even when accurately conducted and typo-free surveys and studies are conducted, however, the pros often misread the numbers or bury the lead. Let’s go back to that doctor’s test. Doctors were asked the following question by statistical researchers: “If a test to detect a disease whose prevalence is 1/1000 has a false positive rate of 5%, what is the chance that a person found to have a positive result actually has the disease?” The vast majority of doctors flunked the test, with half of them stating that the answer was 95% (the actual answer is 2%.)
Similar studies have confirmed these results, which has horrific implications for the world of medicine. The test was fine, but what the doctors didn’t realize was that if the false positive rate is higher than the prevalence of the actual disease, most of the people who test “positive” will be healthy.
You may find this depressing, but I say knowledge is power. Hopefully, the next time you see a headline reading “72% of people…”, you won’t automatically believe it. You’ll check to see if the numbers represent a legitimate study, and you’ll look through the numbers yourself to make sure they aren’t being misapplied or misinterpreted. You’ll be a more informed person for it.
Get real time update about this post categories directly on your device, subscribe now.