Personal Science Week - 240711 Dunning-Kruger

A popular way to dismiss personal scientists is based on bad statistics

Even the most enthusiastic personal scientist must rely on experts, but that doesn’t mean you should stop thinking for yourself.

This week, we’ll refute a common put-down often used by self-proclaimed experts to discourage your questions. Plus, a new comprehensive blood test.

Personal scientists are often dismissed by experts who warn us to “stay in your lane”. Don’t try to pretend you know as much as the experts, they say, and offer as proof the Dunning-Kruger Effect, from a 1999 academic paper that supposedly shows how people new to a field tend to overestimate their knowledge whereas long-time practitioners are more humble:

Incidentally, here’s physicist Neil deGrass Tyson explaining the effect and using it to take down a non-physicist.

But two papers, by Dr. Ed Nuhfer and colleagues in the mathematics journal Numerancy, explain why the DK effect is almost certainly a statistical illusion. Here’s what you get if you generate a similar plot with random data.

They end up looking the same, because self-reported perceived ability is hard to quantify and statistically it’s just not unusual for random numbers to sometimes be over-represented on the high end of a curve. This doesn’t disprove the idea that “dumb” people might tend to exaggerate their intelligence—we can all think of examples—but deGrass Tyson is overly arrogant to suggest that you should trust science only when it comes from his fellow academics.

See more at McGill University’s excellent summary The Dunning-Kruger Effect Is Probably Not Real.

We wrote in PS Week 230126 about different types of overconfidence, some of which are better than others.

Your textbooks might have been wrong

Much of what you remember from school came via textbooks. You may still have some of them that you consult from time to time. Popular textbooks are updated regularly to correct errors or adjust for new findings. Although the differences between editions might appear small, over time they can add up.

Bryan Caplan asks Why Were American Econ Textbooks So Pro-Soviet?

When I was in Econ 1, we actually used the infamous 1989 Samuelson text – the one that said, “the Soviet economy is proof that, contrary to what many skeptics had earlier believed, a socialist command economy can function and even thrive.”  So I was delighted when my colleague David Levy and his co-author Sandra Peart decided to put Samuelson’s Sovietology under a microscope.

Back in 2006, they published some preliminary findings; now we can see Levy-Peart’s full story.  Quick version: Samuelson habitually overestimated Soviet growth, edition after edition.  Here’s a nice table summarizing the numbers relayed to millions of students over the decades:

This is just one example, from an economics textbook, and maybe it’s an extreme case. But I wonder how many self-proclaimed experts, who got an A+ in their economics class long ago, think they still know more than the rest of us, when their “expertise” is based on ideas that have been long since refuted.

Incidentally, I’m sure similar trends showing consistently incorrect information are noticeable in textbooks about medicine, biology, or even advanced physics. Somebody who mastered that material at the time can become overly confident, not realizing that what they learned might not be correct.

New blood tests I’m trying

The best way to learn something is to do it for yourself, so I just signed up for Function Health, now that big VC firm Andreesen Horowitz just dumped $53M on them in hopes of turning them into the biggest company in the world. They offer blood tests—drawn at a real place like Quest Labs—looking at something like 100 parameters. Besides the obvious cholesterol, etc, they report Apo B, T4/TSH, antinuclear antibodies, leptin, sex hormones, nutrients, electrolytes, bioage, and many more. Price $499. They sell additional “add-ons”, like Grail cancer detection ($899), STDs ($49), Celiac ($69) and many more.

We’ve long been fans of Siphox Health for their at-home tests. At $99/test, they’ve been the cheapest comprehensive test I know, but most people won’t learn much new by testing monthly. We also highly recommend Own Your Labs for wholesale pricing on LabCorp tests (PSWeek 220602). If Function turns out to be legit, then maybe the right thing to do is to do one comprehensive test annually, and follow up with single, cheaper tests more frequently if there’s something you need to work on.

I’ll post my results to PS Week in a couple of weeks. If you’d like to try it with me, you can skip the waitlist with this link¹

About Personal Science

Personal scientists prefer as much as possible, to figure things out for ourselves. It’s important to listen to experts, not because they’re necessarily right, but because they’ve had more experience than the rest of us. Someone who’s spent a life studying something will certainly know more than I do, and I’d be missing a learning opportunity if I ignore them.

Still, let’s remember that experts are often wrong. They also often disagree with one another, so you’re left with the question of which expert. So whether you listen to experts or not, you'll still need to make up your own mind.

1. P.S. Just so you know, I get no commission, free tests, or kickbacks of any kind from this service. I’m just a fellow personal scientist trying out new stuff.