Personal Science Week - 26 Jan 2023

"Experts" you shouldn't trust, different kinds of overconfidence, and more

This is a weekly summary, published each Thursday, of ideas we think will be of interest to Personal Scientists, people who want to practice science for personal reasons rather than professional ones. We emphasize personal data with hands-on experiments whenever possible.

This week we look at expertise, and some thoughts about how to decide who or what to trust.

Image generated by Midjourney prompt “Carnival operator, traveling salesman, handlebar mustache, suspicious, black and white photo, 1880s, medicine jars, electrical equipment, cluttered, large wooden collection basket with dollar sign”

“Experts” you shouldn’t trust

It’s easy to find lists of the “best books about X”. Nobody has time to review everything, so almost by definition anything that makes it to a list can be assumed to be at least good enough to criticize. But rather than a list of “the best”, I’d really like a list of “don’t bother”, filtered by somebody I respect so I can avoid wasting my time.

Sheila Kealey has a list of “Nutrition & Health ‘Experts’ You Shouldn’t Trust” that includes the usual suspects like Gwyneth Paltrow and Joseph Mercola — people with obvious conflicts of interest in their sales empires, or who’ve lost or settled lawsuits for libel or false advertising.

But the list also includes a few who seem — at least to my ignorant reading — genuinely interested in the truth. Whatever you think about the evidence in support of Gary Taubes' claims, for example, I find it hard to dismiss him as deliberately trying to mislead anyone. He makes his case clearly and with conviction, whether it’s against sugar, or for keto. Similarly, Peter D’Adamo may be (in)famous for his Blood Type Diet, written early in his career, but he’s spent decades studying this stuff, answering objections of doubters, and honing his ideas. Is it really useful to dismiss him?

New ideas have to come from somewhere. Being wrong sometimes is the price of being right.

Bryan Caplan argues convincingly that “Mainstream Media Is Worse Than Silence”, and “First-Hand Experience Is Less Biased Than the News”. Journalists are humans with their own biases and swim in an industry dependent on attention. Yet, as Richard Hanania points out, we should be able to critique the press without devolving into nihilism. Scott Alexander proposes the idea of Bounded Distrust: reputable organizations rarely tell outright lies, so in general it’s okay to take them at their word that the events they recount are true, but be on the lookout for what they’re not saying.

I don’t know of a solution other than to read widely and find multiple points of view. Our 7-Jul-2022 Week discussed the Red Pen Reviews scientific book review service, and 29-Sep-2022 Week looked at Cochrane Reviews. Let us know if you have other sources you find useful — especially the ones that say “don’t bother”.

Overconfidence isn’t necessarily bad

One of my favorite books about expertise and decision-making is Left Brain, Right Stuff by former Harvard associate professor Phil Rosenzweig. Although it’s written as a business book intended for managers, many of the ideas are useful to Personal Scientists as well.

For example, Rosenzweig warns against confusing the three senses of overconfidence:

1. Overprecision: belief that your prediction is more accurate than it is. This is the error found in those “guess the length of the Nile in miles” tests that underly the literature that supposedly shows that doctors and other experts are overconfident. Mostly this is just a harmless mistake in articulating your level of uncertainty.

2. Overplacement: the belief that we perform better than others.

3. Overestimation: Belief that we can perform at a higher level than what is objectively warranted.

Note that people who suffer from overplacement (e.g. 90% of drivers think they’re above average), aren’t necessarily being haughty. Ask them to rank their drawing or singing ability, for example, and you’ll often find they rank themselves below average.

For things we can’t control (e.g. weather, the S&P 500)  there’s nothing to be gained from overestimation. But when we can influence events, then optimism and a bias for action are good things. A cancer patient who says “I’m going to beat this thing” despite the odds is like a CEO who says “We’re going to double revenue this year”. It may seem impossible, but the point is to change the odds through our actions.

Many other great ideas, all useful to anyone trying to make sense of an overcomplicated world. Highly recommended.

About Personal Science

This is a weekly summary of ideas we think will be useful to anyone who wants to live according to the motto of the Royal Society, established in 1660: Nullius In Verba “take nobody’s word for it”. While weekly summaries like this one will always be free, next week we’ll begin to introduce additional new content for paid subscribers only. We’ll cover topics that won’t be visible to casual readers or to search engines, thus reducing the likelihood that some of our more esoteric ideas will fall into unprepared hands.