Personal Science Week - 250130
Problems with Big Science, plus new tests to consider
A new Administration is proposing radical changes to government agencies that fund science and public health.
Much of the opposition presents these change as “an attack on science”. But what about personal science?
Personal scientists are open-minded, so our first reaction to a proposal is to give its proponents the benefit of the doubt. Whether the topic is fluoride, raw milk, creationism, mood crystals, or reiki, we’ll listen carefully in the hopes of learning something. But the second, equally important attribute is skepticism. We don’t believe anything “just because”. Whether the idea comes from an expert or a lunatic, assume that “the first principle is that you must not fool yourself and you are the easiest person to fool.”
Big Science Has Problems Too
Beginners to personal science often pride on directly reading such-and-such peer-reviewed study, but skepticism applies to the work of professional scientists too:
Only 39% of 100 psychology studies could be reproduced in a 2015 project
A cancer biology project could replicate just 50 experiments out of 193 attempts from high-profile papers
Even in top journals like Nature and Science, only 13 of 21 studies were successfully replicated
What's particularly concerning is that papers that fail to replicate tend to be cited much more frequently - up to 300 times more often for non-replicable papers in prestigious journals. The fact that you heard about something should itself raise your eyebrows.
And as we’ve pointed out many times, any personal scientist who goes beyond popular headlines about food and health knows that virtually all of these studies are a waste of time. This chart and accompanying discussion from FiveThirtyEight are a classic summary.
A Problem of Definitions
The problem with published research is even worse than it looks. When attempting to compile a database of studies for her PhD, Anne Scheel found that for most studies it was nearly impossible to state formally what the researchers were trying to prove in the first place, much less describe whether they succeeded or not.
Just one example: Does screen time affect a person’s well-being? It depends on what you mean by “screen time” and what you mean by “well-being”. Most studies, she notes, don’t bother to define these terms in ways that other, independent researchers could use for replication. Among her suggestions, she concludes that these studies need to move toward more machine-generated hypothesis tests.
Prebiotics’ effect depends on the initial configuration
Even when you have a precise target for research, the final result can be less than actionable.
Duke University researchers tested 28 people with prebiotics over a 6 week period. The prebiotics, specific foods known to feed healthy gut microbes (as opposed to probiotics, which are live microbes taken in pill form) were given out at regular intervals and the people were tested to see the effects. The results? The prebiotics had an effect all right: overall “healthy” microbiome composition changed as expected when taking the prebiotics. But when the researchers looked more carefully at the results, they noticed that the changes depended more on the person’s initial microbiome than it did on the specific prebiotics they happened to take. Some people respond well to prebiotics and some people respond barely at all.
How specific prebiotic compounds affected each test subject. Higher is “better”. Note the wide variance, despite averages being “statistically significant” (p<0.05)
What should you take away from this study? That prebiotics affect the microbiome? Not if you’re one of the (many) people below that yellow line.
Why Personal Science
For the Personal Scientist, the takeaway is to treat all claims with skepticism, including seemingly well-done studies from authoritative sources. But be skeptical and open-minded. Consider the possibility that those authoritative sources may actually be right! It’s too easy to get caught up in a political moment that tries to throw away (or keep!) everything, including much of the hard-won knowledge accumulated by many smart people, including those institutional experts.
The key is to be an active participant in these questions. Listen to the studies and counter-studies, but always ask how does this relate to my situation? Does it conform to your own personal experience? And most of all: can you test it, replicate it?
Whether it’s one of the examples above, like the relationship between screen time and well-being, or prebiotics and gut health, the ultimate test is if something works for you personally.
Let professional scientists decide whether a new government policy is good or bad. Personal scientists should focus on individual consequences—for me, my family, and friends.
Some Tests to Consider
Get a free LP(a) cholesterol test from Endless Health. Sign up at Cholesterol Connect or see Family Health. I sent mine last week and will let you know my results.
Another reminder to please join the reporting system from Outbreaks Near Me, a joint project of Boston Children’s Hospital and Harvard Medical School. I get a brief email each Monday asking if I have any symptoms of cold or flu. Takes less than 15 seconds to respond, and my information is aggregated to a big US map where I can see if there are other people who are suffering illness now.
Here’s this weeks’s map:
Meanwhile Bird Flu Risk is a dashboard that tracks various prediction markets to estimate the risk that bird flu with turn into a pandemic. (Right now the risk is “Little Chance”: 8 out of 100).
Tests Not to Consider
J.E. posted his blood test results proving he’s “healthy as a horse” after months of drinking a 5 lb bucket of horse electrolytes he bought from Tractor Supply. Unclear why he would do this, although the $13 price is maybe a tenth the price of Gatorade.
Never getting hungover again or being iron deficient. I am the #boss . Sipping on half of a horse dose and I feel great. My skin is f*ing tingling.
Not something I would try, but kudos to him for measuring, and then publicizing, his blood test results.
Speaking of stupid things I don’t recommend, survivalists and apocalypse “preppers” who advocate buying ornamental fish antibiotics as an alternative to the prescription kind might feel vindicated by a 2022 study by researchers at the College of Dental Medicine in Arizona who concluded:
Results confirmed that both human-grade prescribed and nonprescribed over-the-counter ornamental fish antibiotics we tested matched USP monogram standards and were pharmacologically indistinguishable”
About Personal Science
We publish every Thursday, but meanwhile if have comments or other ideas you’d like to discuss, please let us know.