The Base Rate Fallacy can get you in trouble

The Base Rate Fallacy comes into play when someone comes to a conclusion without considering all the relevant information. There’s a tendency to over-estimate the value of new information out of context. Consider also, an accurate test is not necessarily a very predictive test. And some facts are provably true but nevertheless can feel false when phrased a certain way. These factors can lead someone to hold misconceptions about medical tests and other data actually mean.

A Common Prediction Mistake

Suppose you’re told that a man named John is extremely well-educated, smokes a pipe, and wears tweed jackets with patches on the sleeve—is he more likely to be a particle physicist or a janitor? A physicist, you immediately think. But you’d likely be wrong, because janitors are common and particle physicists rare. The chances that you’d happen upon a very well-educated, tweed wearing, pipe-smoking janitor are higher than those that you’d meet a physicist who meets the same profile.

Laurie Abraham writing in Slate

The uses and limits of numbers

The statistics can’t capture the true toll of the COVID virus. They can’t tell us what it’s like to work in an intensive-care unit, or how it feels to lose a loved one to the disease. They can’t even tell us the total number of lives that have been lost (as opposed to the number of deaths that fit into a neat category, such as those occurring within twenty eight days of a positive test). They can’t tell us with certainty when normality will return. But they are, nonetheless, the only means we have to understand just how deadly the virus is, figure out what works, and explore, however tentatively, the possible futures that lie ahead.

Hannah Fry writing in The New Yorker

Faith in Numbers

When polls have faltered in predicting the outcome of elections, we hear calls for more and better data. But, if more data isn’t always the answer, maybe we need instead to reassess our relationship with predictions—to accept that there are inevitable limits on what numbers can offer, and to stop expecting mathematical models on their own to carry us through times of uncertainty.

To recognize the limitations of a data-driven view of reality is not to downplay its might. It’s possible for two things to be true: for numbers to come up short before the nuances of reality, while also being the most powerful instrument we have when it comes to understanding that reality.

Hannah Fry writing in The New Yorker

How does this information make me feel?

We don’t need to become emotionless processors of numerical information – just noticing our emotions and taking them into account may often be enough to improve our judgment. Rather than requiring superhuman control of our emotions, we need simply to develop good habits. Ask yourself: how does this information make me feel? Do I feel vindicated or smug? Anxious, angry or afraid? Am I in denial, scrambling to find a reason to dismiss the claim?

Before I repeat any statistical claim, I first try to take note of how it makes me feel. It’s not a foolproof method against tricking myself, but it’s a habit that does little harm, and is sometimes a great deal of help. Our emotions are powerful. We can’t make them vanish, and nor should we want to. But we can, and should, try to notice when they are clouding our judgment.

Tim Harford, How to Make the World Add Up

Pascal’s Wager

Pascal’s argument (written in the 1600’s) went like this: Suppose you concede that you don’t know whether or not God exists and therefore assign a 50 percent chance to either proposition How should you weight these odds when decided whether to lead a pious life? If you act piously and God exists, Pascal argued, your gain – eternal happiness - is infinite. If, on the other hand, God does not exist, your loss, or negative return, is small – the sacrifices of piety. To weigh these possible gains and losses, Pascal proposed, you multiply the probability of each possible outcomes by its payoff and add them all up, forming a kind of average or expected payoff. 

In other words, the mathematical expectation of your return on piety is one-half infinity (your gain if God exists) minus one-half a small number (your loss if he does not exist). Pascal knew enough about infinity to know that the answer to this calculation is infinite, and thus the expected return on piety is infinitely positive. Every reasonable person, Pascal concluded, should therefore follow the laws of God. Today this argument is know as Pascal’s wager. 

Pascal’s wager is often considered the founding of the mathematical discipline of game theory, the quantitative study of optimal decision strategies in games.

Leonard Mlodinow, The Drunkard's Walk: How Randomness Rules Our Lives

Meaningful Relationships

Our predictions may be more prone to failure in the era of Big Data. As there is an exponential increase in the amount of available information, there is likewise an exponential increase in the number of hypotheses’ to investigate. For instance, the U.S. government now publishes data on about 45,000 economic statistics. If you want to test for relationships between all combinations of two pairs of these statistics—is there a causal relationship between the bank prime loan rate and the unemployment rate in Alabama?—that gives you literally one billion hypothesis to test.

But the number of meaningful relationships in the data—those that speak to causality rather than correlation and testify to how the word really works—is orders of magnitude smaller. Nor is it likely to be increasing at nearly so fast a rate as the information itself; there isn’t any more truth in the world than there was before the Internet or the printing press. Most of the data is just noise, as most of the universe is filled with empty space.

Nate Silver, The Signal and the Noise