Normal had Shifted

Regression to the mean is most slavishly followed on the stock market. Wall Street folklore is full of such catch phrases as “Buy low and sell high,” “You never get poor taking a profit.” All are variations on a simple theme: if you bet that today’s normality will extend indefinitely into the future, you will get rich sooner and face a smaller risk of going broke than if you run with the crowd. Yet many investors violate this advice or selling high. Impelled by greed and fear, they run with the crowd instead of thinking of themselves.

Since we never know exactly what is going to happen tomorrow, it is easier to assume that the future will resemble the present than to admit that it may bring some unknown change. A stock that has been going up for a while somehow seems a better buy than a stock that has been heading for the cellar. We assume that a rising price signifies that the company is flourishing and that a falling price signifies that the company is in trouble. Why stick your neck out?

Consider those investors who had the temerity to buy stocks in early 1930, right after the Great Crash, when prices had fallen about 50% from their previous highs. Prices proceeded to fall another 80% before they finally hit bottom in the fall of 1932. Of consider the cautious investors who sold out in early 1955, when the Down Jones Industries had finally regained their old 1929 highs and had tripled over the preceding six years. Just nine years later, prices were double both their 1929 and their 1955 highs. In both cases, the anticipated return to “normal” failed to take place: normal had shifted to a new location.

Peter Bernstein, Against the Gods

22 recent articles about where AI will go in 2024

An OpenAI employee says prompt engineering is not the skill of the future — but knowing how to talk to humans will be – Business Insider  

Generative AI will move from hype to actually being helpful – Semafor

How ‘A.I. Agents’ That Roam the Internet Could One Day Replace Workers – New York Times

Why AI struggles to predict the future – NPR

How AI will upend the customer service industry - Semafor

OpenAI’s chief scientist, on his hopes and fears for the future of AI - MIT Technology Review

Forrester’s 2024 Predictions Report warns of AI ‘shadow pandemic’ as employees adopt unauthorized tools – VentureBeat

2024: The year AI gets real - Axios

The biggest winners — and losers — in the coming AI job apocalypse – Business Insider

Now That Generative AI Is Here, Where Will All The Data Come From? – Forbes

Researchers think there’s a 5% chance AI could wipe out humanity – Semafor

Generative AI a la ChatGPT is pushing investors to new extremes of hype – Axios  

The Generative AI Bubble Will Burst Soon – KD Nuggets  

Wall Street Watchdog Says AI Will Cause 'Unavoidable' Economic Collapse – Gizmodo

Experts Predict the Future of Technology, AI & Humanity – Wired

An English professor long interested in the statistical analysis of literature & he thinks AI is a game-changer in our understanding of texts – Business Insider

How AI Is Impacting Society And Shaping The Future – Forbes

In its own words: The future of AI in sports – Sports Business Journal

iPhone 16 is poised to be an AI superphone — 5 rumors you need to know – Tom’s Guide

Everyone gets an AI agent – The Nieman Lab

Klarna CEO on how AI will make online shopping more 'emotional' – Semafor

Where is AI Heading in 2024? Looking Ahead To AI In 2024 – Forbes

We’re All Lousy Self-Evaluators

A stranger walks into a room and sits down behind a table. He picks up a piece of paper and read aloud a generic-sounding weather report. He completes his “report” in about 90 seconds and walks out of the room.

Next, you’re asked to guess his IQ.

You’re part of a psychological experiment, and you object to the absurdity of the request. I don’t know anything about that guy. He just came into a room and read a report. It wasn’t even his report- you gave it to him to read! How am I supposed to know his IQ?

Reluctantly, you make a wild guess. Separately, Fake Weatherman is asked to guess his own IQ. Who made a better guess?

Amazingly, you did, even though you know nothing about Fake Weatherman. Two (German) psychologists … conducted this experiment, and they found that the strangers’ IQ predictions were better than the predictions of those whose IQ was being predicted- about 66 percent more accurate.

To be clear, it’s not so much that you’re a brilliant predictor; it’s that he’s a lousy self-evaluator. We’re all lousy self-evaluators. College students do a superior job predicting the longevity of their roommates’ romantic relationships than their own.

Savor, for a moment, the preposterousness of these findings. Fake Weatherman has all the information, and you’ve got none. He’s got decades of data- year’s worth of grades, college entrance exams cores, job evaluations, and more. Fake Weatherman should be the worlds foremost expert on Fake Weatherman!

Chip & Dan Heath, Switch

The Secret of Success

Survivorship bias pulls you toward bestselling diet gurus, celebrity CEOs, and superstar athletes. You look to the successful for clues about the hidden, about how to better live your life, about how you too can survive similar forces against which you too struggle. Colleges and conferences prefer speakers who shine as examples of making it through adversity, of struggling against the odds and winning.  

The problem here is that you rarely take away from these inspirational figures advice on what not to do, on what you should avoid, and that’s because they don’t know. Information like that is lost along with the people who don’t make it out of bad situations or who don’t make it on the cover of business magazines – people who don’t get invited to speak at graduations and commencements and inaugurations. 

The actors who traveled from Louisiana to Los Angeles only to return to Louisiana after a few years don’t get to sit next to James Lipton and watch clips of their Oscar-winning performances as students eagerly gobble up their crumbs of wisdom. In short, the advice business is a monopoly run by survivors. As the psychologist Daniel Kahneman writes in his book Thinking Fast and Slow, “A stupid decision that works out well becomes a brilliant decision in hindsight.”

Before you emulate the history of a famous company, Kahneman says, you should imagine going back in time when that company was just getting by and ask yourself if the outcome of its decisions were in any way predictable. If not, you are probably seeing patterns in hindsight where there was only chaos in the moment. He sums it up like so, “If you group successes together and look for what makes them similar, the only real answer will be luck.” 

Entrepreneur Jason Cohen, in writing about survivorship bias, points out that since we can’t go back in time and start 20 identical Starbucks across the planet, we can never know if that business model is the source of the chain’s immense popularity or if something completely random and out of the control of the decision makers led to a Starbucks on just about every street corner in North America. That means you should be skeptical of any book promising you the secrets of winning at the game of life through following any particular example.

David McRaney Read more here

Media Growth Predictions

Here are some takeaways from the annual PricewaterhouseCoopers (PwC) Global Entertainment & Media report:

• U.S. digital newspaper ad revenue expected to surpass print by 2026.

• Online TV’s ad growth (10%) will come at the expense of terrestrial TV’s ad growth, which will decrease from 66.6% in 2021 to 63.1% in 2026. 

• Print still dominates the book market, accounting for 77.4% of total revenue in 2021, with electronic books contributing 22.6%. 

• Virtual reality continues to be the fastest-growing segment of media, albeit from a relatively small base.

• Global internet advertising revenue will expand at an impressive 9.1% CAGR in the next five years to reach $723.6 billion in 2026, at which point 74% of internet ad revenue will be mobile.

• Teenagers are now spending more time in immersive virtual worlds like Roblox and Fortnite than they are on TikTok. 
  

Read more here

A Common Prediction Mistake

Suppose you’re told that a man named John is extremely well-educated, smokes a pipe, and wears tweed jackets with patches on the sleeve—is he more likely to be a particle physicist or a janitor? A physicist, you immediately think. But you’d likely be wrong, because janitors are common and particle physicists rare. The chances that you’d happen upon a very well-educated, tweed wearing, pipe-smoking janitor are higher than those that you’d meet a physicist who meets the same profile.

Laurie Abraham writing in Slate

Who is best at predicting the future

(In a contest involving hundreds of geopolitical questions) a small number of forecasters began to pull clear of the pack: the titular “superforecasters”. Their performance was consistently impressive. With nothing more than an internet connection and their own brains, they consistently beat everything from financial markets to trained intelligence analysts with access to top-secret information.

They were an eclectic bunch: housewives, unemployed factory workers and professors of mathematics. But Philip Tetlock (who teaches at the Wharton School of Business) and his collaborators were able to extract some common personality traits. Superforecasters are clever, on average, but by no means geniuses. More important than sheer intelligence was mental attitude. Borrowing from Sir Isaiah Berlin, a Latvian-born British philosopher, Mr Tetlock divides people into two categories: hedgehogs, whose understanding of the world depends on one or two big ideas, and foxes, who think the world is too complicated to boil down into a single slogan. Superforecasters are drawn exclusively from the ranks of the foxes.

Humility in the face of a complex world makes superforecasters subtle thinkers. They tend to be comfortable with numbers and statistical concepts such as “regression to the mean” (which essentially says that most of the time things are pretty normal, so any large deviation is likely to be followed by a shift back towards normality). But they are not statisticians: unlike celebrity pollsters such as Nate Silver, they tend not to build explicit mathematical models.

But superforecasters do have a healthy appetite for information, a willingness to revisit their predictions in light of new data, and the ability to synthesise material from sources with very different outlooks on the world. They think in fine gradations. 

Most important is what Mr Tetlock calls a “growth mindset”: a mix of determination, self-reflection and willingness to learn from one’s mistakes. The best forecasters were less interested in whether they were right or wrong than in why they were right or wrong. They were always looking for ways to improve their performance. In other words, prediction is not only possible, it is teachable.

Prediction, like medicine in the early 20th century, is still mostly based on eminence rather than evidence. The most famous forecasters in the world are newspaper columnists and television pundits. Superforecasters make for bad media stars. Caution, nuance and healthy scepticism are less telegenic than big hair, a dazzling smile and simplistic, confident pronouncements.

From a review in The Economist of the book Superforecasting: The Art and Science of Prediction by Philip Tetlock and Dan Gardner

Goodhart’s law

Once a useful number becomes a measure of success, it ceases to be a useful number. This is known as Goodhart’s law, and it reminds us that the human world can move once you start to measure it. Deborah Stone writes about Soviet factories and farms that were given production quotas, on which jobs and livelihoods depended.  

Numbers can be at their most dangerous when they are used to control things rather than to  understand them. Yet Goodhart’s law is really just hinting at a much more basic limitation of a data- driven view of the world … there’s a critical gap between even the best proxies and the real thing— between what we’re able to measure and what we actually care about.

Hannah Fry writing in The New Yorker

Faith in Numbers

When polls have faltered in predicting the outcome of elections, we hear calls for more and better data. But, if more data isn’t always the answer, maybe we need instead to reassess our relationship with predictions—to accept that there are inevitable limits on what numbers can offer, and to stop expecting mathematical models on their own to carry us through times of uncertainty.

To recognize the limitations of a data-driven view of reality is not to downplay its might. It’s possible for two things to be true: for numbers to come up short before the nuances of reality, while also being the most powerful instrument we have when it comes to understanding that reality.

Hannah Fry writing in The New Yorker

Predicting the future

How much reliance can we place on regression to the mean in judging what the future will bring? What are we to make of a concept that has great power under some conditions but leads to disaster under others? Keynes admitted that “as living and moving beings, we are forced to act … (even when) our existing knowledge does not provide a sufficient basis for a calculated mathematical expectation.”

With rules of thumb, experience, instinct and conventions – in other words, gut - we manage to stumble from the present into the future … The trick is to be flexible enough to recognize that regression to the mean is only a tool; it is not a religion with immutable dogma and ceremonies. Used to make mechanical extrapolations of the past … regression to the mean is little more than mumbo-jumbo. Never depend upon it to come into play without constantly questioning the relevance of the assumptions that support the procedure. Francis Galton spoke wisely when he urged us to “revel in more comprehensive views” that the average.

Peter Bernstein, Against the Gods

Superforecasters

They are called “superforecasters” and they make surprisingly accurate predictions about world events. Tara Law writes about these semi-professional forecasters in TIME magazine:

Superforecasters tend to share certain personality traits, including humility, reflectiveness and comfort with numbers. These characteristics might mean that they’re better at putting their ego aside, and are willing to change their minds when challenged with new data or ideas…they may also be more flexible than traditional scientists, because they’re not bound to a particular discipline or approach. Their predictions incorporate research and hard data, but also news reports and gut feelings. They tend to be actively open-minded and curious. They’re in “perpetual beta” mode—always striving to update their beliefs and improve themselves. A willingness to change your mind when presented with new information, contend with your biases, challenge one another’s ideas, and break down problems into specific questions are all desirable qualities in people who make big, important decisions.  

Pascal’s Wager

Pascal’s argument (written in the 1600’s) went like this: Suppose you concede that you don’t know whether or not God exists and therefore assign a 50 percent chance to either proposition How should you weight these odds when decided whether to lead a pious life? If you act piously and God exists, Pascal argued, your gain – eternal happiness - is infinite. If, on the other hand, God does not exist, your loss, or negative return, is small – the sacrifices of piety. To weigh these possible gains and losses, Pascal proposed, you multiply the probability of each possible outcomes by its payoff and add them all up, forming a kind of average or expected payoff. 

In other words, the mathematical expectation of your return on piety is one-half infinity (your gain if God exists) minus one-half a small number (your loss if he does not exist). Pascal knew enough about infinity to know that the answer to this calculation is infinite, and thus the expected return on piety is infinitely positive. Every reasonable person, Pascal concluded, should therefore follow the laws of God. Today this argument is know as Pascal’s wager. 

Pascal’s wager is often considered the founding of the mathematical discipline of game theory, the quantitative study of optimal decision strategies in games.

Leonard Mlodinow, The Drunkard's Walk: How Randomness Rules Our Lives

Assume They’re Wrong

From military predictions to technological predictions to sports predictions, when experts foretell the future, it’s always safest to assume they’re wrong. 

Because they are deeply knowledgeable in a particular field, experts are more prone than others to view the world through a too-narrow lens, assuming that the current trends they understand so well are indicators of what is to come. Their expertise reinforces their confidence in their own analysis, blinding them to contrary data or disconfirming evidence. 

As you listen to their smart, persuasive, credible prophecies, just remember: Most of them, most of the time, will be wrong. (You can take my word for it. After all, I’m an expert.)

Jeff Jacoby is a columnist for The Boston Globe