How Numbers Can Lead to Biases

To simplify the world enough that it can be captured with numbers means throwing away a lot of detail. The inevitable omissions can bias the data against certain groups. (Deborah Stone in her book Counting: How We Use Numbers to Decide What Matters) describes an attempt by the United Nations to develop guidelines for measuring levels of violence against women.

Representatives from Europe, North America, Australia, and New Zealand put forward ideas about types of violence to be included, based on victim surveys in their own countries. These included hitting, kicking, biting, slapping, shoving, beating, and choking.

Meanwhile, some Bangladeshi women proposed counting other forms of violence—acts that are not uncommon on the Indian subcontinent—such as burning women, throwing acid on them, dropping them from high places, and forcing them to sleep in animal pens. None of these acts were included in the final list.

When surveys based on the U.N. guidelines are conducted, they’ll reveal little about the women who have experienced these forms of violence. As Stone observes, in order to count, one must first decide what should be counted.

Hannah Fry writing in The New Yorker

Goodhart’s law

Once a useful number becomes a measure of success, it ceases to be a useful number. This is known as Goodhart’s law, and it reminds us that the human world can move once you start to measure it. Deborah Stone writes about Soviet factories and farms that were given production quotas, on which jobs and livelihoods depended.  

Numbers can be at their most dangerous when they are used to control things rather than to  understand them. Yet Goodhart’s law is really just hinting at a much more basic limitation of a data- driven view of the world … there’s a critical gap between even the best proxies and the real thing— between what we’re able to measure and what we actually care about.

Hannah Fry writing in The New Yorker