Intuition vs. Statistical Formulas (The Meehl Challenge)

Key Insight

Paul Meehl's influential 1955 book, 'Clinical vs. Statistical Prediction,' reviewed 20 studies comparing the accuracy of clinical predictions (subjective impressions of trained professionals) with statistical predictions (combining a few scores or ratings according to a rule). In a typical study, a simple formula using limited information, such as high school grades and one aptitude test, was more accurate than 11 out of 14 counselors who conducted 45-minute interviews and had access to more comprehensive data.

Over the past 50 years, this challenge has grown to nearly 200 studies, consistently showing that algorithms are significantly more accurate in about 60% of cases, with ties accounting for the rest. Statistical rules are also much less expensive. This superiority extends across diverse domains including medical diagnoses (e.g., longevity of cancer patients, susceptibility to sudden infant death syndrome), economic measures (e.g., success of new businesses, credit risks), and even predicting the future prices of Bordeaux wines, where a simple weather formula achieved a correlation above 0.90.

Experts are often inferior to algorithms for several reasons. They tend to try to be too clever, considering complex combinations of features that, more often than not, reduce validity, whereas simple combinations perform better. Furthermore, humans are inconsistently reliable in making summary judgments of complex information; for example, experienced radiologists contradict their own diagnoses 20% of the time when re-evaluating the same X-rays. This inconsistency, often influenced by fleeting contextual factors that affect System 1, severely undermines predictive validity in 'low-validity environments.'