Regression to the Mean and Intuitive Predictions

Key Insight

Regression to the mean is a fundamental statistical phenomenon where extreme performances or measurements tend to be followed by less extreme, more average ones. This occurs because extreme outcomes often contain a significant component of luck or random fluctuation that is unlikely to persist. For instance, an exceptionally good golf score on one day is often followed by a slightly worse, but still good, score the next day.

The human mind, however, is strongly biased toward causal explanations and struggles with 'mere statistics,' leading to frequent misinterpretations of regression as a causal effect. For example, flight instructors mistakenly believed that praising cadets for outstanding performance caused subsequent deterioration, and shouting for poor performance caused improvement. In reality, both were simply observing regression: unusually good performance was likely to regress regardless of praise, and unusually bad performance was likely to improve regardless of punishment.

This pervasive causal fallacy manifests in various contexts, such as the 'Sports Illustrated jinx' (attributing an athlete's poor season after a cover feature to overconfidence, rather than regression from an unusually good prior season). The concept of regression itself is deeply counterintuitive and difficult for System 2 to grasp, even for statistically trained individuals, because the mind prefers a satisfying causal narrative over the simpler, mathematically inevitable explanation of chance and imperfect correlation.