Regression to the Mean

Morton and Torgerson, writing in the BMJ, say regression to the mean occurs whenever a nonrandom sample is selected from a population and two imperfectly correlated variables are measured, such as two consecutive blood pressure readings. The less correlated the two variables, the larger the effect. The more extreme the starting value, the more room there is to move back to the mean^[1]. A page list built from last month’s worst results is exactly such a sample.

The idea is old. Francis Galton described it in 1886 in a paper on heredity. He had found that the offspring of large seeds were smaller than their parents, and the offspring of very small seeds larger, and he measured the heights of 930 adult children and their parentages, 205 in number, to test the same pull in people. He called it regression towards mediocrity^[2].

The trap is for the person who acts. Morton and Torgerson write that any intervention aimed at a group very different from the average will appear to be successful because of regression to the mean. A sudden peak in road crashes is often due to chance, so a new enforcement policy will seem to cut crashes, and hospitals at the bottom of a league table that are given more resources will mostly climb it^[1]. A project to fix the pages that dropped has the same shape.

Google’s guide to debugging traffic drops describes the same background. It says search results are dynamic, that this constant change can cause both gains and drops, and that small fluctuations in position can happen at any time, including moving back up without you needing to do anything. Its advice is to avoid radical changes if your page is already performing well^[3].

A ranking of near-equals shows how little a place can mean. In QueryRev’s study, Bytespider is blocked by name on 595 of the sites read, 9.5%, and CCBot on 594, 9.4%: one site apart^[4]. To call one of them “the most blocked” is to pick on a gap of a single site, and a later reading that swaps the two would say nothing yet about a trend.

Record the rule that chose the pages, and the period it used. “The 40 pages whose clicks fell most in March” is a rule. The more extreme the selection, the more room there is to move back, so the rule tells you how large a bounce-back to expect^[1].

Measure the bounce-back with pages you did not touch. Pick a comparison group by the same rule, from the same period, and leave it alone. Its movement is the regression to the mean, with whatever the season adds. A fix has to beat that, not zero. Morton and Torgerson say that policy should be based on evidence from trials where possible, and give randomising poorly performing hospitals as an example^[1].

Start from a steadier level. Instead of one bad week or month, use the average of several periods as the starting point. Morton and Torgerson note that the effect can be reduced by taking serial measurements and calculating the average change^[1].

Illustrative: an invented site, to show the method.

A coffee-equipment shop lists the 40 pages whose clicks fell most in March. A fall that large is partly a bad few weeks, and some of it will reverse. The shop draws 20 of the 40 at random and rewrites them, and leaves the other 20 alone.

In April, 14 of the 20 pages left alone have partly recovered, with no work at all. The rewritten pages have recovered as well. Set against March that looks like a win. Set against the untouched 20 the gain is smaller, and the shop reports that difference as a range, not as the rise from March.