Control Group

A comparison needs something to compare with. In the plain design of a controlled experiment, Kohavi, Henne and Sommerfield describe users randomly assigned to a control, commonly the existing version, or to a treatment. They write that if the experiment is designed and executed properly, the one thing consistently different between the two is the change, so any difference in the measure follows from the assignment^[1]. The control group is what makes that sentence true.

The words “at random” carry the weight. The same authors say that the key is that assignment is random: users cannot be distributed any old way, and no factor can influence the decision^[1]. A group chosen by hand, or because it looks similar, can differ from the changed group in ways nobody measured, and that difference is then read as the effect.

Where the groups are time series, such as clicks by week, the control becomes the input of a model. Google’s CausalImpact package takes a response series, for example clicks, and a set of control series, for example clicks in non-affected markets or on other sites, and predicts the counterfactual: how the response would have evolved had the intervention never happened. It assumes that the control series were not themselves affected, and warns that if they were, the effect may be under- or overestimated, or found where there is none^[2].

A holdout is the same idea in advertising. Google Ads describes Conversion Lift as a controlled experiment that separates an audience into two groups, people who see the ads, the treatment group, and people who do not, the control group, and measures the lift in the conversions each group drives^[3]. The group is held back on purpose, so that the difference can be read as the incremental effect of the ads.

In search the unit is usually the page, a region or a market, not the visitor, because a search engine reads one version of each page; the entry on SEO split testing sets out why. The same care applies to a survey. QueryRev’s study fixes its panel of 9,973 domains from the top places of one published ranking, by rank and not by what a site’s robots.txt says, and reads the same domains each time, so a change between readings is a change on the web and not in the list^[4].

Check the match before the change. Chart both groups’ clicks for the weeks before the date of the change: they should move together. Kohavi and colleagues use an A/A test, in which two groups get exactly the same experience, to test a system: at a 95% confidence level it should reject the null hypothesis about 5% of the time^[1]. Two groups that already differ cannot be compared afterwards.

After the change, compare movement, not level. The measure is how far the changed group moved against the control over the same weeks, set against the gap between them before. Report it as a range with the method named, because groups of pages are noisy and one figure hides that. The method for two groups measured before and after is difference-in-differences.

Check that the control stayed a control. A new internal link from a changed page, a template release or a promotion on those products makes it a second treatment. Read both groups in Search Console for the whole period, and note the date of every change to either.

Illustrative: an invented site, to show the method.

A coffee-equipment shop wants to know whether new titles lift clicks to its grinder pages. It has 120 of them. It pairs each page with the one most like it in traffic, draws one of each pair at random and rewrites it: 60 pages get new titles and 60 are held back. For the 12 weeks before the change the two groups’ clicks move together.

Two other comparisons are open to the shop, and both fail. The espresso-machine pages are a different product with a different buying season. Grinder pages picked after the change because they held steady were chosen by their result. The 60 held-back pages are a control because they were chosen before the change, at random, and nothing else touched them.