SEO Split Testing

Search cannot be split by visitor. Google’s testing guide explains why a test that sends different visitors to different versions fits search badly: Googlebot generally does not support cookies, so it sees only the version that browsers without cookies are given. Showing Googlebot one set of URLs and people another is cloaking, which is against Google’s spam policies whether or not it is a test^[1]. A test of search therefore has to vary pages, not visitors.

The design is that of a controlled experiment. Kohavi, Henne and Sommerfield describe users randomly assigned to a control, usually the existing version, or to a treatment. If the experiment is designed and run properly, they say, the only thing consistently different between the two is the change, so a difference in the measure follows from the assignment, and they stress that the assignment must be random^[2]. In a split test the unit is the page. The paper assumes the units are independent, which pages of one site meet only in part, since they share a domain, links and updates. So the result is an estimate.

Where a clean experiment is out of reach, a model fills the gap. Brodersen and colleagues propose a state-space model that predicts the counterfactual response that would have occurred had no intervention taken place, and discuss the strengths and limits of such models where a randomised experiment is unavailable. The model can take account of seasonality and of covariates that move with the outcome^[3]. A matched group of untouched pages plays the part of such a covariate.

Time is part of the design. Google says some changes take effect in a few days while others take several months, that in general it is worth waiting a few weeks before assessing the result, and that there is no guarantee changes will have a noticeable effect^[4]. It also says that a site running an experiment for an unnecessarily long time may be read as trying to deceive search engines^[1].

Assignment shows its worth in any data set. In QueryRev’s study, the top 1,000 sites of the panel block at least one AI crawler by name more often than the rest, 17.6% against 13.9%^[5]. Those groups were not assigned at random and differ in many other ways, so the gap is not an effect of rank. A split test assigns pages on purpose to avoid that reading.

Build the groups before the change. Choose pages of one kind and split them into two groups matched on traffic, past trend and type, by a rule written down in advance rather than by hand. Check the match: a no-change test, which Kohavi and colleagues call an A/A test, should show no difference, and at a 95% confidence level a null test should reject about 5% of the time^[2].

Compare the groups over the same weeks. The measure is the difference between the changed group and the control after the date of the change, set against the difference before it. Report it as a range with the method named, because a single figure hides the uncertainty. A range that includes zero shows no effect; it does not show that there is none.

Check what else moved. Anything that reaches every page, such as an update or a season, reaches both groups, which is the reason for a control. Anything that reaches one group only, such as a new link block or a template change, breaks the comparison. Read both groups in Search Console for the whole period.

Illustrative: an invented site, to show the method.

A retailer has 200 category pages of one kind. It splits them into two groups of 100, matched on traffic and trend, and checks that the groups moved together for 12 weeks. It then rewrites the titles of one group and leaves the other as it was.

After eight weeks, clicks to the changed pages are 11% above their previous eight weeks, and clicks to the unchanged pages 2% above, the season and an update that touched both. A model that predicts the changed group from the control and from the history gives an effect of +6% to +14%, a range, labelled modeled. The retailer reports it for these pages in these weeks, and tests another page type before extending the change to it.