Counterfactual Baseline

Every claim that a change worked compares two numbers, and one of them never happened. The counterfactual baseline is that number, made explicit: the CausalImpact package describes its job as predicting how the response metric would have evolved after the intervention if the intervention had never occurred^[1].

It turns a result into incremental revenue. Revenue after a change minus the baseline is what the change added; without the baseline, the same revenue says only what search-attributed sales earned, which is a weaker claim.

It also protects the people who did the work. A change that lifted revenue in a falling market looks like a failure against last month and a success against its baseline; a change that rode a seasonal rise looks the other way round.

Analytics uses the same idea. Google Analytics describes its data-driven attribution as a counterfactual approach, comparing the key event probability of users exposed to an ad with that of similar users in a holdback group^[2].

A baseline must say what it leaves out. QueryRev’s study computes its shares over the 6,294 sites it could read and reports the 3,679 it could not, 36.9% of the panel, by cause, instead of counting them as sites that block nothing^[3]. A baseline built on incomplete data says so in the same way.

From history: fit the metric over a long pre-period, with its trend and seasons, and project it over the window after the change. The CausalImpact method models local trends, seasonality and the influence of outside series in this way^[4].

From controls: choose pages of the same kind that the change did not touch, and use how they move after the change to predict the changed pages. The controls must not be affected by the change themselves^[1]. Where no single page is a fair comparison, several series can be combined by regression into one that tracks the changed page before the change, the synthetic control of the CausalImpact paper^[4].

Test it before trusting it. A baseline that cannot predict the pre-period well will not predict the after-period either, and the CausalImpact documentation suggests checking how well the outcome can be predicted before the intervention begins^[1].

Name its inputs beside it: the period it was fitted on, the comparison pages and the method, so that anyone can rebuild it. Write it as a range: its low and high ends carry the uncertainty into every figure built on it. How a forecast from your own last year serves as a baseline is set out in SEO forecasting: how to forecast traffic and revenue.

Sample data: an invented site, to show the method.

A shop changes a product page after week six. For the four weeks that follow, a baseline is fitted on the page’s own twelve months of weekly revenue and on eight similar pages that were not changed, and fixed before the four weeks begin. It predicts $55,100 to $55,900 for the 28 days. The eight comparison pages share neither the product page’s template nor its internal links.

The page records $62,400 of known revenue. Against the baseline, the change added $6,500 to $7,300, a range as wide as the baseline is uncertain. Because the baseline was fixed first, the result could not shape it.