Measurement
Difference-in-Differences
Difference-in-differences is a method that estimates the effect of a change by comparing how a metric moved in the group that got the change with how it moved, over the same period, in a group that did not. Its key assumption is that, without the change, both groups would have followed parallel trends.
- Industry term
By QueryRev ResearchPublished Updated6 min read
Why it matters.
The method is old and plain. Columbia University’s public health methods pages describe difference-in-differences as a quasi-experimental design that uses data over time from treatment and control groups to obtain an appropriate counterfactual. It compares the changes in outcomes between a population that is enrolled in a program and one that is not. The logic, they note, was used as early as the 1850s by John Snow, and some social sciences call it the controlled before-and-after study[1].
It earns its place by removing two biases at once. Columbia explains that it removes biases that come from permanent differences between the two groups, and biases that come from trends over time in the treated group due to other causes of the outcome. The groups may start at different levels, since the method looks at change and not at level[1]. That is why it suits a world in which a perfect control is rare.
The classic use is a minimum wage. On 1 April 1992 New Jersey’s minimum wage rose from $4.25 to $5.05 an hour[2]. Card and Krueger surveyed 410 fast food restaurants in New Jersey and Pennsylvania before and after, and compared the changes in New Jersey with the changes in Pennsylvania, where the minimum stayed at $4.25[2]. Pennsylvania stood in for what would have happened in New Jersey without the law.
Everything rests on the parallel trends assumption. Columbia calls it the most critical of the assumptions and the hardest to fulfil: without the treatment, the difference between the two groups has to stay constant over time. There is no statistical test for it, visual inspection helps when there are many time points, and a violation biases the estimate[1]. A model can relax it. Brodersen and colleagues contrast their state-space approach with classical difference-in-differences schemes, and say it can follow the evolution of an impact over time and allow for local trends, seasonality and covariates[3].
A gap between two groups at one date is not this method. In QueryRev’s study, 17.6% of the sites read in places 1 to 1,000 block an AI crawler by name, against 13.9% of the rest read[4]. That is a difference in level at one reading. The study has one edition, which is its baseline, so no group has a change yet to compare. Difference-in-differences would need the same two bands read again at a second date.
How it is measured.
Take the change in each group: its metric after the change minus its metric before. The estimate is the change in the changed group minus the change in the comparison group. Columbia writes the method as a regression with an interaction term between time and group[1].
Check the assumption with the data you have. Chart both groups over many weeks before the change: they should move in parallel. Columbia advises acquiring more data points before and after to test the assumption, and notes that it has been proposed that the shorter the period tested, the more likely the assumption is to hold[1].
Report a range and name the method. The estimate carries the noise of both groups and depends on the weeks chosen, so one figure overstates what is known. Columbia lists robust standard errors, to allow for correlation over time within a unit, among its best practices[1].
Measured by
- Change in the changed group
- Change in the comparison group
- Difference between the two changes
A worked example.
Illustrative: an invented site, to show the method.
A coffee-equipment shop relaunches its espresso-machine pages in March and leaves its grinder pages as they were. Over the 12 weeks before March the two sets of pages moved in parallel. In the eight weeks after the relaunch, clicks to the espresso-machine pages are 15% above the eight weeks before, and clicks to the grinder pages 10% above.
The grinder pages show what the season and the rest of the market did: they rose by 10% with no change made to them. Subtract it. The effect of the relaunch is a modeled range of 2 to 8 points, not the 15 that the raw rise suggests. The shop reports it as a range, and says it rests on the pages moving in parallel before March.
What to do, step by step.
State the change, the metric, the date and two periods of equal length, one before and one after.
Choose a comparison group of pages that did not get the change and are like the changed ones in kind and trend.
Chart both groups for many weeks before the date. If they do not move in parallel, change the comparison group or stop.
Work out each group’s change over the same two periods, and subtract the comparison group’s change from the changed group’s.
Check that nothing else reached just one of the groups in the period, such as a promotion, a template release or new links.
Report the effect as a range with the method named, and say which pages and which weeks it covers.
Mistakes to avoid.
Comparing two levels, not two changes. A gap at one date, such as the two bands of QueryRev’s panel, is a difference in level[4]. The method needs each group’s movement over time.
Skipping the parallel trends check. Columbia calls the assumption the most critical and the hardest to fulfil, and says a violation biases the estimate[1]. Look at the weeks before the change.
Letting the outcome decide who gets the change. Columbia lists as a limit that the method cannot be used if the allocation of the intervention was determined by the baseline outcome[1]. Choose the groups by a rule, before you look at results.
Using a comparison group the change also reaches. Columbia lists no spillover effects among the assumptions[1]. Links from changed pages to comparison pages, or a shared template change, make the comparison group part of the treatment.
Relying on one pair of dates. Two points cannot show a trend. Columbia advises more data points before and after[1], so that the groups’ paths can be read.
How QueryRev treats it.
QueryRev does not describe any of its checks as this method. A change is verified against a modeled baseline of what would have happened anyway, and where the data and the sample size allow, stronger causal methods are used and named beside the result (the rules). The idea carries over: outside QueryRev, a group of similar pages that were not changed is one way to build such a baseline. The lift above the modeled baseline is a range, labelled incremental. It follows the loop from the revenue a page earns, to the decision to change it, to an outcome that counts once it is verified. See pre-post analysis, counterfactual baseline and the guide to SEO ROI.
Questions people ask.
What is the parallel trends assumption?
It is the assumption that, had the change not been made, the changed group and the comparison group would have moved in parallel. It cannot be tested directly, because the changed group’s untreated future is not observed. Charting both groups over many weeks before the change is the usual check.
How is it different from a before-and-after comparison?
A before-and-after comparison looks at one group against its own past, so a season or an update lands in the result. Difference-in-differences also takes the movement of a comparison group over the same period and subtracts it, which removes what reached both groups.
Can it be used when the change went to every page?
Not as it stands, because it needs a group that did not get the change. When every page changed, a baseline has to come from elsewhere: the pages’ own history, the same period a year earlier, or another site that was not changed.
Sources and citation.
Primary sources only: the search engines, the AI vendors, standards bodies and QueryRev’s own research. Each is cited in the text by its number.
Difference-in-difference estimationColumbia University Mailman School of Public Health ·
Minimum wages and employment: a case study of the fast food industry in New Jersey and Pennsylvania (Card and Krueger, NBER Working Paper 4509, 1993)National Bureau of Economic Research ·
State of AI crawler access, October 2026QueryRev Research ·
Cite this definition
A suggested citation. Its link goes to the definition, whose anchor does not change.
QueryRev Research. “Difference-in-Differences.” QueryRev AI glossary, 25 October 2026. https://queryrev.com/glossary/difference-in-differences#definition
Related terms