Verified Outcome

An executed change is not a result. A tool that reports a task as complete says only that the task finished. For a search team the gap is wide. Google says some changes take effect in a few days and others take several months, that in general it is worth waiting a few weeks before assessing the result, and that there is no guarantee changes will have a noticeable effect^[1]. Publishing starts the clock. It does not end the job.

Each layer of the evidence has to wait for the one before. Google says crawling can take anywhere from a few days to a few weeks, and that requesting a crawl does not guarantee inclusion in results will happen instantly or even at all^[2]. So a change is first live, then read again by search, and only then can the business result be judged. That is the order of QueryRev’s checks: technical, search, business.

Numbers move for reasons that are not the change. Google lists algorithmic updates, technical issues, seasonality and changing interests among the causes of a movement in traffic^[1]. A result therefore needs something to be compared with: a modeled baseline of what the page would have done anyway. That is the counterfactual idea of the causal-impact method, which predicts the response that would have occurred had no intervention taken place^[3].

QueryRev’s term fixes what the word “done” means. A change ends in one of five classes: successful, neutral, monitoring, failed or rollback candidate. A change that did nothing is reported as no effect, and a result inside the baseline’s range is never reported as a small win. Where a lift cannot be separated from the baseline, the result is neutral or still monitoring.

The same discipline applies to a result that was never read. In QueryRev’s study, 3,679 of the 9,973 domains of the panel, 36.9%, could not be read, and they are reported as unread and never as allowing a crawler^[4]. An outcome that has not been measured is not counted as a good one.

By layer. Technical: is the change live, valid and reachable? Search: was the page read again, and did index, position and click-through respond, in AI answers as well as in classic results? Business: did traffic, conversions and revenue move? Each layer gets its own result before the next is asked.

Against a baseline. The business layer compares what happened with a modeled baseline of what the page would have earned anyway. The baseline is a range, so the lift is a range. A result inside the range is no effect. The part above it is the estimated lift, and it is labelled incremental, never added to the other kinds of revenue.

By class, and over time. Each outcome is one of the five classes. Across many changes, count how often the lift lands inside the range the decision expected, by kind of decision. That record is how an estimate earns trust.

Illustrative: an invented site, to show the method.

A publisher merges two guides that competed for one query. On day 0 the technical layer passes: the redirect is live, the canonical is set and the internal links are repointed. On day 9 search has responded: the old address is out of the results and the merged guide ranks higher.

On day 28 the business layer compares sign-ups with a modeled baseline, a range. They stand above it, by an estimated lift of $1.2k to $1.8k a month, itself a range. The change is classified successful, and the figure is written into the entry of the Change Ledger. Had the sign-ups stayed inside the baseline’s range, the outcome would have been neutral, and reported as no effect.