Retrieval-Augmented Generation (RAG)

The paper that named it, by Lewis and colleagues, starts from a limit of language models: they store factual knowledge in their parameters, but cannot easily access it precisely, show where it came from or update it^[1]. Its answer was to pair the model, a parametric memory, with a non-parametric memory, a dense vector index of Wikipedia searched by a neural retriever, and it found that the pair generated more specific, diverse and factual language than the model alone^[1].

For a website the two memories are two routes into an answer. Training puts content into a model’s parameters, slowly and without a link back. Retrieval puts a page in front of the model at the moment it answers, as a source it can cite. Google’s word for the second is grounding: providing content from the Google Search index to the model at prompt time to improve factuality and relevancy^[2]. Microsoft describes grounding as what connects AI to current, authoritative information beyond the model’s training^[3].

The vendors’ controls follow the same split. OpenAI’s GPTBot crawls content that may be used in training, while OAI-SearchBot surfaces sites in ChatGPT’s search features, and each can be allowed or refused on its own^[4]. Refusing a training crawler is a decision about the parametric memory; it leaves retrieval untouched. In QueryRev’s study, CCBot, which builds the open web archive used in many training sets, is blocked by name on 594 sites, 9.4% of those read^[5].

Retrieval is also where pages written for people pay twice. A retriever matches the question against passages, so a section that states its answer plainly can be found on its own. The work is the one described under answer engine optimization, applied to every page an AI search system might read.

From outside, by its traces. An answer that cites pages, quotes facts that appeared after a model’s training or changes when a page changes is drawing on retrieval. Publish a dated change to a page and track when the answers to your prompts begin to carry it: the delay is how long that system takes to read the page again.

Some vendors report part of it. Microsoft points site owners to Bing Webmaster Tools for early insight into how content contributes to AI answers, including citation and grounding signals^[3]. Where such a report exists, it shows the retrieval side directly.

Measure eligibility as well. A page can be retrieved only if the system’s search crawler may fetch it and its index holds it, so check access per AI crawler, indexing, and that the answer sits in the HTML rather than in a script that runs after load.

Sample data: an invented site, to show the method.

An invoicing app raises its starter plan from 12 to 15 dollars a month on 1 September and updates its pricing page the same day. Two weeks later it asks four assistants what the starter plan costs.

Two answers retrieve the pricing page, cite it and give the new price. One answers from memory, with the old price and no link. One cites a review site that still lists the old price. The last is fixed by asking that site to update; the third changes only when the model is trained again.