Embeddings

Computers match exact strings easily and meaning with difficulty. Google’s Machine Learning Crash Course shows why with a food example: a one-hot encoding of 5,000 meals gives every meal its own vector, and nothing in those vectors says that a hot dog is closer to a shawarma than to a salad. Embeddings are its answer, lower-dimensional representations that give dense vectors capturing semantic relationships^[1].

The idea reached language in 2013. Mikolov and colleagues proposed two model architectures for computing continuous vector representations of words from very large data sets, and reported that the vectors performed at the state of the art on their test set for measuring syntactic and semantic word similarities^[2].

It reached retrieval a few years later. Karpukhin and colleagues showed that open-domain question answering could retrieve passages with dense representations alone, using embeddings learned by a dual-encoder from a small number of questions and passages, and reported that it outperformed a strong Lucene-BM25 system largely by 9% to 19% absolute in top-20 passage retrieval accuracy on their datasets^[3]. That is a result on research benchmarks, not a statement about any search engine’s ranking.

Vendors now sell the step as a service. Google’s Gemini API offers embedding models that generate embeddings for text, images, video and other content for tasks such as semantic search, classification and clustering, and says that embeddings play a key role in improving model outputs in retrieval-augmented generation systems. It names cosine similarity as a good measure of how close two embeddings are, because it focuses on direction rather than magnitude, with values from -1 to 1^[4]. Comparing embeddings is one common basis for semantic search.

A site does not have to supply embeddings or a special file. Google says you don’t need new machine-readable files, AI text files, markup or Markdown to appear in Google Search, and that Google Search ignores llms.txt files^[5]. Sites publish them regardless: QueryRev’s study of the top sites found 848 llms.txt files, with a median size of 7 KB and a median of 38 links^[6]. Such a file is a short index of links, and the passages a system compares with a question sit on the pages it points to.

Similarity is measured between two vectors, usually as cosine similarity, a number from -1 to 1 where a higher value means closer in meaning^[4]. The number belongs to the model that produced the vectors. A score from one model cannot be compared with a score from another, and no score is a ranking position.

To judge whether a page answers a question by meaning, embed the question and each passage of the page with the same model, rank the passages by similarity, and read the top few. What matters is whether the passage you meant to answer the question comes first, not the number it gets.

Treat a model of your own as a proxy. Search engines and AI systems use models of their own, so a test run on yours shows how a passage reads to a model, not what an engine will do.

Illustrative: an invented site, to show the method.

A coffee-equipment shop tests one query, “can I send back an espresso machine I have used”, against four passages from its own site with one embedding model. The returns passage, which repeats the words of the query, scores 0.82. The passage about the 100-day trial, which lets a buyer try the machine at home and get a refund, shares none of the query’s key words and scores 0.74, second. A passage on choosing an espresso machine scores 0.41 and one on free delivery 0.12.

The shop sees that the trial passage is matched by meaning although its words differ, and that the delivery passage is not matched at all. It adds the trial’s refund terms to the returns page, so the best passage and the clearest answer are the same one, and it keeps the model name and the date beside the scores.