Meta-ExternalAgent

Meta runs several crawlers for different jobs, and its page lists them. FacebookExternalHit fetches what is shared on Facebook, Instagram or Messenger, to show a title, description and image. Meta-WebIndexer navigates the web to improve the quality of Meta AI search results. Meta-ExternalAds crawls for improving advertising and other business-related products. Meta-ExternalFetcher fetches individual links at a user’s request. Meta-ExternalAgent crawls the web for uses such as training foundation AI models or improving products by indexing content directly^[1].

So the token is not training-only. Meta’s description puts training and product indexing under one name, and a rule for it is a decision about both. A rule for Meta-WebIndexer is a separate decision, about being cited. Meta says allowing that agent helps it cite and link to a site’s content in Meta AI’s responses^[1].

The control is robots.txt. Meta says it makes it easy for site managers to state preferences with industry-standard practices like robots.txt rather than non-standard formats like NoAI tags, and that to block a crawler a site adds a disallow for it. Its example writes the token in lower case; a crawler matches its token against the user-agent lines without regard to case, and one with no group of its own follows the catch-all group^[2].

Two of Meta’s agents may ignore the file. Meta says Meta-ExternalFetcher may bypass robots.txt because it performs fetches that a user requested, and that FacebookExternalHit might bypass it when performing security or integrity checks, such as checking for malware or malicious content^[1]. That fits the protocol: RFC 9309 says its rules are not a form of access authorization^[2]. A rule for Meta-ExternalAgent is therefore not a refusal of every request Meta makes.

In QueryRev’s study, Meta-ExternalAgent is blocked by name on 401 of the 6,294 sites whose robots.txt could be read, 6.4%^[3]. Meta also says to allow up to 24 hours for changes to robots.txt to take effect, because its crawlers may cache the file for up to that long^[1].

In the file, group by group. Fetch robots.txt from each host the site serves and find the group that names meta-externalagent and the group that names meta-webindexer. They answer different questions and are read apart. A rule on the main domain says nothing for a subdomain with its own file.

In the logs, by user agent. Meta’s page gives the strings its agents send, such as meta-externalagent/1.1 and meta-webindexer/1.1^[1]. Count requests for each and the status each one got. Treat a name in a log line as a claim: for FacebookExternalHit Meta calls allowing by IP address the more secure way, and the same caution applies to any agent’s name.

Around a change. Date the edit, wait the 24 hours Meta gives for caches, then compare requests per day for meta-externalagent before and after. They should fall. Requests from the fetcher need not, because it may bypass the file.

Illustrative: an invented site, to show the method.

A software company does not want its documentation collected for training, and does want to be cited when someone asks Meta AI about its product. It writes one group that disallows meta-externalagent for the whole site, and one that allows meta-webindexer.

A day later the logs show meta-externalagent requests have stopped. Requests from meta-externalfetcher continue for a few pages that users have asked about, as Meta says they may. The company does not read that as a failure of the file: the file never covered the fetcher.