noindex

It is the control for keeping a page out of search results. Google says noindex is set with a meta tag or an HTTP response header, and that when Googlebot crawls the page and extracts the rule, Google will drop the page entirely from its results, whether or not other sites link to it. Because it works page by page, it is also useful when a site owner has no root access to the server^[1].

The rule is read by crawling the page, and that is the usual cause of failure. Google says the page must not be blocked by robots.txt and must be otherwise accessible, or the crawler will never see the rule and the page can still appear in results, for example if other pages link to it^[1]. Search Console’s help for the page indexing report says the same from the other side: a robots.txt rule is not recommended for blocking a page, and will actually prevent noindex from being seen^[4].

Robots.txt cannot carry it. Google’s documentation says that specifying noindex in the robots.txt file is not supported^[1]. The change came in 2019: Google announced on 2 July of that year that it would retire the code handling unsupported and unpublished rules, noindex among them, on 1 September 2019, and named noindex in robots meta tags as the most effective way to remove URLs from the index when crawling is allowed^[3].

The header form is for files that have no HTML head. Google says an X-Robots-Tag header can carry the value noindex or none for non-HTML resources such as PDFs, video files and image files^[1]. Google also warns that the rules may not be treated the same by other search engines, and that a rule written for search crawlers does not bind others, such as AdsBot, which may need rules aimed at them^[2].

Removal takes the time a revisit takes. Google says it may take months for Googlebot to revisit a page, depending on the page’s importance, and that a page still in results has probably not been crawled since the rule was added^[1]. The opposite mistake is a site-wide block. In QueryRev’s study, 190 of the 6,294 sites whose robots.txt could be read, 3.0%, close the whole site in their catch-all group, User-agent: *^[5]. On such a site a crawler that obeys the file never reads a noindex tag on any page.

Per URL, read the response as it is served: the robots meta tag in the HTML head and the X-Robots-Tag header. The rule exists only if the response carries it. Then read robots.txt for the same URL. A URL that carries noindex and is also disallowed is a conflict, because the rule cannot be read.

Per site, list the URLs that carry noindex and count them by template. Search Console’s page indexing report names the pages it did not index because of a noindex tag and the pages blocked by robots.txt^[4], and Google points to it for monitoring the pages from which Googlebot extracted a noindex rule^[1].

Over time, date every change to the list. A jump after a deploy usually means a rule meant for a test site shipped to the live one. After a rule is added or removed, count the days until the page leaves or returns to results; for a page that is rarely crawled, that can be a long time.

Illustrative: an invented site, to show the method.

A store wants four URLs out of search results and writes four different rules. A retired pricing guide gets a robots meta tag. A thank-you page gets a meta tag and also a Disallow line, put there by a different team. A draft page gets a Noindex line in robots.txt. A price list PDF gets an X-Robots-Tag header.

Two of the four can work. The guide and the PDF are crawled, their rules are read, and they leave results after the next crawl. The thank-you page is never fetched by an obedient crawler, so its tag is never read and the URL can still be listed. The robots.txt line is not supported at all. The fix is to remove the Disallow line and move the other rule into a meta tag.