Google Documents Retry-After for Slowing Googlebot Crawling

On October 6, 2026 Google updated its guide on reducing crawl rate. It now explains, with examples, how to use the Retry-After HTTP header alongside the 503 and 429 status codes. Support for the header is not new, Google says. It matters to any site whose server struggles when Googlebot crawls too much.

What we know

What changed and when: Google's crawling documentation changelog has an entry dated October 6, 2026. It says the emergency section of the "Reduce the Google crawl rate" guide was restructured, and that information and examples about the Retry-After header were added.
Why: according to that changelog, support for Retry-After is not new. It was already documented in the guide on temporarily pausing or disabling a website. Google moved it into this guide to make it easier to find.
What the guide says now: to reduce crawling urgently and for a short time (a couple of hours, or one to two days), the server can return 500, 503 or 429 instead of 200. With 503 or 429 it can add Retry-After to indicate when Google's crawlers can retry.
Two formats: a delay in seconds, such as "Retry-After: 120", or an absolute UTC date and time, such as "Retry-After: Wed, 21 Oct 2026 07:28:00 GMT". Those are the guide's own examples.
How Google reacts: when it finds a significant number of URLs with those codes, it lowers the crawl rate for the whole hostname, including URLs that return content, and raises it again automatically once the errors drop, according to the guide.
The limit: Google does not recommend doing this for longer than one to two days. If Googlebot sees those codes on the same URL for multiple days, the URL may be dropped from the index, according to the guide.
Who spotted it: Search Engine Roundtable reported it on October 6, 2026.

What changes and what doesn't

The documentation changes, not the crawler's behavior: Google says it already supported Retry-After. What is new is that the instruction and its examples now sit in the guide people open when their server is overloaded.

The guide presents Retry-After as an indication of when a retry is possible. It does not say how precisely the crawlers follow it.

Everything else stands. Reducing crawl rate has a cost: Google discovers fewer new pages, takes longer to reflect price or availability changes, and removed pages stay in the index longer, according to the guide. A 2023 warning from the Google Search Central Blog also still applies: 403 and 404 are not a way to slow Googlebot down. All 4xx codes except 429 cause content to be removed from Search.

How to tell if it affects you

It concerns you if crawlers have ever pushed your server past its limit, or if your site has a maintenance mode. Seven steps:

Check what your site returns in maintenance mode. Next time you turn it on, run "curl -I https://yourdomain.com/" from a terminal. The first line should say 503. If "Retry-After" is there too, even better. If it says 200, 403 or 404, that needs fixing.
Make sure robots.txt does not return 503. Google's guide on pausing a website warns that this blocks all crawling.
Ask your host, your CDN or your firewall provider: "When you rate-limit Googlebot, which status code do you return?" The right answer is 429 or 503. In 2023 Google said it had seen site owners and some CDNs using 404 and other 4xx codes for that.
Before slowing the crawler, find the cause. Google recommends checking your server's access logs with your hosting company. The most common causes, according to the guide: filters and sorting options that multiply URLs (faceted navigation) and calendars with one URL per date.
In an emergency, return 503 or 429 with Retry-After to crawl requests, and only for hours. Set a realistic delay: if the server needs ten minutes, 600 seconds. The guide on pausing a website asks for a best-effort date or duration.
When it is over, go back to 200 and open the Crawl stats report in Search Console, under Settings, to watch the crawl recover.
If your infrastructure cannot serve errors, Google offers a form to request a reduction. It warns that the request can take several days and that you cannot ask for an increase.

Sources

Updates: this note will be extended if Google explains how its crawlers apply the Retry-After delay.