Google-Agent: How to Keep Your Site From Blocking It
Admin
On November 17, 2026, Google published a technical guide to help websites avoid blocking Google-Agent by mistake. Google-Agent is the name of its agents that browse and take actions on a user's behalf. It matters to anyone running a CDN, a web application firewall or a bot-detection service in front of their site.
What we know
- What Google-Agent is: according to Google's documentation, it is the name used by agents hosted on Google infrastructure to navigate the web and perform actions when a user asks.
- What Google published: the changelog of its crawling documentation lists, on November 17, a guide "on how to ensure that Google-Agent has access to your website," aimed at publishers that want to enable it.
- How it identifies itself: each request carries "Google-Agent" in the User-Agent header and comes from an IP address in the public
user-triggered-agents.jsonlist. - Experimental signatures: Google is also testing the Web Bot Auth protocol with the identity
https://agent.bot.goog. Those requests are signed with the HTTP Message Signatures standard (RFC 9421), and the verification keys are public. - What the guide asks: if your service does not recognize it automatically, check the User-Agent, the IP and, when present, the signature; then skip bot challenges for verified requests.
- Where it usually breaks: Google points to rules that block data-center IPs or the "AI agent" category ahead of the allow rule, CAPTCHAs on search and forms, per-IP rate limits (many users share the same egress IPs) and proxies that drop the original IP or headers.
- Not entirely new: according to Search Engine Roundtable, the Google-Agent name already existed in March; the guide is what is new.
What changes and what doesn't
- robots.txt does not stop it. Google-Agent is listed among Google's "user-triggered fetchers," which, Google says, generally ignore robots.txt rules. If you want to limit it, do it in your CDN or firewall.
- It is not Googlebot. Google's documentation lists Google-Agent separately from Googlebot, the crawler that indexes your site for Search. The guide says nothing about ranking effects: allowing one is not allowing the other, so review them separately.
- It is your call. The guide is for publishers that want to enable it. A store or a business that takes bookings may want an agent to finish the purchase; another site may prefer not to let it act.
- What the guide does not say: it gives no traffic figures for Google-Agent and does not name which Google products use it; it speaks of agents acting "upon user request."
How to tell if you are affected
- Look for Google-Agent in your logs. In your host's access logs or your CDN dashboard, filter for a User-Agent containing "Google-Agent" and check the response: a 403, a 429 or a challenge page means it is being blocked.
- Review your bot rules. In Cloudflare or another CDN, check whether you block "AI bots," cloud IPs or data centers. Google asks that the rule allowing Google-Agent run before those blocks.
- Test beyond the homepage. The guide asks that the allowance cover on-site search, filters, product pages, cart, bookings and the site's API calls, not just the first page.
- Do not trust the name alone. Anyone can type "Google-Agent" into a header. If you allow it, verify it against the official IP list or the signature, as Google advises.
- If you would rather keep it out, use a CDN or firewall rule, not robots.txt, and revisit the decision when your sales model changes.
Related: Gemini Tasks Goes Free: What Businesses Should Check
Sources
- Google, "Allow Google-Agent traffic," crawling infrastructure documentation
- Google, crawling documentation changelog, November 17, 2026
- Google, "List of Google user-triggered fetchers"
- Search Engine Roundtable, "New Google Search Help Document On How To Allow Google-Agent Traffic," November 17, 2026
Updates: any changes backed by a primary source will be added here.
Worried about how this affects your website?
We will go over what to change with you. It is a free 30-minute call.