Illustration of a security shield with an open door that a cursor passes through, in front of a browser window

Google-Agent: How to Keep Your Site From Blocking It

Admin

On November 17, 2026, Google published a technical guide to help websites avoid blocking Google-Agent by mistake. Google-Agent is the name of its agents that browse and take actions on a user's behalf. It matters to anyone running a CDN, a web application firewall or a bot-detection service in front of their site.

What we know

  • What Google-Agent is: according to Google's documentation, it is the name used by agents hosted on Google infrastructure to navigate the web and perform actions when a user asks.
  • What Google published: the changelog of its crawling documentation lists, on November 17, a guide "on how to ensure that Google-Agent has access to your website," aimed at publishers that want to enable it.
  • How it identifies itself: each request carries "Google-Agent" in the User-Agent header and comes from an IP address in the public user-triggered-agents.json list.
  • Experimental signatures: Google is also testing the Web Bot Auth protocol with the identity https://agent.bot.goog. Those requests are signed with the HTTP Message Signatures standard (RFC 9421), and the verification keys are public.
  • What the guide asks: if your service does not recognize it automatically, check the User-Agent, the IP and, when present, the signature; then skip bot challenges for verified requests.
  • Where it usually breaks: Google points to rules that block data-center IPs or the "AI agent" category ahead of the allow rule, CAPTCHAs on search and forms, per-IP rate limits (many users share the same egress IPs) and proxies that drop the original IP or headers.
  • Not entirely new: according to Search Engine Roundtable, the Google-Agent name already existed in March; the guide is what is new.

What changes and what doesn't

  • robots.txt does not stop it. Google-Agent is listed among Google's "user-triggered fetchers," which, Google says, generally ignore robots.txt rules. If you want to limit it, do it in your CDN or firewall.
  • It is not Googlebot. Google's documentation lists Google-Agent separately from Googlebot, the crawler that indexes your site for Search. The guide says nothing about ranking effects: allowing one is not allowing the other, so review them separately.
  • It is your call. The guide is for publishers that want to enable it. A store or a business that takes bookings may want an agent to finish the purchase; another site may prefer not to let it act.
  • What the guide does not say: it gives no traffic figures for Google-Agent and does not name which Google products use it; it speaks of agents acting "upon user request."

How to tell if you are affected

  1. Look for Google-Agent in your logs. In your host's access logs or your CDN dashboard, filter for a User-Agent containing "Google-Agent" and check the response: a 403, a 429 or a challenge page means it is being blocked.
  2. Review your bot rules. In Cloudflare or another CDN, check whether you block "AI bots," cloud IPs or data centers. Google asks that the rule allowing Google-Agent run before those blocks.
  3. Test beyond the homepage. The guide asks that the allowance cover on-site search, filters, product pages, cart, bookings and the site's API calls, not just the first page.
  4. Do not trust the name alone. Anyone can type "Google-Agent" into a header. If you allow it, verify it against the official IP list or the signature, as Google advises.
  5. If you would rather keep it out, use a CDN or firewall rule, not robots.txt, and revisit the decision when your sales model changes.

Related: Gemini Tasks Goes Free: What Businesses Should Check

Sources

Updates: any changes backed by a primary source will be added here.

Worried about how this affects your website?
We will go over what to change with you. It is a free 30-minute call.