Which AI crawlers should a merchant allow to access their site?

For maximum agentic-commerce visibility, allow all major AI crawler types. Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Amazonbot, Applebot-Extended) for inclusion in LLM training corpora. Grounding and search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) for real-time answer citations. User-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User) for URL visits triggered by end users. Check both robots.txt and any CDN or WAF (like Cloudflare) — the CDN often overrides robots.txt with its own default AI-bot rules.

What are the three layers of agentic commerce readiness?

Access (whether an agent can reach your site), legibility (whether an agent can read and understand what it finds), and execution (whether an agent can complete checkout). They fail independently, and access almost always fails first, silently.

Why do fraud detection systems block legitimate AI shopping agents?

The behavioral signals that flag a malicious bot are the same as those from a legitimate shopping agent: speed of navigation, headless-browser signature, non-human timing. Some managed CDN and WAF rule sets now block known AI agents by default, returning 403s to agent traffic that never shows up in the merchant's own analytics.

How do you test if your website is blocking AI agents?

Request one of your product pages twice with curl, once with a browser User-Agent and once with an AI agent User-Agent like OAI-SearchBot, and compare the HTTP status codes. If the browser gets 200 and the agent gets 403, 429, or 503, your edge is treating agents differently.

Agentic Commerce Readiness

Q: What's the difference between robots.txt and a web application firewall?

robots.txt is advisory. It is a request an honest crawler chooses to honor, and it enforces nothing. A web application firewall enforces. It sits at the edge, inspects traffic, and can return a 403 before your application ever sees the request. A store can leave robots.txt wide open and still block every agent at the WAF without knowing it.

Q: What is Know Your Agent (KYA)?

One direction being explored for verifiable agent identity. KYA approaches attach a cryptographic credential to agent traffic (built on HTTP Message Signatures, RFC 9421) that links the request, the agent, and the human or enterprise directing it. The question the firewall is asking becomes whether an agent can prove who it is and who it acts for.

What it takes for an AI agent to reach, read, and buy from your store.

Agentic commerce readiness is how prepared a store is for AI agents to shop it. It comes down to three separate requirements: access (can an agent reach the site), legibility (can an agent read the page), and execution (can an agent finish checkout). Each fails independently. When the buyer is software, the same storefront that works for people has to also work for automated systems that read the page, evaluate products, and complete a purchase on someone's behalf.

The failure almost always shows up at the earliest weak layer, which is why the order matters: a perfectly structured catalog is worth nothing to an agent that was turned away at the edge. Being surfaced to an agent before any of this is a separate layer, discovery, covered on the Discovery guide.

Access: can an agent get in?

Access is first because it is where most agents are lost, and they are lost silently. The behavioral signals that flag a malicious bot are the same signals a legitimate shopping agent produces: speed of navigation, a headless-browser signature, non-human timing. A fraud system sees the same thing whether the traffic is an attacker or an agent completing a real task for a real person. The security infrastructure that has protected e-commerce for years is now the main thing standing between agents and the ability to buy.

Two different layers decide who gets in, and they are routinely confused. robots.txt is advisory. It is a request an honest crawler chooses to honor, and it enforces nothing. A web application firewall enforces. It sits at the edge, inspects traffic, and can return a 403 before your application ever sees the request. A store can leave robots.txt wide open and still block every agent at the WAF without knowing it, because the two are configured in different places, often by different people.

The taxonomy matters here, because "AI crawler" is not one thing. Training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended) collect data to train models. Grounding and search crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot) fetch pages at answer time to cite them to a user right now. Blocking the first set is an opt-out of model training. Blocking the second set removes you from the answers agents are giving live, and a single "block AI bots" control on a CDN often does both. A store can opt out of training and, without meaning to, disappear from the assistants its own customers are using.

Crawler	Operator	Type	What blocking it costs
`GPTBot`	OpenAI	Training	Inclusion in ChatGPT training corpus
`OAI-SearchBot`	OpenAI	Grounding / search	Real-time ChatGPT citations
`ChatGPT-User`	OpenAI	User-initiated	ChatGPT fetches when a user asks it to visit a URL
`ClaudeBot`	Anthropic	Training	Claude training data
`Claude-SearchBot`	Anthropic	Grounding / search	Real-time Claude citations
`Claude-User`	Anthropic	User-initiated	Claude fetches when a user asks it to visit a URL
`PerplexityBot`	Perplexity	Grounding / search	Real-time Perplexity citations
`Google-Extended`	Google	Training	Gemini and Vertex AI training corpus
`CCBot`	Common Crawl	Training (upstream)	The training corpus that feeds most major LLMs

Some managed CDN and WAF rule sets now block known AI agents by default, returning 403s to agent traffic that never shows up in the merchant's own analytics. Checking what the edge is already doing to agent traffic comes before optimizing product pages, because for many stores the edge is quietly turning agents away.

The mechanisms for letting the right agents in are still maturing. At the crude end, verified-bot allow-listing matches a known User-Agent and its published IP ranges and lets it through. It is brittle, because a User-Agent string is trivially spoofed and an allow-list is only as current as the last time someone updated it. For legitimate load, a firewall can answer a sudden spike of automated requests (an agent firing off many product lookups in quick succession, which reads like a denial-of-service attempt) with a 429 and a Retry-After header instead of a hard block, telling a well-behaved agent to slow down rather than turning it away. One direction being explored is verifiable identity: Know Your Agent (KYA) approaches attach a cryptographic credential to agent traffic (built on HTTP Message Signatures, RFC 9421) that links the request, the agent, and the human or enterprise directing it. The question the firewall is asking becomes whether an agent can prove who it is and who it acts for.

Legibility: can an agent read you?

Once an agent is through the door, it has to understand what it is looking at, and it leans on machine-readable structure far more than a person does. A human infers from a photo and a price that a product is in stock and how to buy it. An agent needs that stated.

The first requirement is that the content actually exists in the HTML the agent receives. Many storefronts render product data client-side: the initial HTML is close to empty and JavaScript fills it in the browser. Fetch-based agent pipelines often do not run that JavaScript, so they get the empty shell. Server-side rendering (or static generation, or prerendering for agent traffic) is what makes the product legible at all. It is the same failure mode that hides JavaScript-loaded images from AI crawlers unless there is real markup behind them.

On top of that sits the structured layer: schema.org/Product markup for price, availability, and identifiers; semantic HTML with real labels rather than styled div soup; and stable identifiers (SKUs, GTINs) that let an agent refer to exactly one variant. A curated /llms.txt file at the domain root can hand agents a plain-text index of what matters (collections, policies, key pages) instead of making them infer the shape of the site.

Legibility also has a time dimension that human shopping forgives. Software acts immediately on what the page shows, so price and availability need to be current at request-time rather than cached from minutes earlier.

Execution: can an agent complete checkout?

Reading a product is not the same as completing a purchase, and execution is where the open web fights back. Every merchant checkout is structurally unique. A flow an agent completes on one store breaks on the next because a dropdown became a radio-button group, a field moved, or a button's label changed. There is no shared checkout grammar across the web the way there is a shared HTML.

What makes a checkout executable by an agent is boringly consistent: labeled buttons and fields, stable identifiers that hold between sessions, and an unambiguous single path from cart to confirmation. Completion in browser-agent settings turns less on model intelligence than on whether the page exposes plain markup and clear controls the agent can act on without guessing. A capable model still cannot reliably click a control it cannot identify.

This is the layer the emerging commerce protocols are trying to remove entirely, by giving agents a structured checkout endpoint instead of a rendered page to operate. That path is covered on the Protocols guide.

Test your own store

Two of the three layers are observable from the command line, because access and legibility both come down to what your server returns over HTTP. The tool for both is curl, and the useful part is that curl does not execute JavaScript... it sees a page much the way a fetch-based agent does.

Access: is your edge already blocking agents? Request one of your own product pages twice, once as a browser and once as an AI agent, and compare the status codes.

# what a browser sees
curl -sL -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0" https://yourstore.com/a-product

# what an AI agent sees
curl -sL -o /dev/null -w "%{http_code}\n" \
  -A "OAI-SearchBot/1.0 (+https://openai.com/searchbot)" \
  https://yourstore.com/a-product

If the browser request returns 200 and the agent request returns 403, 429, or 503, your edge is treating agents differently. Read your robots.txt directly at the same time:

curl -s https://yourstore.com/robots.txt

...looking for Disallow rules aimed at GPTBot, OAI-SearchBot, or PerplexityBot. One honest limit: this checks User-Agent-string rules only. Some firewalls also verify real crawlers by IP or reverse DNS, so a spoofed User-Agent can read as allowed when a verified bot would be blocked, or the reverse. Treat it as a first signal, not proof.

Legibility: is your product actually in the HTML? Fetch a product page and look for the details an agent needs in order to buy... price, availability, and structured data.

# is the price in the HTML, or injected later by JavaScript?
curl -sL https://yourstore.com/a-product | grep -iE "price|in stock|add to cart"

# is schema.org structured data present in what is served?
curl -sL https://yourstore.com/a-product | grep -i "ld+json"

If price and availability do not appear, they are not in the HTML... they are rendered client-side, and an agent that does not run your JavaScript cannot see them. That one result is the legibility problem made concrete.

Execution is the layer you cannot check this way. A checkout is stateful and usually driven by JavaScript, so no single request exercises it. The realistic test is to run an actual browser agent through a purchase on a test order, or to audit that every control in the flow carries a stable identifier and a real label rather than relying on its position on the page.

Fraud is not the barrier

Fraud is not where agentic commerce breaks. The payments layer runs at very low fraud, because agent credentials are tokenized and scoped in ways covered on the Payments guide. The expensive failures happen higher up, at access and execution, where a legitimate order gets flagged and cancelled after the payment has already cleared. The real barrier to agentic commerce usually sits inside a merchant's own infrastructure, rejecting a real purchase it could not tell apart from an attack.

What's durable, what's in flux

Not all of this ages at the same rate, and it is worth separating the work that will still be true in two years from the parts that are actively moving.

Durable. Server-side rendering, schema.org markup, semantic HTML, and stable identifiers pay off for human shoppers, for search engines, and for every agent model regardless of which one shows up or which protocol wins. None of it is a bet on a specific vendor. It is the layer that was already good practice and simply matters more now.

In flux. Access policy is moving fast: User-Agent allow-lists are a stopgap on the way to verifiable agent identity, and the standards for that identity are not settled. Agent-optimized delivery, serving lightweight markdown or JSON to identified agents rather than a full page, is an emerging optimization rather than a settled practice. And the execution layer may be routed around entirely if structured commerce protocols get adopted.