- AI companies run separate bots for training models, indexing pages for AI search and fetching a page when a user asks. The search and retrieval bots decide whether your products get cited, so keep them allowed.
- Per each company's documentation, blocking a training bot such as GPTBot, ClaudeBot or Google-Extended does not remove you from its search answers. Training is a business decision; search access is a visibility decision.
- If you name a bot in robots.txt, it follows only that group and ignores your general rules, so repeat your cart, checkout and account disallows. On Shopify, all of this lives in the robots.txt.liquid template.
What are AI crawlers, and why do they matter to an online store?
AI crawlers are bots that AI companies use to read the web. Each identifies itself with a user-agent name, and most check your robots.txt file first. The big providers now split their bots by purpose:
- Training crawlers collect content that may be used to train future models.
- Search crawlers index pages so the AI product can find and cite them when answering questions.
- User-initiated fetchers visit a specific page because a person asked the assistant something that needs it.
For a store, the second and third groups matter most. When a shopper asks ChatGPT, Claude or Perplexity for the best running shoes for flat feet, the answer draws on pages those systems can reach. That is the foundation of generative engine optimization, and our guide to getting your store cited in AI search covers the content side.
Which AI user agents should a store know?
The table lists user agents each company documents at the time of writing. Lists change, so check each provider’s crawler page before editing.
| Company | User agent | Job | Follows robots.txt? | Sensible default for a store |
|---|---|---|---|---|
| OpenAI | OAI-SearchBot | Indexes pages for ChatGPT search | Yes | Allow |
| OpenAI | GPTBot | Collects content that may train models | Yes | Your choice |
| OpenAI | ChatGPT-User | Fetches a page when a user asks | OpenAI says rules may not apply | Allow |
| Anthropic | Claude-SearchBot | Indexes pages for Claude’s search results | Yes | Allow |
| Anthropic | ClaudeBot | Collects content that may train models | Yes | Your choice |
| Anthropic | Claude-User | Fetches a page when a user asks | Yes, per Anthropic | Allow |
| Perplexity | PerplexityBot | Indexes pages for Perplexity search, not training | Yes | Allow |
| Perplexity | Perplexity-User | Fetches a page when a user asks | Perplexity says it generally ignores robots.txt | Allow |
| Googlebot | Google Search, including AI Overviews and AI Mode | Yes | Allow | |
| Google-Extended | Robots.txt token for training and grounding in Gemini Apps and Vertex AI; no separate crawler | Yes | Your choice | |
| Apple | Applebot | Search features in Siri, Spotlight and Safari | Yes | Allow |
| Apple | Applebot-Extended | Robots.txt token for training Apple’s foundation models; does not crawl | Yes | Your choice |
A few details from the documentation are worth knowing:
- OpenAI says each setting is independent: you can allow OAI-SearchBot and disallow GPTBot.
- Anthropic says disabling Claude-SearchBot or Claude-User may reduce your visibility in Claude’s search results, and that restricting ClaudeBot signals your content should be excluded from training.
- Google states that Google-Extended does not affect a site’s inclusion or ranking in Google Search. For AI Overviews, Google says a page must be indexed and eligible to be shown with a snippet, and the controls are the usual ones: Googlebot access, nosnippet, max-snippet and noindex.
- Apple says pages that disallow Applebot-Extended can still appear in its search results.
Should an eCommerce store block AI training crawlers?
There is no single right answer, but the trade-off is clearer for stores than for publishers.
A publisher’s product is its text. A store’s product is the thing in the box, and its product names, specs and brand story are written to be repeated. A model learning that your brand makes a well-reviewed hiking boot is not lost revenue.
Reasons a store might still block training bots:
- Your original long-form content, such as research or detailed guides, is a real competitive asset.
- You have a policy or legal position on AI training (confirm it with counsel).
- Training bot crawl load is hurting server performance.
Either way, keep search and retrieval bots allowed on every page you want customers to find.
What should stay blocked for every crawler?
Pages with no value to an outside reader, or with endless URL variations, should be disallowed for every crawler:
- Cart, checkout, order and account pages.
- Internal search results (on-site search URLs such as
/search?q=). - Sort orders and multi-filter combinations from faceted navigation. Our guide to faceted navigation SEO covers which filters deserve to be indexable.
- Admin and staging paths.
Shopify’s default file already disallows most of these, including admin, cart, checkout, orders and account paths, sort parameters and multi-filter collection URLs. WooCommerce and Magento defaults vary, so check yours.
Why can naming an AI bot accidentally unblock your checkout?
This is an easy technical error to make with AI crawler rules. Under the robots.txt standard, a crawler follows the one group that most specifically matches its user agent and ignores User-agent: * entirely. So this file gives OAI-SearchBot access to the cart and account pages:
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
User-agent: OAI-SearchBot
Allow: /
Only name a bot when you want it treated differently, and then repeat the general disallows in its group. Several user agents can share one group:
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /search
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /
Sitemap: https://www.example.com/sitemap.xml
The search bots are not named, so they follow the general group. The two training bots are blocked site-wide. The last line points every crawler to your XML sitemap.
How do you edit robots.txt on Shopify, WooCommerce and Magento?
Shopify and Shopify Plus
To change Shopify’s generated file, add a template named robots.txt.liquid to your theme’s Templates folder; it then replaces the generated output. Shopify’s documentation recommends keeping the default groups through the Liquid robots object and adding new groups as plain text after the loop:
{% for group in robots.default_groups %}
{{- group.user_agent }}
{%- for rule in group.rules -%}
{{ rule }}
{%- endfor -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}
User-agent: GPTBot
Disallow: /
Shopify warns that the rendered output does not always match the generated file. Save your current /robots.txt first, work on a duplicate theme and compare the two line by line.
WooCommerce (WordPress)
WordPress serves a virtual robots.txt. Most SEO plugins include an editor, or you can upload a physical robots.txt to the site root, which takes precedence. Check that security plugins are not adding their own bot rules.
Magento and Adobe Commerce
Edit the custom instructions under Content, Design, Configuration, in the Search Engine Robots section for each website or store view.
Does robots.txt actually stop AI crawlers?
Robots.txt is voluntary. The documented bots say they follow it, with the user-initiated exceptions in the table, but it does not stop scrapers that ignore it. Two other layers can block bots without you noticing:
- Your CDN or firewall. Cloudflare, for example, began blocking known AI crawlers by default for new domains in July 2025. A bot blocked at the network edge is as blocked as a disallow. Check bot settings in any CDN or security tool.
- Rendering. Do not assume every AI crawler runs JavaScript the way Googlebot does. If prices, specs or reviews only appear after scripts load, a bot may see an empty page. Keep core product content and structured data in the initial HTML.
Is llms.txt worth adding?
Llms.txt is a proposal from 2024 for a markdown file at the site root that summarizes a site’s key pages for language models. It is not a standard, and none of the crawler documentation from OpenAI, Anthropic, Perplexity, Google or Apple cited above mentions it. Google’s John Mueller said in January 2026 that Google hosting llms.txt files on some developer docs is not an endorsement.
It is cheap and harmless to add, but it does not replace crawlable HTML, clean product schema and fast pages. Do those first.
Worked example: fixing a store’s AI crawler rules
The scenario below is illustrative.
A Shopify apparel store pasted AI rules from a forum post. Its file named eight AI user agents with Disallow: /, including OAI-SearchBot, Claude-SearchBot and PerplexityBot. The goal was to stop training. The effect was to remove the store from three AI search products. The fix took four steps:
- Separate jobs. The team chose to allow training, since its content was mostly product copy, and removed every named AI group.
- Keep hygiene. The general group kept Shopify’s defaults for cart, checkout, account and filter URLs.
- Check the edge. No bot-blocking rule was active in its CDN or apps.
- Monitor. It tracks referral sessions from chatgpt.com and perplexity.ai in GA4 monthly and checks AI answers to its core product questions by hand.
AI crawler checklist
- [ ] Decide separately on training bots and search or retrieval bots.
- [ ] OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Applebot can reach product, collection and guide pages.
- [ ] Any named bot group repeats the cart, checkout, account and search disallows.
- [ ] Training opt-outs, if wanted, use GPTBot, ClaudeBot, Google-Extended and Applebot-Extended.
- [ ] On Shopify, robots.txt.liquid keeps the default groups via the Liquid loop.
- [ ] The rendered file was compared with the old one before publishing.
- [ ] CDN, firewall and security plugins are not blocking allowed bots.
- [ ] Product names, prices and specs appear in the initial HTML.
- [ ] AI referral traffic is reviewed monthly.
FAQ
Will blocking GPTBot remove my store from ChatGPT search?
According to OpenAI’s documentation, no. GPTBot relates to model training. OAI-SearchBot controls whether your pages can appear in ChatGPT search answers, and the two settings are independent.
Does blocking Google-Extended keep my products out of AI Overviews?
No. Google says Google-Extended does not affect inclusion or ranking in Google Search. AI Overviews draw on pages that are indexed and eligible for a snippet, so the controls are Googlebot access, nosnippet, max-snippet and noindex.
How quickly do robots.txt changes take effect for AI bots?
It depends on how often each bot rechecks your file. OpenAI says about 24 hours for ChatGPT search. Plan for a day or more with other providers.