How to Configure robots.txt for Modern Crawlers and AI Bots

Your robots.txt file now has two audiences. One is traditional search engines. The other is the AI systems that answer questions using your content. A modern configuration keeps your important pages crawlable for Google, Bing, ChatGPT, Claude, and Perplexity. It also gives you deliberate control over model training. Before you change anything, run your current file through our free robots.txt validator so you know exactly which crawlers it allows or blocks today.

This guide is written for eCommerce and content-driven websites. If you’re new to the file itself, start with our explainer on how robots.txt works for search engine crawlers, then come back here for the AI-specific rules.

Key Takeaways

• OpenAI and Anthropic each run separate crawlers for search, user-requested fetches, and training. You can block training without disappearing from ChatGPT or Claude search.
• Some bots fetch pages on a user’s behalf, and robots.txt may not stop them. OpenAI says this about ChatGPT-User, and Perplexity says it about Perplexity-User.
• Google-Extended and Applebot-Extended are control tokens, not crawlers. Neither affects your ranking in Google Search or Apple search.
• A shared AI-training opt-out line (__usage_ai-training) has been announced, but no major provider honors it yet. Keep your per-bot rules for now.
• Allowing a bot in robots.txt doesn’t let it past Cloudflare or another firewall. Check both layers.

1. Understand the different types of crawlers

AI bots fall into three groups, and each group calls for a different decision:

• Search crawlers build the indexes that AI search products cite.
• User-requested fetchers retrieve a page because a person asked about it.
• Training crawlers and control tokens decide whether your content can be used to train models.

Blocking one group has very different consequences from blocking another.

Two details are easy to miss:

• Google-Extended never visits your site. Google’s regular crawlers fetch the pages. The token only tells Google how it may use that content afterward. You won’t find it in your server logs.
• Some tokens are retired. Older block lists often include anthropic-ai and Claude-Web. Anthropic no longer uses them, so rules naming them do nothing. Anthropic’s current tokens are ClaudeBot, Claude-SearchBot, and Claude-User.

2. Recommended robots.txt configuration

This starting point suits an eCommerce or marketing site. It aims for visibility in Google, Bing, ChatGPT, Claude, and Perplexity while restricting two training crawlers.

# Default rules: applies to Googlebot, Bingbot, and any crawler without its own group
User-agent: *
Allow: /
Disallow: /internal-search/
Disallow: /private/

# AI search and user-requested retrieval
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /internal-search/
Disallow: /private/

# Optional: opt out of OpenAI model training
User-agent: GPTBot
Disallow: /

# Optional: opt out of Anthropic model training
User-agent: ClaudeBot
Disallow: /

# Optional: opt out of Apple foundation-model training (does not affect Applebot search crawling)
# User-agent: Applebot-Extended
# Disallow: /

# Optional: opt out of Gemini training AND Gemini grounding (does not affect Google Search)
# User-agent: Google-Extended
# Disallow: /

# XML sitemap
Sitemap: https://www.example.com/sitemap.xml

Before publishing:

• Replace the example directories with real paths on your site.
• Confirm the sitemap URL.
• Keep any existing platform-specific rules.

Why the AI search group repeats the blocked paths. Under Google’s interpretation of the standard, a crawler follows only the most specific group that names it. That group is not merged with User-agent: *. Once a bot has its own group, it ignores your default rules. Any path you want blocked has to appear in its group too.

Why the AI search group exists at all. As written, it matches the * group, so it changes nothing today. Its value is protection. If you later tighten the * group, your AI search visibility won’t change by accident. Listing several user agents above one rule set keeps those paths in a single place to update.

What this configuration does not do. It does not maximize access for every AI system. It deliberately opts out of two training crawlers. If you want unrestricted AI crawling, delete the GPTBot and ClaudeBot groups.

How long changes take. OpenAI says ChatGPT search takes about 24 hours to reflect a robots.txt change. Google generally caches the file for up to a day as well.

A note on user-requested fetchers

ChatGPT-User, Claude-User, and Perplexity-User act on a person’s request rather than crawling on a schedule. The three companies handle robots.txt differently for these bots:

• Anthropic states that all three of its bots, including Claude-User, honor robots.txt.
• OpenAI says robots.txt rules may not apply to ChatGPT-User. It also says ChatGPT-User doesn’t decide whether you appear in ChatGPT search. That job belongs to OAI-SearchBot.
• Perplexity says Perplexity-User generally ignores robots.txt.

The practical takeaway is to treat robots.txt as a reliable control for these companies’ search and training crawlers only. If content must never be fetched, put it behind authentication.

3. Should you block AI training crawlers?

This is a strategic choice, not a universal SEO best practice. Blocking a training crawler signals that your future content shouldn’t be used to train that company’s models. It does not undo past crawling.

Google-Extended needs its own thinking. Blocking it does two things at once:

1. It opts you out of training future Gemini models.
2. It removes your content from grounding in Gemini Apps and in Grounding with Google Search on Vertex AI.

Google’s documentation offers no way to allow one while blocking the other. It also confirms that Google-Extended has no effect on inclusion or ranking in Google Search.

Applebot-Extended is lower risk. In September 2026, Apple added a line to its Applebot documentation stating that Applebot-Extended rules are not considered in Search ranking.

4. What changing in 2027: the shared AI-training opt-out

On October 30, 2026, Google, Microsoft, and OpenAI announced plans to honor one shared robots.txt group for AI-training preferences. The group comes from an individual IETF Internet-Draft submitted by Anthropic’s Kevin Kelley. It looks like this:

User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

What to know before you use it:

• Nobody honors it yet. Google’s crawler documentation targets early 2027. OpenAI says it aims to support it within a few months. Anthropic has said full support will take time.
• Google maps it to Google-Extended. If you use this line and don’t name Google-Extended, Google plans to treat it as a Google-Extended block. That means opting out of training would also remove you from Gemini Apps grounding.
• Microsoft is splitting training from grounding. Microsoft will begin honoring its new training and grounding preferences on February 1, 2027. It will stop treating NoArchive and NoCache as AI controls after March 1, 2027.
• The wildcard can’t match it. A User-agent: * group never applies to this token, and a file without it expresses no preference.

Our recommendation: keep your per-bot rules (GPTBot, ClaudeBot, Google-Extended) as the controls that work today. Add the shared line only once you understand what it will mean for Gemini grounding.

5. Common robots.txt mistakes that affect SEO and AEO

Remember that robots.txt controls crawling, not indexing. Google can still index a blocked URL it discovers through links.

6. How to implement and validate

1. Open https://yourdomain.com/robots.txt and save a copy of the current rules.
2. List your important pages: homepage, product pages, collections, blog posts, and informational content.
3. Identify URL patterns that genuinely need restrictions, such as internal search, carts, and account pages.
4. Add only the bot-specific groups you need, and repeat your blocked paths inside each one.
5. Test important URLs against each crawler’s rules with the robots.txt validator.
6. Confirm the file returns HTTP 200 and is served as plain text at the root of every subdomain you care about.
7. Check your CDN and firewall settings, because robots.txt permission doesn’t get a bot past a security layer. Since September 15, 2026, Cloudflare blocks training and agent crawlers by default on ad-supported pages. This applies to new domains and to free-plan sites that haven’t changed their settings. Search crawlers stay allowed.
8. Monitor Google Search Console, Bing Webmaster Tools, and verified crawler requests in your server logs. User-agent strings can be spoofed, so verify against each company’s published IP ranges.

For Shopify stores: Shopify generates a default robots.txt for you. When you add a robots.txt.liquid template, it replaces that generated output. Build your template on the robots.default_groups Liquid object so Shopify’s default rules stay in place, then add your AI-specific groups below them. Compare the rendered output with your saved copy before publishing.

7. Does allowing AI crawlers improve AI visibility?

It improves access, not guaranteed citations or rankings.

• Crawl access: AI search systems can retrieve the pages you permit.
• Content discovery: Your articles and product information become candidates for AI answers.
• AI citations: These still depend on indexing, relevance, content quality, page accessibility, and how each AI system selects sources.

Our recommendation for eCommerce SEO and AEO is to keep Googlebot, Bingbot, and the major AI search crawlers able to reach your public, indexable content. Treat model-training permissions as a separate policy decision, not a ranking tactic.

Frequently Asked Questions

Does blocking GPTBot or ClaudeBot remove my site from ChatGPT or Claude answers?

No. GPTBot and ClaudeBot only control training. ChatGPT search relies on OAI-SearchBot, and Claude search relies on Claude-SearchBot and Claude-User. As long as those stay allowed, your pages remain eligible to be cited.

Will blocking Google-Extended hurt my Google rankings?

No. Google states that Google-Extended doesn’t affect inclusion or ranking in Google Search. It controls Gemini model training and grounding in Gemini Apps and Vertex AI. Google-Extended is also not the control for AI Overviews or AI Mode. According to Search Engine Journal, Google handles those through a separate generative AI setting in Search Console.

Can robots.txt stop ChatGPT-User or Perplexity-User from fetching my pages?

Not reliably. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. Both bots act on a specific person’s request. Anthropic says Claude-User does follow robots.txt. For content that must stay private, use authentication rather than robots.txt.

How long does a robots.txt change take to work?

Usually about a day. Google generally caches robots.txt for up to 24 hours. OpenAI says ChatGPT search takes roughly 24 hours to reflect an update. Other crawlers apply changes the next time they refetch your file.

Should I add the __usage_ai-training line now?

It’s optional and has no effect yet, because no major provider honors it today. Google plans to read it as a Google-Extended block, which would also remove you from Gemini Apps grounding. Learn those consequences first, and keep your GPTBot, ClaudeBot, and Google-Extended rules in place either way.

Do AI crawlers follow Crawl-delay?

Some do. Anthropic supports the non-standard Crawl-delay directive for its bots. Google ignores it entirely. If crawl load is a problem, use your server’s rate limiting or your CDN’s bot controls instead.

A well-configured robots.txt keeps search and AI search crawlers working for you, restricts training only where you’ve decided to, and never relies on robots.txt alone for anything that must stay private. Validate the file after every change, then recheck it quarterly. The crawler landscape is changing fast, and the shared training opt-out is likely to arrive in 2027.