AI Crawlers List: Every Major AI Bot and What It Does

Last updated: 7 November 2026

The three types of AI crawler

The most common mistake we see is a business blocking every AI bot to protect its content from training, and accidentally removing itself from AI search at the same time. Separating these three types solves that.

AI crawlers list: the main bots

This list covers the crawlers most businesses need to think about. New ones appear regularly, and companies sometimes rename or split their bots. Check each company’s official crawler documentation before you make important changes.

OpenAI crawlers

OpenAI separates its crawlers clearly:

• GPTBot collects content that may be used to train OpenAI’s models. Blocking it opts your future content out of training.
• OAI-SearchBot crawls pages so they can be shown and linked in ChatGPT search. If you want to appear as a source in ChatGPT answers, allow it.
• ChatGPT-User fetches pages when a user’s request in ChatGPT needs them. Because a person asked, OpenAI has said robots.txt rules may not apply to it in the same way.

A common setup is to allow OAI-SearchBot and ChatGPT-User and block GPTBot, which keeps you in ChatGPT search while opting out of training. Our guide to getting indexed by ChatGPT search covers the full process.

Anthropic crawlers

Anthropic, the company behind Claude, uses three bots:

• ClaudeBot collects content that could contribute to training. Blocking it signals that your future content should be excluded from training datasets.
• Claude-SearchBot crawls to improve search results for Claude users. Blocking it stops your content being indexed for Claude’s search, which may reduce your visibility there.
• Claude-User opens pages when a Claude user asks a question. Blocking it may reduce visibility in user-directed searches.

Older guides mention user agents such as “anthropic-ai” and “Claude-Web”. Anthropic’s current documentation focuses on the three bots above.

Perplexity crawlers

• PerplexityBot surfaces and links websites in Perplexity’s search results. Perplexity says it is not used to crawl content for AI foundation models, and it respects robots.txt.
• Perplexity-User fetches pages when a user asks Perplexity a question. Perplexity says that because a user requested the fetch, it generally ignores robots.txt rules.

If you want to appear in Perplexity answers, allow PerplexityBot. Our Perplexity SEO guide explains what else it looks for.

Google crawlers and controls

Google is different from other AI companies because its AI features are part of Google Search.

• Googlebot crawls for Google Search. AI Overviews and AI Mode draw on the Search index, so Googlebot access is what matters for those features.
• Google-Extended is a robots.txt token, not a separate crawler. It lets you control whether content Google crawls can be used to train Gemini models and for grounding in Gemini apps and Vertex AI. It does not affect your inclusion or ranking in Google Search, and it does not remove you from AI Overviews.
• GoogleOther is used by Google teams for research and development crawling outside Search.

Blocking Googlebot to avoid AI Overviews would remove you from Google Search entirely, which is almost never a good idea. See Google AI Mode vs AI Overviews for how those features work.

Microsoft, Apple, Meta and others

Bingbot

Bingbot powers Bing search, which also supports Microsoft Copilot answers and is widely reported to be one of the sources behind ChatGPT search. Blocking Bingbot can hurt visibility in several AI assistants at once.

Applebot and Applebot-Extended

Applebot crawls for Apple’s search features in Siri, Spotlight and Safari. Applebot-Extended is a control token: disallowing it tells Apple not to use your content to train its generative AI models, while Applebot can still crawl for search features.

Meta-ExternalAgent and Meta-ExternalFetcher

Meta-ExternalAgent collects content for Meta’s AI models and products. Meta-ExternalFetcher fetches pages for user-initiated actions. Decide on Meta-ExternalAgent the same way you decide on other training crawlers.

CCBot

Common Crawl builds an open archive of the web that many AI developers use for training. Blocking CCBot reduces the chance your content ends up in datasets built from that archive, but it does not affect any company’s live AI search.

Amazonbot, Bytespider, DuckAssistBot and MistralAI-User

Amazonbot supports Amazon services including Alexa. Bytespider collects content for ByteDance AI products and has been reported in the past to crawl heavily. DuckAssistBot fetches pages for DuckDuckGo’s AI-assisted answers. MistralAI-User fetches pages for Mistral’s Le Chat assistant.

Which AI crawlers should you allow?

There is no single right answer, but here is the decision most of our clients make:

Businesses that want maximum visibility often allow training crawlers too, on the basis that models trained on accurate information about them are more likely to describe them correctly. Publishers that sell content often block them. Both are reasonable. What is rarely reasonable is blocking search crawlers by accident.

For a deeper look at the training decision, see our guide to the AI training opt-out with robots.txt.

Recommended setups by business type

Whatever you decide, keep paid or private content behind authentication. robots.txt alone does not protect it.

How to verify a real AI crawler

Anyone can put “GPTBot” or “ClaudeBot” in a user agent string. If you see suspicious traffic claiming to be an AI crawler, verify it:

• Check published IP ranges. OpenAI and other companies publish the IP addresses their crawlers use. Compare the request IP.
• Use reverse DNS for search engines. Googlebot and Bingbot can be verified by reverse DNS lookup to their official domains, followed by a forward lookup.
• Use your CDN’s verified bot list. Many CDNs label verified bots automatically.
• Look at behaviour. Real crawlers usually request robots.txt and follow consistent patterns. Impostors often hammer specific URLs.

Block impostors at the firewall. Do not change robots.txt for them, because they ignore it anyway.

Platform notes

WordPress

SEO plugins such as Yoast let you edit robots.txt, and some security plugins add AI bot blocking. Check both, and make sure the “discourage search engines” setting is off.

Shopify

Customise rules through the robots.txt.liquid template. Check any bot protection apps.

Webflow

Edit robots.txt in site settings. Make sure staging domains are not indexed.

Cloudflare

Review AI bot blocking options and verified bot settings. A single toggle can block AI search crawlers across your whole site.

Example robots.txt for AI crawlers

This example allows search and user-fetch bots and blocks training crawlers. Adjust it to your own decision.

# AI search and user-triggered fetchers: allow
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# AI training crawlers: block
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
Disallow: /

# Training and grounding control tokens
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Grouping several user agents above one set of rules is valid in the robots.txt standard, but if you are unsure, write a separate block for each bot. Our guide to checking AI crawler access in robots.txt shows how to test that your rules do what you intend.

robots.txt is not the whole story

robots.txt is a request, not a lock. Well-behaved crawlers follow it, but:

• Some user-triggered fetchers may not follow it in the same way, as their operators explain.
• Firewalls, CDNs and hosting security can block bots even when robots.txt allows them.
• Content loaded only by JavaScript may be invisible to some crawlers even when they are allowed.
• Bad actors can fake user agent names. Companies such as OpenAI publish IP ranges so you can verify real requests.

If you need to truly prevent access, use server-level controls. If you want visibility, check that nothing at the server level blocks the bots you allowed.

How to see which AI crawlers visit your site

1. Check server logs. Filter requests by user agent for the names in this list.
2. Check your CDN dashboard. Many CDNs show bot traffic by category and let you see which AI bots were allowed or blocked.
3. Verify IPs for important bots. Compare request IPs against published ranges to rule out impostors.
4. Watch response codes. Repeated 403 or 429 responses to AI bots usually mean a firewall or rate limit is stopping them.
5. Run a quick external check. Our free AI visibility checker reads your robots.txt and tests access the way AI crawlers would.

What happens if you block each crawler

Keeping your AI crawler rules current

The AI crawler landscape changes every few months. Companies launch new bots, split one bot into several, or change what an existing bot does. Anthropic, for example, now documents three separate bots where older guides listed different names. A rule written in 2024 may block the wrong things in 2026.

A simple routine keeps you safe:

1. Review twice a year. Check this list and each company’s official documentation for changes.
2. Check your logs. Look for new AI user agents you have not made a decision about.
3. Retest access. After any change to robots.txt, your CDN or your hosting, run a quick access check.
4. Write down your policy. A short note on which types of bots you allow and why makes future decisions faster and keeps your team consistent.

Common mistakes with AI crawlers

• One rule for all bots. Blocking every AI crawler removes you from AI search along with training.
• Using outdated names. Blocking old user agents that no longer exist while newer bots get through, or the reverse.
• Forgetting the CDN. robots.txt says yes while the firewall says no.
• Blocking Googlebot to avoid AI Overviews. This removes you from Google Search entirely.
• Assuming Google-Extended affects AI Overviews. It does not.
• Setting and forgetting. Review your rules at least twice a year as new crawlers appear.

How Digital Tatva can help

Getting AI crawler access right is one of the fastest wins in AI visibility. Our technical AEO service reviews robots.txt, CDN and firewall rules, server logs and rendering, then sets up crawler access that matches your goals: visible in AI search, with training use under your control. It is part of every AI visibility audit we run.

Frequently asked questions