Is your website blocking ChatGPT? Check in 2 minutes

Your website may be blocking ChatGPT without you knowing it. Open yourwebsite.com/robots.txt in a browser: if you see "Disallow: /" under GPTBot, OAI-SearchBot, ChatGPT-User or "User-agent: *", AI assistants are told not to read your site. Firewalls and hosting settings can block them too. Here's how to check each one and what to change.

Key takeaways

• Each AI company uses several bots: some fetch pages for live answers, others collect training data. You can treat them differently.
• Blocking search bots (like OAI-SearchBot or PerplexityBot) keeps you out of live AI answers. That's usually the one to avoid.
• Blocking only training bots (like GPTBot or ClaudeBot) doesn't remove you from ChatGPT's or Claude's search answers, according to OpenAI and Anthropic.
• Check three places: robots.txt, your firewall or CDN (for example Cloudflare), and whether your content needs JavaScript to appear.

Which bots do AI assistants use?

• OpenAI (ChatGPT): OAI-SearchBot for search answers, ChatGPT-User when a user's question opens a page, GPTBot for training.
• Anthropic (Claude): Claude-SearchBot for search, Claude-User when a user's question opens a page, ClaudeBot for training.
• Perplexity: PerplexityBot for its search index, Perplexity-User when a user's question opens a page.
• Google (Gemini and AI Overviews): Googlebot crawls for Google Search, including AI Overviews. Google-Extended is not a separate bot but a setting that controls whether your content is used for Gemini; blocking it doesn't affect your Google Search ranking.
• Microsoft (Copilot): Bingbot, the same bot that crawls for Bing search.

How to read your robots.txt file

robots.txt is a small public text file at the root of your website. Each block starts with "User-agent" (which bot it's about) followed by rules. "Disallow: /" means "don't read anything". This example blocks ChatGPT's training bot and its search bot:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

And this one blocks every bot that follows the rules, Google included. It sometimes stays behind by mistake after a website was built on a test address:

User-agent: *
Disallow: /

If the page shows "404" or is empty, you have no robots.txt, which means bots are allowed. That's fine.

What should a small business's robots.txt allow?

For most small businesses that want customers to find them, allow the search bots. Whether to allow training bots is your choice: blocking them doesn't hide you from live answers, but allowing them may help AI models know your business in the long run. A simple, open setup looks like this (replace the sitemap address with yours):

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://yourwebsite.com/sitemap.xml

Keep any rules your web developer added for private areas such as /admin or /cart; only remove rules that block the whole site.

Check 2: your firewall, CDN or hosting

A robots.txt file can say "welcome" while a firewall says "no". Cloudflare, for example, has a setting that blocks AI crawlers, and since mid-2025 it has been switched on by default for many new websites. If you use Cloudflare, look in its bot settings for the AI crawler option and decide which bots to allow. Some security plugins and hosting providers have similar "block bots" options, so ask your web person if you're unsure.

Check 3: can bots see your content without JavaScript?

Many AI crawlers read the page as it arrives from the server and don't run JavaScript. If your text, prices or contact details only appear after scripts load, the AI may see an almost empty page. A quick test: open your page, right-click and choose "View page source". If you can't find your main text in there, ask your developer about server-side rendering or pre-rendering.

How long until changes work?

Bots re-read robots.txt regularly, typically within a day or so. Search bots then need to revisit your pages, which can take from days to a few weeks. Unblocking doesn't guarantee a recommendation; it just makes sure you're in the running.

Check it automatically

Every RecoPulse report checks your robots.txt for the main AI bots and tells you, in plain words, which ones are blocked and exactly which lines to change. Start with the free check to see what ChatGPT says about you today.