Iron Goo
P3E02 · Part 3: AI SEO

AI SEO: Letting bots in doesn't mean they can read your site

Learn why AI crawlers can miss your most useful content even when your site loads perfectly in a browser. Separate training access from search visibility, then check the HTML and robots.txt that bots actually receive.

What this episode covers

  • How training, search and live fetch bots differ, and why each needs its own access decision.
  • Why blocking GPTBot doesn't block ChatGPT search, and what OAI-SearchBot controls.
  • How JavaScript can hide important facts, plus a raw HTML check you can run on your pages.
  • Where server rendering helps make prices, dates and answers readable to crawlers.
  • How to compare the robots.txt served through your CDN with the file at your origin.
  • Why Google's training controls and AI search controls need separate decisions.

Key moments

  1. 00:00Intro
  2. 00:55Three kinds of bot
  3. 03:33The GPTBot mistake
  4. 04:44Google's special case
  5. 06:01The JavaScript blind spot
  6. 08:08The robots.txt you never wrote
  7. 10:36Should you opt out?
  8. 11:56Wrap up

Ready to move?

Tell us where your business is today. A written assessment lands in your inbox within two business days.

Talk to us

Transcript

Intro

ReelRadar's film page says the movie is three pounds forty nine to rent. To most AI assistants, that page says nothing at all. Nobody blocked them and nobody made a mistake on purpose. The price simply arrives by JavaScript, and their crawlers don't run JavaScript.

Before anyone talks about getting cited, two gates have to open. The bot has to be allowed in, and your content has to be inside what it fetches. Both fail all the time, and neither failure shows up in your analytics. The traffic chart just stays politely quiet.

So today we sort the bots by job, retire the most common blocking mistake, run the JavaScript test live, and check something almost nobody checks: whether the robots.txt the bots receive is the one you wrote. Everything is checked live, as of October 2026.

Three kinds of bot

One idea clears up most of the confusion. A big AI company doesn't run one crawler. It runs separate bots for separate jobs. One collects pages to train future models. One builds the search index the assistant answers from. And one fetches a page live, the moment a user asks about it.

Here is the matrix as of October 2026, straight from each company's own documentation. Anthropic runs ClaudeBot for training, Claude-SearchBot for search and Claude-User for live fetches. OpenAI runs GPTBot, OAI-SearchBot and ChatGPT-User. Perplexity runs PerplexityBot for search and Perplexity-User for live fetches, and says its search bot doesn't train models.

Each block does something different. Block ClaudeBot, and your future pages stay out of training data. Block Claude-SearchBot, and Claude stops indexing you for search. Block Claude-User, and Claude can't open your page when someone asks about it. You can say yes to one and no to another.

Now read the column most people skip: who actually obeys. Anthropic says its bots honour robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user started the action. Perplexity says its user fetcher generally ignores robots.txt. So for live fetches, robots.txt is a polite request, and two of the three companies tell you so in writing.

Quick quiz, because people really do this. A site blocks OAI-SearchBot and allows GPTBot. What happens? Its pages can still go into training the next model, and it never appears in ChatGPT's search answers. Worst of both worlds, configured on purpose.

Changes aren't instant either. OpenAI says a robots.txt update can take around twenty four hours to reach its search systems, and Perplexity gives the same window. Change a rule, test it ten minutes later, and you are testing yesterday's rule.

Which of these bots actually visit you? Analytics can't tell you, because crawlers never run your analytics script. Your server logs can. In Claude Code we point it at a week of ReelRadar's access logs and ask it to count requests per bot and group them by job.

One warning before you trust that table. A user agent is just a line of text, and anyone can type ClaudeBot into it. Scrapers do it all day. Check each request's IP address against the ranges the company publishes for its bots, and drop what doesn't match. In episode five we do that on a real week of logs, and the fakes are not a rounding error.

The GPTBot mistake

Now the claim we hear every single week. Want to stay out of ChatGPT? Block GPTBot. Its twin is just as common: we blocked GPTBot to protect our content, so why does ChatGPT never recommend us?

OpenAI's own page settles it in two lines. GPTBot is the training crawler. OAI-SearchBot decides whether you appear in ChatGPT search, and sites that block it won't be shown in ChatGPT's search answers. So blocking GPTBot keeps you out of training and does nothing at all for search. Placebo.

The expensive mistake is the opposite one. Someone copies a blanket rule from a forum that blocks every bot with an AI company's name in it. You quietly vanish from the answers you wanted to be in. And because it's one line in robots.txt, nothing errors and nobody gets an alert. The traffic just never arrives.

Here is how we'd set it up for ReelRadar. It wants to be recommended when someone asks an assistant where to watch a film, so the search and user-triggered bots stay open. Whether to allow training is a business decision about your content, and either answer is defensible. Just make it on purpose.

Google's special case

Google is the odd one out, and the most misunderstood. Google-Extended isn't a crawler at all. It's a token you name in robots.txt, with no user agent of its own. It controls whether Google may use your content to train future Gemini models and to ground answers in the Gemini app and on Vertex AI.

What it doesn't touch is Google Search. Google's wording is direct: Google-Extended does not affect inclusion in Google Search and isn't a ranking signal there. AI Overviews and AI Mode are Search features. To limit what they show from a page, Google points you to the normal snippet controls: nosnippet, data-nosnippet, max-snippet and noindex.

There is also no separate Google AI crawler to let in. To appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to show with a snippet, and Google says there are no additional technical requirements. If Googlebot can crawl and index the page, the AI features can use it.

So blocking Google-Extended to keep your pages out of AI Overviews is the GPTBot mistake in a Google costume. Wrong switch, nothing happens. Since the end of August 2026 there is a direct switch for that job, and we get to it at the end.

The JavaScript blind spot

Second gate: can the bot read what it fetched? Googlebot renders JavaScript. Most AI crawlers don't. In a December 2024 study of traffic on Vercel's network, none of the major AI crawlers rendered JavaScript. ChatGPT's and Claude's crawlers even downloaded the script files and then never ran them. For Claude's crawler, almost a quarter of all requests were scripts it didn't execute. An expensive way to read nothing.

Gemini is the exception, because it uses Googlebot's rendering. That study is from 2024 and bots change, so treat it as the safe assumption, not a law of physics. Our working rule doesn't depend on the date: if you want an AI crawler to quote something, put it in the HTML the server sends.

Here is the test from the start, done properly. We ask Claude Code to fetch a ReelRadar film page without running any JavaScript, strip the tags, and look for the UK release date and the rental price the browser shows. Any terminal with curl does the same job. Claude Code just writes the command, runs it and tells you what it found.

Zero matches for both. The page looks perfect in a browser and returns a clean status, yet to most AI crawlers the two facts a person came for don't exist. We've found exactly this on a site we audited: the key figures were injected by a script after load, and the server HTML had none of them. The fix is server rendering for the facts that matter.

You don't need to rebuild the whole site to fix it. Make sure the handful of facts a page exists for are in the first HTML response: the price, the date, the availability, the answer. Server rendering or a static build does that. Interactive extras can stay in JavaScript, because nobody asks an assistant to click your filter button.

Price and date are the obvious ones. Run the same check on whatever your page is for: stock status, opening hours, the rating, the answer tucked into a section that only loads on click. Text that is in the HTML and merely hidden by styling is still readable. Text that only arrives after a script runs is the problem.

The robots.txt you never wrote

Third trap, and the one almost nobody films. You wrote a robots.txt. Is that what the bots receive? If your site sits behind a CDN, not necessarily. The CDN answers before your server does, and some CDNs now serve robots.txt rules on your behalf.

Cloudflare is the clearest example. In July 2025 it switched its default to blocking AI crawlers. That September it began adding its own content signals to the robots.txt it serves for customers, on more than three point eight million domains. And if your site has no robots.txt at all, its documentation says it will create one with disallow rules for known AI crawlers.

Then in July 2026 Cloudflare split AI bots into three groups: Search, Agent and Training. Read the next part twice. From the fifteenth of September 2026, crawlers that do several jobs, which Cloudflare says includes Googlebot, Applebot and Bingbot, get the strictest rule that applies. Cloudflare's own words: customers who block Training also block Googlebot. One switch meant for AI training can switch off the crawler behind Google Search.

We've seen a milder version of this first hand, on a site behind a CDN. The robots.txt being served wasn't the file at the origin. The edge returned its own, and the Sitemap line was gone. Nothing at the origin showed it, because the request never reached the server. A HEAD request even gave a different answer from a normal GET, so always test with GET.

The check takes seconds. Fetch robots.txt the way a bot does: from outside, through the CDN, with a normal GET. Then fetch it straight from your origin server and compare the two. In Claude Code you can ask for exactly that, plus a diff.

If you're on Cloudflare, open its AI crawler settings and look at three things. Is managed robots.txt switched on? What are Search, Agent and Training set to? And is the older block AI bots setting still switched on? Other CDNs have their own bot rules, which is why the outside GET test is the one check that works everywhere.

Verdict: contested, which in plain English means go and look. Your robots.txt is live only if the file served through your CDN matches the one you wrote, on the day you checked. Recheck after any CDN plan change, any new security setting, and once a quarter anyway.

Should you opt out?

Last question: should you opt out of Google's AI features altogether? There is finally a direct switch. The Search generative AI control in Search Console reached every site worldwide on the thirty first of August 2026. It removes a property from AI Overviews, AI Mode and Discover's generative features.

Google's help page answers the obvious worries. The control isn't a ranking or inclusion signal anywhere else in Search. It doesn't affect AI training, which is still Google-Extended's job. It works per property, takes effect within one to two days, and you can switch it back any time.

Our view, from the field. If the answer itself is your product, like a reference figure or a lookup, opting out can be a sensible business decision. If you need to be recommended, like a clinic, a shop or a film site, leaving the place where people now ask for recommendations rarely makes sense. It's reversible, so try it on one property before deciding for all of them.

Keep the three kinds of opt-out apart, because they do different things. Training opt-outs are the training bots plus the Google-Extended token. AI search opt-outs are each assistant's search bot, plus Google's new switch. Live fetches are the least controllable, since two companies say robots.txt may not apply there.

Wrap up

The episode on one screen. Bots come in three kinds, and you block each one for a reason. GPTBot isn't ChatGPT search, and Google-Extended isn't AI Overviews. JavaScript hides content from most AI crawlers. And your live robots.txt may not be the one you wrote.

The AI crawler access checklist and the full bot matrix, with links to each company's own bot page, are free on the episode page at irongoo.com. The bot names will change. The three jobs, and the habit of checking what is actually served, will not.

Free ebook

Put your next steps in order

Build on what you have just read with a free ebook on semantic SEO and AI visibility. Get worked examples, decision rules, checklists and a do-this-week action in every chapter, plus a six-week plan for your site.

By signing up and confirming your address, you consent to receive the free ebook plus one email every two weeks with 3 or 4 new Iron Goo videos, guides and articles, and nothing else; unsubscribe any time with one click.