Transcript
Intro
Ask Google where to stream Harbour of Glass, and ReelRadar's film page answers: here are the services. Ask an AI assistant the same question, it reads the very same page, and it reports back that the page has no streaming information at all. Same URL, two readers. Only one of them ran the JavaScript that builds the answer.
Today is about that gap between what a browser shows and what a crawler reads, and then what Google does with what it read. By the end you'll have a two command test that shows you exactly what each kind of bot sees on your own pages.
How Google renders
Start with Google, because Google is the good news. Googlebot queues every page that returns a 200 status for rendering, unless a robots tag says don't index it. A headless Chrome runs the JavaScript, and Google indexes the rendered result. The page might wait in that queue for a few seconds. Or longer.
How long is longer? A 2024 study tracked over 100,000 Googlebot fetches on one large JavaScript site. Every HTML page they could check got fully rendered, including pages that loaded content through API calls. Half were rendered within 10 seconds of the crawl. The slowest one percent waited about 18 hours.
Does that delay matter? For most pages, no. For pages where timing matters, like a release date announced this morning, it means the version Google first indexes can be hours old if the date only appears after rendering. Time sensitive facts belong in the server HTML, for Google's sake too.
So, first myth. Google can't index JavaScript, so a JavaScript site will never rank.
Google documents its rendering in detail, and that study saw every checkable page rendered. JavaScript sites rank every day. The accurate version of the claim is smaller: Google renders, with a delay, and with a few traps that can stop it cold. Verdict: placebo.
Google's rendering traps
Trap one. If the raw HTML carries a noindex tag, Google says it may skip rendering entirely. So a page that ships with noindex and removes it with JavaScript once the data loads may never get indexed, because Google left before the script ran. If you want a page indexed, keep noindex out of the original code.
Trap two, canonicals. Google says don't use JavaScript to change the canonical to a different URL from the one in the original HTML. Best is to set it in the HTML and leave it alone. If you must inject it with JavaScript, make sure it's the only canonical tag on the page, and that it matches what the server sent.
Trap three, status codes. Pages that return something other than 200 may not get rendered at all. And a single page app that shows a not found message while returning 200 creates soft 404s. Google's fix: redirect to a URL that returns a real 404, or add noindex with JavaScript on those error views.
Trap four, links. Googlebot reads the rendered HTML for links again and queues what it finds, so links added by JavaScript can be discovered. They still have to be real a elements with an href. A menu built from click handlers works fine for people and gives crawlers nothing to follow. We covered that one in the internal linking episode.
The AI crawlers
Now the second audience. In December 2024, Vercel and MERJ published what AI crawlers did across Vercel's network. None of the major ones rendered JavaScript. That covered OpenAI's crawlers, including the one behind ChatGPT search, Anthropic's ClaudeBot, and the crawlers from Perplexity, Meta and ByteDance. Only Google's Gemini, which runs on Googlebot's infrastructure, and Apple's crawler rendered pages.
And these are not hobby crawlers. In one month on that network, GPTBot made 569 million requests and Anthropic's crawler 370 million, together about a fifth of Googlebot's volume. Both even downloaded JavaScript files, about 11.5 and 24 percent of their requests. Then didn't run them. Like buying a cookbook and eating the cover.
One nuance from the same study. Content inside the initial HTML can still be read even when it isn't visible text, like a block of JSON data a framework embeds in the page. Don't build on that. Readable text in the HTML is the version every crawler understands.
Which gives us the second myth. If a page ranks on Google, AI assistants can read it too.
Ranking on Google proves Google rendered the page. It says nothing about a crawler that reads only the raw HTML. Anything your page builds in the browser, like prices, specs, review scores, tab content or a where to watch box, can rank in Google and be missing for ChatGPT or Claude. Google's own docs put it in six words: not all bots can run JavaScript. Verdict: placebo.
A freshness note, because crawlers change. That study is from December 2024. As of October 2026, we haven't found any of these companies documenting JavaScript rendering for their crawlers, but check again before you bet on it. The safe design doesn't care about the answer: put the content that matters in the HTML your server sends.
What about Google's own AI features? Google says that to be a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to show with a snippet. No extra technical requirements. Those features work from Google's index, which is built from the rendered page. So the JavaScript gap mostly bites with assistants outside Google.
The two-command test
Here's the test we run on every important template. Command one fetches the raw HTML, the way a non rendering crawler gets it, and counts how often the answer text appears. Command two loads the page in headless Chrome, lets the JavaScript run, and counts again. If the first number is zero and the second isn't, most AI crawlers can't see that content.
A few tips from running it on real sites. Search for a phrase that only appears in the answer, not one that's also in the menu. Test one URL per template, because pages built the same way fail the same way. And some sites serve different HTML by country or user agent, so run it from where the crawlers come from if you can.
On ReelRadar, our fictional film site, the result was exactly that: zero in the raw HTML, one after rendering. The where to watch box was fetched from an API after the page loaded. The page source held an empty placeholder and a script tag. Google saw a full answer. A crawler that skips scripts saw a film page with nowhere to watch it.
The fix was to render the box on the server, so the HTML arrives with the answer already in it. The extra streaming details stay collapsed for people, using a native details element instead of loading the content only when someone clicks. Rerun the test: one in the raw HTML, one after rendering. Same page for every reader.
So what belongs in the server HTML? The study's own priority list is a fair one: the main content, the meta information like titles and descriptions, and the links that connect your pages. For a film site, that's the dates, the where to watch answer, the cast and links to related films. Live widgets like view counters can stay in the browser.
You can check Google's side in Search Console too. URL Inspection shows the HTML Google got, a rendered screenshot and the JavaScript output. If the answer is missing from Google's rendered view as well, you have a Google problem too, usually a blocked script or an API the crawler couldn't reach.
One more check while you're in there. Rendering only matters if the crawler gets in at all. OpenAI says the crawler behind ChatGPT search is OAI-SearchBot, and sites that block it won't be shown in ChatGPT search answers. Blocking GPTBot, its training crawler, is a separate choice. Read your live robots.txt and your CDN's bot settings before you debug anything fancier.
And the CDN deserves a second look. Since July 2025, Cloudflare has blocked known AI crawlers by default on new domains, unless the owner chooses otherwise at setup. Plenty of site owners never saw that choice, because someone else set the site up. Your browser loads the page fine. The crawler gets a challenge page. Check the setting, then decide on purpose.
There's also a third OpenAI agent, ChatGPT-User, which visits a page when a user asks ChatGPT something that needs it. OpenAI says it doesn't crawl automatically, and that robots.txt rules may not apply to it, because a person triggered the visit. It was one of the crawlers the 2024 study saw skipping JavaScript, so the raw HTML rule holds here too.
Crawled vs discovered
Now indexing, starting with the two statuses everyone mixes up. Crawled, currently not indexed means Google fetched the page, looked at it, and passed for now. Discovered, currently not indexed means Google knows the URL exists but hasn't fetched it yet, typically because crawling it then was expected to overload the site.
So the fixes point in opposite directions. A pile of discovered, not indexed URLs points at crawling: a slow server, or thousands of junk URLs fighting for attention, which is our crawling episode. A pile of crawled, not indexed pages points at the pages themselves. Google saw them and wasn't impressed. Look for thin, duplicate or near empty pages.
Got a long crawled, not indexed list? Export it and group it by template before you touch anything. It's almost never random. It's all tag pages, or all filter pages, or all product variants that differ by one word. Fix the template, or decide that template shouldn't be indexed at all, and the list shrinks in one move.
On ReelRadar, the list looked scary until we sorted it. Most of it was tag pages, one per actor, each holding two or three film links and nothing else. Google crawled them, shrugged and moved on, and honestly, so would you. The photo galleries were next, thin pages that belonged inside the film page. Two template decisions, and most of the list was handled.
Which leads to the third myth. If a page is crawled but not indexed, hit Request Indexing in Search Console, and that fixes it.
Google's own status description says there's no need to resubmit this URL for crawling. Its recrawl page adds that requesting it many times won't get it crawled faster, and a crawl doesn't guarantee inclusion anyway. Google already read the page and made a call. Pressing the button again doesn't change the page it judged. Verdict: placebo.
Request Indexing is for something else: a page you just published or substantially changed, so Google picks up the new version sooner. For crawled, not indexed pages, change the page. Merge thin pages into one stronger page, add what the other results lack, or remove the page if it shouldn't exist.
Canonicals and shortcuts
Canonicals next. Your canonical tag is a request, not an order. Google ranks the signals: a redirect is strong, a rel canonical tag is strong, and sitemap inclusion is weak. They stack, so the more of them agree, the more likely Google picks your URL. When they disagree, Google picks for you.
Remember that indexing is selection, not storage. Google says plainly that it doesn't serve duplicate pages, so in every cluster of near identical URLs, one gets chosen and the rest are reported as duplicates. The goal is never how many pages you can get indexed. It's whether the right one in each cluster wins.
Google also lists a couple of quieter preferences. It prefers HTTPS over HTTP versions as canonical, unless something like an invalid certificate gets in the way, and it uses hreflang clusters as a signal. If your old HTTP pages still answer with a 200 instead of redirecting, you're leaving that choice to Google as well.
Here's a disagreement on ReelRadar. URL Inspection shows the declared canonical is the clean film URL, but Google picked the version with a sort parameter. We traced it, and the site's own internal links and its sitemap both pointed at the sort version. Two signals against one tag. Google went with the majority, which is fair.
The fix was to make every signal say the same thing: internal links to the clean URL, the clean URL in the sitemap, and the sort version kept out of crawling. When signals agree, Google tends to agree. We checked that on real data: across the top 60 pages of two client sites, a dental clinic and a patent and trademark firm, 110 indexed pages declared a canonical, and Google chose the same URL for all 110.
Last stop, the Indexing API, because people sell it as a shortcut. Google says it can only be used for pages with job postings, or livestream videos marked up as a broadcast event. Every submission goes through spam detection, and abuse can get your access revoked.
So pushing blog posts or product pages through it breaks the rules it was released under and risks the account, for no documented gain. For normal pages, the real ways to get indexed are the boring ones: crawlable links from strong pages, an accurate sitemap, and a page worth indexing.
Wrap up
The scorecard. Google can't index JavaScript: placebo, it renders, with traps. If it ranks on Google, AI can read it: placebo, most AI crawlers skip scripts. Request Indexing fixing crawled, not indexed: placebo, the page has to change. The Indexing API for normal pages: placebo, and against Google's rules.
If you do one thing this week, take your three most valuable templates and run the two command test on one URL from each. Then open URL Inspection on the same URLs and compare the declared and selected canonicals. Those two checks cover both halves of this episode in about twenty minutes.
The two command test is free on the episode page at irongoo.com, including a version that loops over a list of URLs and flags every page where the answer only exists after rendering. Run it on your money templates first.
