Iron Goo
P3E09 · Part 3: AI SEO

Programmatic SEO: test indexation before you scale

Learn how to test whether a programmatic SEO template deserves more pages. Find out why polished copy and higher word counts can still leave pages unindexed.

What this episode covers

  • Google's scaled content abuse and doorway rules
  • A data advantage test for every template
  • Sampling URLs with the URL Inspection API
  • When to improve, merge, noindex or delete pages
  • Auditing structured data, canonicals and modification dates
  • Preventing indexable copies of your pages
  • Using embeddings to suggest internal links for human approval

Key moments

  1. 00:00Intro
  2. 00:56Where Google draws the line
  3. 02:32The data advantage test
  4. 04:27The site we turned around
  5. 06:16Measure indexation by template
  6. 08:58The machine-readable layer
  7. 11:47Internal links at scale
  8. 13:15Verdicts

Ready to move?

Tell us where your business is today. A written assessment lands in your inbox within two business days.

Talk to us

Transcript

Intro

We diagnosed a content site whose six hundred or so templated directory pages passed every quality check in its pipeline. Facts, copying, duplicates, floors, opener variety. All green. Then we asked Google, page by page, what it had actually indexed. Of thirty directory pages we sampled, it had kept six. That's where the turnaround started.

That's the real middle ground of programmatic SEO. Building many pages from a template and a database is neither dead nor safe. It works when every page carries something real the reader needs. When it doesn't, it fails quietly, however polished the text.

Today: where Google draws the line, a test for whether a template earns its pages, how we diagnosed and turned around that site, and the checks that came out of it.

Where Google draws the line

Start with the policy. Google defines scaled content abuse as many pages generated mainly to manipulate rankings and not to help users, and then it adds: no matter how it's created. Its examples include using generative AI to churn out pages without adding value, scraping feeds, and stitching other people's content together.

Read what that line is made of: purpose and value. Not the tool, and not templates. Google has said for years that automation makes plenty of helpful content, and its own examples are sports scores, weather forecasts and transcripts. Those are programmatic pages. Nobody calls the weather page spam.

Google's guidance on generative AI isn't hostile to the tool either. It says AI is useful for research and for adding structure to original content. The risk it names is the same one: many pages without added value. So the question is never which tool wrote the text. It's what each page adds.

Google even hands you the self-test. Are you producing lots of content on many different topics, hoping some of it performs? Are you using extensive automation to produce content on many topics? If the honest answer is yes, the template is the problem, not the wording.

Next door sits doorway abuse: pages made to rank for similar searches that funnel people to something less useful than the real destination. A page per city, all pointing at one booking form, is the classic. Programmatic builds drift into doorways when the variable in the template is the search query, not the data.

The data advantage test

So here's the test we use before building any template. Every page must carry a data advantage: a fact the reader needs that a generic page doesn't have. For ReelRadar, a where-to-watch page per film and country is legitimate only if the availability is real, checked and dated. Every film times every country, with nothing checked? That's the trap, in a nice font.

Templates that pass on ReelRadar: availability per film and country with the date it was checked, a rental price history per film, a timeline of when each trailer dropped. Templates that fail: best films about some topic in every city, or a page per actor that repeats a filmography any database already shows. The question is always the same. Does this page know something the reader can't get in one click elsewhere?

Three questions per template. Does each page carry the fact the reader came for? Would anyone miss it if it were deleted? Does the first sentence give the answer type the query expects, the services, the date, the price? Fail the first question, and no amount of prose saves it.

A data advantage also goes off, like milk. Availability changes monthly, so a page that was right in spring can be confidently wrong by autumn. If data is your reason to exist, re-checking it on a schedule is part of the template. Next episode covers how we schedule those checks, and what happens when one quietly stops.

And audit the claim that is your product harder than anything else. On that same site, our audit found every directory page saying availability was verified for the United States, while the check behind it compared streaming links from a catalogue and never opened a US catalogue at all. The copy was rewritten to describe exactly what the check does. And it became a rule: the claim that is the product gets the hardest audit, not the gentlest.

The site we turned around

Now the full diagnosis. The site had two kinds of pages: editorial pieces like reviews and watch-order guides, and that templated directory. Everything ran through quality checks backed by 575 passing tests, and on paper it was spotless. The question nobody had asked yet was which of those pages Google actually wanted.

First we cleaned up everything the prose checks could see. Pages under the substance floor: zero. Naming defects: zero. Directory pages reached a median of 530 words. Then we stopped polishing and asked Google directly.

We took a stratified sample of sixty six URLs by page type, thirty of them from the directory, and ran each one through Google's URL Inspection API. Watch-order guides: six of six indexed. Reviews: four of six. Deep dives and lists: three of six each. Character pages: none. The directory: six of thirty.

There it was. Google was keeping the editorial and passing on the directory. So we put the effort where Google was already saying yes. We moved finished editorial pieces forward in the schedule and raised the editorial share of the site from 13.5 percent to roughly 30.

That's the lesson that changed how we build. Prose checks tell you a page is well made. Only Google tells you whether it's wanted. So our rule since then: a template earns more pages by getting indexed in a sample first, not after we've built six hundred of them.

Which settles the claim that adding three hundred AI words per page makes programmatic pages safe. The directory pages we diagnosed had a median of five hundred and thirty words and still landed at six of thirty. Length is a floor, never a pass. Verdict: placebo.

Measure indexation by template

The single most useful diagnostic for a programmatic site is indexation rate per template. Search Console's API lets you inspect up to two thousand URLs a day per property, at up to six hundred a minute. So a proper sample costs nothing but a script, and Claude Code can write that script in minutes. You decide the sample.

Sample by page type, not at random across the site, or the biggest template drowns out the rest. Take the same number from each editorial type, a bigger slice from the template you're worried about, and spread the picks across page age, not just the newest.

Building the sample is a nice first job for an agent. Point Claude Code at your sitemap, have it sort every URL into a page type by its path, and draw the sample into a file. Then check the buckets yourself before anything runs. One wrong pattern, and your reviews get graded as directory pages.

One thing the API can't tell you: whether an indexed page is any good. Indexed means Google kept it, not that it ranks or earns a click. So the sample answers one question, does Google want this template at all, and your performance data answers the rest. Ask them in that order.

For each URL the API returns a verdict, a coverage state and two canonicals: the one you declared and the one Google chose. Crawled, currently not indexed means Google fetched the page and decided not to keep it. A Google-chosen canonical that differs from yours means Google thinks the page is a copy of something else.

Don't panic over crawled, not indexed. Google's own help page says such a page may or may not be indexed later, and there's no need to resubmit it. One page in that state is noise. Most of a template in that state is a verdict.

Then do what we do in every episode: check three rows by hand in the Search Console interface. The agent's table is a claim until those three match. And it's a snapshot. Run the same sample next month and compare like with like.

Act per template, not per page. A template Google mostly keeps can grow. A template it mostly declines gets one of four decisions: give it a real data advantage, merge it into fewer stronger pages, noindex it, or delete it. The one decision that's never right is publishing more of it.

Here's what merging looks like on ReelRadar. Instead of one thin page per film per country, each film gets one page with a country table: service, subscription or rental, and the date each row was checked. Same data, a fraction of the URLs, and every page now has a reason to exist. Readers in any country land on the same useful page.

The machine-readable layer

Back to that site, because the sample had a second story to tell. Prose checks read the visible text. They never look at the page head, the response headers or the structured data. And on that site, the machine-readable layer was telling Google a different story from the page.

Here's what the audit found. The structured data named every directory page by its romanised Japanese title, while the headline used the English one. A video upload date carried a 2013 premiere date, on a site launched in 2026. And date modified changed every day on every page, because a nightly check rewrote a timestamp whether or not anything had changed. Fresh as a daisy, every single morning.

The audit kept going. A plain-text copy of every page sat at its own address, indexable, linked from every page, and listed 705 times in the site's llms.txt file: a complete second copy of the site. Every link out to a streaming service carried a sponsored tag, declaring paid placements nobody had paid for. The homepage had no canonical, and the www version served a full duplicate.

Google's own documents cover every one of these. Mark up only content readers can actually see. Last-modified dates count only if they're consistently and verifiably accurate. And the review Google asks for on generated content also covers titles, meta descriptions, structured data and image alt text.

About that llms.txt file. Google's AI features guide says Google Search doesn't use llms.txt or similar files. It also warns that Google may crawl and index many kinds of files besides HTML. So a file that does nothing for your rankings can still put a second copy of your site in the index. Bit of a lopsided trade.

Two quick fixes. Sponsored is for links that are ads or paid placements, so putting it on your main call to action tells search engines that your call to action is an ad. And a plain-text mirror needs a noindex in its response header and a canonical pointing back to the real page. Otherwise you've published your site twice.

All of it is fixed and live. The bigger change is the habit: the machine-readable layer gets its own audit, separate from the prose checks. Pull the structured data from a sample of pages and compare every value with what the page shows: name, date, price. Check the headers on every alternate copy. Then compare last-modified dates with real content changes.

If you run the audit with Claude Code, make it print the mismatches, not a score. A line that says this page's markup name differs from its headline is something a person can check in seconds. An overall quality score is something nobody can check at all.

Last piece: internal links at scale, which programmatic sites need and usually get wrong. Our method: turn every page into an embedding, a list of numbers that captures what it's about. For each page, find its closest neighbours that don't already link to it. Then have Claude propose a sentence and a few anchor options for each suggested link.

On tools: Anthropic doesn't make an embedding model. Its docs point to Voyage AI and tell you to compare vendors on your own data. Several crawlers can produce embeddings for you too. Which provider you pick matters less than the next step.

The next step is a person approving every link. Close in meaning is not the same as useful. Two ReelRadar pages about films with similar plots sit side by side in embedding space, while the reader actually wants the director's other films. A human reads the suggestion the way a reader would.

Two limits keep it safe at scale. Each page gets a handful of approved links, not every possible neighbour. And the anchors vary, because a template that stamps the same exact anchor on hundreds of pages is writing for the crawler, not for the reader.

Google's advice on anchors is short: descriptive, reasonably concise, and relevant to both pages. That's why Claude proposes several options and a person picks one. And never set a link floor without checking what it measures. Last episode, a links floor turned out to be measuring how long a show ran.

Verdicts

Two verdicts. Claim one: programmatic SEO is dead.

Google's own examples of helpful automation are programmatic pages: scores, forecasts, transcripts. The policy targets purpose and value, not templates. On the site we diagnosed, the directory wasn't punished. It just wasn't kept, because it added too little. Verdict: placebo. Pages with a real data advantage still have a place.

And claim two, from earlier: add three hundred AI words per page and you're safe. Placebo, measured on the directory we diagnosed. The line is purpose and value, however the page is made.

The indexation-by-template script and the machine-readable audit checklist are free on the episode page at irongoo.com. Next episode: the whole pipeline end to end, including the job that failed eight mornings in a row, and the alert that means it can't fail quietly again.

Free ebook

Put your next steps in order

Build on what you have just read with a free ebook on semantic SEO and AI visibility. Get worked examples, decision rules, checklists and a do-this-week action in every chapter, plus a six-week plan for your site.

By signing up and confirming your address, you consent to receive the free ebook plus one email every two weeks with 3 or 4 new Iron Goo videos, guides and articles, and nothing else; unsubscribe any time with one click.