Iron Goo - AI Upgrade Agency for Ambitious Companies

---
title: "AI Agency Red Flags: Six Claims to Check"
seoTitle: "AI Agency Red Flags: Six Claims to Check"
description: "Spot AI agency red flags with six sales claims checked against evidence, a verdict for each, and questions to ask before signing a contract."
datePublished: "2026-11-01T18:42:00Z"
dateModified: "2026-11-01T19:33:00Z"
category: "business"
imageAlt: "Business owner highlights a contract beside a sales brochure, with six marked cards on a pinboard"
tags: ["ai-agency", "red-flags", "vendor-claims", "ai-accuracy", "buyer-decision"]
faq: true
headQuery: "ai agency red flags"
featuredImageFile: "/blog/art/en/ask-before-sign-hero.d0cd6e40.webp"
---

AI agency red flags are often described as bad manners: jargon, pressure or vague answers. The claims that can cost you money are more specific and checkable. The FTC said Workado promoted its AI Content Detector as 98 percent accurate, while independent testing found 53 percent accuracy on general-purpose content; according to the FTC, the model had been trained or fine-tuned only to classify academic content. [The FTC's Workado release](https://www.ftc.gov/news-events/news/press-releases/2025/04/ftc-order-requires-workado-back-artificial-intelligence-detection-claims) shows why a number without the right test cases can mislead a buyer.

| The claim | What the evidence says | Verdict | Ask this |
|---|---|---|---|
| We train a custom model for you | Instructions and retrieval do not train a model | Mostly false | Which base model, on whose account, and what happens when it retires? |
| It is 95 percent accurate | Accuracy depends on the cases and grading | Unproven until tested on your cases | 95 percent of what: how many of our cases, graded by whom, and which errors count? |
| It runs fully autonomously | Agents struggle with multistep tasks | False for multistep work | What share of cases will a person review, and what happens when the system is unsure? |
| It needs no maintenance | Models and business information change | False | How many hours a month of upkeep, and who re-runs the test set when the model or our data changes? |
| It works with any system | Integration paths differ by system | Partly true | For each of our systems, name the API or connector; which ones will you drive through the screen? |
| The savings are guaranteed | A promise needs a baseline and remedy | False as a promise | Guaranteed against which measured baseline, over what period, and what do you pay back if you miss? |

These are claims an agency makes in a pitch or contract, checked against evidence at the contract stage. The [questions to ask an AI consultant before paying for discovery work](/blog/five-questions) are a separate checklist.

## “We train a custom model for you”

A custom AI assistant can use your documents without training a custom model. A proposal may call the result “your model” even when the agency has configured an existing vendor model with instructions and access to your information.

OpenAI's [model optimization guide](https://developers.openai.com/api/docs/guides/model-optimization) says prompt engineering may be all a use case needs. Fine-tuning is a distinct process that changes a model through training; other vendors and open models still support it. The same OpenAI guide says the company is winding down its self-serve fine-tuning platform, with fine-tuned models available for inference until their base models are deprecated.

OpenAI [notified developers on 7 May 2026](https://developers.openai.com/api/docs/deprecations) that active existing customers can no longer create self-serve fine-tuning jobs from 6 January 2027. GPT-4o, GPT-4o mini, GPT-4.1 and GPT-4.1 mini are scheduled for API removal on 14 January 2027, with fine-tuned versions on the same shutdown schedule. Our inference is that a model fine-tuned on GPT-4.1 or GPT-4.1 mini stops answering on that date.

**Verdict: Mostly false.** An assistant tailored with instructions and retrieved documents is custom-configured, but those steps do not train a model.

**Ask this:** “Which base model, on whose account, and what happens to our model on that base model's retirement date?” A straight answer names the model and distinguishes actual fine-tuning from configuration. For how an assistant answers from your documents without training a model, see [grounding AI on your business data](/guides/ai-automation/grounding-ai-on-your-business-data).

## “It is 95 percent accurate”

A 95 percent accuracy claim describes a result on a particular set of cases, not a known error rate for your company's work. Without the cases and grading rules, you cannot tell whether the number applies to an invoice, a support answer or an AI-content detector.

Workado's claimed 98 percent and the FTC's reported 53 percent on general-purpose content show what can happen when the application differs from the content used to train the detector. Legal research tools offer another warning: LexisNexis promoted “hallucination-free” citations, and Thomson Reuters said its products “avoid” hallucinations. A [Stanford HAI story on a RegLab study](https://hai.stanford.edu/news/ai-trial-legal-models-hallucinate-1-out-6-or-more-benchmarking-queries) reported that, on a pre-registered dataset of more than 200 open-ended legal queries, Lexis+ AI and Ask Practical Law AI produced incorrect information more than 17 percent of the time, while Westlaw AI-Assisted Research hallucinated more than 34 percent of the time. Those are tests of legal research tools, not agencies, but the lesson applies to a vendor's accuracy claim.

A hypothetical company processes 2,000 items a month and gets the stated accuracy on that volume. Our arithmetic shows the remaining errors:

:::formula{label="Wrong items at 2,000 items per month"}
95% accuracy: 2,000 × 5% = 100 wrong per month, or 1,200 per year
98% accuracy: 2,000 × 2% = 40 wrong per month
:::

:::figure{src="/blog/art/en/ask-before-sign-errors.cb26a931.webp" alt="Accuracy rates compared with the number of wrong items in a monthly workload" width="1440" height="810"}
An accuracy percentage leaves a count of wrong items in every batch. Put the monthly volume beside the percentage to see that count.
:::

**Verdict: Unproven until tested on your cases.** Even a correctly reported score may describe a different workload.

**Ask this:** “95% of what: how many of our cases, graded by whom, and which errors count?” A straight answer identifies the test set, its size, the grader and the rule for marking an item wrong.

## Can an AI agent run fully autonomously?

A multistep AI agent cannot be assumed to run fully autonomously across a company's work. A narrow single step, such as classification, extraction or drafting for review, can run unattended once its performance has been measured for the intended cases.

In the [10 September 2025 version of TheAgentCompany](https://arxiv.org/abs/2412.14161), the most competitive agent completed 30 percent of simulated workplace tasks autonomously in a simulated software company. The researchers found that harder, long-horizon tasks remained beyond the systems they tested.

The [tau-bench study from June 2024](https://arxiv.org/abs/2406.12045) found that function-calling agents such as the gpt-4o-class models tested succeeded on fewer than 50 percent of tasks. In retail, `pass^8` was under 25 percent: that measure counts the share of tasks an agent gets right on all eight repeated tries. An unattended process needs that kind of consistency across repeated cases. Both studies are dated, and newer models may score higher.

A sales label is not evidence of competence. In September 2024, the FTC alleged that DoNotPay, which called its service “the world's first robot lawyer,” had not tested whether its output matched a human lawyer's; the FTC announced a proposed order of $193,000 in [the FTC's September 2024 announcement](https://www.ftc.gov/news-events/news/press-releases/2024/09/ftc-announces-crackdown-deceptive-ai-claims-schemes).

**Verdict: False for multistep work.** The evidence does not support a blanket promise that an agent can handle a multistep workflow without supervision.

**Ask this:** “What share of cases will a person review, and what happens when the system is unsure?” A straight answer gives a review share and describes where uncertain cases go.

## “It needs no maintenance”

An AI automation needs maintenance because its model and the business information it uses can change. A system that passed its acceptance test once has not thereby passed every future version of the same test.

Chen, Zaharia and Zou compared two hosted GPT-4 versions on the same prime-versus-composite questions. Their [2023 study](https://arxiv.org/abs/2307.09009) measured 84 percent accuracy for the March version and 51 percent for the June version. That result describes a particular task and pair of versions; it does not mean every model change causes the same decline.

Pinning a dated model version can avoid changes behind a model name, but a pinned version can be retired. The scheduled 14 January 2027 removal described above makes that a contract issue: someone must select a replacement and test whether the automation still behaves acceptably.

Your own prices, products and policies can change too. When a change affects an automation's instructions or source material, someone needs to update the relevant information and re-run the test cases. A maintenance promise should account for that work even when the underlying model stays available.

**Verdict: False.** A working automation has dependencies that can change after delivery.

**Ask this:** “How many hours a month of upkeep, and who re-runs the test set when the model or our data changes?” A straight answer identifies an estimated allowance and the person responsible for testing, including what happens when the estimate is exceeded.

## “It works with any system”

An AI automation can connect to many systems, but each connection needs a workable path. A proposal that says “any system” leaves out whether the agency will use an API, an existing connector, a file exchange or the application's screens.

[Zapier's app directory](https://zapier.com/apps) listed more than 9,000 apps when read on 1 November 2026. That gives many systems a ready connector, but it does not establish that every action you need is available through that connector. A system without an API or connector may need custom integration, file export and import, or screen automation.

Screen automation operates through the interface a person sees. [Anthropic's computer use documentation](https://platform.claude.com/docs/en/agents-and-tools/tool-use/computer-use-tool) says models can act as though an action succeeded without checking the result, and that dropdowns and scrollbars can be difficult to manipulate. The documentation recommends human confirmation for decisions with meaningful real-world consequences.

Our rule is to name one integration path for each system in the proposal. A screen path should also have a confirmation step and a higher upkeep estimate, because a changed screen can require the automation to be revised and tested.

**Verdict: Partly true.** Many systems can be connected, but the available path and its upkeep differ.

**Ask this:** “For each of our systems, name the API or connector; which ones will you drive through the screen?” A straight answer identifies the path for each system and flags any file exchange or screen work.

## Are guaranteed AI savings real?

Guaranteed AI savings are false as a sales promise and real only as a contract clause with a measured baseline, a measurement method, a period and a remedy. A forecast can be useful, but a percentage in a proposal cannot tell you how much your company will save.

In the March 2026 revision of [NBER Working Paper 33777](https://www.nber.org/papers/w33777), Humlum and Vestergaard studied 25,000 workers at 7,000 Danish workplaces. Two years after ChatGPT's launch, their analysis ruled out earnings effects larger than 2 percent; adopters reported saving about 3 percent of their work hours. Those findings do not predict the result for your company, but they give you a reason to question a large saving presented as automatic.

The [FTC alleged](https://www.ftc.gov/news-events/news/press-releases/2024/09/ftc-announces-crackdown-deceptive-ai-claims-schemes) that FBA Machine falsely promised guaranteed income through online storefronts using AI-powered software, that sales agents called it risk-free, and that it cost consumers more than $15.9 million. The [public enforcement record on inflated AI claims](/blog/ai-theater) puts that case in context.

**Verdict: False as a promise.** In our view, a savings guarantee becomes a deliverable only when the contract specifies a measured baseline, a measurement method, a period and a remedy such as a refund or fee reduction.

**Ask this:** “Guaranteed against which measured baseline, over what period, and what do you pay back if you miss?” A straight answer gives each term in writing. The [AI automation ROI guide](/guides/ai-automation/measuring-ai-automation-roi) explains how to measure the baseline a savings clause needs.

Our decision rule is to sign only claims the agency will restate in the contract with their number, test set or remedy. [Iron Goo's operations service](/services/operations) builds and runs AI automations and chatbots, and you can hold our proposal to the same six checks.