Views: 1
Every AI agent demo looks intelligent. That’s the problem.
A demo is, almost by definition, a curated set of inputs the builder already knows the system handles well. The interesting question was never “does it work on the demo” — it’s “what happens the first time a real user does something the demo never covered.” Most teams find out the answer in production, from an angry support ticket, instead of finding out on purpose, beforehand, on their own terms.
There’s a rigorous way to ask this question before your users ask it for you. It comes from a simple reframing of what “intelligent” even means.
The reframe: stop measuring skill, start measuring generalization
The instinctive way to evaluate an agent is to measure skill: does it get the right answer? Skill is necessary, but it’s a dangerously incomplete signal, because skill can be bought two completely different ways:
- Genuine capability — the system has learned something that transfers to situations it hasn’t specifically seen before.
- Memorization dressed as capability — the system (or, more often, its prompt engineering and few-shot examples) has been narrowly shaped around exactly the cases it’s being tested on.
Both produce identical-looking demo performance. They produce wildly different production performance. This distinction has a name in AI research — François Chollet formalized it in “On the Measure of Intelligence” (2019): stop asking how skilled a system is, and start asking how skilled it is relative to how much prior knowledge and specific preparation went into producing that skill. A system that needed a thousand hand-tuned examples to handle a narrow band of inputs is doing something categorically different from one that handles novel variations correctly having seen only a handful.
This isn’t academic. It’s the exact question every “should we ship this agent” decision is actually asking, whether or not anyone phrases it that way.
Five tests that separate the two
None of these require special tooling — just deliberate design of your test set, done before you trust the demo.
1. The paraphrase test. Take a task your agent handles well. Rephrase the request five different ways — same intent, completely different wording, different level of formality, different order of information. A system that understood the task keeps performing. A system that pattern-matched surface phrasing degrades, sometimes sharply, the moment the wording drifts from what it was tuned against.
2. The compositional test. If your agent can handle sub-task A and sub-task B independently, can it handle a request that requires combining A and B in a way it’s never explicitly been shown? Genuine understanding tends to compose. Memorized patterns tend not to — the system will often either ignore one half of the request or produce something that superficially addresses both without actually integrating them.
3. The irrelevant-perturbation test. Change a detail in the input that shouldn’t matter — swap a name, change a date, add a harmless aside — and check that the output changes only where it should. If a functionally irrelevant edit produces a materially different (or broken) response, that’s a strong signal the system is keying on surface patterns rather than the actual structure of the request.
4. The distractor test. Insert plausible-looking but irrelevant information into the context — an extra document, an unrelated instruction, a red herring — and see whether it derails the agent. Systems relying on shallow pattern completion are disproportionately vulnerable to being pulled off course by things that merely look relevant.
5. The cost-of-preparation audit. This is the one teams skip, and it’s the one Chollet’s framework centers on. Honestly tally what it took to get the current performance level: how many hand-written examples, how many prompt-engineering iterations, how much fine-tuning data, how narrowly scoped the eval set is. Then ask — plainly — whether that investment was building a general capability or hand-carving a very elaborate path around a specific set of known cases. Neither answer is shameful; a narrow, hand-tuned system can be exactly the right engineering choice for a bounded task. The failure mode is not knowing which one you built, and shipping it as if it were the other.
Why this matters more for agents than for one-shot models
A single LLM call that hallucinates is a bad answer. An agent that hallucinates — plans, calls tools, takes actions, chains steps together — compounds a wrong assumption at step one into a wrong trajectory across every subsequent step, often invisibly, because each individual step can look locally reasonable. Agentic systems don’t just inherit an LLM’s generalization gaps; they amplify them, because a plausible-looking but wrong first decision doesn’t get corrected — it gets built on.
This is exactly why “did it pass the demo” is such a weak bar for anything agentic, and why the five tests above are worth running as a standing practice, not a one-time pre-launch check — new capabilities and new integrations reopen all five questions every time.
The takeaway
None of this requires exotic tooling — it requires treating “does it generalize” as a first-class question with its own test suite, separate from “does it pass the happy path.” The gap between a genuinely capable agent and an elaborately narrow one is invisible in a demo and very visible in production. The five tests above are a way to find that gap on your own schedule, not your users’.
Leave a Reply