Why Your AI Agent's Tests Are Lying to You
Golden tests pass. Production fails. Here's why snapshot-based testing gives you false confidence in AI systems — and what to do instead.
You've built an AI agent. You've written tests. They pass. You ship to production. And then it breaks in ways your tests never predicted. This isn't a testing failure — it's a testing philosophy failure. Most AI agent test suites are built on golden tests, and golden tests are lying to you.
The Golden Test Trap
A golden test captures a known-good output and asserts that future runs match it. For deterministic code, this is fine. For AI agents, it's actively misleading. LLM outputs vary between runs. A golden test that passes today may fail tomorrow with an identical input — not because something broke, but because the model generated a slightly different phrasing. Teams respond by loosening assertions until the tests pass consistently. At that point, the tests aren't testing anything.
What Actually Breaks in Production
The failures we see in production aren't "the output changed slightly." They're structural: the agent called the wrong tool, passed malformed parameters to an API, hallucinated a field name that doesn't exist in the schema, or got stuck in a retry loop. Golden tests don't catch these because they test the output, not the behavior. A test that asserts "the email body contains a greeting" passes even when the agent sent the email to the wrong recipient.
Fuzz Testing for Agents
Instead of golden tests, we use fuzz testing: throw a wide range of inputs at the agent and assert on structural properties of the output. Does the response contain valid JSON? Did the agent call the expected tools in the right order? Are all referenced IDs present in the database? Does the output stay within token budget? These assertions hold regardless of the specific LLM output. They test the system's behavior, not its exact words.
Contract Tests Over Content Tests
For agent-to-agent communication, we use contract tests. Agent A promises to send a message with fields X, Y, Z. Agent B expects those fields. The test validates the contract, not the content. This catches the real production failures: schema changes, missing fields, type mismatches. We've found more bugs with 20 contract tests than clients found with 200 golden tests.
A Practical Testing Stack
Our recommended approach: fuzz tests for input handling, contract tests for agent communication, integration tests for tool usage (does the API call actually work?), and a small set of smoke tests for end-to-end flows. Skip golden tests entirely. If you need to validate output quality, use an LLM-as-judge pattern with structured rubrics — but that's evaluation, not testing. Keep them separate.
Ready to build your AI workforce?
Get Started →