Golden dataset for AI regression testing: what should it contain?

A golden dataset for AI agents should not be a folder of “nice prompts”. It is a curated set of representative cases that protects specific behaviours from regression and lets you compare versions under controlled conditions.

1. Store expected behaviour, not always exact wording

Traditional tests often have a single expected value. In LLM applications it can be more useful to encode expectations across layers: intent, allowed/prohibited behaviour, tool call, arguments, knowledge source and final backend state.

The agent can then produce a different but valid sentence and still pass if all required invariants hold.

2. Where test cases should come from

• production incidents and known defects,
• support tickets and negatively rated conversations,
• critical business flows,
• domain-specific edge cases,
• multi-turn corrections and stale-state cases,
• tool/API failures, timeouts, retries and race conditions,
• safety or guardrail cases relevant to the deployment.

The set should grow from real failure modes instead of becoming a one-off benchmark that nobody updates.

3. What to store with each scenario

Also record prompt/model version, configuration, knowledge-base/RAG version and release identifier. That makes the baseline reproducible and candidate differences explainable.

4. Match the scorer to the criterion

Not every criterion needs an LLM judge. Hard requirements are better checked in code: tool name, arguments, status, entity ID, record count and business invariants. Semantic criteria can use a judge/scorer with a clear rubric.

If behaviour is stochastic, a single run may be weak evidence. Repeated runs and a risk-appropriate threshold can be more meaningful. There is no universal pass-rate threshold that fits every agent.

5. Do not optimise only for dataset size

A thousand low-risk examples can be less useful than eighty cases that cover critical processes and historical failures. Coverage should follow the risk map: what matters most, where the agent mutates state and where a false confirmation can cause real harm.

6. Treat the dataset as a living artifact

1. Capture a new failure mode.
2. Turn it into a reproducible scenario.
3. Add expectations and severity.
4. Run it against baseline and candidate.
5. After the fix, keep the case in the suite so it cannot silently return.

This connects evaluation datasets with practical regression testing. Instead of relying on one aggregate score, you protect a set of behaviours that must continue to work.