One of the most dangerous agent failures happens when the conversation looks correct but the action underneath is wrong. Tool-calling regression testing should therefore go beyond checking whether the model “called a tool”.
Test the action contract: the correct tool, correct arguments, satisfied preconditions, correct result handling and the expected final system state.
1. A tool call has a contract, not just a name
For every critical tool, define observable criteria. For reschedule_appointment() that can include the correct appointment, requested time, user permissions, atomic behaviour and what happens if the new slot becomes unavailable during the operation.
- selection: the intended tool was chosen,
- arguments: types, values, units and no stale slots,
- preconditions: business rules were checked before mutation,
- result handling: tool output was interpreted correctly,
- state: backend state matches what the agent told the user.
2. High-value regression cases
Stale argument after a user correction
The user changes 14:00 to 16:00. The response sounds correct, but the tool still receives 14:00. This is a conversation-state failure that can be caught with a deterministic assertion.
Timeout with an unknown operation outcome
A timeout does not prove the remote operation failed. If the remote system committed the change, a blind retry can create a duplicate or mutate state twice.
Partial failure
In a multi-step mutation, one write can succeed and the next can fail. The test must check the final invariant, not only individual HTTP statuses.
3. Retry and idempotency belong in the suite
Mutation tools should be exercised under retry, lost responses and duplicate events. Not every integration supports an idempotency key, but the system still needs a strategy for determining whether an operation already happened.
This matters most in bookings, payments, cancellations, ticket creation and any workflow where repeating a mutation has a real cost.
4. Much of tool QA can be deterministic
Many critical tool criteria do not need an LLM judge. You can assert the tool name, schema-valid arguments, entity identifier, response status, created-record count and backend invariant directly. A semantic judge becomes useful for criteria such as whether an escalation was justified by the conversation context.
5. Record the version behind every regression run
Baseline → candidate comparisons are only useful if you know what changed: prompt, model/provider, tool schema, orchestration code, RAG/data, feature flags and environment. Otherwise a failure is hard to reproduce.
Stochastic behaviour may require repeated runs and thresholds; backend invariants can usually remain hard assertions.
6. Make quality gates severity-aware
Four cosmetic wording issues do not necessarily justify a release block. One critical false confirmation or data-loss case might. The gate should follow business risk and pre-defined rules rather than an arbitrary overall score.
Sources and standards
These are technical references behind the methodology described above. Their inclusion does not imply certification against those standards.