A voice agent can pass a polished demo and still fail in production. The reason is simple: the user is not interacting with an LLM alone. The request moves through audio, speech recognition, turn-taking, conversation state, knowledge, tools, APIs and the final business state.
Core rule: a voice-agent test is not complete until you verify the final business outcome. A natural spoken confirmation is not evidence that a booking, cancellation or update actually happened.
1. Test layers, not only the transcript
The common mistake is reducing QA to “did the bot answer correctly?”. Voice systems add several independent failure surfaces. Not every architecture has every layer, so scope should follow system design and business risk.
- Audio / ASR: critical names, dates, times, amounts and identifiers under realistic conditions.
- Turn-taking: detecting when the user is done, avoiding premature responses and handling interruptions.
- Conversation state: corrections replace stale values and context survives digressions.
- Knowledge / RAG: retrieval matches the current intent and uses appropriate sources.
- Tools / API: the right tool, arguments and timing.
- Business outcome: the backend finishes in the expected state.
2. Voice has failure modes of its own
Voice introduces issues that do not exist in text chat: speech pace, pauses, interruption, background noise and mid-sentence corrections. VAD and turn-detection settings directly influence when the agent decides a turn has ended and whether an ongoing response is interrupted when the user starts speaking.
I would not use one universal latency threshold for every voice agent. Acceptable latency depends on the process, channel and user expectations. What matters is measuring it consistently and exercising degraded conditions instead of relying on one ideal run.
3. Critical slots need stronger validation
Not every ASR error has the same impact. A harmless wording error is different from misrecognising a date, amount, surname, order number or appointment time. Scenarios should identify critical values and define when the agent must confirm, reject or escalate.
4. Exercise integration failures and recovery
Test what happens when a tool or API behaves badly: timeouts, resource conflicts, 4xx/5xx responses, partial writes and retries. A timeout does not prove that an operation did not happen, so a blind retry can create a duplicate unless the design uses idempotency or verifies state first.
Example: the agent calls book(); the connection times out, but the backend committed the appointment. Retrying blindly can create a second booking. The test should verify idempotency or state reconciliation before retry.
5. Do not require identical wording on every run
AI systems may be non-deterministic. Business invariants can still be asserted deterministically: selected tool, arguments, API status, number of bookings or final state. Semantic quality may be better evaluated with criteria, scorers or repeated runs and thresholds rather than exact string matching.
Keep “did the process succeed?” separate from “how well did the agent communicate it?”. That separation prevents a fluent response from hiding a broken operation.
6. A minimum pre-launch set
- The highest-value happy path through to backend state.
- A failed precondition or business-rule case.
- A multi-turn correction of a critical slot.
- A failed tool/API call and recovery path.
- User interruption and turn recovery.
- Low-confidence or incorrect recognition of a critical value.
- A known scenario that previously regressed after a prompt/model change.
The complete suite should be risk-based: the higher the cost of a failure, the stronger the criteria, scenario coverage and observability should be.
Sources and standards
These are technical references behind the methodology described above. Their inclusion does not imply certification against those standards.