REQUIREMENTS
Requirements must be testable
“The bot handles bookings” is not enough. I turn requirements into observable criteria: preconditions, action, expected state and recovery behaviour.
Independent AI Acceptance Testing · Voice · Chat · Agents
Independent AI acceptance testing before you sign off the system.
A vendor demo proves that the happy path can work. Acceptance testing answers the harder question: does the delivered system actually satisfy your business requirements, use integrations correctly and behave safely when the process does not go as planned?
independent validationbusiness rules + integrationsGO / FIX / NO-GO
ACCEPTANCE CASE · DEMO
AT-014 · ATOMIC_RESCHEDULE
REQUIREMENTNew slot unavailable → old appointment remainsDEFINED
VENDOR DEMOReschedule · happy pathPASS
EDGE CASESlot disappears before writeFAIL
CRITICAL FINDING
NEW SLOT409 SLOT_TAKEN
OLD APPOINTMENTDELETED
EXPECTEDold appointment kept
ACTUALuser has no appointment
ACCEPTANCE DECISIONNO-GO · FIX REQUIRED
Beyond the happy path
An AI system can look ready and still fail acceptance.
A good-looking response is only one layer. During acceptance I also check business criteria, conversation state, knowledge, tool calls, APIs and what actually happened in the system when the process ended.
INDEPENDENCE
A vendor demo is not an acceptance test
I validate the agreed process independently, including edge cases that are unlikely to appear in a prepared demo but can expose integration or business-logic failures.
BUSINESS OUTCOME
The final system state matters
If the agent says “done” but the backend did not execute the operation, the result is FAIL. Acceptance is about system behaviour, not only response quality.
EVIDENCE
Sign-off should be evidence-based
Important findings include the input, expected behaviour, actual result, evidence and severity so the team knows what must be fixed before acceptance.
Acceptance flow
From requirements to a GO / FIX / NO-GO recommendation.
01
Define acceptance criteria
We identify the most important processes, business rules, integrations and behaviours the system must never get wrong. Criteria can be binary or threshold-based when behaviour is stochastic. We start with the highest-risk processes instead of documenting everything at once.
02
Design risk scenarios
Beyond the happy path: multi-turn corrections, incomplete data, conflicting rules, empty RAG results, API timeouts, retries, context loss, race conditions and other failure modes that matter to your process.
03
Run independent tests
Black-box through UI, phone number or API and, when access allows, with logs and backend state. I also record the system version and relevant test-environment conditions. For voice systems the scope can include ASR, barge-in and critical slots.
04
Capture evidence and severity
PASS / FAIL is not a subjective impression. Findings include the information needed to reproduce the issue plus business severity, so a critical blocker does not get buried next to a cosmetic language problem.
05
Provide a recommendation
The report shows which criteria passed, what must be fixed and whether the tested scope looks ready for acceptance. GO / FIX / NO-GO supports your decision; it does not replace contractual or legal sign-off requirements.
Finding example
The most serious failures often appear between the conversation and the backend.
Criterion · atomic reschedule
The old appointment must not be deleted until the new slot is successfully reserved.
The happy path works, so the feature appears ready. In the acceptance test the new slot disappears between search and write. The implementation deletes the existing booking first and only then tries to create the new one.
This is not a “worse LLM response”. It is a broken business rule that leaves the user with no appointment.
CRITERIONold appointment remains on failure
DELETE OLD200 OK · too early
BOOK NEW409 SLOT_TAKEN
FINAL STATE0 appointments
SEVERITYCRITICAL
DECISIONNO-GO · FIX REQUIRED
Deliverables
Material for an acceptance decision, not a loose list of observations.
CRITERIA MATRIX
Acceptance criteria matrix
Key requirements and scenarios with a clear PASS / FAIL / NOT TESTED result.
FINDINGS
Issue list
Problem description, preconditions, steps, expected vs actual behaviour and business impact.
EVIDENCE
Evidence
Conversation excerpt, API response, tool call, log or system state needed to support the finding.
SEVERITY
Fix priority
CRITICAL / HIGH / MEDIUM so an acceptance blocker is clearly separated from a cosmetic issue.
TRACEABILITY
Requirement traceability
Where possible, each finding points back to the criterion or business process the system failed to satisfy.
DECISION
GO / FIX / NO-GO
A clear recommendation for the tested scope together with blockers that should be closed before acceptance.
Who it is for
Most useful when your team is expected to say: “we accept this system”.
GOOD FIT
- you are accepting a voicebot, chatbot or AI agent from an external vendor
- the system performs a process with a real business outcome
- you need an independent checkpoint before go-live
- CRM, calendar, HIS/ERP or API integrations are part of the solution
- the happy path works but edge cases and failure recovery remain uncertain
It can sit alongside the vendor's QA as an independent acceptance layer, not replace it.
LESS USEFUL AT THIS STAGE
- the product is still an early prototype without stable business flows
- there are no criteria for a correct business outcome
- the goal is only to compare models on one aggregate score
- there is no safe environment or authorization to test the system
In that case an AI Agent Audit or a small Free AI Quality Check may be a better first step.
Who runs the tests
Adam Stankiewicz · AI Quality Engineer
I work hands-on with production voicebot and chatbot evaluation. Before moving into AI quality, I spent more than three years in QA for energy-market systems where software failures had real business consequences.
FAQ
Common questions
Do AI acceptance tests require source-code access?
Not always. A large part of the testing can be performed black-box through the interface, phone number or API. Log access is useful when diagnosing integrations, tool calls and final backend state.
Can you independently test a system delivered by an external vendor?
Yes. The goal is to independently verify agreed processes and acceptance criteria in an environment you are authorized to test. The acceptance decision does not have to rely only on the vendor's demo.
Does acceptance testing replace the vendor's QA?
No. Vendor QA and independent acceptance testing have different roles. The vendor tests its product; independent validation asks whether the delivered solution satisfies your business requirements and is ready to be accepted or released.
What is the output of AI acceptance testing?
You receive an acceptance report with a criteria matrix, PASS/FAIL results, evidence, severity and a GO / FIX / NO-GO recommendation for the tested scope. It supports an acceptance decision but is not a formal legal certification.
Do you test voice agents as part of acceptance testing?
Yes. For voice systems the scope can also cover ASR, barge-in, turn-taking, endpointing, critical slots, confirmations and recovery after recognition errors.
Start with one process
Let's test one critical process before you accept the system.
In the Free AI Quality Check I will select five targeted scenarios and show whether the system merely sounds correct or actually completes the business process as expected.