01 · PROMPT
Instruction change
It can improve tone or routing while changing an escalation rule or the order of questions.
AI Regression Testing · Voice · Chat · Agents
AI regression tests that show what the last change actually broke.
A prompt, model, RAG, tool or integration change can improve one scenario and silently break another. I build repeatable PASS / FAIL tests that check not only the model response, but also actions and the final business state.
regression after changesbusiness rules + backend stateoptional CI quality gate
Regression run · example
RELEASE_CANDIDATE_013
BASELINEprompt_v12 · model_A132 / 132 PASS
CANDIDATEprompt_v13 · model_B128 PASS · 4 FAIL
Critical regression · RESCHEDULE_APPOINTMENT
RESPONSEPASS
TOOL / API409 SLOT_TAKEN
BACKEND STATENOT CHANGED
BUSINESS OUTCOMEFAIL
SEVERITY · CRITICALQUALITY GATE FAILED
Why AI regression is different
A small configuration change can alter the behaviour of the whole process.
In a conventional test, checking one deterministic output is often enough. In an LLM application you may also need to verify context, retrieval, tool arguments, retry behaviour and whether the backend actually completed the action.
02 · MODEL
Model swap
The same prompt does not guarantee the same behaviour with a different model or provider.
03 · RAG
Knowledge update
Changing sources, chunking or retrieval can replace the right context with a plausible but wrong one.
04 · TOOLS
Schema / integration
The response can sound correct even when arguments are wrong, an API times out or an action fails.
05 · VOICE
Voice layer
ASR, barge-in, endpointing or latency can break a scenario that passes perfectly in text.
Method
A regression suite starts with the definition of PASS, not a random list of prompts.
A scenario passes when the system behaves correctly from the perspective of the business process. A good-looking model response is not enough.
01
Define PASS criteria
Process, preconditions, invariants and final state. We define what the system must do, must never do, and which failures are release-critical.
02
Build scenarios
Happy paths, edge cases, multi-turn corrections, previous bugs, bad data, timeouts, retries and cases derived from real usage.
03
Choose evaluators
Deterministic assertions, business rules, tool-call validation and backend-state checks. Where behaviour is semantic or stochastic, I use an appropriate scorer and, when needed, repeated runs rather than expecting identical wording every time.
04
Establish a baseline
I record the prompt, model, configuration and data version that form the reference point. Hard invariants can use binary assertions; stochastic behaviour may need a threshold or a distribution observed across repeated runs.
05
Re-run after changes
Manually before a release or optionally in CI/CD. The report shows which scenario broke, where it broke and how severe it is.
One scenario, several layers
The bot can answer correctly and still fail the test.
Example · appointment reschedule
“Done” is not evidence that the action happened.
The user asks to reschedule an appointment. The response is natural and aligned with the intent. The regression test does not stop at text: it verifies the tool call, API response and final appointment state.
In this example, one CRITICAL regression is enough for the quality gate to fail.
USERMove my appointment to Thursday at 4 PM.
RESPONSEPASS · “Done...”
TOOL CALLreschedule_appointment()
API409 SLOT_TAKEN
STATEappointment NOT changed
FINALFAIL · FALSE CONFIRMATION
Deliverables
Not a dashboard with one score. A suite you can run again.
SCENARIOS
Scenario suite
Critical flows, edge cases and known failure modes captured in a repeatable format.
GOLDEN DATASET
Test inputs and expectations
Inputs, context, preconditions and expected behaviours for release-protecting scenarios.
EVALUATORS
Assertions and evaluators
Checks for responses, tool calls, arguments, business rules and the final system state.
DIFF
Baseline → candidate
A clear view of what still passes, what improved and which behaviour came back as a regression.
SEVERITY
Failure severity
CRITICAL / HIGH / MEDIUM rather than treating every FAIL as equally important.
OPTIONAL CI
Quality gate
When the architecture allows it, the suite can return PASS / FAIL in the pipeline before release.
Who it is for
It creates the most value when the AI system changes regularly.
GOOD FIT
- software houses or AI agencies delivering agents to clients
- product teams changing prompts, models or RAG frequently
- agents using CRM, calendars, APIs or other tools
- manual regression takes engineering time every release
- a failure can create a false confirmation or wrong business state
Available white-label as an independent QA layer before client handover.
LESS USEFUL AT THIS STAGE
- a very early prototype with no stable critical process
- no agreed definition of the correct business outcome
- you only need a one-off usability review
- the goal is a generic “LLM score” without testing system actions
In that case, an AI Agent Audit or a small Quality Check may be the better first step.
Who runs the tests
Adam Stankiewicz · AI Quality Engineer
I work hands-on with production voicebot and chatbot evaluation. Before moving into AI quality, I spent more than three years in QA for energy-market systems, where software failures had real business consequences.
FAQ
Common questions
Does a regression suite require source-code access?
Not always. Many scenarios can be tested black-box through the UI, a phone number or an API. Logs and code access become useful when we need to validate a specific retriever, tool call, argument or backend state.
Do you regression-test voice agents as well?
Yes. For voice systems the suite can include ASR, barge-in, turn-taking, endpointing, latency and recovery after a critical slot is misrecognized.
Should every FAIL block a release?
No. A quality gate should account for scenario importance and severity. A critical false confirmation of a business action may block a release, while a minor wording issue may not.
Can the tests run in CI?
Yes, when the architecture and access allow it. The suite can be run manually before a release or wired into CI/CD and return PASS / FAIL as a quality gate.
How many scenarios should a regression suite contain?
There is no universal number. I start with the highest-risk business processes and known failure modes. The Free AI Quality Check uses five targeted scenarios around one process as a small sample.
Start with one process
Let's check whether the last change broke something outside the happy path.
In the Free AI Quality Check I select five targeted scenarios around one process and return a short set of PASS / FAIL findings.