How many real questions do I need in a test set to trust the results?

50-100 real anonymized questions is a reasonable starting point for most small businesses — enough to surface real variety in phrasing and topic without requiring enterprise-scale data collection.

What's worse: the AI giving a wrong answer, or refusing to answer at all?

A confidently wrong answer is almost always worse than a safe refusal — a refusal at least signals uncertainty and can route to a human, while a wrong answer stated with confidence can mislead a customer into a bad decision before anyone catches it.

How often should I re-test an AI agent after it's live?

At minimum whenever your products, pricing, or policies change meaningfully, and as a general cadence, quarterly re-testing against fresh real examples catches drift that a one-time launch test wouldn't reveal.

Can I use synthetic (AI-generated) test questions instead of real customer questions?

Synthetic questions can supplement a test set but shouldn't replace real ones — real customer phrasing tends to be messier and more varied than what an AI generates when asked to imagine customer questions, and that messiness is exactly what you need to test against.

Who should score the test set results — the developer or someone else?

Ideally someone with real domain knowledge of what a correct answer actually looks like — often the business owner or a knowledgeable staff member, not solely the developer who built the agent, for the same reason a builder shouldn't be the only one testing their own work.

Agentic AIMay 12, 2026

How Do You Know an AI Agent Is Ready for Real Customers?

An AI agent is ready for real customers when it's been scored against a real test set built from your actual past enquiries — not when it performed well in a handful of scripted demo questions — because a scripted demo tests what you expect customers to ask, while real customers ask things you didn't think to script, and that gap is exactly where an unready agent fails in public.

Why Isn't a Successful Demo Enough to Trust an AI Agent?

A demo, even an honest one, tends to use questions the builder anticipated the agent handling well. Real customers ask oddly phrased questions, combine two requests in one message, or ask something entirely outside scope. An agent that's never been tested against that real variety hasn't actually been tested for the environment it's about to operate in.

How Do You Build a Real Test Set From Anonymized Enquiries?

Pull 50-100 real past customer questions or messages — from email, chat logs, or call transcripts — and strip any identifying customer information, keeping the actual question intact. This gives you a test set built from what customers genuinely ask, not what you assumed they'd ask, which is a meaningfully different and more useful thing to test against.

What Are the Three Outcomes to Score Separately?

Outcome	What It Means
Correct answer	The agent answered accurately and completely — the ideal outcome
Safe refusal	The agent correctly recognized it didn't have enough information or wasn't equipped to answer, and said so clearly instead of guessing
Escalation	The agent correctly routed the question to a human rather than attempting an answer it shouldn't

Why Score These Three Separately Instead of Just "Pass/Fail"?

A safe refusal and a correct answer are both good outcomes; a wrong answer stated confidently is a bad outcome, and a missed escalation on a question that genuinely needed a human is also a bad outcome, even if the agent's answer sounded reasonable. Lumping all of this into one pass/fail score hides which specific failure mode is actually happening, and you need to know that to fix it.

What Score Should You Actually Require Before Going Live?

There's no universal number, but the real threshold is that the combined rate of correct answers plus safe refusals plus correct escalations should be very high — and critically, the rate of confidently wrong answers should be as close to zero as achievable, since that specific failure mode is the one that damages trust and creates real problems with real customers.

How Do You Keep Testing After Launch, Not Just Before?

Continue logging real customer interactions after launch and periodically re-run the scoring process against new, real examples — your business changes, your products or pricing change, and an agent that scored well at launch can drift as the world around it changes without the agent being updated to match. This is how we structure every Agentic AI engagement: a defined test set before launch, and a defined re-testing cadence after.

What's the Minimum Viable Version of This for a Small Business?

Even without formal tooling, manually running 50 real anonymized questions through the agent and scoring the three outcomes by hand in a spreadsheet is enough to surface the real gaps before customers do. It doesn't require enterprise QA infrastructure — it requires actually doing it, deliberately, before go-live.