How to Evaluate Voice AI for Production: A Buyer's Checklist for Observability, Call Quality and Reliability

A voice agent that handles twenty scripted test calls well tells you very little about call two thousand. Real callers hesitate, correct themselves, and ask about parking halfway through a booking. Carriers drop packets, and booking systems time out at the worst moment.

That's why a demo can't be your evaluation. Production exposes three things a demo hides: visibility, behavior under load, and recovery from failure.

This checklist covers observability, call quality, and reliability. For each one, you'll get the questions to ask and the evidence to request. Use it in vendor calls, in your pilot, and in contract review.

What changes between a demo and production

A demo runs on a clean line with a rehearsed caller. Production runs on mobile phones in cars, on hotel lobby lines, and on callers who change their minds mid-sentence. The system is the same, but the conditions aren't.

People don't speak in clean intents. They think out loud, correct themselves, ask a side question, and then return to the main topic. A good system handles that messy input and still delivers a fast, transactional outcome.

The cost of getting this wrong is quiet. A failed call rarely triggers an alarm. It becomes a lost booking, an unanswered tenant, or a Guest who never calls back.

So the evaluation has to find failures that nobody reports. That's the difference between buying a voice demo and buying operational infrastructure.

The checklist at a glance

The table below summarizes six areas. Each one is covered in detail in the sections that follow.

1. Observability: Can you see why a call went wrong?

Observability means you can ask a system why something happened without knowing its internals in advance. OpenTelemetry's observability primer frames it around traces, metrics, and logs. Voice AI needs the same idea, applied to conversations.

A dashboard that shows call counts isn't observability. It shows that calls happened. It doesn't show which ones failed or why.

The minimum you should see

Ask the vendor to show you each of these on a real call, not a slide:

• A recording and a transcript for every conversation, stored where you can find them.

• A verdict against your own criteria, with a written reason, so a failure comes with an explanation.

• A record of what the agent did, including which tools it called and what came back.

• A searchable timeline per caller across voice and text, so you can follow a repeat Guest.

• Pass rates alongside not-applicable counts, so a percentage isn't presented without context.

• A way to export results for your own review or reporting.

If a vendor can only show you the first item, you can't debug. You can only listen.

Grade every call, not a sample

Traditional quality assurance listens to a handful of calls a month. That leaves most failures unseen. With AI, you can grade every conversation.

Voicetta's AI Evaluations work this way. You write the criteria, such as "each time the Guest wanted to book a stay, the agent booked it." The system grades each call and text thread as pass, fail, or not applicable, with a one-sentence rationale.

Three limits are worth knowing, and a good vendor will state its own. Grading is transcript-based, so it doesn't score tone of voice. It only covers conversations that end after you enable it, and "not applicable" isn't a failure.

Ask for a post-mortem

Pick one failed call and ask the vendor to walk you through it. Start from the transcript and the verdict. Then trace the cause and show the fix.

A mature team does this routinely. It reads the rationale and finds the fault, whether that's the prompt, the knowledge base, or an integration. Then it changes only that part.

A weaker team talks about the model. The language model is one component inside a controlled execution layer. Failures usually sit in the layer around it.

Check where the evidence lives

Evidence should sit in one place, close to the conversation. Voicetta's History view opens each conversation with its transcript, summary, recording, and cost. You can search and filter from there.

Ask whether the same applies to text channels. A thread should be graded as a whole, not message by message. Otherwise a fail can flip to a pass without anyone noticing.

2. Call quality: Measure outcomes, not just how human it sounds

Sounding human is easy to demo and hard to turn into revenue. A pleasant voice that books the wrong room is still a failed call. So define quality as outcomes first, then speed, then sound.

Response time under real conditions

Latency decides whether a call feels like a conversation. The ITU-T G.114 recommendation on one-way transmission time is the usual reference. It treats roughly 150 milliseconds as comfortable for most applications and delays beyond 400 milliseconds as generally unacceptable.

That figure covers network delay only. An agent's thinking time sits on top of it. We explain why under 800 milliseconds is the working target for a full AI response.

Ask for latency under concurrent load. A single quiet test call proves little. Ask for the slowest responses too, since an average hides the awkward pauses callers remember.

What to test on live calls

Call the system yourself, and use the messy patterns real callers produce:

• Interruptions. Talk over the agent and see whether it stops and recovers.

• Self-correction. Say "Friday, no, Saturday" and check which date it books.

• Implied requests. Ask "is it quiet there?" and see whether it hears a question about sleep.

• Side questions. Ask about parking halfway through a booking, then return to the booking.

• Background noise. Call from a car or a busy room.

• Silence. Pause for several seconds and see how it behaves.

Score each result against an outcome, such as booking captured or callback offered. Don't score it on how natural it felt.

Measuring quality after launch

Quality drifts. Menus change, staff change, and callers ask new questions. The discipline is continuous measurement, which we cover in our guide to monitoring AI phone agents.

Pick three or four standards that matter to your business. Grade every conversation against them. Watch the pass rate weekly, along with the not-applicable count.

3. Reliability: What happens when something fails?

Every production system fails sometimes. A reliable one fails in a controlled way. That's the difference between "the agent froze" and "the agent offered a callback and logged the reason."

Ask about failure containment

Failure containment means one broken part doesn't take the whole call down. Ask the vendor to walk through these scenarios:

• The booking system is slow or down. Does the agent retry, tell the caller honestly, and log the failure?

• The language model times out. Is there a fallback, or does the caller hear silence?

• The transfer destination doesn't answer. Does the caller get a callback offer instead of a dead end?

• The caller hangs up mid-process. Is there a follow-up for an incomplete booking?

• Call volume spikes. Are there concurrency limits, and what happens above them?

Voicetta covers the hang-up case with a disconnect follow-up. It's an optional SMS or WhatsApp nudge after an incomplete or hung-up voice call.

Recovery follow-ups for failed threads are capped at one per thread and one per Guest per day. Opted-out Guests never receive them.

Run failure drills in the pilot

Don't accept a written answer alone. Make the vendor fail on purpose during the pilot. The table below lists drills worth running.

Don't skip the plumbing

Ask what the carrier setup looks like. Can you keep your existing number? Voicetta supports routing a number you already own, so callers never see a change.

That's routing, not porting. The number stays billed on your own carrier account.

Ask about uptime too. Request the commitment in writing, along with the history of past incidents. A vendor that can't show either hasn't measured it.

4. Control: Who changes the agent, and how?

An agent that rewrites itself silently is a liability. You need to know what changed, why, and who approved it. This is where many "self-improving" claims fall apart.

Look for approval, not autonomy

Good systems propose changes and wait. Voicetta's Optimize reads recent conversations, finds recurring patterns, and proposes prompt and knowledge fixes. Every recommendation is backed by real Guest interactions, and nothing changes until you approve it.

There are practical limits. Optimize needs roughly 20 completed conversations before the analysis means much.

It works from summaries and intent, not raw transcripts. Manual runs are limited to one per 60 minutes.

Those limits are a good sign. A vendor that admits constraints is easier to trust than one that promises everything.

Check the knowledge pipeline

Ask how new information reaches the agent. Documents, FAQs, and website content should be staged and approved before the agent uses them. Otherwise one wrong page on your website becomes a wrong answer on the phone.

Finally, ask whether you can test before real traffic. Voicetta offers test calls by browser or phone. A pilot should start there.

5. Compliance and data handling

Compliance questions are boring until they aren't. Ask them early, and ask for documents rather than reassurance.

Start with recordings and consent. Ask where recordings are stored, how long they're kept, and how callers are informed. Rules differ by state and country, so get your own counsel's view.

Outbound calling adds another layer. In the US, the FCC has ruled that AI-generated voices count as artificial under the Telephone Consumer Protection Act. Consent and calling-hours rules apply, so ask how the vendor enforces them.

If callers can pay by phone, ask how card data stays out of recordings and transcripts. The PCI Security Standards Council sets the standards for protecting payment account data. A vendor should be able to explain its approach in plain language.

For healthcare, treat compliance claims as claims until you've seen documentation. Ask for the paperwork and the subprocessor list. Verify before you rely on any badge.

6. Cost at production volume

Pricing pages show the best case. Production shows the real one. The way to compare vendors is to ask each for the same number.

Ask for one comparable number

Request the all-in cost of 1,000 inbound minutes. That should include voice infrastructure, telephony, the language model, speech recognition, voice synthesis, quality checks, and messaging. Then ask what changes at ten times that volume.

Watch for add-ons that live on other pages:

• Concurrency fees beyond a base number of lines.

• Compliance surcharges for regulated industries.

• Transfer minutes billed at a different rate.

• Evaluation and QA priced as a separate product.

• Per-channel fees for text, as opposed to one shared system.

What a clear rate card looks like

Voicetta's BYOK model charges about 1.2¢ per minute for voice infrastructure, plus roughly 0.4¢ per minute for US inbound telephony. You pay your own AI providers directly. Managed AI starts near 12¢ per minute and removes the provider accounts.

Evaluations, Optimize, and shared memory are included in both models. SMS is about 5¢ per message and WhatsApp about 1¢. Every figure carries a "~", and the public page lists main rates only.

The point isn't that these numbers are lowest. It's that you can calculate your own total from them.

How to run a pilot that tests production

A good pilot is short, specific, and uncomfortable. It should try to break the system before your Guests do.

1. Write your standards first. Choose three or four outcomes, such as bookings captured and callbacks offered.

2. Build a scenario list from real calls. Include the messy ones, such as corrections, side questions, and angry callers.

3. Test before live traffic. Run browser or phone test calls and fix obvious gaps.

4. Go live on a small slice. Route one number or one time window, such as after-hours.

5. Run the failure drills. Use the table in the reliability section.

6. Review graded results weekly. Read the failures, not just the pass rate.

7. Decide with evidence. Compare vendors on outcomes, cost per completed outcome, and how quickly they fixed problems.

Keep the pilot honest by involving the people who normally answer the phone. They know which calls are hard. Their judgment is a better test than any script.

Questions buyers ask when evaluating voice AI

How many test calls are enough?

Enough to cover each scenario several times, plus a live slice. As a rule, aim for at least the volume a tool needs to produce useful patterns. Voicetta's Optimize, for example, needs about 20 completed conversations.

Is sub-second response time required?

It's a strong target for natural conversation. Callers notice long pauses. What matters is consistency under load, so test with concurrent calls, not a single line.

Should I trust a vendor's uptime claim?

Treat it as a claim until you see it in writing, with incident history. Ask what the service credit or remedy is. A vendor that has measured reliability can answer quickly.

Do I need transcripts or audio?

You need both. Transcripts make grading and search practical. Recordings let you hear tone, timing, and noise that a transcript can't capture.

What does done-for-you change about evaluation?

It shifts the burden of building. The vendor designs and maintains the execution layer, and you operate it. You still evaluate the same things, but you spend less time on integration and more on outcomes.

Buy the system, not the demo

Evaluate voice AI the way you'd evaluate a new hire for a revenue-critical role. Watch it under pressure. Check how it handles mistakes, and ask who's accountable when something goes wrong.

Observability tells you what happened, and call quality tells you whether it worked for the caller. Reliability tells you whether it'll keep working when conditions turn rough. You need all three.

Want to see how that looks on your own calls? Start free at app.voicetta.com, and a specialist will contact you and prepare the setup. You can also call +1 650 550 1955 any time and test it yourself.