A multi-LLM stack for voice calls and follow-up messages

A customer calls a property maintenance company after hours to report a leak. The agent needs to collect an address, assess urgency and arrange a callback. If the caller hangs up, the follow-up may move to SMS or WhatsApp. That is one workflow, but it is not one model task: speech needs quick turns, while a detailed message may benefit from more deliberate reasoning.

A practical multi-LLM voice architecture separates those jobs without splitting the customer record. Voicetta handles customer conversations across voice, SMS and WhatsApp, qualifies leads, books appointments and follows up. Its shared timeline keeps interactions together. The model-routing plan around it should preserve that continuity rather than treat each channel as a new conversation.

Start with the path, not the model list

Draw the flow first: call arrives, agent gathers facts, urgency is classified, an appointment or callback is arranged, then the customer receives confirmation. Mark which steps must happen during a live call and which can wait for a message. A spoken answer has a short window before silence feels like failure. A text reply can take longer if it is more accurate and useful.

Voicetta lists OpenAI, Anthropic and Gemini among provider options, and says teams can use their own providers. That makes a BYO-provider design possible, but it does not mean a particular fallback policy is built in. Confirm which routing and recovery behavior your chosen setup supports before putting calls through it. XBert’s guide to inbound communications architecture is a useful prompt to evaluate the whole call path, not just the model name.

Build two model paths

1. Voice path: Choose a primary model for short, routine turns: greeting, collecting contact details, clarifying the issue and confirming the next step. Test it with real examples from your business. Measure the delay from the end of a caller’s turn to the start of a useful response, not just a model’s published speed. Voicetta also lists Deepgram and ElevenLabs as provider options, so evaluate speech components separately from the language model.
2. Message path: Use a separate route for incoming SMS or WhatsApp that need more interpretation, such as a long description of damage or several questions in one message. Compare OpenAI, Anthropic and Gemini on your own message set for accuracy, response time and whether the answer follows your escalation rules. Do not assume the most capable answer is worth its extra delay on every message.
3. Fallback path: Define what should happen if the primary provider times out or returns an unusable response. A secondary model can try a bounded retry; if that fails, the workflow should collect a callback request or hand the case to a person. Set a limit on retries. Repeating the same request across providers can add delay, cost and duplicate actions.

A routing layer may sit between the agent workflow and model providers. ZES’s field test of 9Router and LiteLLM in agent workflows gives teams a concrete starting point when comparing routing options. Treat a router as infrastructure to test, not as proof that every provider switch will be invisible to the caller. Check how it handles timeouts, provider errors, logging and the context passed to the next model.

Keep actions and context out of the fallback gamble

Models should propose or interpret an action; the business system should remain the source of truth for whether that action succeeded. Before retrying a booking or callback request, check whether the first attempt already created it. Give each action a stable identifier if your integration supports it, and make duplicate prevention part of the workflow. A fallback model that repeats a tool request can otherwise create two appointments or two follow-up messages.

Pass the minimum useful context to each model: the customer’s stated issue, relevant conversation history and the rules for escalation. Keep the shared guest record and interaction history in the communication layer or connected business system, not in one provider’s session alone. Then a move from voice to messaging, or from one model to another, need not erase what the customer already said.

Test failure cases before launch

Run the same scenarios through each route: a clear routine request, an ambiguous address, a distressed caller, a provider timeout and a message that contradicts earlier information. Verify the response, the recorded conversation, the appointment or callback result, and the human handoff. Test an interruption mid-call too; a fast model is no help if it loses the thread after the customer speaks over it.

Keep a monthly check of spoken-response delay, sensitive data handling and total cost per completed conversation. The same discipline appears in Voicetta’s voice AI benchmark checklist: measure your own calls rather than relying on a single headline number. Compare the full route, including telephony and any routing layer, and include cases where the system escalates instead of resolving the request.

The trade-off is straightforward. Multiple providers can give a team options when one path is slow or unavailable, but they add configuration, testing and operational responsibility. Start with one voice route and one message route. Add fallback only when a measured failure justifies it, and keep a human path for cases the models should not resolve. That is a more durable stack than sending every conversation to whichever model seems strongest this month.