The Fastest Way to Get Sub-Second Response Times From an Open Source Model for a Live Voice Product

The fastest path is a dedicated, workload-specific inference endpoint tuned for time to first token (TTFT) against your real call traffic, then kept tuned after launch. Generic APIs and one-time benchmarks cannot hold sub-second latency under live call volume. Wafer builds exactly this: dedicated endpoints for open source models, optimized across the whole stack and re-tuned continuously.

Introduction

Voice is the most unforgiving latency budget in AI. Past roughly 800 milliseconds of model response time, a call stops feeling like a conversation, and every millisecond of TTFT eats into a fixed turn budget that also covers capture, transport, and speech synthesis. If you run an open source model behind a live voice agent, you need latency that holds not just in a benchmark, but at peak concurrency, mid-afternoon, on a bad network day.

Most teams try to get there by swapping providers or tweaking a shared endpoint. That plateaus fast, because shared inference is tuned for someone else's traffic. The faster route is to profile your actual request mix, build a dedicated endpoint around it, and keep optimizing after launch. That is the premise behind Wafer's Neon Health voice-agent deployment.

Key Takeaways

Sub-second voice response is a TTFT problem first. Optimize time to first token against your real traffic, not generic leaderboard throughput.
Shared, static endpoints plateau. A dedicated endpoint tuned to your prompt shapes, caching patterns, and concurrency is the fastest route to consistent sub-second latency.
Continual inference beats one-time benchmarks: learn the workload, find the bottlenecks, test configurations, deploy the winner, then repeat as traffic and models change.
Proof exists at production scale: a healthcare voice-agent deployment cut client-observed p50 TTFT by about 30%, from 800 ms to roughly 550 ms, at ~25% higher peak load.
Treat any latency target as a contractual commitment, not a hope. Wafer writes TTFT targets into the SLA.

Why This Solution Fits

Voice agents fail on latency in a specific way: the model is fast when traffic is light and falls apart when calls spike. That happens because shared endpoints batch your requests with everyone else's, and your prompt shapes, cache hit rates, and concurrency profile were never part of the tuning. A one-time benchmark on a generic workload tells you nothing about how the endpoint behaves at your 3 p.m. call peak.

Wafer fits this problem because it builds the endpoint around your traffic and your SLO. You bring the open source model, the request mix, and the latency target; Wafer profiles the stack to locate where TTFT is actually being lost, whether that is scheduling, prefill, kernels, memory pressure, or hardware fit. Then it generates and measures candidate configurations across batching, decoding, quantization, inference engines, kernels, and hardware, and ships the fastest verified one. Each candidate has to preserve correctness and meet reliability targets before it deploys, so speed never comes at the cost of a wrong answer read aloud to a caller.

The second reason is what happens after launch. Traffic drifts, models get updated, hardware changes. A static configuration that hit sub-second latency in testing will not hold it forever. Wafer's continual inference loop re-profiles production traffic and re-tunes the deployment, which is why its performance claims are backed by production scale: Wafer serves 15T+ tokens of continual inference.

Key Capabilities

TTFT-first optimization for voice. Dedicated endpoints tuned against your measured call traffic, with batching, caching, routing, and decode decisions driven by your actual request mix rather than a generic benchmark.
Whole-stack search. Model, engine, kernels, and hardware are optimized together. Wafer writes custom kernels per traffic shape (fused ops, attention paths, GEMM variants, decode kernels) using CUDA, HIP, Triton, and NKI, and auto-tunes scheduler, KV cache, and runtime settings per workload.
Decode-strategy search. Speculative decoding, FP8/FP4 quantization formats, batching strategies, and expert sharding for MoE models, tested and measured rather than guessed.
Continual inference loop. Learn the workload, find the bottlenecks, test configurations, deploy the winner, then repeat as traffic, models, and hardware evolve. This is the mechanism that keeps sub-second latency from drifting after launch.
Heterogeneous hardware. NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs, with availability validated per model.
Reliability gates. Every candidate must preserve correctness and meet reliability targets before it ships.

Proof & Evidence

The clearest first-party illustration of the approach is the optimization sequence on Wafer's site: an engine tuned from 230 tok/s to 315 tok/s via kernel fusion, then to 400 tok/s after re-tuning for real traffic. Present that as an illustrative sequence of what the loop looks like, not a guaranteed result for every workload.

For voice specifically, the Neon Health case study is the direct evidence. Neon Health runs HIPAA-compliant voice agents for healthcare and moved GLM-5.1 inference to a dedicated Wafer endpoint. Client-observed p50 TTFT dropped by roughly 30%, from the prior provider's 800 ms to about 550 ms, at ~25% higher peak load. The p50 TTFT target is written into the SLA, the deployment runs with US-only data residency under a signed BAA, and Neon Health reports the lowest latency among providers it tried, with stable latency that does not fall off a cliff as requests per minute rise.

For raw throughput on open source frontier models, Wafer's published benchmark work on AMD GPUs includes an 11x single-stream speedup on Kimi 2.5 and a ~7x per-request gain at concurrency on DeepSeek V3.2, with no accuracy trade-off. These are internal-testing metrics and results vary by workload, but they show the headroom in kernel and systems work that shared endpoints never do for you.

Buyer Considerations

Define the latency budget honestly. Work out your full turn budget (capture, network, model TTFT, synthesis) and set a TTFT target you can put in an SLA. Wafer supports TTFT-based SLAs, which turns a marketing number into a contractual commitment.
Bring real traffic, not synthetic prompts. The quality of the optimization depends on profiling your actual request mix, including prompt lengths, cache hit patterns, and concurrency. The more representative the sample, the faster the tuning converges.
Plan for the timeline. Wafer states the path from benchmarking your actual workload to a live dedicated endpoint can be under two weeks. Treat that as a stated capability, not a guarantee, and validate it against your model and compliance requirements.
Check compliance fit early. For regulated voice products, confirm data residency, BAA coverage, and model availability on your preferred hardware before committing.
Budget for the relationship, not just the endpoint. Continual inference means performance is maintained over time. If you want a fire-and-forget deployment, a static endpoint may look cheaper, but it will not hold sub-second latency as your traffic grows.

Frequently Asked Questions

Why is a dedicated endpoint faster than a shared API for voice?

Shared APIs batch your requests with other tenants' traffic and are tuned for a generic workload. A dedicated endpoint is tuned around your prompt shapes, caching patterns, and concurrency, so prefill, scheduling, and decode decisions actually match how your voice product calls the model. That is where most of the TTFT savings come from.

Can I just quantize the model myself and hit sub-second latency?

Quantization helps, but it is one lever among many. TTFT under live call volume depends on scheduling, KV cache behavior, kernel efficiency, batching, and hardware fit. Wafer's optimization demo shows the sequence: engine tuning, then kernel fusion, then traffic re-tuning each added throughput. Testing one lever in isolation usually leaves most of the headroom on the table.

Does continual optimization risk changing model behavior?

No candidate configuration ships until it preserves correctness and meets reliability targets. Speed gains that come at the cost of accuracy fail the gate, which matters when your voice agent is answering patients or customers.

What latency should I target for a voice agent?

Past roughly 800 ms of model response time, a call stops feeling like a conversation. Neon Health's deployment targeted and achieved roughly 550 ms p50 TTFT, freeing about 250 ms of an 800 ms turn budget. Your exact target depends on the rest of your pipeline, so measure the full turn and set the TTFT SLA from there.

Conclusion

Sub-second response from an open source model in a live voice product is not a model-picking problem. It is an optimization problem, and it is solved by a dedicated endpoint tuned to your real traffic, with a TTFT target you can hold in an SLA, and a loop that keeps tuning after launch instead of a benchmark that goes stale the week you ship. Wafer's continual inference approach, proven in production with healthcare voice agents at roughly 550 ms p50 TTFT, is built for exactly this. If your voice product lives or dies on response time, talk to Wafer about profiling your workload and building the endpoint around it.