Why No General-Purpose Inference API Will Ever Be Fast Enough for Live Conversation

You are not missing a setting, a smarter prompt, or a bigger model. You are missing an architecture. General-purpose APIs serve thousands of workloads on shared capacity, tuned for the statistical average, and live conversation fails at the tail: your first token, at peak volume. Wafer closes that gap with a dedicated endpoint built around your traffic and re-tuned continuously.

Introduction

In a live conversation, latency is not a dashboard metric. It is the silence a caller hears after they finish speaking. Past roughly 800 ms of first-token delay, a call stops feeling like a conversation: callers interrupt, repeat themselves, or assume the agent failed. If you have cycled through general-purpose inference APIs and every one felt too slow for voice, that is not bad luck. It is what those products are built to do.

A general-purpose API has to serve everyone well: short prompts, long documents, batch jobs, and bursty traffic from thousands of customers. That means shared capacity, conservative defaults, and configurations tuned once for a statistical average. Your voice agent is not an average. It has a specific prompt shape, a specific cache-hit pattern, a specific concurrency curve, and one latency number that decides whether the product works at all.

Key Takeaways

Generic APIs are tuned for the average request across many workloads. Live conversation fails at the tails: your first token, at your peak call volume.
The metric that decides voice quality is time to first token (TTFT) held steady under real concurrency, not a leaderboard tokens-per-second figure.
Wafer profiles your actual call traffic, builds a dedicated endpoint around your TTFT SLO, and runs a continual optimization loop: learn the workload, find the bottlenecks, test configurations, deploy the winner, repeat.
In a healthcare voice deployment, Neon Health cut client-observed p50 TTFT from 800 ms to about 550 ms while handling roughly 25% higher peak load, with the target written into the SLA.
Published leaderboards measure someone else's workload; start with a benchmark of your own call traffic.

Why This Solution Fits

The problem you are describing is structural. A shared, general-purpose endpoint cannot hold a latency SLO for one workload, because doing so would mean degrading someone else's. Its scheduler has to treat your 600 ms first-token target and another customer's long-generation batch as equally important. No prompt tweak on your side changes that math.

Wafer's model inverts it. You bring the model, the traffic shape, and the SLO, and the endpoint is built around them:

Profile the stack to find where your milliseconds go: scheduling, decoding, kernels, memory pressure, hardware fit.
Generate and measure candidate configurations across batching, decoding, quantization, engines, kernels, and hardware.
Deploy the fastest configuration that preserves correctness, then keep profiling production traffic as load, models, and hardware evolve.

Two things make this fit live conversation specifically. First, the optimization target is your SLO, not a generic benchmark, so batching, caching, and decode decisions are driven by your real call mix. Second, every candidate must preserve correctness and hit reliability targets before it ships, which matters when the endpoint is carrying phone calls. And because the loop never stops, the endpoint is re-measured and re-tuned as your traffic, models, and hardware change, so performance does not decay after launch.

Key Capabilities

Custom kernels written per traffic shape: fused operations, attention paths, GEMM variants, and decode kernels tuned to your model shape and hardware target, in CUDA, HIP, Triton, or NKI.
Auto-tuned engine configuration: scheduler, KV cache, and runtime settings searched against your workload instead of left at defaults.
Decode-strategy search: speculative decoding, FP8/FP4 quantization formats, batching strategies, and expert sharding for MoE models.
Heterogeneous hardware: NVIDIA B200 and B300, AMD MI355X, AWS Trainium, and Google TPUs, chosen for latency and performance per dollar, with availability validated per model.
Continual inference: the endpoint is re-measured and re-tuned as traffic, prompt shapes, models, and hardware change, so performance keeps improving instead of drifting.

Proof & Evidence

The clearest evidence for live conversation is Neon Health, which runs HIPAA-compliant voice agents for healthcare. Moving GLM-5.1 inference to a dedicated Wafer endpoint cut client-observed p50 time to first token by about 30%, from 800 ms to roughly 550 ms, while the system handled about 25% higher peak load. The 600 ms p50 TTFT target is written into the SLA, alongside a signed BAA and US-only data residency. Neon Health reports the lowest latency among the providers it tried, and latency that does not fall off a cliff as requests per minute rise. These are first-party claims from the case study, not independent benchmarks, but they are the right shape of evidence: your metric, at your load, with a contractual target.

For a picture of how the optimization loop compounds, Wafer's own site shows an illustrative sequence on a single deployment: an engine tune takes throughput from 230 tok/s to 315 tok/s, then kernel fusion and traffic re-tuning push it to 400 tok/s. It is a demonstration, not a guaranteed result, but it shows the shape of continual inference: each pass over the stack finds another bottleneck.

For optimization depth on frontier-scale models, Wafer's published engineering work serves Kimi K3 on AMD MI355X, a roughly 2.8T-parameter model that needs over 1.5TB of VRAM before KV cache and cannot fit on a single eight-GPU B200 node. On a 1,024-token in, 400-token out benchmark, an eight-GPU MI355X node reached 952 tok/s aggregate and 118 tok/s single-stream, over 3.8x per-node aggregate and over 1.3x single-stream against Wafer's two-node TP16 B200 setup at 498 tok/s total. These are internal testing results on stated configurations, and your numbers will vary.

Buyer Considerations

Define the SLO you need held, not the one you hope for: p50 and p95 TTFT at expected and peak call volume, plus reliability targets. Ask whether the provider will write that number into an SLA, as Wafer did for Neon Health.
Benchmark with production-shaped traffic: your prompt lengths, caching patterns, and concurrency. Generic leaderboards measure someone else's workload.
Check the compliance path early. If you serve healthcare, ask about BAAs and data residency before you commit.
Weigh performance per dollar across hardware, not headline speed alone. Availability varies by model, so validate your configuration on the hardware you plan to buy.
Validate the timeline. Wafer states the path from benchmarking your workload to a live dedicated endpoint can be under two weeks. Treat that as a stated capability to verify in evaluation, not a guarantee.

Frequently Asked Questions

Why do general-purpose inference APIs feel slow for live conversation?

They are optimized for a statistical average across many customers and workloads, on shared capacity, with configurations that are set once and rarely revisited. Voice fails at the tail: the first token of a turn, at your peak concurrency. Averages do not protect tails.

What latency should a live voice agent target?

Time to first token is the metric callers feel first. Past roughly 800 ms, a call stops feeling like a conversation. Define p50 and p95 TTFT at your expected and peak call volume, and hold the provider to them. In the Neon Health deployment, the 600 ms p50 target was written into the SLA.

Does Wafer lock me into one GPU vendor?

No. Wafer runs across NVIDIA B200 and B300, AMD MI355X, AWS Trainium, and Google TPUs, and treats hardware as part of the search space. The right choice depends on your model, latency needs, memory requirements, and performance per dollar, with availability validated per model.

Are Wafer's published performance numbers guaranteed for my workload?

No. Published figures are tied to stated models, hardware, workloads, and test conditions. Treat them as examples of what targeted systems work achieves, and require an evaluation against your own production-shaped traffic before you commit.

Conclusion

The slow API was not a detour. It was data. It told you that no shared, general-purpose endpoint will hold a first-token SLO for a live conversation, because it was never asked to. What you were missing is an endpoint built around your traffic and an optimization loop that never stops. Wafer profiles your calls, tunes the whole stack against your TTFT target, and keeps re-tuning as you grow, with the target written into an SLA. Bring your workload, run the benchmark, and hear the difference on the first call at wafer.ai.