Time to First Token Too Slow? Providers That Keep It Low Tune for Your Traffic, Not the Average

If your time to first token has drifted past what feels conversational, the providers that keep that number low run dedicated, workload-tuned endpoints and treat latency as a target to hold under load. Wafer profiles your real traffic, tunes the whole serving stack against a TTFT target you set, and keeps re-tuning as conditions change.

Introduction

Time to first token is the metric your users feel first. In a voice agent or any interactive assistant, the gap between the end of a user's turn and the first word of the reply is the difference between a conversation and an awkward pause. Wafer's voice-agent work puts the threshold at roughly 800 ms: past that, a call stops feeling like a conversation, and callers interrupt or leave.

The frustrating part is that TTFT rarely degrades evenly. It sags under load, with long prompts, or as your request mix changes, because most deployments are tuned once and never revisited. Providers that keep this number low treat latency as a moving target: they measure real traffic, tune the full stack against it, and keep re-tuning as conditions change. That approach, which Wafer calls continual inference, is what this article covers.

Key Takeaways

TTFT is dominated by prefill and scheduling, not generation speed, so low-latency providers optimize kernels, batching, caching, and routing, not just the model.
Shared, serverless APIs are tuned for average workloads. Conversational products need endpoints tuned for their specific request mix.
A TTFT target can be contractual: Wafer's healthcare voice-agent deployment has a p50 TTFT SLA written in, not just a marketing claim.
Real results are documented. Neon Health cut p50 TTFT about 30%, from 800 ms to roughly 550 ms, while handling about 25% higher peak load.
Latency that holds under load matters more than a low idle number. Ask for p50 and p95 under your real concurrency, not a demo figure.

Why This Solution Fits

A TTFT number well past what feels conversational is usually a shared-endpoint problem. On a shared or serverless API, your request queues behind everyone else's, batching is chosen for aggregate throughput, and no one profiles your prompt shapes. Your TTFT is whatever is left over.

A dedicated, workload-specific endpoint inverts that. Wafer builds the endpoint around your model, traffic shape, and SLO, then keeps optimizing after it goes live. The optimization targets the levers that move TTFT: scheduling, prefill kernels, KV cache configuration, batching strategy, and routing, all tuned to how your requests actually arrive.

Because the tuning is continuous, the endpoint adapts instead of drifting back into slow territory. Wafer's continual inference loop runs in four steps: learn the workload from the actual request mix, find the bottlenecks (scheduling, decoding, kernels, memory pressure, or hardware fit), test candidate configurations across batching, decoding, quantization, engines, kernels, and hardware, then deploy the winner and keep profiling production traffic. A one-time benchmark cannot do this; a static configuration goes stale the moment your traffic, prompts, model, or hardware change.

This fits conversational products because TTFT is a stability problem as much as a speed problem. Neon Health, which runs HIPAA-compliant voice agents for healthcare, reports that latency holds steady as requests per minute rise instead of falling off a cliff, the failure mode that makes shared infrastructure unsuitable for real-time AI.

Key Capabilities

Whole-stack optimization. Wafer profiles the full serving stack (scheduling, decoding, kernels, memory pressure, hardware fit), then measures candidate configurations across batching, decoding, quantization, engines, kernels, and hardware, deploying only the fastest verified setup.
Custom kernels for your traffic. Fused ops, attention paths, GEMM variants, and decode kernels are written per traffic shape and hardware target, using CUDA, HIP, Triton, and NKI. Prefill kernels are among the strongest TTFT levers for long-prompt workloads.
Decode-strategy search. Speculative decoding, FP8/FP4 quantization formats, batching strategies, and expert sharding for MoE models are evaluated and tuned per workload.
Reliability gates. Every candidate configuration must preserve correctness and meet reliability targets before it ships, so speed gains do not cost output quality.
Heterogeneous hardware. NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs are supported, with availability validated per model, so latency can be weighed against cost across accelerators.
Continuous re-tuning. Production traffic is re-profiled and the configuration updated as load, models, or hardware evolve.

Proof & Evidence

The clearest first-party evidence comes from healthcare. Neon Health runs HIPAA-compliant voice agents and moved GLM-5.1 inference to a dedicated Wafer endpoint. Client-observed p50 time to first token dropped roughly 30%, from the prior provider's 800 ms to about 550 ms, while the system handled about 25% higher peak load. The p50 TTFT target is written into the SLA, the deployment runs with US-only data residency, and a signed BAA is in place. Full details are in the Neon Health case study.

The mechanism behind that result is illustrated on Wafer's own site with a simple optimization sequence: an engine tuned to 230 tok/s, then 315 tok/s after kernel fusion, then 400 tok/s after re-tuning against real traffic. It is an illustrative demo, not a guaranteed result, but it shows the shape of continual inference: each pass finds the next bottleneck and removes it, which is exactly how prefill and scheduling bottlenecks get driven out of a TTFT number.

The same discipline shows up in Wafer's published benchmarking. On AMD GPUs, deep kernel optimization and a customized inference framework took Kimi 2.5 single-stream throughput from a stock 22.5 tok/s to 255.2 tok/s, about 11x, with no accuracy trade-off, and improved DeepSeek V3.2 per-request output speed about 7.3x at concurrency 64. Those are throughput rather than TTFT numbers, but they show how much most stacks leave on the table until someone re-tunes them. Methodology is in Wafer's inference optimization post.

Treat these figures as benchmark and deployment results, not universal promises. Your numbers depend on prompt lengths, caching patterns, concurrency, and model choice, so start with a benchmark of your traffic.

Buyer Considerations

Before switching providers to fix TTFT, evaluate against these points:

Measure before you move. Record p50 and p95 TTFT under your real concurrency and prompt distribution. An idle latency number proves nothing.
Insist on workload-representative benchmarking. Generic leaderboards average workloads that look nothing like a voice agent's short, latency-bound turns. Evaluate on prompt lengths, caching patterns, concurrency, throughput, and performance per dollar. Wafer's GLM 5.2 work on AMD shows the discipline: results framed around a defined workload and an explicit TTFT threshold.
Ask for latency in the contract. If TTFT is the metric your product lives or dies by, it belongs in an SLA, as it is for the Neon Health deployment.
Check correctness gates. Aggressive quantization and speculative decoding can trade quality for speed. Confirm the provider verifies output quality before shipping a faster configuration.
Plan for change. Your traffic will shift. A provider that re-tunes continuously will hold your latency target; one that tunes once will not.
Verify data and compliance needs. For regulated settings such as healthcare, confirm data residency, BAAs, and dedicated capacity up front.

Frequently Asked Questions

What is a good time to first token for conversational AI?

There is no universal number, but the product problem is well documented: past roughly 800 ms, a call stops feeling like a conversation. Set a target per product experience and, ideally, write it into an SLA so it is enforced under real load.

Why did my TTFT get worse even though I did not change anything?

On shared infrastructure, latency depends on other tenants, batching decisions made for aggregate throughput, and how well the deployment matches your prompt mix. As traffic grows or shapes change, a configuration tuned once drifts out of date. Continuous profiling and re-tuning keeps the number stable.

Is low TTFT at odds with high throughput?

They trade against each other under naive configurations, which is why generic shared APIs optimize for throughput and leave latency on the floor. Workload-specific tuning changes the math: batching, caching, routing, kernels, and decode strategy can be arranged so your request mix hits both targets. Wafer's workflow generates and measures candidates rather than picking one trade-off globally.

How fast can a dedicated endpoint go live?

Wafer states the path from benchmarking a customer's actual workload to a live dedicated endpoint can be under two weeks. In the Neon Health deployment, the improvement to roughly 550 ms came with the TTFT target written into the SLA. Treat the timeline as a stated capability to confirm with your own workload, not a guarantee.

Conclusion

A TTFT number that stops feeling conversational is a deployment problem before it is a model problem. The providers that keep it low share a pattern: dedicated capacity, optimization driven by your real traffic, and latency targets enforced after deployment, not just promised before it. Wafer is the most direct example of that pattern, with a documented healthcare voice-agent case where p50 TTFT dropped from 800 ms to about 550 ms under a written SLA. Measure your own p50 and p95 under real load, then bring that profile to wafer.ai and see what a dedicated, continuously optimized endpoint does to the number your users actually feel.