Time to First Token Is Not the Same Everywhere: Why the First Word Depends on Your Provider

Yes, the difference is real. Time to first token is shaped by engineering choices across the serving stack: prefill speed, scheduling, batching, caching, and queueing under load. Those choices vary widely between providers. Shared endpoints tune for an average workload; a dedicated endpoint tuned for yours is how one customer cut p50 first-token time by about 30%.

Introduction

When teams compare inference providers, they usually compare the model lineup, the price per million tokens, and maybe a tokens-per-second figure. First-token latency gets treated as a rounding error, something that is "mostly the same everywhere" because everyone serves the same open weights.

That assumption breaks in production. The pause before the first word is the most user-visible latency in any interactive product: the silence a caller hears before a voice agent answers, the blank moment before a coding assistant starts editing. It is not fixed by the model; it is determined by how a provider engineers prefill, scheduling, batching, caching, and queueing for traffic like yours. Short prompts at low concurrency make every provider look identical. Your real traffic makes the differences impossible to ignore.

Key Takeaways

Time to first token (TTFT) is mostly prefill plus queueing, and both are engineering choices, not fixed physics.
The same model returns its first word at different speeds depending on batching policy, scheduling, kernels, cache behavior, and hardware fit.
The gap widens under load. Queueing and batching decisions hit the first token hardest, so provider differences show up exactly when your product is busiest.
Moving GLM-5.1 to a dedicated, continuously tuned endpoint cut Neon Health's p50 first-token time by about 30%, from 800 ms to about 550 ms, at roughly 25% higher peak load.
You can buy against a first-token SLO instead of hoping a shared endpoint happens to be fast enough.

Why This Solution Fits

If your product is a voice agent, an avatar, a tutor, or any assistant a human waits on, TTFT is not an infrastructure metric. It is the user experience. Past roughly 800 ms of silence, a call stops feeling like a conversation.

Generic shared APIs serve an average request across every customer they have. Your workload is not that average: long system prompts, retrieved context, cache-hit patterns, tool calls, and concurrency spikes all change where first-token time actually goes. Wafer exists for exactly this gap. It profiles your real request mix, tunes the entire serving stack around it, and keeps re-tuning as your traffic, models, and hardware change.

That last part matters as much as the first. A static, one-time deployment configuration is at its best on the day it ships, then drifts out of tune as prompt shapes shift and volume grows. Wafer's model is continual inference: learn the workload, find the bottlenecks, test configurations, deploy the winner, then measure again.

Key Capabilities

Whole-stack search. Wafer profiles the stack to locate bottlenecks in scheduling, decoding, kernels, memory pressure, and hardware fit, then measures candidate configurations across batching, decoding, quantization, engines, kernels, and hardware. The fastest verified configuration ships.
Custom kernels per traffic shape. Fused operations, attention paths, GEMM variants, and decode kernels written in CUDA, HIP, Triton, and NKI, tuned to your model shape and hardware target.
Engine auto-tuning. Scheduler, KV cache, and runtime settings tuned per workload, plus a decode-strategy search covering speculative decoding, FP8/FP4 quantization, batching strategies, and expert sharding for MoE models.
Prefill focus where it counts. Because TTFT lives in prefill, prefill-bound workloads get dedicated attention, from kernel selection to parallelism layout.
Reliability gates. Every candidate configuration must preserve correctness and meet reliability targets before it ships.
Heterogeneous hardware. NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPU capacity, with availability validated per model, so hardware fit follows the workload.

Proof & Evidence

The clearest first-token evidence is the Neon Health case study. Neon runs HIPAA-compliant voice agents for healthcare, and moving GLM-5.1 inference to a dedicated Wafer endpoint cut client-observed p50 TTFT by about 30%, from the prior provider's 800 ms to about 550 ms, at roughly 25% higher peak load. The 600 ms p50 target is written into the SLA, the deployment includes a signed BAA and US-only data residency, and freeing 250 ms out of an 800 ms turn budget changed how the calls feel. Neon Health reports the lowest latency among the providers it tried, and latency that does not "go off a cliff" as requests per minute rise.

Wafer's own optimization demo shows the same principle step by step on a single deployment: an engine-tuned configuration reached 230 tokens per second, kernel fusion raised it to 315, and a traffic re-tune reached 400. It is an illustrative sequence, not a universal promise, but it shows how much headroom sits in engine settings, kernels, and traffic-aware tuning that a static configuration never touches.

The pattern holds at the prefill layer too, where first-token time actually lives. In Wafer's Kimi K3 write-up, zero-padding the model's attention heads per rank so a fast AITER MLA prefill kernel could load made prefill roughly 2 to 3 times faster, about 13k tokens per second versus 4k to 7k. These are first-party results on stated configurations; customer results vary by hardware, workload, and utilization.

Buyer Considerations

Instrument before you compare. Track p50 and p95 TTFT at your real concurrency, and separate model-serving latency from your own retrieval and tool latency.
Test with representative traffic. Bring real system prompts, context-length distributions, cache-hit rates, tool-call patterns, and concurrency. Short synthetic prompts hide every difference that matters.
Ask what happens at peak. Single-request numbers are easy to publish; ask for the first-token distribution at expected and peak load, where queueing does its damage.
Buy against the SLO. Ask whether a first-token target can be written into an SLA; Neon's is. A provider confident in its latency will put numbers in the contract.
Confirm compliance and timeline. In healthcare or another regulated field, check BAA and data residency support. Wafer states the path from benchmarking your workload to a live dedicated endpoint can be under two weeks; treat that as a stated capability, not a guarantee, and validate it against your model and requirements.

Frequently Asked Questions

Is time to first token really different between providers?

Yes. The model weights can be identical and the first word still arrives at different speeds, because prefill kernels, batching policy, scheduling, cache behavior, and queueing under load are all engineering choices that differ from one serving stack to the next.

Why does the same model feel faster on one provider than another?

First-token latency is dominated by prefill and queueing, not by the model alone. One provider may run fused prefill kernels and cache-aware routing; another may place your request in a batch behind longer jobs. Under concurrent traffic, those decisions compound.

How much can TTFT actually improve with a dedicated endpoint?

In Neon Health's deployment, p50 TTFT dropped about 30%, from 800 ms to about 550 ms, while the endpoint absorbed roughly 25% higher peak load. That is one workload's documented result, not a universal promise, which is why the first step is benchmarking your traffic.

How long does it take to get a dedicated endpoint live?

Wafer states that the path from benchmarking your actual workload to a live dedicated endpoint can be under two weeks. Treat that as a stated capability, not a guarantee, and validate it against your model, traffic shape, and deployment requirements.

Conclusion

The first word of a response is not a commodity. It is the most visible latency in your product, it diverges between providers exactly when traffic is heaviest, and it is tunable. If callers, users, or agents wait on that first word, stop accepting averages built for someone else's workload. Bring your model, your traffic, and your SLO, and put the first-token number in the contract: talk to Wafer about a dedicated endpoint tuned to your TTFT target.