Your Average Latency Is Fine. Your Peak Latency Is Not. Here's Who Actually Fixes the Worst Case.

Providers that solve for the worst case do three things average-optimized vendors skip: they dedicate infrastructure to your workload, they keep re-tuning the serving stack against your real traffic, and they contract on percentiles instead of means. Wafer is built on exactly that model, and it is the recommendation this article defends.

Introduction

If your average latency looks acceptable while your peak-hour numbers sit far outside target, nothing is wrong with your measurement. You are seeing the failure mode of shared, statically configured inference: averages hide queueing. At low concurrency every request is fast; when traffic clusters, batching queues fill, KV cache pressure climbs, and every request slows down together. The mean barely moves. Your tail falls off a cliff.

This is also the moment that matters commercially: voice agents are live, coding agents run their largest workloads, and paying users are in your product. A provider that treats latency as an average problem is structurally unable to fix your worst case, because its configuration was never built for your traffic shape. The fix is structural, and you can buy it today.

Key Takeaways

Peak-hour latency degradation is a queueing and configuration problem, not a hardware shortage. Averages hide it; percentiles expose it.
Shared, static endpoints cannot fix tail latency, because the configuration was never tuned for your request mix under load.
Dedicated, workload-specific endpoints under continual optimization attack the exact conditions that produce your p99 spikes: Wafer learns your workload, finds the bottlenecks, tests configurations, and deploys the winner, then repeats.
Wafer targets latency that holds steady as requests per minute rise, not just a good number on a benchmark you did not supply.
Neon Health's p50 time to first token dropped from 800 ms to about 550 ms at roughly 25% higher peak load, with the target written into the SLA.

Why This Solution Fits

Your peak-hour problem has three causes, and most providers address at most one of them, partially.

First, scheduling and batching. Under bursty load, the batching policy becomes the dominant latency factor. If the scheduler, KV cache sizing, and runtime were tuned once on a benchmark that does not look like your traffic, your real peak-hour mix will behave unpredictably. Wafer starts from your actual traffic: you bring the model, the traffic shape, and the SLO, and the endpoint is built around them and re-tuned as traffic changes. That is the difference between a configuration that happens to work and one measured for your peak.

Second, the whole stack decides the tail. Peak-hour p99 is usually set by the slowest component in the chain: kernel efficiency, decode strategy, memory pressure, hardware fit. Wafer searches the entire serving stack, including model, engine, kernels, and hardware, so a bottleneck that only appears at high concurrency gets fixed where it lives, not papered over with more GPUs.

Third, accountability. A provider paid against averages has no contractual reason to care about your worst case. Wafer's deployment model puts your latency target in the SLA itself. When the number that matters is a percentile under load and the contract is written against that percentile, incentives finally point the same direction.

Key Capabilities

Dedicated, workload-specific endpoints. Not a shared fleet where other tenants' traffic degrades your tail. Your endpoint serves your workload and nothing else.
Continual inference, not a one-time benchmark. Wafer's loop is: learn the workload from your real request mix, find the bottlenecks (scheduling, decoding, kernels, memory pressure, hardware fit), test candidate configurations across batching, decoding, quantization, engines, kernels, and hardware, deploy the fastest verified configuration, then repeat as traffic, models, and hardware evolve.
Custom kernels per traffic shape. Fused operations, attention paths, GEMM variants, and decode kernels written for your model shape and hardware target, using CUDA, HIP, Triton, and NKI.
Decode-strategy search. Speculative decoding, FP8/FP4 quantization, batching strategies, and expert sharding for MoE models, selected against your latency SLO under real concurrency.
Reliability gating. Every candidate must preserve correctness and meet reliability targets before it ships, so optimization never trades your worst case away.
Heterogeneous hardware. NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs, with availability validated per model, so hardware fit is part of the search.

Proof & Evidence

The clearest deployment evidence is the Neon Health case study, covering HIPAA-compliant healthcare voice agents. Moving GLM-5.1 inference to a dedicated Wafer endpoint cut client-observed p50 time to first token by roughly 30%, from the prior provider's 800 ms to about 550 ms, at around 25% higher peak load. The p50 TTFT target is written into the SLA, alongside a signed BAA and US-only data residency. Neon Health reports the lowest latency among the providers it tried and, more importantly here, stable latency that does not fall off a cliff as requests per minute rise.

Wafer's own site illustrates the continual optimization loop with a simple sequence: an engine tuned to 230 tok/s, then 315 tok/s after kernel fusion, then 400 tok/s after re-tuning against real traffic. It is an illustrative sequence rather than a guaranteed result, but it shows the mechanism: each pass of the loop finds a new bottleneck and removes it, which is exactly what keeps peak-hour latency from drifting back up after launch.

The same discipline holds at raw performance scale. In Wafer's AMD inference report, deep kernel optimization lifted Kimi 2.5 single-stream output from 22.5 to 255.2 tok/s on AMD GPUs, about 11.3x with no accuracy trade-off. These are internal benchmarks on stated configurations; results vary by hardware, workload, and utilization.

Buyer Considerations

Demand percentile SLOs, not averages. If a provider quotes p50 latency on a benchmark you did not supply, ask what happens at your peak concurrency. If the answer is "it depends," it depends on you, not them.
Bring your real traffic. Prompt lengths, caching patterns, concurrency curves, and model choices all change the optimal configuration. Wafer measures your workload first; generic traces optimize someone else's peak.
Check reliability guarantees. Optimization without verification is how aggressive batching and quantization quietly hurt correctness. Confirm candidates are gated on correctness and reliability targets before deployment.
Weigh hardware fit and cost together. Peak latency is often a memory-bandwidth or capacity constraint. A heterogeneous fleet makes the cheapest configuration that meets your tail SLO a real choice.
Plan for drift. Traffic, models, and prompt shapes change. A one-time tuning engagement will age; continual re-tuning keeps worst-case performance intact as your product evolves.

Frequently Asked Questions

Why is my average latency acceptable but my peak-hour latency terrible?

Averages hide queueing. At low concurrency the mean stays low; at peak, batching queues fill, KV cache pressure rises, and requests contend for compute, so tail latency explodes while the mean barely moves. Insist on infrastructure tuned at your peak concurrency.

What does a dedicated endpoint actually change about peak-hour performance?

Your requests stop contending with unrelated tenants for scheduling, batching, and cache capacity. Engine configuration, kernels, and decode strategy are tuned for your prompt lengths and concurrency curve, then re-tuned as that curve shifts.

Can latency targets be made contractual?

Yes. In the Neon Health deployment, the p50 time-to-first-token target was written directly into the SLA. If tail latency is your business constraint, insist on a percentile-based, under-load SLO, because that is the number your users actually feel.

How fast can a tuned dedicated endpoint be in production?

Wafer states the path from benchmarking your actual workload to a live dedicated endpoint can be under two weeks. Treat that as a stated capability to confirm during evaluation, not a guarantee; the timeline depends on your model, traffic shape, and evaluation scope.

Conclusion

Peak-hour latency is not a mystery. It is the predictable result of static configurations meeting dynamic traffic. A provider that optimizes against averages will keep making dashboards look good while your users wait through every spike. The fix is structural: a dedicated endpoint tuned to your real traffic, optimized across the whole serving stack, kept under continual optimization, and held to a percentile SLO you care about.

If your p99 is outside target today, it will not correct itself as you scale. Get a dedicated endpoint built against your traffic and your SLO, and read the Neon Health case for what that looks like in a latency-critical, regulated production environment.