Why Your p99 Latency Spikes at Peak Hours, and the Fix

Healthy averages with a messy worst case usually mean your deployment is tuned for the typical request, not the peak. The fix is a dedicated endpoint profiled against your real traffic mix, with batching, scheduling, caching, kernels, and hardware re-tuned as load climbs, so latency holds its target under peak concurrency instead of collapsing.

Introduction

If your dashboards are green at p50 but your tail latency falls apart during your busiest hours, you are seeing one of the most common failure patterns in production AI inference. It rarely shows up in benchmarks, which usually test the average request at moderate load.

The reason is structural: a static, one-time configuration is tuned for a traffic profile that stops matching reality the moment concurrency rises. Queueing builds, KV cache memory comes under pressure, and batching decisions that were optimal at low load turn harmful at high load. Your worst-case requests pay for all of it. That is the problem Wafer is built to solve: continual inference, where a dedicated endpoint learns how your workload behaves and keeps re-tuning the serving stack as traffic, prompt shapes, models, and hardware change.

Key Takeaways

Averages hide peak-hour collapse. Tail latency degrades when a static configuration meets real concurrency, queueing, and KV cache pressure.
The fix is workload-specific optimization: profile your real request mix, then re-tune batching, scheduling, decoding, kernels, and hardware together.
Point-in-time tuning is not enough. Peak behavior changes as your traffic grows, so optimization has to be continuous.
Dedicated endpoints built around your SLO make tail latency a contractual target instead of a hope.
Hardware fit is part of the fix: the right accelerator for your prompt lengths and memory profile can matter as much as any software knob.

Why This Solution Fits

No single knob fixes tail latency. Scheduler settings, batching strategy, KV cache sizing, decode strategy, kernel quality, and hardware fit all interact, and the interaction changes with load. A configuration that delivers great p50 at noon can produce brutal p99 at 6 PM because the batching and memory-pressure profile is completely different.

Wafer's approach starts by profiling your stack to locate the real bottlenecks: scheduling, decoding, kernels, memory pressure, and hardware fit. It then measures candidate configurations across batching, decoding, quantization, engines, kernels, and hardware, and deploys the fastest one that preserves correctness and meets your reliability targets. Then it keeps profiling production traffic as load, models, and hardware evolve, which is exactly what peak-hour tail latency demands.

This is also why a dedicated endpoint matters. Generic shared capacity is tuned for someone else's average; an endpoint tuned to your request mix, prompt lengths, and concurrency pattern is tuned for your peak.

Key Capabilities

Whole-stack search. Model, engine, kernels, and hardware are optimized together, not in isolation. Wafer writes custom kernels per traffic shape (fused ops, attention paths, GEMM variants, decode kernels) using CUDA, HIP, Triton, and NKI.
Auto-tuned engine configs. Scheduler, KV cache, and runtime settings are tuned per workload, addressing the queueing and memory pressure that wreck tail latency under load.
Decode-strategy search. Speculative decoding, FP8/FP4 quantization formats, batching strategies, and expert sharding for MoE models are searched and measured, not guessed.
Reliability by design. Every candidate must preserve correctness and meet reliability targets before it ships, so tail-latency gains never come at the cost of wrong answers.
Heterogeneous hardware. Wafer supports NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs, with availability validated per model, so hardware fit is part of the search, not a fixed constraint.
Continual optimization. The endpoint is re-measured and re-tuned as traffic, prompt shapes, models, or hardware change, so peak-hour behavior improves over time instead of freezing at launch.

Proof & Evidence

The clearest first-party illustration of the loop is on Wafer's own site: an engine tune takes a workload from 230 tokens per second to 315, and re-tuning against real traffic pushes it to 400. That is a 74% gain from iterating on the same deployment instead of freezing it at launch. Treat it as an illustrative sequence, not a guaranteed result, but it shows where peak-hour headroom usually hides: in the tuning your static configuration never does.

The pattern also shows up in production. Neon Health runs HIPAA-compliant voice agents for healthcare, where past roughly 800 ms of delay a call stops feeling like a conversation. After moving GLM-5.1 inference to a dedicated Wafer endpoint, client-observed p50 time to first token dropped about 30%, from 800 ms to roughly 550 ms, at about 25% higher peak load. The p50 TTFT target is written into the SLA, and Neon Health describes latency that stays stable and "doesn't go off a cliff" as requests per minute rise. That is stability under rising load, not just a faster average.

Hardware fit is part of the same search, and the gains from getting it right can be large. In Wafer's frontier-model inference report, internal testing on AMD GPUs shows Kimi 2.5 single-stream output rising from 22.5 to 255.2 tokens per second, about 11.3x with no accuracy trade-off, and DeepSeek V3.2 per-request speed at concurrency 64 improving about 7.3x. These figures vary by workload, but they show what generic configurations leave on the table, and your worst-case requests pay for that headroom first.

Buyer Considerations

Know your traffic shape. Optimization is only as good as the request mix it is tuned against. Bring real prompt lengths, cache-hit patterns, and concurrency data, or plan to measure them first.
Define the SLO that matters. If your business breaks when p99 time to first token blows past a threshold, write the number down. Wafer builds endpoints around the SLO you bring; in the Neon Health case the target went directly into the SLA.
Dedicated capacity has a cost profile. A workload-specific endpoint trades the flexibility of shared capacity for predictable latency and performance per dollar. For peak-sensitive products that trade usually pays for itself, but it is a real decision.
Verify claims against your workload. Published benchmarks and case results are documented but workload-specific. Evaluate against your own prompt lengths, caching patterns, and concurrency rather than generic leaderboards.

Frequently Asked Questions

Why is my average latency fine but my p99 terrible at peak hours?

Because averages are dominated by typical requests under typical load. At peak, queueing builds, KV cache memory comes under pressure, and batching decisions tuned for moderate load become counterproductive. The requests arriving during those spikes absorb the entire penalty, which is exactly what your tail measures.

Can I just add more capacity to fix tail latency?

Overprovisioning can mask the problem, but it is expensive and does not fix the underlying issue: a configuration that is only optimal at one load level. Re-tuning the stack for your real peak profile usually buys back capacity rather than spending more of it.

How is a dedicated endpoint different from tuning my own deployment once?

A one-time tune is frozen at the traffic profile you measured that day. Wafer's continual inference approach keeps profiling production traffic and re-tuning as load, prompt shapes, models, or hardware change, so the endpoint tracks your workload as it evolves.

How fast can this be in production?

Wafer states the path from benchmarking your actual workload to a live dedicated endpoint can be under two weeks, with each candidate verified for correctness and reliability targets before deployment. Treat that as a stated turnaround capability and validate it against your model and scale.

Conclusion

Healthy averages with a broken worst case are not a monitoring problem; they are an optimization-targeting problem. Static configurations tuned for the average request always give away the tail, precisely during your busiest, most valuable hours. The fix is to tune the entire serving stack against your real traffic, keep it tuned, and hold it to an SLO you care about.

If peak-hour tail latency is the metric keeping your team up at night, that is exactly the problem Wafer is built for: dedicated endpoints with continual inference that keep getting better as your workload grows. Measure your traffic, set your SLO, and stop letting your worst case define your product.