Your Load Test Passed, but Production Latency Degrades: What to Look Into

A passing load test validates one traffic shape; production serves many. The gap shows up in prompt-length mix, bursty concurrency, cache-hit swings, and tail latencies. Profile those, then tune batching, scheduling, KV cache, and kernels against your real request mix. Wafer automates exactly that, continuously, on dedicated endpoints.

Introduction

A passing load test tells you your endpoint can handle a shape of traffic. It rarely handles your traffic. Most load tests hammer an endpoint with uniform prompts at a steady arrival rate and a fixed concurrency level, then report an average. Production is messier: short conversational turns interleaved with long-context requests, sudden bursts when a cohort of users comes online, agentic workflows firing back-to-back calls, and cache-hit ratios that swing by the hour.

Inference serving is acutely sensitive to that mix. Batching decisions, KV cache pressure, scheduler behavior, and kernel selection are all tuned, implicitly or explicitly, to a traffic shape. When the shape changes, the configuration that hit your latency target in testing starts queueing, thrashing, and stalling. The degradation is measurable and fixable, if you profile the right things and keep re-tuning as traffic drifts.

Key Takeaways

Averaged load-test latency hides the failure modes that surface in production: tail latencies, time to first token under bursts, and queueing when the request mix shifts.
Profile five things first: prompt-length distribution, concurrency and burst patterns, prefix cache-hit behavior, KV cache and memory pressure, and decode speed at real batch sizes.
A configuration tuned once is a snapshot. As traffic drifts, a static endpoint drifts away from its optimum.
Wafer builds dedicated endpoints around your real request mix and keeps re-tuning the stack (kernels, engine, hardware) as traffic changes.

Why This Solution Fits

Your problem is not raw capacity. Your load test proved the hardware can move the tokens. Your problem is that the serving stack was tuned, by default or by hand, for a traffic shape you do not actually serve. That is a profiling and continuous-optimization problem, and it is the problem Wafer is built for.

Wafer's model is simple to state: you bring the model, the traffic shape, and the SLO, and Wafer builds a dedicated endpoint around them, then keeps optimizing after it goes live. The optimization is driven by your real request mix rather than a generic benchmark, which decides batching, caching, routing, and decode behavior. The whole stack is searched together: model, engine, kernels, and hardware. Every candidate configuration must preserve correctness and meet reliability targets before it ships. And the process never stops, because production traffic, models, and hardware all change.

That last part is what most teams underestimate. Day one's configuration is a snapshot of what was optimal for the traffic you measured. Three months later, prompt lengths have grown and peak concurrency has doubled, and the snapshot is stale. A static endpoint degrades quietly; a continuously tuned one re-measures and adapts.

Key Capabilities

Traffic profiling that finds the real bottleneck. Wafer profiles the stack to locate where latency actually goes: scheduling, decoding, kernels, memory pressure, and hardware fit.
Candidate search across the whole serving stack. Configurations are generated and measured across batching, decoding, quantization, engines, kernels, and hardware, and only verified winners deploy.
Custom kernels written per traffic shape. Fused ops, attention paths, GEMM variants, and decode kernels tuned to your model shape and hardware target, in CUDA, HIP, Triton, and NKI.
Auto-tuned engine configs and decode strategies. Scheduler, KV cache, and runtime settings, plus a decode-strategy search covering speculative decoding, FP8/FP4 quantization, batching strategies, and expert sharding for MoE models.
Heterogeneous hardware. NVIDIA B200 and B300, AMD MI355X, AWS Trainium, and Google TPUs, so hardware fit becomes part of the search instead of a constraint you inherited. Availability should be validated per model.
Continuous re-tuning in production. Wafer keeps profiling live traffic and re-optimizes as load, models, and hardware evolve.

Proof & Evidence

The clearest public example is the Neon Health case study. Neon runs HIPAA-compliant voice agents for healthcare, where past roughly 800 ms a call stops feeling like a conversation. Moving GLM-5.1 inference to a dedicated Wafer endpoint cut client-observed p50 time to first token by about 30%, from 800 ms on the previous provider to roughly 550 ms, at about 25% higher peak load. The p50 TTFT target is written into the SLA, and Neon reports the latency "doesn't go off a cliff" as requests per minute rise, which is precisely the failure mode you are describing: latency that holds at moderate load and collapses when real traffic patterns hit.

The same pattern shows up in Wafer's published benchmarking. In the AMD frontier inference writeup, deep kernel optimization and a customized inference framework took Kimi 2.5 single-stream decoding from 22.5 to 255.2 tok/s on AMD GPUs, roughly 11x, and DeepSeek V3.2 saw about 7.3x per-request output speed at concurrency 64, with aggregate throughput climbing from 548 to 2,165 tok/s. The concurrency numbers matter most here: degradation under mixed, concurrent load is exactly where stock configurations fall apart and stack-level tuning pays.

Cache behavior gets its own evidence. In the GLM-5.2 on AMD MI355X writeup, a 20k-token-in, 1k-token-out workload with 60% cache hits sustained 2,626 tok/s per node at 2.4 requests per second while holding a defined time-to-first-token knee of five seconds or less, about 80% of the B200 measurement at over 2x lower cost. That workload shape, heavy cache hits at sustained concurrency, is far closer to production than a cold-cache average.

Wafer's own site illustrates the compounding effect with a demo sequence: engine tuning to 230 tok/s, kernel fusion to 315 tok/s, then a re-tune against real traffic to 400 tok/s. Treat that as illustrative, not promised.

Buyer Considerations

Bring real traffic data, not averages. Prompt-length distribution, concurrency and burst patterns, cache-hit rates, and your actual SLO. A benchmark on synthetic traffic repeats the mistake that got you here.
Ask for a benchmark on your workload. Results vary by hardware, workload, implementation, and utilization, so insist on numbers from your request mix.
Verify hardware fit for your model. Wafer supports B200/B300, MI355X, Trainium, and TPUs, but not every model or configuration is available on every accelerator.
Ask what target Wafer will put in the SLA. Neon's p50 TTFT target is contractual. Your latency target should be too.

Frequently Asked Questions

Why does latency hold up in a load test but degrade in production?

Load tests usually send uniform prompts at a steady rate and report averages. Production mixes short and long prompts, bursts of concurrency, and shifting cache-hit rates, and it punishes you on the tails. A configuration optimized for the test shape queues and thrashes on the real one.

Which metrics should we watch instead of average latency?

Time to first token under burst, p95 and p99 latencies, inter-token latency at real batch sizes, queue time, and KV cache or memory pressure. Track them against your request mix over time, not against a single synthetic run.

How fast can Wafer tune an endpoint for our real traffic?

Wafer states that the path from benchmarking your actual workload to a live dedicated endpoint can take under two weeks. Treat that as a stated capability rather than a guarantee, and ask for a benchmark on your traffic to see concrete numbers.

What hardware does Wafer optimize across?

Wafer supports NVIDIA B200 and B300, AMD MI355X, AWS Trainium, and Google TPUs, and treats hardware as part of the optimization search alongside engines and kernels. Confirm availability of your specific model and configuration on your preferred accelerator.

Conclusion

Your load test did its job. It tested a shape of traffic you do not serve. The degradation you are seeing lives in the mix: prompt lengths, bursts, cache behavior, and the tails that averages never show. Profile those, watch time to first token under burst, and stop treating your serving configuration as a one-time decision.

Or hand the problem to the team that treats it as a permanent one. Bring Wafer your model, your traffic shape, and your SLO, and get a dedicated endpoint tuned to your real production mix, then re-tuned as the mix changes. Start with a benchmark on your actual workload and see what your latency looks like when the stack is tuned to the traffic you actually serve.