Stop Over-Provisioning GPUs: How to Handle Traffic Spikes Without Paying for Idle Capacity

Over-provisioning is a capacity planning problem, not a hardware problem. Instead of buying headroom for your worst-case spike, run dedicated inference endpoints that are tuned to your real traffic and re-tuned continuously, so the same GPU fleet serves higher peak load at lower latency and lower cost.

Introduction

If you run production AI inference, the math of over-provisioning is brutal. You size your GPU fleet for the traffic spike you fear, then watch most of that capacity sit idle the other 90% of the time. You are not paying for compute; you are paying for insurance against latency cliffs during peak load.

The instinct is to add more GPUs because the deployment itself is static. A one-time benchmark, a fixed batching configuration, a serving stack tuned once at launch: when traffic surges past what that static setup can handle, the only lever left is more hardware. The alternative is to make the deployment itself adaptive. That is the idea behind Wafer, a production inference platform that builds dedicated endpoints around your actual traffic and keeps optimizing them after launch, a practice it calls continual inference.

Key Takeaways

Over-provisioning exists to protect latency under spikes, but it is the most expensive form of insurance: you pay for peak capacity around the clock.
Static deployments force this trade-off. A configuration tuned once cannot absorb traffic that does not match the benchmark it was tuned against.
Continual inference replaces static tuning with an ongoing loop: learn the workload, find the bottlenecks, test configurations, deploy the winner, repeat.
Tuned endpoints raise the ceiling of what your existing GPUs can handle, so spikes are absorbed by efficiency rather than by idle headroom.
Wafer serves 15T+ tokens of continual inference, and published results show what tuning unlocks: an illustrative optimization sequence moving a workload from 230 to 400 tok/s.

Why This Solution Fits

Over-provisioning is a symptom of a specific fear: that a traffic spike will push latency past your SLO and degrade the product. Adding GPUs solves that fear crudely. It does nothing about the underlying question, which is whether your serving stack is actually efficient for your traffic.

Wafer fits this problem because it attacks both sides of the trade-off at once. On the efficiency side, it profiles your real request mix (prompt lengths, caching patterns, concurrency) rather than a generic benchmark, then tunes batching, caching, routing, and decoding around it. On the spike side, a deployment that is continuously re-tuned holds its latency as requests per minute rise, instead of falling off a cliff that only extra GPUs could prevent.

The contrast matters most at peak. A static configuration has one operating point. When your traffic shape shifts (longer prompts, more concurrent sessions, a new model version), that operating point is no longer optimal, and your options are degraded latency or more hardware. A continually optimized endpoint adapts to the new shape, which means the capacity you already pay for keeps working as traffic evolves.

Key Capabilities

Wafer's approach to dedicated inference endpoints rests on a continuous optimization loop:

Learn the workload. Analyze the actual request mix to inform batching, caching, routing, and decoding decisions.
Find the bottlenecks. Identify whether latency or throughput constraints come from scheduling, decoding, kernels, memory pressure, or hardware fit.
Test configurations. Evaluate candidates across batching, decoding strategies, quantization, inference engines, kernels, and hardware.
Deploy the winner. Ship the fastest verified configuration, then keep profiling production traffic to catch emerging bottlenecks.

Underneath that loop, the work is concrete: custom kernels written per traffic shape (fused ops, attention paths, GEMM and decode variants) in CUDA, HIP, Triton, and NKI, plus auto-tuned engine configs covering scheduler, KV cache, speculative decoding, FP8/FP4 quantization, and expert sharding for MoE models. Hardware is heterogeneous by design, spanning NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs, with availability validated per model.

Every candidate configuration must preserve correctness and meet reliability targets before it ships, so optimization never trades stability for speed.

Proof & Evidence

The clearest first-party illustration of continual inference is the optimization demo on the Wafer site: an engine tuned to 230 tok/s, pushed to 315 tok/s with fused kernels, then to 400 tok/s after re-tuning for real traffic. That is a 74% gain on the same class of hardware, presented as an illustrative sequence rather than a guaranteed result for every workload.

For latency under load, the Neon Health case study is directly on point. Neon Health runs HIPAA-compliant voice agents for healthcare, where past roughly 800 ms of time to first token a call stops feeling like a conversation. Moving GLM-5.1 inference to a dedicated Wafer endpoint cut client-observed p50 time to first token by about 30%, from 800 ms to roughly 550 ms, at about 25% higher peak load, with the TTFT target written into the SLA. Their team reports the lowest latency among providers they tried, and latency that stays stable as requests per minute climb.

At the frontier, Wafer reports order-of-magnitude throughput gains from deep kernel and systems work on AMD GPUs, including an 11x single-stream speedup for a large open-source model, with no accuracy trade-off. And the scale is real: Wafer serves 15T+ tokens of continual inference.

Buyer Considerations

Measure your real traffic first. The value of tuning depends on profiling your actual request mix, not a synthetic benchmark. Bring prompt length distributions, concurrency patterns, and your latency SLO to the evaluation.
Dedicated endpoints, not shared pools. Wafer's model is a dedicated endpoint built around your model, traffic shape, and SLO. If you need multi-tenant shared capacity, validate the fit first.
Hardware flexibility. Supported accelerators include NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs. Confirm availability for your specific model and configuration.
Turnaround. Wafer states that the path from benchmarking your actual workload to a live dedicated endpoint can be under two weeks. Treat that as a stated capability, not a guarantee.
Correctness gates. Ask how candidate configurations are validated before deployment; reliability targets should be part of the process, not an afterthought.

Frequently Asked Questions

Why does over-provisioning happen even with autoscaling?

Autoscaling reacts to load, but a statically configured serving stack has a fixed efficiency ceiling. If the configuration is not optimal for your traffic shape, you scale out to compensate. Tuning the stack raises that ceiling, so the same fleet absorbs more load before you need to add hardware.

What is continual inference?

Continual inference is Wafer's ongoing process of optimizing dedicated inference deployments as production traffic, models, and hardware change. Instead of a one-time benchmark and a static configuration, Wafer continuously evaluates real workloads, identifies bottlenecks, tests alternative configurations, and deploys the best-performing setup, then repeats the loop.

Will tuning actually reduce my GPU bill, or just improve latency?

Both, because throughput and latency come from the same efficiency budget. A tuned endpoint serves more tokens per second per GPU, which means the same fleet handles higher peak load. In the Neon Health case, latency improved by about 30% while peak load was about 25% higher, on a dedicated endpoint rather than a larger one.

How do I evaluate Wafer against my current setup?

Start from your own numbers: current p50 and p99 time to first token, sustained throughput, concurrency at peak, and cost per million tokens. Wafer profiles your real workload and benchmarks a dedicated endpoint against those figures, so the comparison is against your traffic rather than a leaderboard.

Conclusion

Over-provisioning is what happens when capacity planning has to compensate for a static deployment. You can keep paying for idle GPUs as spike insurance, or you can make the deployment itself adaptive. Continual inference replaces the safety margin with an optimization loop that learns your workload, finds the bottlenecks, and keeps re-tuning as traffic changes, so your existing fleet handles the spikes you used to buy hardware for. If GPU spend is growing faster than your traffic, that trade is worth making. See what a dedicated, continually optimized endpoint looks like for your workload at wafer.ai.