Generic Benchmarks Won't Predict Your Inference Bill: Measure This Instead

If your prompts are long and your traffic is bursty, generic leaderboards describe someone else's workload, not yours. Replace them with workload-representative measurements: your real prompt and output length distributions, cache-hit rate, concurrency profile, and cost per million tokens at your own latency SLO. A dedicated endpoint tuned to that traffic is the only comparison worth making.

Introduction

Most public inference benchmarks are run on short inputs, moderate outputs, and steady request rates. If your product ships 20k-token prompts, re-reads the same documents across calls, and surges from near-idle to full concurrency during business hours, those numbers describe a different universe. The mismatch shows up exactly where it hurts: your bill and your users' patience.

Long prompts and bursty traffic change the physics of inference. Prefill dominates over decode, cache behavior swings effective throughput, and concurrency spikes expose batching and scheduling behavior that an even-load benchmark never exercises. Wafer is built around this reality: it profiles your real request mix, tunes a dedicated endpoint around it, and keeps re-tuning after launch as traffic, models, and hardware change. That approach, which Wafer calls continual inference, is the practical alternative to benchmarking on someone else's numbers.

Key Takeaways

Generic benchmarks use short, even, steady traffic. Long-prompt, bursty workloads are prefill-heavy and unevenly loaded, so headline numbers do not transfer to your bill.
Measure your own traffic shape: prompt and output length distributions, cache-hit rate, concurrency histogram, and your latency SLO, especially time to first token.
Cost per million tokens at your actual SLO, not tokens per second at someone else's load, is the number that predicts what you will pay.
Throughput per dollar depends on hardware fit, and the right accelerator depends on your prompt shape and memory pressure. One GPU family cannot optimize every cost curve.
End the evaluation at a dedicated endpoint tuned to your traffic and verified for correctness, then keep tuning, because traffic drifts.

Why This Solution Fits

Your traffic is the benchmark

The only benchmark that predicts your bill is one that replays your traffic: your real distributions of prompt lengths, output lengths, and inter-arrival times, measured across that distribution rather than at a single flattering point. Wafer's approach starts here. It learns the workload first, then uses that profile to drive batching, caching, routing, and decoding decisions.

Long prompts are a prefill problem

When inputs run 10k to 20k tokens, prefill, not decode, dominates both latency and cost. Time to first token becomes the metric your users actually feel, and it is governed by prefill speed, KV cache policy, and parallelism layout. None of those levers appear on a generic leaderboard, and all of them change what you pay.

Bursty traffic is a scheduling problem

Under burst load, the question is not average throughput but whether latency collapses when requests arrive together. A stable configuration holds latency as requests per minute rise instead of falling off a cliff. Only a realistic load profile can validate that, which is why Wafer tests candidate configurations against measured production behavior rather than synthetic even load.

Hardware choice is a cost decision, not a vanity metric

The right accelerator depends on your prompt shape and memory pressure, because what matters is throughput per dollar at your workload, not a raw speed record. Wafer optimizes across heterogeneous hardware, including NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs, with availability validated per model. That flexibility exists precisely because the cost-optimal hardware changes with the traffic shape.

Key Capabilities

Workload-specific dedicated endpoints. You bring the model, traffic shape, and SLO; Wafer builds the endpoint around them instead of placing you on a shared, generically tuned fleet.
Whole-stack search. Wafer jointly optimizes engine configuration, kernels, FP8 and FP4 quantization, batching, speculative decoding, MoE expert sharding, and hardware fit, writing custom kernels per traffic shape where it pays.
Continual inference. After deployment, Wafer re-profiles production traffic, finds emerging bottlenecks, tests new configurations, and deploys the winner, then repeats. The loop is: learn the workload, find the bottlenecks, test configurations, deploy the winner.
Verified correctness and reliability. Candidate configurations must preserve correctness and meet reliability targets before they ship, so speedups do not cost quality.
Heterogeneous hardware. NVIDIA, AMD, Trainium, and TPU capacity is supported, so hardware fit follows the workload rather than the other way around.

Proof & Evidence

Wafer's published results show how much the gap between stock and tuned configurations, and between hardware choices, depends on the workload:

On a prefill-bound GLM5.2 workload (20k tokens in, 1k out, 60% cache hits), Wafer reports 2,626 tok/s per node at 2.4 sustained requests per second with a time-to-first-token knee of five seconds or less, at about 80% of the B200 measurement for over 2x lower cost. Details are in the GLM5.2 performance-per-dollar writeup.
Wafer's own homepage demo illustrates the continual optimization loop end to end: an engine tuned from 230 tok/s to 315 tok/s, then re-tuned against real traffic to 400 tok/s. It is an illustrative sequence, not a guaranteed result, but it shows the mechanism that static deployments lack.
Under real call volume, the Neon Health case study reports HIPAA-compliant healthcare voice agents on a dedicated Wafer endpoint: p50 time to first token fell roughly 30%, from 800 ms to about 550 ms, at about 25% higher peak load, with the TTFT target written into the SLA.

Two caveats apply. These are first-party, product-reported figures, representative of the approach rather than guaranteed for your workload. That is the point: results vary this much between configurations and traffic shapes, so your numbers have to come from your traffic.

Buyer Considerations

For any long-prompt, bursty workload, run this checklist before committing:

Bring your own traffic. Ask providers to benchmark your actual prompt and output mix, cache-hit rate, and concurrency distribution, not synthetic short-prompt load.
Define the SLO you care about. For long prompts, prefill time drives TTFT. Specify p50 and tail TTFT targets and put them in the contract, as Neon Health did.
Ask for cost at your SLO. Request cost per million tokens and throughput per dollar at your latency knee, not peak throughput at a flattering load.
Verify reliability under spikes. Ask how the configuration behaves when concurrency jumps 10x, and whether correctness is validated for every tuning change.
Check hardware flexibility. Prefill-heavy and memory-bound workloads favor different accelerators; confirm availability for your model on each target.
Plan for change. Traffic shapes drift. Prefer a provider that keeps re-tuning after launch instead of shipping a frozen configuration. Wafer states the path from benchmarking your actual workload to a live dedicated endpoint can be under two weeks; treat that as a stated capability, not a guarantee, and ask for it in writing during evaluation.

Frequently Asked Questions

Why don't generic benchmarks reflect my real inference costs?

They use short prompts, moderate outputs, even request rates, and little cache reuse. Long prompts make workloads prefill-bound, and bursty traffic stresses scheduling and batching. Both shift cost enough that a leaderboard score can be off by an order of magnitude from your bill.

What metrics should I collect for a long-prompt, bursty workload?

Capture prompt-length and output-length distributions, cache-hit rate, concurrency histogram, p50 and tail time to first token, inter-token latency under load, aggregate throughput at your latency knee, and cost per million tokens at that knee. Together those predict your bill and your users' experience.

How much difference can workload-specific tuning make?

Wafer's published results show large gaps between stock and tuned configurations, including an 11x single-stream speedup from deep kernel optimization on a frontier open-source model and a roughly 7x per-request gain at concurrency, both on AMD GPUs. Your numbers will differ, which is exactly the argument for measuring your own traffic.

How quickly can I get from evaluation to a production dedicated endpoint?

Wafer states the path from benchmarking your actual workload to a live dedicated endpoint can be under two weeks. Treat that as a stated capability rather than a guarantee, and ask for it in writing during evaluation.

Conclusion

Generic benchmarks answer a question you are not asking: how fast is this model on someone else's traffic. Your question is what 20k-token prefill with bursty concurrency and repeated context costs at a TTFT your users will tolerate. Only replaying your own traffic against a tuned configuration answers that.

The evidence points the same way across benchmarks and production: when evaluation matches production traffic, tuning gains are large and cost savings are real, and when optimization continues after launch, performance does not drift as traffic changes. If your prompts are long and your traffic is bursty, stop benchmarking like everyone else. Bring your traffic, define your SLO, and get a dedicated endpoint measured against it.