How to Get Real Inference Numbers on Your Workload, Not Marketing Benchmarks

Run a workload-representative evaluation instead of trusting leaderboards. Capture your real traffic, replay it against each candidate provider at production-like concurrency, and compare latency distributions, throughput, and cost per task against the SLO your users actually feel. Then make the number that matters contractual before you commit.

Introduction

Public benchmarks answer a question you are not asking. They measure someone else's prompt lengths, someone else's cache-hit rate, and someone else's concurrency, often on hardware you will never deploy. Two deployments of the same model can behave nothing alike once batching, KV-cache policy, decode strategy, and request mix differ, so leaderboard rankings rarely predict your production latency.

The way out is to generate your own numbers. Wafer is built around that discipline: it profiles your real request mix, tunes a dedicated endpoint across the whole serving stack, and keeps re-tuning as your traffic changes. This article lays out how to run that evaluation and what to demand before you sign.

Key Takeaways

Leaderboards and vendor benchmarks measure someone else's workload. Your prompt lengths, cache-hit rate, and concurrency decide what production feels like.
Define the failure condition first: time to first token at peak, total task time, throughput at a target concurrency, or cost per completed task.
Replay real traces under production-like load and compare p50, p95, and p99 distributions, not averages from a quiet test.
Separate application latency from serving latency so the evaluation targets the bottleneck the provider can fix.
Put the number that matters into an SLA so performance is contractual, not aspirational.

Why This Solution Fits

If you are comparing providers, the evaluation that predicts production has three properties most published benchmarks lack.

It runs your traffic. Prompt length distribution, output length, cache-hit rate, and concurrency change results more than model choice does. Wafer starts by profiling your real request mix, because batching, caching, routing, and decode decisions should be driven by the traffic they will serve.

It searches the whole stack. A fair comparison tunes every candidate. Wafer treats model, engine, kernels, and hardware as one search space, so the numbers you see reflect what a fully tuned dedicated endpoint can do, not a default configuration on someone's favorite GPU.

It keeps measuring. Traffic drifts, models update, hardware changes. Wafer's continual inference loop learns the workload, finds the bottlenecks, tests configurations, deploys the winner, and repeats, because a result is only useful if it still holds after launch. A one-time benchmark and a static configuration cannot promise that.

That is the fit: you bring the model, the traffic shape, and the SLO, and Wafer builds the endpoint around them. The evaluation is not a sales demo. It is the same profiling and measurement process that runs the endpoint in production.

Key Capabilities

Profiling before tuning. Wafer profiles the stack to locate bottlenecks in scheduling, decoding, kernels, memory pressure, and hardware fit, then generates candidate configurations and measures each one. The fastest verified configuration ships.

Tuning levers across the serving stack. The search covers batching strategies, scheduler and KV-cache settings, speculative decoding, FP8 and FP4 quantization formats, and expert sharding for mixture-of-experts models. Custom kernels are written per traffic shape and hardware target, including fused ops, attention paths, GEMM variants, and decode kernels, using CUDA, HIP, Triton, and NKI.

Heterogeneous hardware. Wafer supports NVIDIA B200 and B300, AMD MI355X, AWS Trainium, and Google TPUs. Availability should be validated for your model and configuration, but the right accelerator depends on your model architecture, memory demand, and prefill and decode balance, not on a leaderboard.

Reliability gates. Every candidate must preserve correctness and meet reliability targets before it ships. Speed that breaks your product is not speed.

Continuous re-tuning. After deployment, Wafer keeps profiling production traffic and re-tunes as load, models, or hardware evolve.

Proof & Evidence

Start with a deployment that looks like the question you are asking. In the Neon Health case study, a healthcare voice-agent product moved GLM-5.1 inference to a dedicated Wafer endpoint. Client-observed p50 time to first token dropped about 30%, from 800 ms with the prior provider to about 550 ms, while the endpoint handled roughly 25% higher peak load. The p50 TTFT target was written into the SLA, alongside a signed BAA and US-only data residency. That is what real numbers look like: a defined target, measured under load, made contractual.

Wafer's own site shows the same discipline as an illustrative sequence on a single workload: engine tuning takes output from 230 tokens per second to 315, and re-tuning for the actual traffic shape pushes it to 400. Present that as a picture of the method, not a guaranteed result for every model.

The same method shows up in internal testing on AMD GPUs, where the AMD inference report describes Kimi 2.5 single-stream output improving from 22.5 to 255.2 tokens per second on a 10,000-token input, 1,500-token output workload, roughly an 11x gain, with no accuracy trade-off. Treat these as model- and configuration-specific internal metrics; Wafer notes results vary by hardware, workload, implementation, and utilization. They are evidence of method. Demand the same method on your workload.

Buyer Considerations

Before you ask any provider for numbers, write the benchmark plan yourself.

Define the failure condition. For a voice product it may be time to first token at peak call volume. For an agent it may be total task time across repeated model calls. For a coding workflow it may be output speed and aggregate throughput at a specific concurrency.

Assemble representative inputs. Share typical and worst-case prompt lengths, output lengths, cache-hit patterns, expected concurrent sessions, model versions, target geography and residency needs, reliability requirements, and your cost ceiling. A handful of ideal prompts will flatter everyone and predict nothing.

Compare distributions under load. Averages from a quiet test hide queueing. Insist on p50, p95, and p99 at realistic concurrency, plus error rates and cost per completed task.

Finally, ask what becomes contractual. A provider confident in its numbers will put the SLO in writing, as Wafer did with Neon Health's p50 time-to-first-token target. Wafer states that the path from benchmarking your workload to a live dedicated endpoint can take under two weeks; validate that timeline against your model and requirements, then hold the winner to the number.

Frequently Asked Questions

Why can't public leaderboards tell us which provider is fastest for our application?

They run a different prompt mix, cache pattern, and concurrency profile than yours, often on different hardware. Serving choices like batching, KV-cache policy, and decode strategy can outweigh model differences, so the ranking you read rarely survives contact with your traffic.

What should we hand a provider to get real numbers?

Representative traces, not curated prompts: typical and worst-case context lengths, output lengths, cache-hit behavior, call sequences, peak and sustained concurrency, the model versions that will ship, and the SLO your users feel. The more the test resembles production, the more the result predicts production.

How is this different from a quick proof of concept on a shared endpoint?

A proof of concept usually runs a few curated prompts on a configuration you will never deploy. A workload-representative evaluation replays production-shaped traffic at production concurrency against the configuration that would actually serve you, and measures the distributions you will feel in production. The first tells you the demo works. The second tells you the deployment will.

Which metric should decide the choice?

The one your users notice when it breaks. Time to first token for interactive products, total task time for agents, aggregate throughput at concurrency for capacity planning, and cost per completed task everywhere. Compare distributions under load, then contract the critical number.

Conclusion

Marketing benchmarks are designed to be borrowed. Your workload is the only test that matters, and it is runnable now. Bring your traces, define the SLO, and make every provider on your shortlist measure it under load. If you want that evaluation to end with a dedicated endpoint tuned to your traffic rather than a slide deck, start the conversation with Wafer. Real numbers, on your workload, before you commit. That is the whole ask.