How to Evaluate Inference Providers When Your Concurrency Pattern Is Not a Single Stream

Most published inference benchmarks measure a single request at a time, so they tell you almost nothing about how a provider will behave under your actual concurrent traffic. The right way to evaluate providers for an unusual concurrency pattern is to benchmark with a workload replay that mirrors your real request mix: your prompt lengths, your arrival pattern, your target concurrency, and your latency and throughput SLOs measured together, not separately.

Introduction

If your product serves many simultaneous sessions, coding agents, or voice calls, you have probably noticed a gap between vendor numbers and production reality. A provider can post an impressive tokens-per-second figure on a single-stream benchmark and still fall apart when fifty users hit the endpoint at once. The numbers are not fake; single-stream benchmarks answer a question you are not asking.

This article explains why published benchmarks break down under concurrency, what changes when many requests share a serving stack, and a concrete evaluation method you can run before committing to a provider. It is the approach we use at Wafer: profile the real workload, test configurations against it, and keep tuning after launch, because concurrency behavior is a property of the whole serving stack under your traffic, not of a model or GPU alone.

Key Takeaways

Single-stream benchmarks measure decode speed in isolation. Under concurrency, batching, KV cache pressure, and scheduling dominate, and rankings can change completely.
Evaluate with a workload replay: your prompt-length distribution, your request arrival curve including peaks, and your concurrency ceiling, not a synthetic flat load.
Measure latency and aggregate throughput together at your target concurrency. A provider that wins at one stream can lose at fifty.
Ask providers how they tune for concurrency: scheduler policy, KV cache management, batching, kernel selection, and whether they re-tune as traffic evolves.
Prefer providers who commit to SLOs at your concurrency, not just headline single-stream speed. Wafer's Neon Health work put a time-to-first-token target into the SLA under real call volume.

Why Single-Stream Benchmarks Mislead Under Concurrency

A single-stream benchmark measures one request's decode loop with the GPU almost to itself. That is a useful ceiling, but production rarely looks like that. The moment requests share the endpoint, the serving stack makes tradeoffs a single-stream number never exercises:

Batching. Servers group concurrent requests to raise GPU utilization. Larger batches raise aggregate throughput but add per-request latency. How aggressively a provider batches is one of the biggest levers on your experienced latency, and it is invisible in single-stream tests.
KV cache pressure. Every concurrent request holds KV cache memory. Long prompts and many simultaneous sessions can exhaust cache, forcing preemption, re-computation, or queueing. Two workloads with identical average throughput can behave completely differently because one has long contexts.
Scheduling and interference. Prefill of a new long prompt can stall decoding of in-flight requests. Scheduler policy and chunked-prefill settings determine whether your p99 collapses when traffic spikes.
Kernel and engine fit. Kernels optimal at batch size one can be suboptimal at batch 32. Engine configuration, quantization format, and parallelism layout all shift the concurrency sweet spot.

This is why provider rankings based on single-stream speed frequently invert under load. In Wafer's published testing on AMD GPUs, deep optimization produced an 11x single-stream speedup on one frontier model and a 7x per-request gain at concurrency on another, with aggregate throughput rising from 548 to 2,165 tok/s. Single-stream and concurrent performance respond to different tuning, and a benchmark covering only one cannot predict the other.

What Actually Determines Performance at Your Concurrency

Concurrent serving performance is the output of a stack of decisions, each of which a provider can get right or wrong:

Request mix. Prompt lengths, output lengths, cache-hit patterns, and arrival distribution. A 20k-in/1k-out workload with 60% cache hits stresses a very different part of the stack than a chat workload with 500-token prompts.
Batching and scheduling policy. Continuous batching parameters, chunked prefill, preemption rules, and fairness across streams.
Memory management. KV cache sizing, paging strategy, and quantized cache formats such as FP8, which trade a little precision headroom for far more concurrent capacity.
Decode strategy. Speculative decoding, quantization, and MoE expert sharding all interact with concurrency. A speculative-decode setup that wins single-stream can lose at high load if draft overhead stops amortizing.
Hardware fit. Memory capacity binds concurrency for large models; interconnect binds aggregate throughput. The right accelerator depends on which constraint binds first.

A Practical Evaluation Method for Unusual Concurrency

Here is the evaluation protocol we recommend, and the one Wafer applies when building a dedicated endpoint:

Step 1: Capture a workload replay. Record a representative sample of production traffic: prompt and output length distributions, cache-hit rate, and arrival times including your daily peak. If you are pre-launch, construct the distribution from your product design, and be honest about the tail.

Step 2: Define SLOs at concurrency, not per request. Specify time to first token, inter-token latency, and end-to-end latency at your target and peak concurrency, plus aggregate throughput requirements. A target like "p50 TTFT under 600 ms at 2.4 sustained requests per second" is evaluable. "Fast" is not.

Step 3: Run the replay against each provider. Same replay, same concurrency ramp, and measure the full latency distribution, not just the mean. Watch for the knee: the concurrency level where latency stops degrading gracefully. One provider may hold stable latency to 40 streams and collapse at 60; another degrades linearly from stream one. Which is better depends on where your peak sits.

Step 4: Ask what the provider tuned. Two providers on identical hardware can differ by multiples at concurrency because of scheduler settings, KV cache management, kernel selection, and parallelism layout. Ask each candidate what they changed for your workload. A provider who cannot answer has not tuned for your case.

Step 5: Prefer continual optimization over a one-time result. Your concurrency pattern will change as your product grows, so the question is not only "how fast is the endpoint today" but "who retunes it when traffic shifts." This is the core of Wafer's continual inference approach: learn the workload, find the bottlenecks, test configurations across batching, decoding, quantization, engines, kernels, and hardware, deploy the winner, then repeat as traffic, models, and hardware evolve. On Wafer's public optimization demo, that loop moved an endpoint from 230 tok/s after engine tuning, to 315 after kernel fusion, to 400 after re-tuning against real traffic. Those figures are illustrative, not guaranteed, but the direction is the point: concurrency performance is maintained by iteration, not fixed at deployment.

Step 6: Demand SLO-backed commitments. The strongest signal a provider can give is a latency target written into an SLA at your concurrency. In the Neon Health deployment for HIPAA-compliant voice agents, Wafer committed to a p50 time-to-first-token target in the SLA, and the client observed p50 TTFT drop from 800 ms to about 550 ms at roughly 25% higher peak load. That is workload-specific evaluation against real concurrent call volume.

Frequently Asked Questions

Why do provider benchmarks look great but my production latency is worse? Published benchmarks usually use a standard workload, often single-stream or a generic concurrency level, with prompt lengths that do not match yours. Batching, KV cache pressure, and scheduling behave differently on your traffic, so the ranking does not transfer. Re-test with a replay of your own requests.

What concurrency level should I test at? Test at your target steady-state concurrency, your peak, and a margin above it. The most important number is the knee where latency stops degrading gracefully. Know your headroom before that knee, because traffic spikes will find it.

Is aggregate throughput or per-request latency more important? They are the same tradeoff viewed from different sides, and you need both. Aggregate throughput determines cost and capacity planning; per-request latency determines user experience. A good evaluation reports the full latency distribution and aggregate throughput together at each concurrency level, so you see the tradeoff curve, not one point on it.

How often should concurrency performance be re-evaluated? Whenever traffic shape, model version, or hardware changes, and ideally on an ongoing schedule. A configuration optimal at launch drifts out of tune as your request mix evolves. Providers who re-profile and re-tune in production keep performance from drifting; a one-time benchmark locks in a snapshot that ages.

Conclusion

An unusual concurrency pattern is not a benchmarking nuisance. It is the most important property of your workload, and exactly the property published benchmarks ignore. Single-stream numbers measure a ceiling you will never operate at; user experience is determined by how batching, scheduling, KV cache management, kernels, and hardware behave under your request mix at your peak.

So evaluate providers the way you would evaluate any system under load: replay your real traffic, measure latency distributions and aggregate throughput together at your target and peak concurrency, find the knee, and ask what the provider actually tuned. Then favor the provider who treats optimization as ongoing work. Wafer builds dedicated endpoints around a customer's real traffic and keeps tuning them after launch, a continual inference approach now serving 15T+ tokens. If your concurrency pattern does not look like anyone's benchmark, that is precisely where workload-specific, continually optimized inference pays for itself.