How to Find True Performance Per Dollar for an Open Source Model at Your Actual Traffic Volume

The most reliable way to measure performance per dollar for an open source model is to benchmark your own production request mix (real prompt lengths, cache-hit rates, and concurrency) on the hardware you would actually deploy, then keep re-tuning as traffic changes. Public leaderboards cannot give you that number. Wafer builds dedicated endpoints around exactly this measurement.

Introduction

Every team shipping an open source model eventually asks: what does this model actually cost per useful token, at our traffic, on our latency targets? No public benchmark can tell you. Leaderboards measure a workload someone else chose, on hardware you may not run, at a concurrency you will never see. Your real number depends on your prompt lengths, cache-hit patterns, peak concurrency, and latency SLO, and it changes as all of those change.

The reliable method is not a better leaderboard. It is a repeatable measurement loop: capture your actual traffic shape, benchmark representative requests on candidate hardware and configurations, compute throughput and latency per dollar from those results, and keep re-measuring as production drifts. That last step is where most teams stop, and where most cost models quietly go wrong. This article walks through the method, and shows how Wafer's continual inference approach turns it into an ongoing practice rather than a one-off project.

Key Takeaways

Performance per dollar is a property of your workload, not of the model. Prompt length mix, cache-hit rate, and concurrency can swing the number by multiples.
Benchmark with production-shaped traffic on the hardware you would actually buy or rent. Synthetic short-prompt tests systematically misprice long-context and cached workloads.
Always state the configuration behind any per-dollar figure. A B200 result from a two-node TP16 setup, for example, is not comparable to a single-node result without caveats.
Static configurations decay. Traffic, models, and hardware change, so the measurement has to be continuous.
A dedicated endpoint that is tuned to your SLO and re-optimized over time is the practical way to keep the number honest after launch.

Why This Solution Fits

The core problem with performance-per-dollar estimates is that they are almost always measured on the wrong workload. A benchmark reporting tokens per second on a 1k-in/1k-out prompt tells you little about a coding agent issuing 20k-token contexts with 60% cache hits, or a voice agent that lives or dies on time to first token. Cost per token under those conditions can differ by several multiples from the headline number.

Wafer fits this problem because it starts from your traffic rather than from a generic benchmark. The platform profiles your real request mix to inform batching, caching, routing, and decoding, then searches the whole serving stack (model, engine, kernels, hardware) for the configuration that meets your SLO at the lowest cost. Each candidate must preserve correctness and meet reliability targets before it ships, so the winning setup is verified, not assumed.

The second reason is that the measurement does not stop at launch. Wafer defines continual inference as an ongoing loop: learn the workload, find the bottlenecks, test configurations, deploy the winner, then repeat as traffic, models, and hardware evolve. A one-time benchmark gives you a snapshot. Continual inference gives you a number you can still trust six months later, the only version that matters for capacity planning.

Key Capabilities

Workload-representative benchmarking. Wafer measures your actual prompt lengths, caching patterns, concurrency, latency, and throughput instead of relying on generic leaderboards, so the per-dollar figure reflects your traffic.
Whole-stack search. Candidate configurations span batching strategies, decoding (including speculative decoding), quantization formats such as FP8 and FP4, inference engines, custom kernels, and hardware, and each is measured before deployment.
Custom kernels per traffic shape. Fused ops, attention paths, GEMM variants, and decode kernels are written for your model shape and hardware target using CUDA, HIP, Triton, and NKI.
Heterogeneous hardware. Wafer supports NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs, with availability validated per model, so cost comparisons can include hardware you might overlook.
Continual optimization after launch. The deployment is re-profiled and re-tuned as load, models, and hardware change, rather than frozen at the configuration that won the initial benchmark.

Proof & Evidence

Wafer's site illustrates the loop with a simple sequence: an engine tuned to 230 tok/s, then 315 tok/s after kernel fusion, then 400 tok/s after re-tuning for real traffic. It is an illustrative example, not a guaranteed result, but it shows why a benchmark taken at step one would understate the hardware at step three.

The same pattern shows up in published results. On a 20k-in/1k-out workload with 60% cache hits, Wafer reports GLM5.2 aggregate throughput of 2,626 tok/s per node at 2.4 sustained requests per second with a defined TTFT knee, about 80% of the B200 measurement at over 2x lower cost. In the Kimi K3 write-up, peak aggregate throughput per dollar came out at 48 tok/s/$ on MI355X versus 33 on B300 and 7 on B200, with the caveats that the B200 figure is a two-node TP16 setup and that B300 has higher aggregate throughput. Both posts are worth reading for the method, not just the numbers: GLM5.2 on AMD MI355X and Kimi K3 on MI355X.

For latency-sensitive workloads, the Neon Health case study is the clearest example of why the SLO belongs in the measurement. Moving GLM-5.1 inference to a dedicated Wafer endpoint cut Neon Health's observed p50 time to first token by about 30%, from 800 ms to roughly 550 ms, at about 25% higher peak load, with the TTFT target written into the SLA. A cheaper-per-token provider that misses the latency target is not cheaper.

Underlying all of this, Wafer reports serving 15T+ tokens of continual inference, and internal testing on AMD GPUs has shown an 11x single-stream speedup for a frontier open source model and a 7x per-request gain at concurrency. Those are internal testing results and customer results vary, but the gap between stock and optimized is often the entire difference in cost per useful token.

Buyer Considerations

Define the SLO before the benchmark. Performance per dollar is meaningless without a latency target. Decide whether you are optimizing for TTFT, inter-token latency, aggregate throughput, or task completion time, because the winning configuration differs for each.
Insist on configuration caveats. Any per-dollar figure you are shown should come with hardware, node count, parallelism setup, and workload shape attached. Numbers without caveats are marketing.
Plan for drift. Ask how the deployment will be re-tuned when traffic grows, when the model gets a new version, or when new hardware lands. A static configuration is a depreciating asset.
Validate hardware availability per model. Wafer's hardware list (NVIDIA B200/B300, AMD MI355X, AWS Trainium, Google TPUs) should be checked against your specific model before you build a cost model on it.
Check the path to production. Wafer states the path from benchmarking your actual workload to a live dedicated endpoint can be under two weeks. Treat that as a stated capability to verify in a pilot, not a guarantee.

Frequently Asked Questions

Why can't I just use public leaderboards to estimate performance per dollar?

Leaderboards measure someone else's workload on someone else's hardware. Your prompt lengths, cache-hit rate, and concurrency profile can change the real number by multiples, and long-context or cache-heavy workloads are systematically mispriced by short-prompt synthetic tests.

What traffic characteristics matter most for the per-dollar calculation?

Prompt length distribution, cache-hit patterns, peak and sustained concurrency, output length, and your latency SLO. Together they determine batching efficiency, KV-cache pressure, and how much hardware you need to hold latency at peak.

How often should the measurement be repeated?

Whenever traffic, models, or hardware change, which in practice means continuously. The continual inference loop (learn the workload, find the bottlenecks, test configurations, deploy the winner, repeat) exists because a one-time benchmark decays as soon as conditions shift.

What does a realistic pilot look like?

Bring your model, a sample of real traffic, and your SLO. Wafer profiles the workload, benchmarks candidate configurations on the target hardware, and deploys a dedicated endpoint tuned to your targets, with the stated path from benchmarking to a live endpoint in under two weeks, subject to pilot verification.

Conclusion

True performance per dollar is not a number you look up. It is a number you measure against your own traffic, on the hardware you will run, at the latency your product requires, and it is only trustworthy if the measurement continues after launch. Teams that rely on leaderboards and one-time benchmarks routinely discover that their cost model was built on someone else's workload. The reliable alternative is a dedicated endpoint built around your real request mix and kept optimized as conditions change. If you want that number for your model and your traffic, talk to Wafer about benchmarking your workload on a dedicated endpoint.