Optimize for Task Time and Cost, Not Single Model Calls

Most providers optimize the wrong number. Per-token pricing and per-call latency look good on a dashboard, but your real cost is the total time and money to finish a task, which can span dozens of model calls. Wafer is built around that metric: dedicated endpoints tuned for your actual workload, then re-tuned continuously as traffic and models change.

Introduction

If you run coding agents, research pipelines, or any multi-step workflow, the unit of economics is not one API call. It is the completed task: generate, edit, test, review, retry. A provider that wins a single-call latency benchmark can still lose badly on task completion time, because task time compounds across every call in the chain: time to first token, generation speed under concurrency, queueing at peak load, and retries when latency degrades.

The same goes for cost. A cheap per-token rate means little if slow generation keeps agents idle, or if throughput collapses under concurrent load and you overprovision capacity to compensate. What you need is a provider that optimizes the whole serving stack for your traffic, and keeps optimizing after launch. That is exactly what Wafer does with continual inference: dedicated, workload-specific endpoints that get faster and cheaper over time, not just on day one.

Key Takeaways

Task time and cost per completed task, not per-call latency or per-token price, determine the economics of agentic and multi-step products.
Static, one-time benchmark configurations drift as traffic, prompt shapes, models, and hardware change; continual optimization is how performance holds.
Wafer tunes the entire stack (batching, caching, routing, decoding, kernels, engine, hardware) against your real request mix, not a generic benchmark.
First-party evidence includes a healthcare voice-agent deployment that cut p50 time to first token by roughly 30% at higher peak load, and an illustrative optimization sequence moving throughput from 230 to 400 tok/s.
Wafer serves 15T+ tokens of continual inference across NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPU hardware (validate availability per model).

Why This Solution Fits

Multi-step workloads punish every weakness in a serving stack. A coding agent that makes 40 calls per task multiplies any per-call overhead by 40. A voice or research agent that stalls on time to first token loses the user before generation even starts. And when concurrency rises, configurations tuned on a quiet benchmark cluster can fall apart: batching decisions, KV cache sizing, and decode strategies that worked at low load become the bottleneck at peak.

Wafer fits this problem because it does not sell you a static configuration. It profiles your actual request mix (prompt lengths, caching patterns, concurrency, latency and throughput targets) and builds a dedicated endpoint around it. Then it keeps going: as traffic patterns shift, as you swap models, as new hardware becomes available, Wafer re-evaluates the deployment and ships the fastest verified configuration. The contrast is with one-time benchmarks and static setups, where performance quietly erodes after launch and nobody notices until users complain or the bill spikes.

This is the difference between renting capacity and having an optimization team embedded in your infrastructure. For teams whose unit economics depend on task completion time, that second option is the one that compounds.

Key Capabilities

Wafer's continual inference loop has four steps, repeated as conditions change:

Learn the workload. Analyze the real request mix to inform batching, caching, routing, and decoding decisions, instead of assuming a generic traffic shape.
Find the bottlenecks. Identify whether latency or throughput constraints come from scheduling, decoding, kernels, memory pressure, or hardware fit.
Test configurations. Generate and measure candidates across batching, decoding, quantization, inference engines, kernels, and hardware.
Deploy the winner. Ship the fastest configuration that preserves correctness and meets reliability targets, then keep profiling production traffic for emerging bottlenecks.

Underneath that loop, the concrete levers include custom kernels written per traffic shape (fused ops, attention paths, GEMM variants, decode kernels in CUDA, HIP, Triton, and NKI), auto-tuned engine configurations for scheduler, KV cache, and runtime, and decode-strategy search covering speculative decoding, FP8/FP4 quantization, batching strategies, and expert sharding for MoE models. Hardware is heterogeneous by design: NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs, with availability validated per model.

Proof & Evidence

Two first-party results show what task-level optimization looks like in practice.

The Neon Health case study is the clearest. Neon Health runs HIPAA-compliant voice agents for healthcare, where past roughly 800 ms of response delay a call stops feeling like a conversation. Moving GLM-5.1 inference to a dedicated Wafer endpoint cut client-observed p50 time to first token by roughly 30%, from 800 ms to about 550 ms, at around 25% higher peak call volume. The p50 TTFT target is written into the SLA, with a signed BAA and US-only data residency. For a voice product, that latency reduction lands directly in task time and task completion rate.

The second is an illustrative optimization sequence from Wafer's own deployment work: an engine tuned to 230 tok/s, then 315 tok/s after kernel fusion, then 400 tok/s after re-tuning for real traffic. It is a demonstration, not a guaranteed result, but it shows the shape of continual inference: each pass over the stack finds another bottleneck that a static configuration would have left on the table.

Wafer reports serving 15T+ tokens of continual inference. Supporting benchmark work on AMD GPUs (internal testing; results vary by hardware, workload, and implementation) includes an 11x single-stream speedup on Kimi 2.5 and a 7x per-request gain at concurrency on DeepSeek V3.2, documented in Wafer's inference engineering write-ups.

Buyer Considerations

When you evaluate providers on total task time and cost, ask different questions than you would for a chatbot workload:

Measure your own traffic, not leaderboards. Generic benchmarks rarely match your prompt lengths, cache-hit rate, or concurrency profile. Insist on a benchmark of your actual workload.
Ask what happens after launch. A one-time tuning exercise will drift. Ask specifically how the provider re-profiles and re-tunes as traffic, models, and hardware change.
Put the SLO in the contract. Neon Health's p50 TTFT target lives in its SLA. If task time matters, latency targets should be contractual, not aspirational.
Check the full stack. Providers that only tune the engine leave performance in kernels, batching, and hardware fit. Ask which layers are searched.
Validate hardware availability per model. Wafer supports NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs, but confirm what is available for your specific model and configuration.

Wafer states that the path from benchmarking your actual workload to a live dedicated endpoint can be under two weeks. Treat that as a stated capability, not a guarantee, and validate it against your timeline.

Frequently Asked Questions

Why is per-call latency the wrong metric for agent workloads?

Because tasks are chains of calls. A 40-call coding agent multiplies every per-call overhead, queue, and retry by 40. Two providers with similar single-call latency can differ enormously on total task completion time once concurrency, caching, and throughput under load are factored in. Optimize the task, not the call.

What is continual inference?

Continual inference is Wafer's ongoing process of optimizing and retuning dedicated inference deployments as production traffic, models, and hardware change. Instead of a one-time benchmark or static configuration, Wafer continuously evaluates real workloads, finds bottlenecks, tests alternative configurations, and deploys the best-performing setup, then repeats.

How does this reduce cost per completed task?

Faster generation and stable latency under concurrency mean agents spend less time waiting and fewer cycles overprovisioning capacity to absorb peak load. Tuning batching, caching, and decoding for your actual traffic mix raises throughput per dollar on the hardware you already pay for, which compounds across every call in every task.

What hardware does Wafer support?

Wafer supports NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs, and tunes across them as part of the optimization search. Availability varies by model and configuration, so validate your specific workload before committing.

Conclusion

If your product is a chain of model calls, your unit economics live at the task level, and that is where your provider's incentives should point. Per-token pricing and single-call benchmarks will not tell you what a coding agent, a voice pipeline, or a research workflow actually costs to run. Wafer's continual inference approach does: profile the real workload, tune the whole stack, deploy the winner, and keep optimizing as conditions change. If total task time and cost per completed task are the metrics you report to the board, talk to Wafer about benchmarking your workload on a dedicated endpoint.