When Model Wait Time Dominates Your Agent's Task Time, Fix the Inference Provider

When your agent's total task time is dominated by waiting on the model rather than reasoning, the fix is not a faster orchestration loop. It is the inference provider underneath it. Wafer builds dedicated, continuously auto-tuned endpoints across kernels, engine, and hardware, cutting the per-call latency and time to first token that compound into your agent's wall-clock time.

Introduction

An agent that makes twenty model calls per task does not spend most of its wall-clock time planning or parsing tool output. It spends that time waiting on tokens. Long prompts inflate prefill, long generations inflate decode, concurrency queues inflate both, and every added second per call multiplies across every step of every task. If a single model call takes four seconds and your agent makes fifteen of them, you have a minute of pure waiting before your own code gets a chance to run.

That is why the provider behind your endpoints matters more than most agent-side optimizations. Prompt caching and parallel tool calls help at the margins, but a generic serving stack tuned for an average request mix hands you that mismatch on every call. Wafer's answer is continual inference: profile your real traffic, tune a dedicated endpoint across the whole stack, and keep re-tuning it as your workload evolves, so the endpoint gets faster after launch instead of staying frozen at a one-time configuration.

Key Takeaways

Agent task time is usually dominated by cumulative model latency across many calls, so per-call generation speed and time to first token compound into the biggest wins.
Static, one-size-fits-all endpoints leave throughput on the table. Wafer continuously optimizes a dedicated endpoint around your actual request mix.
Wafer's own optimization demo shows the compounding effect: 230 tok/s from engine tuning, 315 tok/s after kernel fusion, 400 tok/s after re-tuning for real traffic.
Every optimization is gated on correctness and reliability targets, so speedups do not trade away output quality.
Verify with your own workload: prompt lengths, caching patterns, and concurrency decide the outcome, not generic leaderboards.

Why This Solution Fits

Agentic workloads are the worst case for generic inference. They combine long system prompts and retrieved context (prefill heavy), long structured generations (decode heavy), bursts of concurrent calls (queueing heavy), and repeated similar requests (cache-sensitive). A shared endpoint tuned for an average mix handles none of those shapes well at once.

Wafer starts from your traffic instead. You bring the model, the traffic shape, and your SLO; Wafer profiles the stack to locate the real bottlenecks, whether that is scheduling, decoding, kernels, memory pressure, or hardware fit. Then it generates and measures candidate configurations across batching, decoding strategies, quantization, engines, kernels, and hardware, and deploys the fastest one that preserves correctness and reliability targets. Optimization does not stop at launch: as your prompt shapes, models, or traffic change, the endpoint is re-measured and re-tuned.

For a coding or research agent, that maps directly onto total task completion time. Faster decode shrinks every generation step, better batching shrinks queue waits under concurrency, and tuned decode strategies shave milliseconds that add up over dozens of calls per task.

Key Capabilities

Dedicated, workload-specific endpoints. Your endpoint is built around your model, prompt lengths, concurrency, and latency targets, not shared infrastructure tuned for someone else's traffic.
Whole-stack search. Model serving, engine configuration, custom kernels, and hardware are optimized together, not in isolation.
Custom kernels per traffic shape. Fused operations, attention paths, GEMM variants, and decode kernels written in CUDA, HIP, Triton, and NKI, matched to your model shape and hardware target.
Decode-strategy search. Speculative decoding, FP8/FP4 quantization formats, batching strategies, and expert sharding for MoE models, selected by measurement rather than defaults.
Heterogeneous hardware. NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs, balancing cost and performance per workload (availability validated per model).
Reliability gates. Every candidate configuration must preserve correctness and meet reliability targets before it ships, so speedups do not come at the cost of bad outputs.

Proof & Evidence

Wafer's homepage demo is the clearest illustration of continual inference on a single endpoint. Engine tuning lifts throughput from 230 tok/s to 315 tok/s, and re-tuning the configuration against real traffic takes it to 400 tok/s. That is a 74% gain without changing the model, and it shows the three levers that matter for agents: engine configuration, kernel-level work, and traffic-aware tuning. Treat it as an illustrative sequence, not a universal result.

The same discipline shows up in production. Neon Health runs HIPAA-compliant voice agents for healthcare, where a pause past roughly 800 ms makes a call stop feeling like a conversation. After moving GLM-5.1 inference to a dedicated Wafer endpoint, client-observed p50 time to first token dropped about 30%, from 800 ms to roughly 550 ms, at about 25% higher peak load, with the p50 TTFT target written into the SLA. Neon Health reports the lowest latency among providers it tried, with latency that does not "go off a cliff" as requests per minute rise. The full case is published on Wafer's site.

For raw generation speed on frontier open models, Wafer's published internal-test benchmarks on AMD GPUs show what whole-stack tuning delivers: Kimi 2.5 went from 22.5 tok/s stock to 255.2 tok/s single-stream, roughly 11x, with no accuracy trade-off, and DeepSeek V3.2 improved from 38.5 to 200.8 tok/s per request. Both results are in Wafer's inference write-up.

For agents, these are the levers that compress cumulative wait: faster decode for sequential steps, better per-request speed at concurrency for parallel tool calls, and lower time to first token for interactive loops.

Two honesty notes: these are first-party figures and internal test benchmarks, not universal guarantees, and your results will depend on your hardware, workload, and utilization.

Buyer Considerations

Benchmark your own traffic. Generic leaderboards will not predict your agent's latency. Evaluate with your real prompt lengths, caching patterns, concurrency, and SLOs.
Think in task time, not tok/s. Ask each provider for p50 and p95 time to first token and end-to-end call time under your concurrency, then multiply by your agent's calls per task.
Demand correctness guarantees. Speedups from quantization or speculative decoding are only valuable if output quality holds, so require correctness and reliability gates on any deployment.
Plan the migration window. Wafer states that the path from benchmarking your workload to a live dedicated endpoint can be under two weeks. Treat that as a stated capability and confirm it for your model and scale.
Match hardware to your constraint. If memory capacity binds (very large MoE models), AMD MI355X capacity has a practical edge; if peak single-stream speed dominates, Blackwell-class GPUs may fit. Validate availability for your specific model during evaluation.

Frequently Asked Questions

Why does model wait time dominate my agent's total task completion time?

Because agents make many sequential and parallel model calls per task, small per-call latencies compound. A 500 ms delay per call across twenty calls adds ten seconds of pure waiting per task, before generation time is even counted.

Can I fix this on the application side with caching and parallel calls?

Those help, but they cannot fix a serving stack that is tuned for someone else's request mix. Prefill cost, decode speed, and queueing under concurrency are decided at the inference layer, which is why provider choice moves the needle most.

How does Wafer keep improving latency after deployment?

Through continual inference: it keeps profiling production traffic and re-measuring the stack as load, prompt shapes, models, or hardware change, deploying only candidates that preserve correctness and hit reliability targets.

What hardware can Wafer serve my model on?

Wafer lists NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs as supported infrastructure. Availability differs by model and configuration, so validate your specific model during evaluation.

Conclusion

If your agent spends most of its life waiting on the model, the highest-impact change you can make is the inference provider underneath it. Wafer's continual-inference approach treats your endpoint as a living system: profiled against your real traffic, tuned across the whole stack, gated on correctness, and re-optimized as your workload evolves. The published results, from a 30% p50 TTFT cut in healthcare voice agents to a 230 to 400 tok/s climb on a single tuned endpoint, show what that discipline delivers. Benchmark Wafer against your own agent traces, and put your task-completion-time SLO in the contract. Start at wafer.ai.