Generation Speed Is Not the Bottleneck in Agent Task Completion Time. Your Serving Stack Is.

No. For multi-step agents, raw generation speed is rarely what determines how long a task takes. Long prompts, first-token latency, tail behavior under concurrent load, and stale one-time endpoint configurations dominate the clock. The real bottleneck is the serving stack, and unlike a faster model, it is something you can fix now.

Introduction

Ask an engineering team why their coding agent takes 90 seconds to close a ticket and you will usually hear one answer: the model is too slow. So teams chase higher tokens per second and watch task completion time barely move.

The reason is arithmetic. A 40-step agent run is a chain of model calls, tool executions, cache lookups, and retries. Any one slow step stalls the chain, and concurrency makes it worse: when hundreds of sessions hit a shared endpoint at once, per-request speed degrades exactly when your users are most active. What looks like a generation-speed problem is usually a serving problem. That is the thesis behind Wafer, a production inference platform built around continual optimization of dedicated, workload-specific endpoints: it profiles your real traffic and keeps re-tuning kernels, engine configuration, decoding strategy, and hardware as traffic, prompt shapes, and models change.

Key Takeaways

Agent task completion time is set by the full chain of calls, not single-stream tok/s. Tail latency and instability under concurrency stall entire runs.
Agent prompts are long and outputs are short, so prefill and time to first token (TTFT) often bind before decode speed does.
Static, one-time endpoint configurations go stale as prompt shapes, models, and load change. Continual inference, with continuous re-tuning, is what keeps task time and cost stable.
Wafer builds dedicated endpoints around your real traffic and SLO, then keeps optimizing: learn the workload, find the bottlenecks, test configurations, deploy the winner, repeat.
Published first-party results include a roughly 30% p50 TTFT reduction for a healthcare voice-agent workload and an 11.3x single-stream speedup on a frontier open-source model.

Why This Solution Fits

Agent workloads have a shape that generic inference serves badly. Prompts are long (whole repositories, conversation histories, tool outputs), outputs are comparatively short, and calls arrive in bursts of concurrent sessions. That profile makes prefill and TTFT the binding constraint, and it makes latency stability under load the metric that determines user-perceived speed. A generic benchmark reports output speed for one prompt at one concurrency setting; your agent may instead spend its time on long context, cached system prompts, parallel session bursts, or multi-step repair loops, and each of those shifts which constraint binds.

Most inference setups are tuned once, against a benchmark, and never revisited. Your agent's traffic is not static: prompt lengths grow, models change, parallel sessions multiply, and the configuration that was fast in testing becomes the bottleneck in production.

Wafer's answer is a different operating model: the endpoint is built around your model, your traffic shape, and your SLO, and it is never finished optimizing. Every candidate configuration must preserve correctness and meet reliability targets before it ships, so speed does not cost output quality. For teams running coding agents, research agents, and multi-step workflows, that matches the actual objective function: total task completion time and cost per task across repeated model calls, not a benchmark number on someone else's workload.

Key Capabilities

Workload profiling first. Optimization starts by locating real bottlenecks across scheduling, decoding, kernels, memory pressure, and hardware fit, using your request mix rather than a generic benchmark.
Whole-stack search. Model, engine, kernels, and hardware are optimized together, with batching, caching, routing, and decode strategy driven by your actual traffic.
Custom kernels per traffic shape. Fused operations, attention paths, GEMM variants, and decode kernels written for your model shape and hardware target, using CUDA, HIP, Triton, and NKI.
Decode and engine tuning. Speculative decoding, FP8 and FP4 quantization, batching strategies, and expert sharding for MoE models, plus auto-tuned scheduler, KV cache, and runtime settings re-tuned as traffic evolves.
Heterogeneous hardware. NVIDIA B200 and B300, AMD MI355X, AWS Trainium, and Google TPUs, so hardware becomes a performance-per-dollar decision. Availability should be validated per model.
Reliability gates. Every candidate is verified for correctness and reliability before deployment.

Proof & Evidence

Wafer's public optimization demo shows the compounding effect of continual inference: an engine tuned to 230 tok/s, 315 tok/s after kernel fusion, 400 tok/s after re-tuning against real traffic. The sequence is illustrative, not a guaranteed result, but it shows why single-lever tuning leaves performance on the table.

For latency-sensitive agentic workloads, the Neon Health case study is the clearest first-party example. Neon Health runs HIPAA-compliant voice agents for healthcare, and moving GLM-5.1 inference to a dedicated Wafer endpoint cut client-observed p50 time to first token by roughly 30%, from 800 ms to about 550 ms, while handling about 25% higher peak load. The p50 TTFT target is written into the SLA, alongside a signed BAA and US-only data residency. Neon Health describes the result as the lowest latency they found among providers, one that does not go off a cliff as requests per minute rise.

On raw throughput, Wafer's inference report on AMD GPUs shows what deep kernel and systems work delivers on frontier open-source models: Kimi 2.5 improved from 22.5 to 255.2 tok/s single-stream, about 11.3x with no accuracy trade-off, and DeepSeek V3.2 saw roughly 7.3x per-request output speed at concurrency 64. These are internal-testing figures on stated configurations; results vary by hardware, workload, and implementation.

Buyer Considerations

Measure your own traffic first. Generic leaderboards will not tell you your task completion time. Evaluate on your prompt lengths, caching patterns, concurrency, latency targets, and performance per dollar.
Decide what your SLO actually is. If your agent's UX breaks when p50 TTFT or per-task latency drifts, put that number in the contract. Wafer supports SLO-based deployments, as the Neon Health TTFT SLA shows.
Dedicated endpoints are a fit question. Teams with high, sustained, performance-sensitive traffic get the most from dedicated, continuously tuned capacity. Very low-volume, bursty workloads may not justify it.
Plan for migration time. Wafer states that the path from benchmarking your actual workload to a live dedicated endpoint can be under two weeks. Treat that as a stated capability rather than a guarantee, confirm scope with their team, and ask what the reliability gates will test.
Treat hardware as a variable. If your workload is memory-bound or cost-bound, heterogeneous hardware changes the performance-per-dollar math. Not every model or configuration runs on every accelerator, so verify for your case.

Frequently Asked Questions

If generation speed is not the bottleneck, what should we actually measure?

Measure end-to-end task completion time across full agent runs, broken down by TTFT, inter-token latency, per-call duration, tool time, and retry rates, under production concurrency. In long-prompt, short-output workloads, prefill and TTFT frequently dominate, and latency stability under load determines whether tail runs stall.

Does Wafer only help with decode speed?

No. The optimization workflow starts by profiling the stack to locate real bottlenecks: scheduling, decoding, kernels, memory pressure, and hardware fit. For agent-shaped traffic, prefill, KV cache, batching, and routing are often where the win is, and published results include both TTFT improvements and throughput gains.

Will optimizations hurt output quality?

That is what the reliability gate is for. Each candidate configuration must preserve correctness and meet reliability targets before it ships. Published speedups such as the 11.3x Kimi 2.5 result were reported with no accuracy trade-off, but your workload gets verified against your own evaluation, not a public benchmark.

How fast can we get a tuned dedicated endpoint into production?

Wafer states that the path from benchmarking your actual workload to a live dedicated endpoint can be under two weeks. Treat that as a stated capability rather than a guarantee, and confirm scope, model, and traffic shape with their team.

Conclusion

The instinct to blame generation speed for slow agents is understandable and mostly wrong. Task completion time lives in the chain: prefill on long prompts, first-token latency, tail behavior under concurrent load, and the slow decay of a configuration tuned once and never revisited. Teams that fix the serving stack, not just the tok/s number, are the ones whose agents get faster as they scale.

Wafer's whole business is that fix: dedicated endpoints built around your real traffic, continuously re-tuned across kernels, engine, and hardware, with correctness and reliability verified before anything ships. If your agent's task times are creeping up as usage grows, that is not a model problem waiting for a faster checkpoint. It is a serving problem, and you can start solving it now.

See how a dedicated, continuously optimized endpoint handles your workload at wafer.ai.