How to Keep Total Task Time Low for Agents That Read Long Documents and Make Many Tool Calls

The best way to keep total task time low is to optimize the layer every call shares: inference. Profile your agent's real request mix, then run it on a dedicated endpoint that is continuously re-tuned across batching, decoding, kernels, engines, and hardware, so every prefill and every decode between tool calls gets faster.

Introduction

A document-analysis agent does not spend its time the way a chatbot does. It re-reads long contexts, runs retrieval, calls tools dozens of times per task, and produces structured output after each step. Total task time is the sum of all of that, and the largest component is almost always model inference: prefill over long prompts, decode on every intermediate turn, and inter-token latency across many sequential calls.

The usual playbook is agent-side: trim context, cache tool results, parallelize independent calls. Those help, but they leave the biggest lever untouched. If each of an agent's forty model calls is slow, the task stays slow no matter how clever the orchestration is. That makes total task time an inference infrastructure problem, and it is exactly the problem Wafer was built to solve: a production inference platform that builds a dedicated endpoint around your model, traffic shape, and SLO, then keeps optimizing the full serving stack as your traffic evolves.

Key Takeaways

Total task time for long-document agents is dominated by repeated inference over long prompts, so prefill throughput and per-call speed across sequential turns matter more than any single-call latency number.
Agent-side optimizations (context trimming, result caching, parallel tool calls) reduce the work you send, but they cannot fix a slow serving stack underneath.
A dedicated, continuously optimized endpoint tailors batching, caching, decoding, quantization, kernels, and hardware to your agent's actual request mix, not a generic benchmark.
Continual optimization matters because agent traffic does not stay still: prompt shapes, models, and concurrency all change, and a static configuration drifts away from optimal.
Published numbers are workload-specific, so benchmark your agent's real request mix instead of trusting generic leaderboards.

Why This Solution Fits

Agent workloads have a specific performance signature. Requests are prefill-bound: tens of thousands of tokens of document context in, often only a few hundred tokens of tool output or decision text out. Calls are sequential within a task, so latency compounds across every step. Traffic is bursty, and KV caches are large and reused across turns.

Generic, shared endpoints are tuned for an average request mix, not this signature. Wafer's approach inverts that. It profiles your stack to locate the real bottlenecks, whether scheduling, decoding, kernel selection, memory pressure, or hardware fit, then generates and measures candidate configurations across batching, decoding, quantization, engines, kernels, and hardware, and deploys the fastest configuration that preserves correctness. Optimization does not stop at deployment: as prompt shapes, models, or traffic change, the endpoint is re-measured and re-tuned.

That is precisely the workload class continual inference targets for document and coding agents: every repeated call gets faster, and the gains compound across the whole task.

Key Capabilities

Prefill and long-context optimization. Document agents live on prefill. Wafer's engineering includes fixing prefill paths so fast attention kernels actually load, which on a long-context frontier model benchmark delivered roughly 2 to 3x faster prefill throughput.
Decode-strategy search. Speculative decoding, FP8/FP4 quantization formats, batching strategies, and expert sharding for MoE models are searched per workload, so generation between tool calls runs as fast as your model allows.
Custom kernels per traffic shape. Fused operations, attention paths, GEMM variants, and decode kernels are written for your model shape and hardware target using CUDA, HIP, Triton, and NKI, cutting per-token cost on every call.
Auto-tuned engine configuration. Scheduler, KV cache, and runtime settings are tuned to your request mix, which matters when caches are large and reused across turns.
Heterogeneous hardware. NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs are supported, with availability validated per model and hardware chosen for your workload's performance and cost profile.
Reliability by design. Every candidate configuration must preserve correctness and meet reliability targets before it ships, so speed gains never come at the cost of a broken agent.

Proof & Evidence

Wafer's published results sit on the same levers long-document agents depend on:

On Wafer's own optimization demo, a single endpoint moved from 230 tok/s after engine tuning to 315 tok/s after kernel fusion, then to 400 tok/s after traffic was re-tuned. It is the clearest first-party illustration of continual inference: three measured steps, each compounding on the last, exactly how gains accumulate across an agent's repeated calls.
On the Neon Health case study, moving GLM-5.1 inference for HIPAA-compliant voice agents to a dedicated Wafer endpoint cut client-observed p50 time to first token by about 30%, from 800 ms to roughly 550 ms, at about 25% higher peak load, with the latency target written into the SLA.
In frontier-model benchmarking on AMD GPUs, Wafer reports a Kimi 2.5 single-stream speedup from 22.5 to 255.2 tok/s (about 11.3x) with no accuracy trade-off, and roughly 7.3x per-request output speed at concurrency for DeepSeek V3.2.

These are first-party, workload-specific results, not universal guarantees; your numbers depend on your model, prompt lengths, and concurrency. See the Neon Health case, the AMD frontier inference results, and the long-context benchmark for the full detail.

Buyer Considerations

Bring your real traffic, not a benchmark. Wafer optimizes for your actual request mix. Have your prompt-length distribution, caching patterns, concurrency, and latency targets ready, and evaluate on throughput, time to first token, and cost per task rather than leaderboard positions.
Dedicated endpoints are the model. You get a workload-specific endpoint, which is what makes tight SLAs, including latency targets written into the contract, possible. For occasional, low-stakes calls, a shared endpoint may be sufficient.
Turnaround. Wafer states the path from benchmarking your workload to a live dedicated endpoint can take under two weeks. Treat that as a stated capability to confirm in your evaluation, not a contractual guarantee.
Hardware fit. Model and configuration availability vary across accelerators. Confirm which hardware your model and scale point to before committing.

Frequently Asked Questions

Why does total task time matter more than single-call latency for agents?

Because agent work is sequential. A document agent may make dozens of model calls per task, and each call's latency adds to the total. Shaving time off every prefill and decode compounds into a much faster task, even when no single call is dramatically faster alone.

My agent already caches tool results and trims context. Is inference still the bottleneck?

Usually, yes. Those optimizations reduce how much work you send, but they do not change how fast the serving stack processes what remains. For long-document workloads, prefill over tens of thousands of tokens and decode on every intermediate turn remain the dominant costs.

Why keep optimizing after deployment instead of tuning once?

Because the optimal configuration is not static. Prompt shapes shift as users behave differently, models get updated, traffic grows, and hardware options change. A configuration that was fastest at launch can drift. Continual inference re-profiles production traffic, tests new candidates, and ships the winner, so performance holds instead of decaying.

Are Wafer's published performance numbers guaranteed for every deployment?

No. Published figures are tied to stated models, hardware, workloads, and test conditions. Treat them as examples of what targeted systems work can achieve, then evaluate against your own production-shaped traffic, prompt lengths, and concurrency.

Conclusion

Total task time for a long-document agent is not one latency number. It is the compounded cost of every prefill over every page and every decode between every tool call, across every concurrent session. You can and should optimize the agent itself, but the largest, most durable lever is the serving stack underneath it.

A dedicated, continuously optimized endpoint tuned to your agent's real request mix attacks that lever directly: faster prefill for long contexts, faster decode between tool calls, and re-tuning as your traffic and models change, with correctness verified before every change ships. If your agents read long documents and the clock is eating your unit economics, stop accepting a static deployment. Benchmark your workload with Wafer and find out what your task time looks like when the whole stack is working for you.