How to Speed Up a Research Agent That Makes Dozens of Model Calls

A research agent's runtime is the sum of its model calls, so dozens of sequential steps turn small per-call delays into minutes. Profile where each call spends time: time to first token on long, growing contexts, decode speed, cache hit rate, and queueing. Then move the workload to Wafer, which builds dedicated endpoints tuned continuously to your traffic.

Introduction

When an agent runs through dozens of steps, every inefficiency in a single call gets multiplied. A 400 ms wait before the first token is barely noticeable once; repeated forty times it is 16 seconds of pure waiting, before slow decode or queueing add more. Most teams respond by trimming prompts or cutting steps. That helps, but it ignores the other half of the equation: how fast each call is served.

The productive path is to measure the anatomy of a single call, find which phase dominates, and then fix the serving stack rather than only the prompt. That is the problem Wafer was built for: dedicated endpoints tuned to a specific workload instead of a generic, one-size-fits-all configuration, and re-tuned as that workload changes.

Key Takeaways

Agent latency compounds. Dozens of sequential calls mean time to first token, inter-token latency, and queueing multiply into total task time.
Watch four numbers per call: time to first token, output tokens per second, prompt-cache hit rate, and tail latency as concurrent sessions rise.
Separate retrieval and tool latency from model-serving latency so you fix the real bottleneck instead of the convenient one.
A dedicated endpoint tuned to your traffic (batching, caching, kernels, decode strategy, hardware) is the biggest serving-side lever, and continual optimization keeps it fast as your agent evolves.

Why This Solution Fits

Multi-step research workflows are one of the workload classes Wafer prioritizes explicitly, where total task time and cost matter across repeated model calls. The fit comes from four parts of the approach.

First, the endpoint is tuned to your real traffic, not a generic benchmark. A research agent has a recognizable shape: long contexts that grow step by step, repeated system and tool prefixes, parallel sessions, and long generations before each tool call. Wafer profiles that mix and drives batching, caching, routing, and decode around it.

Second, the whole stack is searched together: model, engine, kernels, and hardware optimized as a system, not defaults that were never chosen for your agent.

Third, changes are reliable by design: every candidate must preserve correctness and meet reliability targets before it ships, which matters when quality depends on consistent serving across dozens of calls.

Fourth, optimization never stops. Wafer follows a continual inference loop: learn the workload, find the bottlenecks, test configurations, deploy the winner, then repeat as traffic, models, or hardware change. That is exactly how agent workloads evolve, and it is the opposite of a one-time benchmark or a static configuration that drifts out of date.

Key Capabilities

Full-stack profiling. Bottlenecks are located in scheduling, decoding, kernels, memory pressure, and hardware fit, so effort goes where the time goes.
Candidate search across the stack. Configurations are generated and measured across batching, decoding, quantization (FP8 and FP4 formats), engines, kernels, and hardware.
Custom kernels per traffic shape. Fused operations, attention paths, GEMM variants, and decode kernels are written for your model shape and hardware target in CUDA, HIP, Triton, and NKI.
Auto-tuned engine configs and decode strategies. Scheduler, KV cache, and runtime settings are tuned per workload, with a decode search covering speculative decoding, FP8/FP4 formats, batching, and MoE expert sharding.
Heterogeneous hardware. Dedicated endpoints run on NVIDIA B200 and B300, AMD MI355X, AWS Trainium, and Google TPUs, with availability validated per model, so hardware fit is part of the search.
Continuous optimization with gates. After launch the endpoint keeps being profiled and re-tuned, with every candidate clearing correctness and reliability targets first.

Proof & Evidence

The closest published case is a voice-agent deployment, but the mechanism is the same one that slows research agents. Moving GLM-5.1 for Neon Health's HIPAA-compliant healthcare voice agents to a dedicated Wafer endpoint cut client-observed p50 time to first token by about 30%, from 800 ms to roughly 550 ms, at about 25% higher peak load, with the latency target written into the SLA. An agent that waits on a first token at every step compounds the same effect across its run, as described in the Neon Health case study.

On raw generation speed, Wafer's internal testing on AMD GPUs reports Kimi 2.5 single-stream output improving from 22.5 to 255.2 tok/s, about an 11.3x gain, through deep kernel optimization and a customized inference framework with no accuracy trade-off. Both the result and the methods behind it are in the AMD inference write-up.

For cost per task, the GLM5.2 benchmark post reports 2,626 tok/s per node of aggregate throughput on a 20k-token-input, 1k-token-output workload with 60% cache hits, about 80% of the B200 measurement for over 2x lower cost. The B200 reference there is a two-node TP16 setup, and B300 delivers higher aggregate throughput at a proportionally higher price. The Kimi K3 post adds a memory-bound frontier model: on a 1,024-token-input, 400-token-output benchmark, an eight-GPU MI355X node reached 952 tok/s of aggregate throughput and 118 tok/s single-stream, against 498 tok/s total for the two-node TP16 B200 setup, while B300 won about 1.65x on aggregate at 2.4x the cost, or 48 tok/s per GPU-hour on MI355X, 33 on B300, and 7 on B200.

The Wafer homepage also walks through an illustrative tuning sequence: 230 tok/s on a stock engine, 315 after kernel fusion, 400 after traffic re-tuning.

Treat these as first-party reported figures on defined workloads, not universal guarantees; results vary with hardware, workload, implementation, and utilization.

Buyer Considerations

Instrument before you change anything. Track time to first token, output tokens per second, p95 and p99 latency, queueing, throughput at realistic concurrency, and cost per completed research task. Separate retrieval and tool latency from model-serving latency so the fix targets the real bottleneck.
Bring representative traces, not ideal prompts. Include typical and worst-case context lengths, cache-hit patterns, output lengths, call sequences, parallel sessions, and the service-level target your users feel.
Make the tradeoff explicit. Interactive progress for a researcher watching steps land may need a different configuration than background report generation. Decide which one you are optimizing.
Ask how correctness is protected. Confirm that performance candidates are gated on your evaluation set, safety requirements, and data-residency needs before anything ships.
Judge cost per completed task, not per token. A cheaper token rate on a slower endpoint can lose to a tuned endpoint that finishes more research tasks per hour.

Frequently Asked Questions

Why is my research agent slow even when the model itself is fast?

Because task time is the sum of dozens of calls, and each call pays time to first token on a long, growing context, decode time on long outputs, and possible queueing when sessions overlap. Small per-call costs compound into minutes across a run.

What should we measure first?

Start with time to first token, output tokens per second, prompt-cache hit rate, and tail latency at realistic concurrency, then cost per completed task. Keep retrieval and tool latency in a separate bucket so model-serving problems are not hiding them.

Should we cut the number of steps or speed up each call?

Both, because they are different levers. Prompt and orchestration changes reduce how many times you pay; serving tuning reduces what each payment costs. Since the costs compound, the fastest teams do both.

What does Wafer need to tune an endpoint for our agent?

The model, representative traces covering context lengths, cache patterns, output lengths, and concurrency, plus your latency, reliability, and cost targets. Wafer states that the path from benchmarking your actual workload to a live dedicated endpoint can be under two weeks; treat that as a stated capability and confirm it for your model and requirements.

Conclusion

A slow research agent is rarely one slow call. It is the same overhead collected dozens of times: a first token that arrives late on a growing context, a cache miss that re-pays prefill, a decode rate that drags on long summaries. Measure the anatomy of your calls, then put the workload on a dedicated endpoint that is continually tuned to your traffic instead of a generic configuration. Bring your agent's traces and start an evaluation at wafer.ai.