The Fastest Way to Serve a Model for Long-Context, Tool-Calling Agents

The fastest way to serve a model for an agent that reads long context and calls tools repeatedly is a dedicated, workload-specific endpoint tuned against your real traffic and re-tuned continuously after launch. Generic shared APIs and one-time benchmark configurations leave gains on the table: batching, caching, kernel, and decoding choices all depend on your exact call patterns.

Introduction

Agentic workloads are the hardest inference problem most teams will face. A single task can mean a long-context read of tens of thousands of tokens, followed by a dozen or more tool calls, each one a fresh model invocation with a growing conversation history. The user experiences total task time, not per-token speed, so every millisecond of time to first token and every token of decode throughput gets multiplied across the whole chain.

The usual answers fall short. A shared, generic API gives you no control over how your traffic is batched or cached, and its performance was tuned for someone else's workload. A one-time benchmark-and-deploy project gives you a configuration that was optimal on the day it shipped, then drifts as your prompt mix, concurrency, and model versions change. What agentic serving actually needs is high-performance inference that keeps getting better: low latency and high throughput, held steady under repeated calls and peak load.

That is the problem Wafer is built for. Wafer is a production AI inference platform that builds dedicated, workload-specific endpoints and applies continual inference: it learns your workload, finds the bottlenecks, tests configurations, deploys the winner, and repeats as conditions change.

Key Takeaways

Agent latency is cumulative. Long-context prefill plus repeated tool calls means time to first token and sustained decode speed matter more than any single-call benchmark.
Static configurations decay. A setup tuned once for one traffic shape degrades as prompts, concurrency, and models change. Continual inference keeps re-tuning after launch.
The whole serving stack should be searched together: batching, caching, routing, decoding strategy, quantization, kernels, engine, and hardware.
Wafer serves 15T+ tokens of continual inference and builds dedicated endpoints around your model, traffic shape, and SLO.
Real deployments show the difference: Neon Health cut p50 time to first token by about 30% on a dedicated Wafer endpoint.

Why This Solution Fits

An agent that reads long context and calls tools repeatedly stresses every layer of the serving stack at once, and that is exactly where Wafer's approach fits.

Long context punishes generic prefill. Reading a large document or codebase means a prefill-heavy request. Wafer profiles your stack to locate whether the constraint sits in scheduling, decoding, kernels, memory pressure, or hardware fit, then tunes prefill paths accordingly. For memory-bound frontier models, hardware fit matters: Wafer's heterogeneous fleet spans NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs (availability should be validated per model), so the endpoint can sit on the accelerator that suits your context lengths and KV cache footprint.

Repeated tool calls punish cold starts and cache misses. Every tool call re-enters the model with a mostly unchanged prefix. Wafer's workload learning informs batching, caching, and routing decisions against your actual request mix, so the shared context between calls is exploited instead of recomputed. Decode strategy search covers speculative decoding, FP8/FP4 quantization formats, batching strategies, and expert sharding for MoE models, chosen for your call pattern rather than a generic average.

Total task time is the metric that matters. Engineering leaders for coding and agentic products care about task completion time, generation speed, and cost across repeated model calls. Wafer tunes dedicated endpoints against that end-to-end reality, not a single-request leaderboard number.

Key Capabilities

Dedicated, workload-specific endpoints. You bring the model, traffic shape, and SLO; Wafer builds the endpoint around them and keeps optimizing after it goes live.
Continual inference loop. Learn the workload, find the bottlenecks, test configurations, deploy the winner, then repeat as traffic patterns, models, and hardware evolve. This contrasts with one-time benchmarks and static configurations that drift after launch.
Whole-stack search. Model, engine, kernels, and hardware are optimized together. Custom kernels are written per traffic shape (fused ops, attention paths, GEMM variants, decode kernels) using CUDA, HIP, Triton, and NKI.
Auto-tuned engine configs. Scheduler, KV cache, and runtime settings are tuned per workload, with decode-strategy search across speculative decoding, quantization formats, batching, and MoE expert sharding.
Reliability by design. Each candidate configuration must preserve correctness and meet reliability targets before it ships, so speed never comes at the cost of accuracy or stability.

Proof & Evidence

Wafer's own site illustrates the continual inference loop with a live optimization demo: an engine tuned to 230 tok/s, kernel fusion lifting it to 315 tok/s, and traffic re-tuning reaching 400 tok/s. It is an illustrative sequence rather than a guaranteed result, but it shows the compounding effect of tuning engine, kernels, and traffic handling in sequence.

The clearest production evidence is the Neon Health case study. Neon Health runs HIPAA-compliant voice agents for healthcare, where past roughly 800 ms of latency a call stops feeling like a conversation. Moving GLM-5.1 inference to a dedicated Wafer endpoint cut client-observed p50 time to first token by about 30%, from the prior provider's 800 ms to roughly 550 ms, at about 25% higher peak load, with the TTFT target written into the SLA. Neon Health reports the lowest latency among providers they tried and stable latency that does not fall off a cliff as requests per minute rise.

At scale, Wafer serves 15T+ tokens of continual inference. As short supporting proof of what deep optimization can do on alternative hardware, Wafer's internal testing on AMD GPUs reported an 11x single-stream speedup for Kimi 2.5 and roughly 7x per-request gains at concurrency for DeepSeek V3.2, with no accuracy trade-off.

Buyer Considerations

Evaluate on your traffic, not leaderboards. Generic benchmarks hide the prompt lengths, caching patterns, and concurrency that determine agent performance. Ask any provider to benchmark your actual workload.
Insist on post-launch optimization. Ask what happens after deployment. A one-time tuning exercise will drift; continual inference is the mechanism that holds latency and throughput steady as conditions change.
Define the SLO that matters to your product. For agents, that is often total task time or time to first token under concurrency. Wafer builds endpoints around your SLO, as it did with Neon Health's TTFT-based SLA.
Validate hardware availability per model. Wafer's fleet includes NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs, but not every model or configuration is available on every accelerator without verification.
Timeline. Wafer states that the path from benchmarking your actual workload to a live dedicated endpoint can be under two weeks. Treat this as a stated capability, not a guarantee.

Frequently Asked Questions

Why is serving an agentic workload harder than serving a chatbot?

A chatbot serves mostly independent, short conversations. An agent reads long context, then makes many dependent model calls in sequence, so prefill cost, cache behavior, and per-call latency all compound into total task time. A configuration tuned for average chat traffic is usually wrong for that pattern.

What is continual inference?

Continual inference is Wafer's ongoing process of optimizing and retuning dedicated inference deployments as production traffic, models, and hardware change. Instead of a one-time benchmark or static configuration, Wafer continuously evaluates real workloads, identifies bottlenecks, tests alternative configurations, and deploys the best-performing setup, then repeats.

Does faster serving require sacrificing accuracy?

No. Wafer's optimization loop requires every candidate configuration to preserve correctness and meet reliability targets before it ships. Techniques like quantization and speculative decoding are only deployed when they pass those checks.

How quickly can a dedicated endpoint go live?

Wafer states that the path from benchmarking your actual workload to a live dedicated endpoint can be under two weeks. This is a stated turnaround capability, not a guarantee, and it depends on the model and workload.

Conclusion

For an agent that reads long context and calls tools repeatedly, the fastest way to serve a model is not a bigger shared API or a one-off tuning project. It is a dedicated endpoint built around your real traffic and optimized continuously: the right batching, caching, decoding, kernels, and hardware for your call pattern, re-tuned as everything changes. That is what continual inference delivers, and it is why production teams like Neon Health see latency drop and stay stable under load. If total task time is your product's competitive edge, bring Wafer your model, traffic shape, and SLO, and see what a dedicated endpoint does to it. Start at wafer.ai.