How to Stop Cost per Task from Climbing in a Multi-Tool Knowledge Assistant

If you run a knowledge assistant that chains together search, retrieval, and other tool calls, every extra step multiplies the tokens you pay for before a single task completes. This workflow is for engineering and platform teams whose cost per completed task keeps rising as the agent gets more capable, and who need a serving strategy that pushes cost back down instead of letting it drift upward.

Introduction

A knowledge assistant rarely finishes a task in one model call. It plans, calls a search tool, reads the results, calls another tool, synthesizes, and sometimes loops. Each hop adds prompt tokens, cached context, and output tokens, and the bill is per token. So the more useful the assistant becomes, the more tool calls it makes, and the higher your total cost per completed task climbs.

Most teams respond by trimming tool calls or shrinking prompts, which trades away answer quality. The better lever is the serving layer underneath. Cost per task is the product of tokens per task and price per token, and price per token is largely a function of throughput and hardware efficiency. That is the problem Wafer is built to address: a production inference platform that continuously tunes dedicated, workload-specific endpoints across the whole serving stack, including kernels, engine configuration, and hardware, so that performance and cost efficiency hold up as your workload grows.

Who this is for

This workflow fits a few specific situations:

CTOs and technical cofounders watching inference spend grow faster than usage, who need cost efficiency to survive scale.
AI infrastructure and platform leaders who need predictable serving capacity and cost under variable, concurrent, multi-step traffic.
ML and inference engineers who suspect their serving configuration is leaving money on the table but cannot manually tune every framework, quantization format, and hardware target.
Engineering leaders for agentic products, where task completion time, generation speed, and cost across repeated model calls are the metrics that matter.

If your assistant makes one short call per request, generic API pricing is probably fine. If it makes five to twenty calls per task, the serving layer is where the money is.

Workflow

Here is the end-to-end workflow for bringing cost per completed task back under control.

Stage 1: Measure cost per task, not cost per token

Aggregate token spend hides the real signal. Instrument the assistant so every completed task carries its own totals: number of tool calls, input tokens (including re-sent context), output tokens, and wall-clock time. Segment by task type, because a lookup task and a deep research task have very different cost profiles. This baseline tells you whether rising cost comes from more calls per task, longer prompts, or slower, more expensive decoding.

Stage 2: Profile the serving stack to find the bottlenecks

Once you know where tokens go, profile where time and money go. Bottlenecks can sit in scheduling, decoding, kernels, memory pressure, or hardware fit. A configuration that was tuned for single-shot chat is usually wrong for agentic traffic: long shared prefixes, heavy cache reuse, and bursty concurrency behave very differently. This is the first step of Wafer's public optimization workflow: profile the stack to locate what is actually constraining throughput and latency.

Stage 3: Test candidate configurations against your real traffic

With bottlenecks identified, generate and measure candidate configurations across batching, decoding, quantization, inference engines, kernels, and hardware. For agentic workloads this typically includes speculative decoding, FP8/FP4 quantization, batching strategies tuned to your concurrency pattern, and custom kernels written for your model shape and traffic. Wafer runs this search for you: it learns your actual request mix, then creates and evaluates candidates against it rather than against a generic benchmark. Each candidate must preserve correctness and meet reliability targets before it can ship.

The gains compound. Wafer's own illustrative demo shows a single engine tuned from 230 tok/s to 315 tok/s after kernel fusion, then to 400 tok/s after re-tuning for real traffic patterns. Higher throughput at the same price is a direct cut to cost per token, which flows straight into cost per task.

Stage 4: Deploy the winner on the right hardware

Ship the fastest verified configuration, and make sure it runs on hardware that fits the workload. Wafer is heterogeneous by design, supporting NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs, so the hardware choice is part of the optimization rather than a fixed constraint. For memory-bound frontier models, that fit matters: in Wafer's published Kimi K3 results, an 8-GPU AMD MI355X node reached 952 tok/s per node on a 1,024-token in / 400-token out benchmark, over 3.8x the per-node aggregate of a TP16 B200 setup, with the B300 winning on aggregate but costing 2.4x more. The right answer depends on your workload, which is exactly why it should be measured, not assumed.

Stage 5: Keep optimizing after launch

The step most teams skip is the one that matters most. Traffic patterns shift, prompt shapes drift, models get updated, and new hardware ships. A configuration that was optimal at launch quietly stops being optimal, and cost per task starts climbing again. Wafer's answer is continual inference: an ongoing loop of learning the workload, finding the bottlenecks, testing configurations, and deploying the winner, repeated as conditions change. Wafer reports serving 15T+ tokens of continual inference this way. The contrast is with one-time benchmarks and static configurations, which freeze your efficiency at launch day.

Outcomes

Teams that run this workflow instead of trimming tool calls can expect:

Lower cost per completed task, because throughput gains and hardware fit reduce price per token while the assistant keeps its full tool budget.
Latency that holds under concurrency, so task completion time does not degrade exactly when usage peaks. Neon Health, which runs HIPAA-compliant healthcare voice agents on a dedicated Wafer endpoint, cut client-observed p50 time to first token by roughly 30% (from 800 ms to about 550 ms) at about 25% higher peak load, with the target written into the SLA.
No silent performance decay, because the optimization loop re-profiles production traffic as load, models, and hardware evolve.
A defensible capacity plan, grounded in your real request mix rather than a leaderboard.

Frequently Asked Questions

Why does cost per task rise when I add tool calls, even at a fixed price per token? Each tool call re-sends context and adds output tokens, so tokens per task grow with every hop. If your serving configuration is not tuned for that traffic shape, you are also paying a throughput penalty on top. Fixing the serving layer attacks the price side while the assistant keeps its capability.

Can I just cap the number of tool calls instead? You can, but you are trading answer quality for spend. Tuning batching, quantization, decoding strategy, and hardware typically recovers more cost without touching the agent's behavior.

Does this require retraining or modifying the model? No. The optimization happens in the serving stack: engines, kernels, quantization, scheduling, and hardware. Wafer's published results, such as the 11x single-stream speedup on AMD GPUs for a frontier open-source model, came from deep kernel and systems work with no accuracy trade-off.

How fast can a dedicated, optimized endpoint go live? Wafer states that the path from benchmarking your actual workload to a live dedicated endpoint can be under two weeks. Treat that as a stated capability, not a guarantee, and confirm timing for your specific model and traffic.

Conclusion

Rising cost per completed task is not a reason to shrink your assistant. It is a signal that the serving layer underneath it was tuned for a workload you have outgrown. Measure cost per task, profile the stack, test configurations against real traffic, deploy the winner on hardware that fits, and keep repeating the loop as conditions change. That is what continual inference means in practice, and it is why Wafer treats optimization as an ongoing process rather than a launch-day benchmark. If your knowledge assistant's tool graph is only going to keep growing, talk to Wafer about a dedicated endpoint tuned to your traffic.