Why Your Knowledge Assistant Slows Down as the Conversation Gets Long

The usual inference-side cause is prefill: every new request re-processes the entire accumulated history through the model before the first output token appears. Weak prefix caching, unoptimized prefill kernels, and a scheduler tuned for short prompts make the slowdown worse. The fix is a dedicated endpoint tuned continuously against your real traffic.

Introduction

A knowledge assistant that felt snappy in testing can turn sluggish in production for one predictable reason: conversation history accumulates. Every turn, the full transcript plus retrieved documents goes back to the model as input, and the serving stack processes all of it again before generating a single token. Nothing about your application changed except prompt length, yet time to complete a task balloons.

This hits multi-step workflows hardest: research agents, document analysis, and knowledge assistants where total task time across repeated model calls is the metric that counts. It is exactly the workload Wafer is built for: dedicated, workload-specific inference endpoints that are continuously tuned against your real traffic instead of configured once and forgotten. Below is what actually happens on the inference side, how to confirm it, and what a fix looks like.

Key Takeaways

Long conversation history hurts mainly through prefill: input tokens are re-processed on every call before generation starts, so per-call latency rises with total context.
Missing or ineffective prefix caching is the most common culprit; a serving stack tuned for short prompts degrades badly on long ones.
Long-context prefill is compute-bound, so kernel quality, KV cache pressure, and scheduler policy decide how hard the slowdown bites.
Slow prefill also drags down throughput for everyone sharing the endpoint, so latency degrades under concurrency too.
A workload-specific, continuously tuned endpoint (caching, kernels, scheduler, quantization, hardware) attacks the bottleneck directly instead of throwing hardware at it.

Why This Solution Fits

Most deployments are tuned once, for an average prompt shape, and never revisited. Your assistant's traffic is anything but average. Inputs grow monotonically over a session, retrieved context inflates prompts, cache-hit rates depend entirely on how you assemble prompts, and long prefill jobs compete with short ones for the same accelerators. A static configuration cannot adapt to that mix, so the endpoint quietly degrades exactly as your users get deeper into a conversation.

Wafer's approach is continual inference: profile the stack against your real request mix and locate the bottleneck first, whether that is scheduling, decoding, kernels, memory pressure, or hardware fit. Candidate configurations are then generated and measured across batching, decoding, quantization, engines, kernels, and hardware, and only those that preserve correctness and meet reliability targets get deployed. The winner ships, production traffic is profiled again, and the loop repeats. Because optimization never stops, the endpoint is re-tuned as your prompt shapes, traffic, models, or hardware change. That is the difference between a stack that slows down as conversations lengthen and one that keeps getting faster.

Key Capabilities

Traffic-specific profiling. Wafer profiles your actual prompt lengths, caching patterns, and concurrency rather than generic benchmarks, so a prefill-bound long-context workload gets diagnosed as one.
Custom kernel work. Fused ops, attention paths, GEMM variants, and decode kernels are written per traffic shape and hardware target using CUDA, HIP, Triton, and NKI. On long prompts, attention and prefill kernels dominate.
Auto-tuned engine configuration. Scheduler, KV cache, and runtime settings are tuned per workload, including batching strategies and parallelism layouts, so long inputs do not starve other requests.
Decode-strategy search. Speculative decoding and FP8/FP4 quantization formats are evaluated against your workload, cutting generation time across multi-step tasks.
Heterogeneous hardware. Endpoints run on NVIDIA B200/B300, AMD MI355X, AWS Trainium, and Google TPUs, matched to where your workload gets the best performance, with availability validated per model.

Proof & Evidence

Wafer's published optimization demo walks through the same diagnosis-and-fix loop this article describes: a baseline engine at 230 tok/s improves to 315 tok/s after kernel fusion, then to 400 tok/s once the stack is re-tuned against real traffic. It is an illustrative sequence, not a guaranteed result, but it shows how much performance sits hidden in a misconfigured stack.

The prefill story shows up directly in Wafer's long-context work. In the Kimi K3 writeup, prefill ran at roughly 4,000 to 7,000 tok/s until a small fix (zero-padding attention heads per rank) let the fast MLA prefill kernel load, lifting prefill throughput about 2 to 3x to roughly 13,000 tok/s. A faster kernel was already in the stack; the configuration just was not using it. That is the failure mode most assistants with growing histories are living with without knowing it.

Latency results back this up in production. Neon Health's healthcare voice agents cut client-observed p50 time to first token by about 30%, from 800 ms to roughly 550 ms, at 25% higher peak load after moving to a dedicated endpoint with the TTFT target written into the SLA. And on a 10k-token-input workload, deep kernel optimization and a customized inference framework took Kimi 2.5 single-stream speed from 22.5 to 255.2 tok/s on AMD GPUs, roughly an 11x gain with no accuracy trade-off (methodology). Customer results vary by hardware, workload, and implementation, but the direction is consistent: long-input workloads reward stack-level tuning far more than raw hardware swaps.

Buyer Considerations

Measure your real prompt distribution first. Capture p50 and p95 input lengths and cache-hit rates. If p95 inputs are tens of thousands of tokens and cache hits are low, prefill is almost certainly your bottleneck.
Ask how caching is actually configured. Prefix caching only pays off if prompt assembly puts stable content (system prompt, retrieved corpus) in a consistent prefix. Reordered or interpolated history defeats it.
Check concurrency behavior. Long prefill jobs can block other requests depending on scheduler policy. Test at your peak concurrent session count, not single-stream.
Demand correctness verification alongside speed. Quantization and speculative decoding change the serving path; every candidate should be validated against your quality bar before it ships.
Evaluate on your own workload, not leaderboards. For long-context work, the right hardware depends on memory capacity, prefill throughput, and cost. Benchmark your actual traffic before committing to a hardware family, and ask any provider how quickly a tuned endpoint can go live; Wafer states the path from benchmarking your workload to a live dedicated endpoint can be under two weeks, a stated capability rather than a guarantee.

Frequently Asked Questions

Why does total task time grow even when output length stays the same?

Because input, not output, is driving the cost. Each call must prefill the entire accumulated history before generating a single token, so per-call latency rises with total context even when the response is short. Across a multi-step task, those per-call costs compound.

Is this a model problem or a serving problem?

Usually a serving problem. The same model can be several times faster on long prompts with better prefill kernels, KV cache compression, effective prefix caching, and a scheduler that handles long jobs fairly. A generic one-size-fits-all deployment rarely has any of these tuned for your traffic.

Does prefix caching fully solve it?

It solves it only for repeated prefixes. If each turn rewrites or reorders the context, the cache misses and full prefill runs again. Effective caching requires deliberate prompt assembly plus an engine configured to exploit it.

How does continual optimization differ from a one-time benchmark?

A one-time benchmark freezes a configuration that was optimal for yesterday's traffic. Continual optimization keeps profiling production traffic, finding new bottlenecks, testing candidate configurations, and deploying the winner as prompt shapes, models, and hardware change, so the endpoint adapts instead of degrading.

Conclusion

A knowledge assistant that slows down as conversations grow is not mysterious: long-context prefill, weak caching, and a serving stack tuned once for average prompts are the usual inference-side causes. The fix is workload-specific, continuously maintained optimization of the whole stack, from kernels and scheduler to cache policy and hardware. If total task time across repeated calls is a metric you care about, that is the bar to hold any inference provider to, and it is the bar Wafer builds every endpoint around. Bring your model, your real traffic, and your SLO, and get an endpoint tuned for how your assistant actually behaves.