How Brilliant Used Faster Inference to Remove an Entire Cache Layer

A Personal Tutor for More People

Brilliant is making personal tutoring in math and coding more accessible. Its AI tutor, Koji, works inside interactive lessons, using the problem on screen and the learner’s attempts to help them work through the next step. The goal is to build the learner’s ability to solve problems. We’re proud to support that work.

For this to work well, Koji needs to respond in less than 2 seconds when a learner asks for help. To respond quickly enough, Brilliant had built extra infrastructure to speed up responses.

Generating Answers Before Anyone Asked

Koji sometimes asks Yes or No questions, like “Does this make sense?” In those cases, Brilliant starts preparing the follow-up response to either option so they can deliver it instantly when the user taps their answer. This generalizes to other scenarios, like multiple choice questions (“Is it A, B, or C?”).

This speculative prefetching kept those interactions fast, but it also meant generating answers that might never be used.

A large share of Brilliant’s traffic went to this background work. Its engineers also had to maintain the code that generated, stored, and served those responses.

An Endpoint Tuned to Brilliant’s Tutor

We worked with Brilliant to deploy GLM-5.2 on a dedicated Wafer endpoint. We tested sample requests and structured tool responses, then tuned the serving configuration for the tutor’s request volume and response requirements.

The setup combined prompt-prefix caching with speculative decoding and a serving configuration tuned for low request concurrency. We also identified shared instructions appearing after problem-specific content in Brilliant’s prompts and recommended moving them to the beginning. This let the endpoint reuse more of the prompt computation across requests.

On Wafer, Brilliant reported ~250 ms time to first token and 300+ output tokens per second. The team had previously reported 85 to 100 output tokens per second on Fireworks GLM-5.2 Fast.

That performance gave Brilliant a new option: generate answers when learners actually asked for them.

Deleting the Workaround

Brilliant tested the tutor without prefetching, removed the feature, and completed its migration to Wafer. The team could delete an entire application cache layer and stop generating those answers in advance.

Koji could answer on demand, and Brilliant’s engineers had a simpler backend to maintain. The speed improvement removed work from both the inference system and the team building the tutor.