token cost optimisation.
Token cost optimisation reduces the spend of a language model system through model selection, prompt reduction, caching, batching and routing, without degrading the output quality that matters.
Where token cost optimisation fits on a long engagement.
Routing by task is usually the largest available saving. Most systems send every request to their most capable model, while a substantial proportion of those requests — classification, routing, simple extraction — are handled identically by a model costing a fraction as much.
Prompt caching is the second lever and is frequently unused. Systems with a large fixed preamble pay for those tokens on every call, and providers offering caching reduce that substantially for no quality change. It is close to free money for anyone with a long system prompt.
What an assigned team does with token cost optimisation.
Cost work needs quality measurement beside it, or it becomes quality reduction with a cost chart. A cheaper model that is adequate on most cases and poor on the ones that matter is not a saving.
Running both together is why cost work belongs with the evaluation harness rather than separately, agreed as part of how the monthly fee is built.
What we use token cost optimisation for.
- Requests routed by difficulty Simple tasks handled by a cheaper model, which is usually the largest saving available.
- Fixed preambles cached Repeated context billed once rather than on every call.
- Quality measured alongside cost Evaluation run with every reduction, so savings are not quality loss.
How token cost optimisation capacity is assigned.
Cost optimisation is assigned inside AI capacity alongside the evaluation harness, because a saving without a quality measure is not a saving.
Tell us what your roadmap needs token cost optimisation for.
A service delivery manager replies with the disciplines we would assign, the monthly capacity and what the first month looks like.
Loading the contact form… You can also email hello@azendo.co.
We reply within one working day. No obligation, and no newsletter.