How We Reduced LLM Costs by 70%
When you run several AI products, the model bill stops being a rounding error and starts being a real line on the P&L. Over a couple of quarters we cut our own LLM spend by roughly 70% without degrading output quality, and almost none of it came from a single clever trick. It came from measuring, then stacking unglamorous reductions. Here is exactly what moved the number.
Step zero: cost per request, or you are guessing
You cannot optimize what you cannot attribute. Before anything else, we instrumented every model call to record input tokens, output tokens, the model used, and the feature that triggered it, then rolled it up to cost-per-request and cost-per-SKU. The first time we looked, two surprises jumped out: one low-revenue feature was our biggest cost center, and a 'cheap' flow was silently routing to the strong model. You find those only with per-request attribution.
Model tiering: default down, escalate on need
The biggest single lever was routing. We had been defaulting everything to the strong model 'to be safe.' Most tasks did not need it. We split work into tiers:
- Classification, extraction, short structured answers -> cheap, fast model.
- Long-form reasoning and nuanced narrative -> strong model.
And we default to the cheap tier, escalating deliberately:
```js let out = await cheapModel(task); if (!passesQualityCheck(out)) { out = await strongModel(task); // escalate only on failure } ```
This cascade means most requests are served by the cheap model and only the ones that fail a programmatic check pay for the strong one. Routing alone was the largest chunk of our savings.
Prompt caching: stop re-billing your boilerplate
Our system prompts and grounding instructions are long and stable. Without caching, every single call pays for those tokens again. Prompt caching lets the provider reuse a stable prefix at a steep discount. We restructured prompts so the large invariant block (instructions, schema, examples) comes first and the small variable block (the user's question and data) comes last, maximizing the cacheable prefix. For high-frequency endpoints this was a quiet, large saving.
Output caching: never generate the same thing twice
A lot of AI output is deterministic given its inputs. A report for the same birth details, an OG image for the same page, a summary of the same document — there is no reason to regenerate it. We hash the full input and cache the output:
```js const key = hash(model + version + serialize(inputs)); const cached = await store.get(key); if (cached) return cached; // zero tokens const out = await generate(inputs); await store.set(key, out); return out; ```
The cheapest token is the one you never send. Output caching turned repeat traffic — which is a lot of traffic — into near-zero marginal cost.
Context trimming and RAG over giant prompts
It is tempting to dump everything into the prompt and let the model sort it out. You pay for that on every call. Instead we ground on computed, relevant facts. In our astrology chat we do not paste an entire reference text; we compute the user's chart and send only the facts relevant to their question. Choosing focused RAG over a giant prompt cut input tokens dramatically and improved answer quality at the same time, because the model was not wading through irrelevant context.
Concretely: audit your longest prompts and ask, for each block, 'does the model need this for this request?' Usually a third of it is dead weight.
Structured outputs: pay for answers, not prose
When you need data, ask for JSON, not an essay. Free-form responses waste output tokens on framing ('Sure! Here is...') and ramble. Constraining the model to a schema makes outputs shorter, cheaper, and parseable, which also removes a class of downstream errors. Output tokens often cost more than input tokens, so trimming output length is direct margin.
Batching where latency allows
For non-interactive work — nightly audits, bulk content generation — we batch requests rather than firing them one at a time with full overhead per call. Anything the user is not actively waiting on is a candidate for batching, which amortizes fixed costs and lets us use cheaper async pathways.
What the 70% was actually made of
No single tactic got us there. Roughly: model tiering was the largest share, output and prompt caching together were the next, context trimming and structured outputs were the rest. The meta-lesson is that LLM cost is a portfolio problem. Instrument first, then stack reductions, and re-measure after each one so you know what actually worked rather than what felt clever.
And keep quality gated the whole way — every one of these changes ran against our eval harness so we knew we were cutting cost, not corners.
If your AI bill is climbing faster than your revenue, GrahAI Systems can help you find the 70% — reach us at support@grahai.com.