AI inference cost optimisation: capacity, utilisation and architecture
Almost every inference cost programme starts at the prompt and works outward. That is the wrong end of the problem. Average GPU utilisation across enterprise Kubernetes clusters is 5%, which means the largest single line in most AI infrastructure budgets is capacity that was bought, provisioned, and then left running with nothing on it.
Two levers do nearly all of the work here. This is what they return, and why your organisation will refuse to pull the second one.
- Cast AI's 2026 report puts average GPU utilisation at 5%, CPU at 8% and memory at 20%, across tens of thousands of clusters. An idle GPU costs dollars an hour where an idle CPU costs cents.
- Reclaiming idle capacity is the first lever because it costs no accuracy, no model change and no application work. It is also the one nobody is selling you.
- Routing is the largest software lever: 30 to 70% on mixed traffic, and under 10% where the traffic is uniformly difficult. Know which you have before you build a router.
- Caching pays back at roughly 1.4 reuses on a five-minute cache, which makes it the highest return per unit of effort on the list.
- Routing is blocked by a verification problem, not an engineering one. You cannot safely send work to a cheaper model unless something tells you when the cheaper model got it wrong.
Start with the bill you already pay and never use
Cast AI's 2026 State of Kubernetes Optimization Report, drawn from tens of thousands of clusters, puts average GPU utilisation at 5%. CPU sits at 8% and memory at 20%. Read the first number again: on average, 95% of GPU capacity is doing nothing.
It is the least interesting finding in AI cost and by some distance the most valuable. Reclaiming it requires no model change, no accuracy trade and no prompt engineering. It is scheduling and capacity management, which is unfashionable work that no vendor has a product page for, which is roughly why it stays undone while three teams argue about context windows.
The economics are lopsided in a way that makes the neglect expensive rather than merely untidy. The same 8% CPU utilisation has been tolerated in enterprise Kubernetes for years because an idle core costs cents an hour. An idle GPU costs dollars. Identical operational habit, two orders of magnitude difference in what it burns.
Why it persists is not mysterious. AI infrastructure was procured under scarcity. When GPUs were genuinely hard to get, holding capacity you were not using was the rational move, and provisioning habits formed under that constraint have not updated now that the constraint has moved. Most enterprises are still paying the insurance premium on a risk that has changed shape.
One honest qualifier, because it decides whether this section applies to you at all. If you run entirely on metered APIs, there is no idle capacity to reclaim and this lever is not yours. Go straight to the software levers below. If you self-host anything, measure utilisation before you optimise a single prompt.
The levers, ranked by what they actually return
Reported savings, with what each one costs you. The ranges are wide because workload mix decides almost everything, and a lever that returns 70% on one traffic profile returns single digits on another.
| Lever | Reported return | Where it applies | What it costs you |
| Reclaiming idle capacity | From a ~5% utilisation baseline | Self-hosted serving only | Scheduling discipline |
| Model tiering and routing | 30 to 70% on mixed traffic | Anywhere | A verification step |
| Prompt and context caching | 40 to 90% on cached tokens | High-reuse workloads | Close to nothing |
| Async batching | 50% off list on batch APIs | Non-interactive work | Latency |
| Continuous batching | Up to 23x throughput against naive static batching | Self-hosted serving | Engineering effort |
| Quantization | 30 to 50% with INT8 | Self-hosted serving | 1 to 3% accuracy at INT8 |
| The full stack together | 60 to 80%, or 20 to 40% on uniformly hard traffic | Depends entirely on mix | All of the above |
Every figure is sourced at the end of this article.
Two things fall out of that table that contradict how these programmes are usually sequenced. Caching is the best return per unit of effort on the list, near-zero risk and reported break-even at about 1.4 reuses on a five-minute cache, yet it routinely loses the first planning session to a quantization debate. And quantization, which is where engineering instinct goes first, is a middle-ranked lever that trades accuracy for money. It belongs in the programme. It does not belong at the front of it.
Routing is the biggest software lever, and nobody switches it on
Tiered routing returns a reported 30 to 70% on mixed traffic. Almost no enterprise runs it. They serve everything, the trivial classification and the contract question alike, on the most capable model they have access to.
The usual diagnosis is cost blindness. It is not. It is risk, and the instinct behind it is sound. Nobody wants to be the person who routed the compliance question to the cheap model and found out afterwards. Faced with a saving they cannot bound and a failure they cannot detect, engineering leaders correctly choose the expensive default.
So the instinct is right and the conclusion is wrong, and the thing in between is detection. You cannot route because you have no independent way of knowing when the cheaper model got it wrong. Solve that and routing stops being a gamble and becomes a policy.
Which puts Greenlight in an odd position in a cost article, because it is usually described as a governance model: an agent proposes, an independent check tests the result against the standard it has to meet before anyone acts on it, and a person clears what matters. Structurally, that middle step is also the precondition for tiering. The check does not need to run on the frontier model and usually should not. A smaller model testing a smaller model's output against an explicit standard is cheap, and it is what converts "we could route" into "we route, and here is what the check caught."
The reflex, when a cost programme starts, is to look at verification as overhead. More calls, more tokens, an obvious line to trim. It is exactly backwards. Verification is what unlocks the largest lever on the list, and cutting it to save tokens spends 30 to 70% to save single digits.
Four things not to optimise
Cost programmes go wrong in predictable places, and all four of these are defensible decisions made one context too late.
Do not batch an interactive experience. Continuous batching returns 3 to 5x on high-QPS background jobs and hits a 1.2 to 1.5x ceiling under a sub-100ms p99 requirement. Aggressive batching behind an interactive assistant or a voice agent buys a modest saving with a user-visible regression.
Do not treat quantization as free. FP8 has been reported as effectively lossless across more than 500,000 evaluations, while INT8 carries 1 to 3% degradation. On a task where 1 to 3% matters, that is not a cost saving. It is a quality decision wearing a cost costume, and it should be taken by whoever owns the quality bar.
Do not build a router for uniform traffic. Routing returns under 10% where difficulty does not vary, before the routing overhead. If genuinely everything you serve is hard, the lever is not yours and the effort belongs elsewhere.
Do not optimise without a per-task baseline. Without one you cannot distinguish a saving from a regression that happened to reduce volume, and the failure mode is silent: a compression change that quietly drops answer quality shows up as a cost win on every dashboard you have.
The test, before anyone writes a prompt
Two questions decide whether an inference cost programme is worth starting. What is your GPU utilisation across the estate you self-host? And what share of your requests are served by your most expensive model? If the honest answers are "we do not measure that" and "all of them", nothing you do at the prompt layer will matter, because you are optimising the third-largest term while the first two sit untouched.
Before either, get a baseline you can compare against: the cost per completed task calculator turns your own volumes and prices into the one number every lever on this page is trying to move. The reason to ask the two questions in that order is that neither requires a model change, an accuracy trade or a vendor conversation. They are both operating-model questions, and they are both answerable this quarter. Everything further down the list is genuine engineering work with genuine trade-offs, and it is worth doing after, not instead.
For Australian enterprises the first lever is usually larger than the global average implies, for an unglamorous structural reason. Data residency requirements pin workloads to specific regions where instance availability is thinner, which pushes teams to hold capacity rather than risk not getting it back. That is a rational response to scarcity and it produces precisely the 5% utilisation pattern described above. The premium is real. It is worth knowing what you are paying for it.
This article is the second half of a pair. Enterprise AI token costs covers what the work costs and the one number worth governing, which is cost per completed task. This one covers how to cut it. Both sit under AI FinOps, which is where the levers get owned after go-live rather than won once in a review, and if the bill in front of you is infrastructure rather than inference, cloud optimisation is the same discipline one layer down.
Sources
Figures in this article are compiled from the following published sources. Each should be read in full before a number is used in a business case.
- Cast AI. 2026 State of Kubernetes Optimization Report, 21 April 2026. GPU utilisation 5%, CPU 8%, memory 20%. cast.ai
- GMI Cloud. Cutting LLM Inference Costs in 2026: Where Caching, Batching, and Smart Routing Actually Pay Off. Routing and batching ranges, and the full-stack 60 to 80% figure. gmicloud.ai
- Digital Applied. AI Inference Cost Optimization: FinOps Playbook 2026, 26 May 2026. Cache break-even, batch API discount, continuous batching throughput, and the FP8 and INT8 accuracy figures from the Red Hat AI and IST Austria evaluations. digitalapplied.com
- NeuralTrust. LLM Cost Reduction: 12 Strategies to Cut AI Inference Costs. Quantization, caching and routing savings ranges. neuraltrust.ai
- Atlan. LLM Cost Optimization Strategies: Complete Tactics List. Prompt compression and batch discount figures. atlan.com
