What AI costs in production: getting inference spend under control

The paradox: tokens get cheaper, the bill grows

Both sides of the paradox are well documented. In his November 2024 analysis “LLMflation”, Guido Appenzeller of a16z calculated that the price of inference for a model of comparable quality falls roughly tenfold every year: a GPT-3-level model cost $60 per million tokens in November 2021, while three years later a model costing $0.06 achieved the same score on the MMLU benchmark. The author himself cautions that the pace may slow and that the benchmark is not a perfect yardstick.

Gartner describes the other side. In a press release dated August 17, 2026, it predicts that inference costs per agentic workflow will increase more than fivefold through 2028. It calls the phenomenon the “inference paradox”: better unit economics drive up total AI costs without offering a clear path to commensurate, predictable value. According to the same release, handing a task to an agentic reasoning model raises the provider’s inference costs at least fivefold, and considerably more for complex tasks.

State of FinOps 2026 shows this is no fringe topic: of 1,192 respondents, together representing more than $83 billion in annual cloud spend, 98% say they already manage AI spend as well. A year earlier it was 63%, two years earlier 31%. According to the report, AI cost management is the number one skill teams need to add, and the most requested tool capability is granular tracking of AI spend – tokens, LLM requests and GPU utilization.

The Czech context: according to the Czech Statistical Office (ČSÚ), 18% of enterprises with ten or more employees used at least one AI technology in 2025, and among large companies with more than 250 employees the figure reached 54% for the first time. So in large companies AI is running in production at the majority – and with it comes a bill that someone has to explain.

Where the consumption comes from

Three mechanisms explain most surprising invoices. All three are visible in usage data, provided someone looks at it.

The agent loop sends the context again and again

The model is stateless. An agent that calls tools therefore sends the entire history so far at every step: the system prompt, the tool definitions, all previous outputs. Input tokens thus grow not linearly with the length of the task but roughly quadratically.

An illustrative calculation (a model, not a measurement): an agent with a system prompt and tools totalling 5,000 tokens takes 20 steps, and each step adds 2,000 tokens to the history. The last request has 43,000 input tokens, but over the whole task you pay for 480,000 input tokens – eleven times more than the final context. The same task with 40 steps already comes to 1.76 million.

Reasoning tokens are invisible, but you pay for them

The OpenAI documentation says it plainly: reasoning tokens are not visible via the API, yet they still occupy space in the context window and are billed as output tokens. Output tokens tend to be more expensive than input tokens in price lists, so a task with a high reasoning effort can cost a multiple of the same task without it, without the visible answer growing by a single word.

Long contexts and data bundled in “just in case”

A large context window tempts you to send the model a whole document, a whole repository, or ten RAG results instead of three. Every such token is paid for on every call. For a one-off query it is negligible; for a service with thousands of calls a day it becomes the main cost item.

Measure first: cost per task, not cost per token

The price per million tokens is a figure from a price list. For management you need a different unit: how much one completed task costs – a resolved ticket, a reviewed pull request, a processed invoice. Only that can be compared with the value the task delivers, and only that shows whether an optimization works.

Three things are enough:

• Usage data from every response. The API returns counts of input, output, cached and reasoning tokens. Store them together with the task, team, application and model identifiers.
• A single point all calls pass through. A gateway or proxy in front of the model providers. Without one, you learn about consumption from the invoice – a month late and with no breakdown by team.
• Attribution at the cloud level. Separate deployments, API keys or projects per application and environment, plus consistent tagging so that costs can be attributed in the billing export.

State of FinOps 2026 also points to a weak spot of the entire discipline: according to respondents, nobody can yet answer whether AI delivers value. Measuring cost per task does not settle that question, but it is the necessary half of the equation. The same principle, applied to infrastructure in general, is described in the article on getting your infrastructure in order before adopting AI.

Five techniques that reduce the bill

1. Prompt caching

The provider stores the repeated beginning of a prompt (system instructions, tool definitions, reference documents) and bills it at a discount on subsequent calls. In an agent loop, where the prefix repeats at every step, this is the single most effective measure.

The specific rules vary by vendor. For example, the Anthropic documentation (as of October 2026) lists cache writes with a five-minute lifetime at 1.25× the base input price, with a one-hour lifetime at 2×, and cache reads at 0.1× as standard (even less for some of the newest models). That leads to a simple calculation: with a five-minute cache, the write pays for itself on the second call (1.25 + 0.1 versus 2.0), with a one-hour cache on the third.

The cache only works on an exactly matching prefix. The order is tools → system prompt → messages, and a change at a higher level invalidates everything below it. In practice this means:

• stable content at the beginning, variable content (time, user name, the current query) at the end;
• do not change tool definitions between steps of the same task;
• track the share of tokens read from cache in your usage data – a low share for an agentic workload is a flaw in the prompt design, not a property of the model.

2. Batch processing

Anything that does not need to be done immediately does not belong on the synchronous API. Anthropic’s Message Batches API charges 50% of the standard price for both input and output; according to the documentation, most batches finish within an hour, the maximum is 24 hours, and the discount can be combined with prompt caching. Cloud platforms offer a similar mode – Azure has a discounted Batch deployment type, Amazon Bedrock a Flex tier with a discount for workloads that can tolerate longer processing.

Typical candidates: overnight classification and summarization, model evaluations, bulk generation of descriptions, retroactive processing of an archive.

3. Routing to smaller models

In the cited release, Gartner recommends “inference-tiering, routing and orchestration” – that is, sending routine tasks to cheaper models and reserving the most expensive reasoning for tasks that genuinely need it. The simplest form is static: classification, extraction and short summaries go to a small model, planning and complex debugging to a large one. A dynamic router that decides based on the content of the query only makes sense once you have an evaluation set and can verify that quality has not dropped.

The same applies to reasoning effort: set it per task, not globally to the maximum.

4. Context compression and maintenance

• Return fewer but better-ranked chunks from RAG; reranking is cheaper than having a large model read unnecessary pages.
• Trim tool outputs at the source – the agent does not need the whole log, it needs the relevant fifty lines.
• In long sessions, summarize the history as you go or remove old tool outputs.
• Limit output length and require a structured response wherever a machine reads it.

Watch out for the conflict with caching: every change to the history alters the prefix and invalidates the cache. So trim in larger blocks and less often, not after every step.

5. Application-level cache

The cheapest token is the one you never send. Identical queries (FAQs, repeated classification of the same input) can be served from your own response cache. A semantic cache built on embeddings also catches rephrased queries, but carries the risk of returning the answer to a different question or someone else’s data – so the cache key must include the tenant and permissions.

Limits and budgets: insurance against a weekend loop

Optimization lowers the average. Only hard limits protect against the extreme – an agent stuck in a retry loop, a forgotten test script, a leaked key. They belong at three levels:

• per task: maximum number of agent steps, maximum number of output tokens, a time limit;
• per key and team: a budget per period and limits on tokens and requests per minute;
• per account: budget alerts and anomaly detection in cloud billing.

Most LLM gateways support per-team budgets. An example for the open-source LiteLLM proxy, where the budget resets automatically once the chosen period elapses:

curl 'http://litellm.internal:4000/team/new' \
  --header 'Authorization: Bearer <master-key>' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "team_alias": "platform-team",
    "max_budget": 500,
    "budget_duration": "30d",
    "tpm_limit": 200000,
    "rpm_limit": 500
  }'

Pay-as-you-go or reserved capacity

Alongside pay-per-token, the hyperscalers also offer prepaid capacity. The principle is the same as with reserved instances: a fixed payment in exchange for guaranteed performance and a lower unit price at high utilization.

On Azure, provisioned throughput is billed at an hourly rate per PTU regardless of the number of tokens processed; a monthly or annual reservation lowers the rate. The documentation flags two things that are easy to miss: a reservation does not guarantee capacity, so you should create the deployment first and only then buy the reservation; and when PTUs are exhausted, the deployment returns a 429 error unless spillover to a standard deployment is configured.

Amazon Bedrock has an equivalent in its service tiers: Reserved, with a fixed price per thousand tokens per minute billed monthly, a 99.5% availability target and automatic overflow into the Standard tier, followed by Priority, Standard and Flex.

When optimization does not make sense

Not every AI bill is a problem. A few situations where it is better to leave things alone:

• Small volume. If inference costs a fraction of one engineer’s salary, a week spent building a router will not pay off. Introduce measurement and limits, and postpone the rest.
• Smaller model, more attempts. A cheaper model that completes the task on the first try less often can be more expensive per completed task – because of the retries and because of the time of the people who fix its output. Hence cost per task, not cost per token.
• Savings that make development more expensive. Limits on AI developer tools set too low save tokens and burn hours. A budget should stop an anomaly, not ordinary work.
• Commitment in a fast-changing market. An annual reservation for a specific model is a bet that you will still want to use it a year from now. Given how fast models and price lists change, a shorter commitment is often the wiser choice, even if it costs more per unit.
• Forecasts are not certainties. Gartner’s fivefold figure is an analyst firm’s prediction, not a measurement; a16z’s estimate of a tenfold annual decline dates from 2024 and rests on a single benchmark. Plan based on your own data.

What to do on Monday

Limits and model routing are best kept as configuration in version control and deployed the same way as the rest of the platform – see deployment automation in the AI era.

Key takeaways

• The problem is consumption, not the price list. Tokens are getting cheaper, but according to Gartner agentic workflows consume so many of them that the cost per workflow will increase more than fivefold through 2028.
• The agent loop grows quadratically. Every step resends the entire history; without caching and context maintenance you pay for the same tokens many times over.
• Reasoning is billed as output. It does not show in the answer, but it shows on the invoice. Set the effort level per task.
• Measure cost per completed task. It is the only way to tell whether a cheaper model really saves money.
• Caching and batch are the cheapest savings. They require changes to prompts and calls, not to the architecture.
• Limits protect against the extreme. Caps on steps and tokens and a per-team budget belong in every production agent deployment.
• Reserved capacity only with data. A fixed payment only pays off at consistently high utilization; on Azure, a reservation does not even guarantee capacity on its own.