What Is Agentic AI Context Caching? Stop Resending the Codebase

What Is Agentic AI Context Caching?

Agentic AI context caching is the runtime control that lets an agent pay once to process a large, stable block of context. On every later turn, the agent reuses the provider’s stored computation at about a tenth of the price. The block is usually the system prompt, the tool schemas, the retrieved documents, or the codebase itself. Without caching, an agent re-sends and re-pays for all of that on every turn of its loop.

You cannot shrink the codebase. You can stop resending it.

That is the whole post in two sentences. But the numbers behind it are worth the next ten minutes. Caching is now the single largest cost lever in agentic AI, and most enterprises still leave it on the default setting.

Why agents re-pay for the same context

Agentic AI, by my definition, is a governed, goal-driven software layer where agents plan and execute multi-step work. Every step is a model call. Every call carries the full context: the instructions, the tools, the history, and whatever the agent read since the last turn.

Anthropic’s engineering team measured what that loop costs. In its post on building a multi-agent research system, it reported that agents use about four times the tokens of a chat. Multi-agent systems use about fifteen times. Anthropic’s later piece on context engineering for agents calls context a finite resource with diminishing returns. Both points lead to the same place. Most of the tokens an agent buys are input tokens, and most of those are the same tokens it bought one turn ago.

So the cost problem is not that the codebase is big. The cost problem is that the agent pays full price for it fifty times a session.

How context caching works

Every major provider now sells a version of the same mechanism. The model processes a prompt prefix once and stores the intermediate state. A later request that begins with the exact same prefix skips that work, so the provider bills the repeated tokens at a discount and returns the first token faster.

Anthropic’s prompt caching uses explicit breakpoints. You mark where the stable prefix ends, and cache reads bill at one tenth of the input price. A five-minute cache costs 1.25 times input to write; a one-hour cache costs twice input to write. Google’s Vertex AI caching charges 10 percent of the input price for cached tokens on Gemini 2.5 and later. It runs an implicit mode by default and an explicit mode you manage yourself. OpenAI caches automatically for prompts over 1,024 tokens, and Amazon Bedrock offers up to 90 percent off cached reads with checkpoints you place.

The details differ. The shape does not. You pay a small premium to write once, then you pay a tenth to read many times. The condition is that the prefix stays byte-for-byte stable and the cache has not expired.

Where caching sits in the architecture

In my six-layer agentic AI architecture, caching is a runtime-layer control. It sits next to the harness and next to context compaction, and the three interact. The harness decides what goes into the context. Compaction decides how much of it survives. Caching decides what that context costs.

One distinction matters here, because I see architects get it wrong. Caching does not buy context-window space. A cached token still occupies the window and still counts against the limit. Caching changes the price and the latency of that token, nothing else. If the problem is that the window is full, the answer is compaction. If the problem is the bill, the answer is caching. Most production agents need both.

The enterprise proof: Uber’s one-hour cache

Uber published the most detailed public account of this in August. In its post on running a software factory efficiently, Uber reported that weekly agent requests grew 9.4 times between February and mid-August 2026. Total AI spend, in Uber’s words, relatively stabilized since April. I wrote up the full case yesterday. Here I want the caching decision alone.

Uber found that engineers walk away from the keyboard. A five-minute cache expired while they got coffee, so the next turn rebuilt the full context at full price. Uber moved interactive sessions to the one-hour cache and paid the double write premium, because the alternative was a full-price rebuild worth far more. Subagents stayed on the five-minute cache, since their tasks finish in seconds.

Uber then tracked the result per person. One session dashboard in the post shows $4,162 spent across 433 sessions, a 95 percent cache hit rate, and $1,214 of savings identified. Cache hit rate is a line on Uber’s unit-economics dashboard, next to cost per merged pull request. That is what a governed metric looks like.

The break-even math

The write premium means caching is a bet that the agent will read the prefix again before it expires. The math is short, so I will show it.

Set the normal input price at one. A five-minute cache write costs 1.25, and every read costs 0.1. The first request pays a quarter more than an uncached request. The second request already saves 0.9. So a five-minute cache pays for itself on the second turn.

A one-hour cache write costs 2.0. The extra premium over an uncached request is 1.0, and each read saves 0.9. That needs roughly three uses within the hour to come out ahead. Any real agent loop clears that bar in its first minute. Below it, though, caching is a hidden tax. A breakpoint placed on content that changes every turn pays the premium forever and never reads.

That is the number a FinOps lead should ask for: not the cache hit rate alone, but the hit rate against the write premium, per agent, per week.

What breaks the cache

Caching fails silently. The bill goes up and nothing else changes, so the failure modes are worth naming.

Any change before the breakpoint invalidates everything after it. A timestamp in the system prompt, a user name, a tool schema that loads in a different order, even a whitespace change. Anthropic’s docs add that changing the thinking parameters invalidates cached messages too.

Compaction resets the prefix. Uber compacts at 400,000 tokens even on a million-token model, and it describes that threshold as a trade between model performance and cache bursts. Tune the two together or one will undo the other.

Then there is the tenant question. A Stanford audit of prompt caching, published at ICML 2025, detected cache sharing across users at seven of eight providers that cached. The timing side channel could reveal what another user had asked. Providers have since moved to per-organization isolation. But inside one organization the cache is shared. A vendor that runs many customers through one provider account has to design tenant isolation itself, because the provider will not do it for them.

What transfers to regulated loan origination

In regulated loan origination, where every agent turn has to be auditable and every dollar has to be attributable. Three practices transfer directly.

I put the stable content first and the volatile content last, and I make that a platform rule rather than an engineer’s choice. The policy text, the tool schemas, and the reference documents sit above the breakpoint. The applicant’s file sits below it.

I set the cache TTL by how the agent is used, not by the default. Interactive review sessions get the one-hour cache. Batch verification agents that finish in seconds get the five-minute cache.

And I put cache hit rate on the same dashboard as cost per completed task. A hit rate below 80 percent on a stable workload means someone changed the prefix, and I want to know who and why before the invoice tells me.

The roadmap diagnostic: if nobody in your organization can tell you the cache hit rate of your busiest agent, you are still at Piloted, whatever the demo said.

The bottom line

On my Five-Stage Agentic AI Roadmap, caching discipline is a Governed-stage control. It arrives when the platform team owns the prefix layout, sets the TTL policy, and reports the hit rate per agent. Before that stage, caching is whatever each engineer’s tool does by default. After it, an unattended agent at the Assured stage cannot run up full-price context rebuilds nobody sees.

Gartner predicted in June that AI coding costs will pass the average developer’s salary by 2028. It named token discipline as something that will not emerge from developer choice alone. I agree. The discipline has to live in the platform. Instructions in, results out was IT. Intent in, outcomes out is agentic AI, and the outcome includes the invoice. I wrote the longer version in Intent In, Outcomes Out and the case for earned autonomy in Earned Autonomy.

What is the cache hit rate of your busiest agent this week, and who owns the number?

My books go deeper on both: Intent In, Outcomes Out and Earned Autonomy.

Go deeper

Agentic AI Architect: control design for enterprise agents.
Agentic AI Case Studies: deployment evidence, one teardown at a time.
Agentic AI P&L: cost, payback, and risk in a CFO’s voice.
Agentic AI SDLC: the lifecycle that builds the agents.
Substack first access: https://agenticaiarch.substack.com/subscribe

Dr. Harish Kotadia, Ph.D., is an Enterprise AI Architect with 20+ years of IT consulting experience serving Fortune 100 clients, specializing in agentic AI systems built on Anthropic Claude, AWS Bedrock, and Google Vertex AI.

Disclaimer:

This blog post is based on publicly available academic publications, vendor documentation, open standards, and news items from reputed media sources linked above. This post is intended for educational purposes, to help the enterprise agentic AI community build a shared vocabulary from public, authoritative sources.

Views and opinions expressed here are my own and do not represent those of any employer or client, past or present. The analysis presented is my independent interpretation of the published sources linked above and does not constitute legal, financial, or consulting advice of any kind.