Prompt Caching Explained: Why a Cache Hit in Claude Code Costs 10x Less

Every message you send to a large language model carries hidden baggage. The model does not just see your latest message. It reprocesses the system prompt, the available tools, and the entire conversation history on every single turn. In this 72-second explainer, Arjay McCandless walks through how prompt caching eliminates that repeated work, then goes a layer deeper to show what is actually being cached: the KV cache state inside the model.

The video is a living-room whiteboard session. McCandless alternates between couch-side narration and standing in front of a TV showing hand-drawn diagrams of conversation prefixes, cache reads, and query-key-value vectors.

The Problem: Recomputing the Same Prefix Every Turn

The video opens with what inference looks like without caching. A diagram on the TV breaks a ChatGPT-style conversation into blocks: History, System Prompt, and Tools. Two stacked rows show that both a short reply ("hello chat") and a long one ("not much") drag the same full prefix along with them.

As McCandless puts it, this is true every time you send a message in a conversation. The model has to recompute all of it from scratch.

The Fix: Cache the Prefix, Compute Only the New Stuff

The name gives it away. On every turn, the unchanged prefix is read directly from the cache, and expensive computation only happens on the new tokens. Once that turn finishes, the new content gets added to the cache for the next turn. A slide in the video titled "Each turn resends the full context, and the unchanged prefix reads from cache" walks through three turns, showing how each turn's cache read grows while only the new message gets processed. It also flags the catch: if the system prompt changes, the prefix changes, and the cache breaks from that point forward.

What Is Actually Cached: The KV Cache

The second half goes one layer deeper. At inference time, every token in your prompt is turned into a query, key, and value vector. To process the next token, the model has to attend over all previously computed keys and values. Those keys and values are stored in the KV cache to speed up that operation.

So prompt caching is not storing your prompt text. It is storing the state of the KV cache itself. When the next turn arrives, the entire prefix is reloaded directly into the KV cache from the prompt cache, and none of those key and value vectors need to be recalculated.

Why It Matters: 10x Cheaper Inference

The payoff is cost. McCandless closes with a concrete number: in a tool like Claude Code, a cache hit is 10 times cheaper than a cache miss. Agentic tools resend large system prompts, tool definitions, and long histories on every turn, which is exactly the workload prompt caching was built for.

Key Takeaways

• LLMs reprocess the full prompt (system prompt, tools, history) on every conversation turn unless caching intervenes.
• Prompt caching stores the unchanged prefix so only new tokens require expensive computation each turn.
• What gets cached is not the prompt text but the KV cache state: the key and value vectors computed for every prefix token.
• Reloading the prefix into the KV cache skips recomputing those vectors entirely.
• In Claude Code, a cache hit is 10 times cheaper than a cache miss, which is why caching matters most for long agentic sessions.

Resources

• Claude Code -- named in the video as the example where cache hits are 10x cheaper than cache misses