LLM Prompt Caching: Where to Draw the Cache Boundary to Cut API Costs
Run an agent in production for a while and something odd shows up on the bill: input tokens outnumber output tokens by an order of magnitude or more. Of course they do. Every turn resends the entire system prompt, the entire tool schema, and the entire conversation history. The user types "yes" and a 20,000-token prompt flies along with it.
Prompt caching targets exactly this. If the front of your prompt is identical to last time, the provider reuses the computation for that portion and charges dramatically less. The concept is simple; actually getting a high hit rate comes down to where you draw the boundary. This article is about that boundary.
Caching Is Prefix Matching
First thing to internalize: prompt caching works on an exact match running from the front. Think of the prompt as one long string — the cache asks "how many characters from the start are identical to last time?" Change a single character in the middle and everything from that point on is a cache miss.
That leads to a brutal conclusion. Put a frequently changing value near the front and you disable caching entirely. If the current timestamp is on line two of your system prompt, the 20,000 tokens after it get recomputed every single time.
// Bad — a value that changes every turn sits at the very front
system prompt:
Current time: 2026-10-15 14:32:07 <- different every call
You are ... (20,000 tokens of agent description) <- all cache misses
// Good — stable content first, changing content last
system prompt:
You are ... (20,000 tokens of agent description) <- cache hit
Current time: 2026-10-15 14:32:07 <- only this recomputedMark the Boundary Explicitly
When prompt assembly is scattered across a codebase, a human has to decide "is this section above or below the cache line?" every time. That's an easy thing to get wrong. So embed a boundary marker as a string inside the prompt and let the provider adapter split there and attach the cache directive.
// Marker separating the stable region from the dynamic one.
// The prompt builder and the provider adapter share this constant.
const CacheBoundaryMarker = "<!-- CACHE_BOUNDARY -->"
// Split the system prompt into two blocks at the boundary.
// Marker present: stable (cached) + dynamic (not cached).
// Marker absent: one block, cached wholesale (backwards compatible).
func splitSystemPromptForCache(content string) []map[string]any {
ephemeral := map[string]any{"type": "ephemeral"}
idx := strings.Index(content, CacheBoundaryMarker)
if idx == -1 {
return []map[string]any{
{"type": "text", "text": content, "cache_control": ephemeral},
}
}
stable := strings.TrimSpace(content[:idx])
dynamic := strings.TrimSpace(content[idx+len(CacheBoundaryMarker):])
blocks := []map[string]any{
{"type": "text", "text": stable, "cache_control": ephemeral},
}
if dynamic != "" {
// The dynamic block gets no cache_control
blocks = append(blocks, map[string]any{"type": "text", "text": dynamic})
}
return blocks
}There's a reason the marker is shaped like an HTML comment. If it ever survives unsplit and reaches the model verbatim, that's the form least likely to be misread as a meaningful instruction.
What Goes Above and What Goes Below
This is the practical core, and the test is a single question: "does this value change from turn to turn within the same session?"
Above the boundary (cached)
- Agent role and persona definitions
- Behavioral rules and response style guides
- Agent-level config files (team rules, tool usage conventions, capability definitions)
- Tool definition schemas — usually the single largest chunk in the whole prompt
- Skill catalog summaries
Below the boundary (not cached)
- Current time and date
- Per-user profile files (they differ when the user differs)
- Memory context retrieved and injected for this turn
- Runtime figures like cumulative session tokens
- Channel and chat-room details that vary per run
The code that splits config files into two groups stays this simple:
// Agent-level config — rarely changes -> above the boundary
var stableFiles = map[string]bool{
"AGENTS.md": true,
"TOOLS.md": true,
"CAPABILITIES.md": true,
}
// Per-user and per-session files fall into the dynamic group automatically
func splitStableDynamic(files []ContextFile) (stable, dynamic []ContextFile) {
for _, f := range files {
if stableFiles[filepath.Base(f.Path)] {
stable = append(stable, f)
} else {
dynamic = append(dynamic, f)
}
}
return
}Cache Your Tool Definitions Too — the Chunk People Miss
Plenty of teams cache the system prompt and stop there. But once you have 30 tools, the tool schemas can be larger than the system prompt. JSON Schema is verbose.
With the Anthropic API, attaching the cache directive to the last tool in the array pulls every preceding tool definition into the cached prefix.
// Set a cache breakpoint on the last tool to cache all tool definitions
if len(tools) > 0 {
tools[len(tools)-1]["cache_control"] = map[string]any{"type": "ephemeral"}
}There's a precondition, though: the tool list must be in the same order on every request. If you build the array by ranging over a map, Go randomizes iteration order and your cache breaks every call. Sort it. This bug shows up constantly in practice.
Providers Differ — Branch on a Capability Flag
Caching mechanisms vary by provider. Some use block-level directives, some take a cache key in the request body, some don't support it at all or handle it server-side. None of that should leak to the call site.
type ProviderCapabilities struct {
Streaming bool
ToolCalling bool
Thinking bool
CacheControl bool // supports block-level cache directives?
MaxContextWindow int
}
// Middleware for providers that take a cache key in the request body.
// Passes through silently for endpoints that don't support it.
func CacheMiddleware(body map[string]any, cfg MiddlewareConfig) map[string]any {
cacheKey, hasKey := cfg.Options[OptPromptCacheKey]
if !hasKey {
return body
}
if !isNativeEndpoint(cfg.APIBase) { // skip when going through a proxy
return body
}
body["prompt_cache_key"] = cacheKey
return body
}The "skip when going through a proxy" line matters. Proxies claiming API compatibility frequently return a 400 when handed a field they don't know. Caching is an optimization, not a requirement — when in doubt, don't send it.
Always Measure the Effect
Once caching is wired up, you have to confirm it's actually hitting. The usage field on the response reports cache creation tokens and cache read tokens separately. Leave those out of your tally and you won't notice when the cache stops working.
if resp.Usage != nil {
total.PromptTokens += resp.Usage.PromptTokens
total.CompletionTokens += resp.Usage.CompletionTokens
total.CacheCreationTokens += resp.Usage.CacheCreationTokens // written to cache (expensive)
total.CacheReadTokens += resp.Usage.CacheReadTokens // read from cache (cheap)
}
// Hit rate — if this doesn't climb, your boundary is in the wrong place
hitRate := float64(total.CacheReadTokens) /
float64(total.CacheReadTokens+total.CacheCreationTokens)Watch out: cache creation costs more than ordinary input. Build a cache you never reuse and you've lost money. Which means caching may not pay off for short conversations that end after a turn or two. The wins come from long conversations, repeated calls, and many users sharing the same agent.
Five Ways People Quietly Break Their Cache
- Timestamps near the front — the most common and most damaging. Time and date always go below the boundary.
- Tool order shifting between calls — building an array by ranging a map gives random order. Sort it.
- Unstable JSON serialization order — if key order wobbles, the bytes differ and the cache breaks.
- Prompt mode changing mid-session — if your architecture toggles prompt sections by situation, pin the mode within a session. A mode change alters the stable region itself.
- Misjudging cache lifetime — most caches expire after a short idle period. A batch job running every few minutes may find the cache already gone and pay creation costs every time.
Summary
- Caching is exact prefix matching — one changed character up front misses everything after it
- Embed a boundary marker in the prompt and let the adapter split and attach the directive
- One test decides placement: "does it change from turn to turn within a session?"
- Tool definitions are cacheable too — as long as you pin the ordering
- Hide provider differences behind a capability flag, and don't send fields you're unsure about
- Track cache read and creation tokens separately to measure hit rate — unmeasured caching fails invisibly
Prompt caching isn't an algorithm problem, it's a layout problem. You're not writing more code, you're reordering strings you already have. And that one reordering removes most of your input cost.
댓글
댓글 쓰기