Prompt Caching in Production: A Complete Setup Guide for Claude, OpenAI & Gemini (2026)

Why prompt caching matters — the actual numbers

Every LLM call has two token buckets: input tokens (what you send) and output tokens (what the model generates). Input tokens are usually cheaper per-token, but for most production workloads they're the majority of your bill because system prompts, tool definitions, and conversation history all get re-processed on every call.

Consider a customer support agent with a 5,000-token system prompt (instructions + few-shot examples + tool definitions). A single user turn might involve 5 model calls (initial reasoning, tool call, tool result processing, another tool call, final response). That's 25,000 input tokens paid for what's essentially the same 5,000-token prefix being re-processed five times.

Prompt caching solves this by having the provider save the intermediate computation state after processing a prefix. On subsequent calls with the same prefix, the model reuses the cached computation instead of re-processing from scratch. The billing consequence:

• Anthropic Claude: cached input tokens billed at 10% of the standard input rate (90% discount)
• OpenAI: cached input tokens billed at 50% of the standard input rate (50% discount)
• Google Gemini: cached tokens billed at 25% of the standard rate plus a small hourly storage fee (~75% discount)

Rough real-world impact: teams with high-context workloads (agents, RAG, long conversations) typically see 40-70% reduction in input token bill after enabling caching properly. On a $10,000/month AI spend where 80% is input tokens, that's $3,200-5,600/month back.

How each provider's caching architecturally differs

Before writing any code, understand how each provider's caching works architecturally — because the mental models are different, and mixing them up is the #1 source of misconfiguration.

Claude: explicit breakpoints, ephemeral by default

Claude uses cache_control markers you place explicitly in your request. You mark up to 4 breakpoints across your system prompt, tool definitions, and messages. Everything from the beginning of your request up to a breakpoint gets cached. Cached content lives for 5 minutes by default (extendable to 1 hour with the ttl parameter). You control what's cached; the provider does the work.

OpenAI: automatic, prefix-based

OpenAI does prompt caching automatically for any prompt over 1,024 tokens. You don't mark anything — the provider hashes your input prefix and caches it. Cache hits reduce billed input tokens by 50%. Cache lifetime is 5-10 minutes typically (no explicit TTL control on standard tier). What you cache is entirely determined by your prompt structure — put stable content first.

Gemini: explicit CachedContent objects

Gemini's model is closer to a full database cache — you create a CachedContent object with your context and get back a cache ID. You then reference that ID in subsequent requests. Minimum cacheable size is 32,768 tokens (Gemini 3.5 Pro; lower for Flash). You explicitly set TTL (default 1 hour, up to your project's max). Different from Claude/OpenAI: this is a first-class stored object, not an ephemeral compute artifact.

Comparison at a glance

None is universally better. Use the one from the provider you're already on; if you're multi-provider, use each provider's flavor for that provider's traffic — don't try to abstract across.

Setting up prompt caching on Claude

Claude's caching gives you the most control and the biggest discount, but it requires explicit setup. Here's the complete working pattern.

The basic pattern

from anthropic import Anthropic

client = Anthropic()

LARGE_SYSTEM_PROMPT = """You are a customer support agent for AcmeCorp.

[... 5000 tokens of instructions, examples, and rules ...]
"""

TOOLS = [
    {
        "name": "search_orders",
        "description": "Search customer orders by ID, date, or status",
        "input_schema": {
            "type": "object",
            "properties": {
                "query": {"type": "string"}
            },
            "required": ["query"]
        }
    },
    # ... more tools ...
]

def chat_with_cache(conversation_messages):
    return client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        system=[
            {
                "type": "text",
                "text": LARGE_SYSTEM_PROMPT,
                "cache_control": {"type": "ephemeral"}   # ← breakpoint
            }
        ],
        tools=TOOLS,   # Cached automatically because it's before the breakpoint's scope
        messages=conversation_messages,   # NOT cached — changes each turn
    )

The cache_control marker at the end of the system block tells Claude: cache everything from the start of the request up through here. The first call is a "cache write" (billed at 1.25× normal input rate). Every subsequent call within 5 minutes that has the same prefix is a "cache read" (billed at 10% of normal input rate).

Multi-breakpoint pattern for agents

For agent workloads with reusable context, use multiple breakpoints strategically. Claude caches up to 4 breakpoints, and each cache is checked independently:

system=[
    {
        "type": "text",
        "text": BASE_SYSTEM_PROMPT,           # ~3000 tokens, rarely changes
        "cache_control": {"type": "ephemeral"}
    },
    {
        "type": "text",
        "text": USER_SPECIFIC_CONTEXT,        # ~1500 tokens, changes per user
        "cache_control": {"type": "ephemeral"}
    }
],
tools=[
    *TOOLS_STABLE,                            # Cached with system prompt breakpoint
    {
        **TOOL_DYNAMIC,
        "cache_control": {"type": "ephemeral"} # Third breakpoint
    }
],
messages=conversation_messages,               # Not cached

Extended TTL (1-hour cache)

For contexts you'll use across longer sessions, extend the TTL:

system=[{
    "type": "text",
    "text": LARGE_SYSTEM_PROMPT,
    "cache_control": {"type": "ephemeral", "ttl": "1h"}   # 1 hour instead of 5 min
}]

1-hour caches cost 2× the normal input rate to write (vs 1.25× for 5-min), so only use if you'll reuse the prefix for at least ~10 requests over an hour.

Reading the response

The usage object tells you exactly what got cached:

response = chat_with_cache(messages)
print(response.usage)
# Usage(
#     input_tokens=45,              # Fresh input tokens
#     cache_creation_input_tokens=0, # Zero because this was a cache read
#     cache_read_input_tokens=5142,  # Cache hit! 5,142 tokens at 10% price
#     output_tokens=234
# )

If you see cache_creation_input_tokens repeatedly on requests you expected to be reads, your cache is not being hit — usually because content before the breakpoint is drifting. See the cache breakpoint limit error page for related pitfalls.

Setting up prompt caching on OpenAI

OpenAI's automatic prompt caching requires almost no code changes — but it requires attention to prompt structure. Put stable content first, variable content last, and OpenAI does the rest.

The zero-config setup

from openai import OpenAI

client = OpenAI()

# System prompt (large, stable) MUST come first for caching to work
SYSTEM_PROMPT = """You are a helpful customer service agent...
[... 3000+ tokens of instructions ...]
"""

def chat(user_message, conversation_history):
    return client.chat.completions.create(
        model="gpt-5-6",
        max_completion_tokens=1024,
        messages=[
            # Stable prefix: system + any few-shot examples
            {"role": "system", "content": SYSTEM_PROMPT},
            # Variable content: conversation history + new user message
            *conversation_history,
            {"role": "user", "content": user_message},
        ]
    )

Any prompt with 1,024+ tokens in a stable prefix gets automatically cached. No flag to set, no explicit breakpoint. First call is a cache write (billed normally). Subsequent calls with the same prefix within ~5-10 minutes get the 50% discount on the matched prefix tokens.

Verifying cache hits

response = chat(user_msg, history)
print(response.usage)
# CompletionUsage(
#     prompt_tokens=4823,
#     completion_tokens=234,
#     total_tokens=5057,
#     prompt_tokens_details=PromptTokensDetails(
#         cached_tokens=4096   # ← how many were cache hits
#     )
# )
cache_hit_rate = response.usage.prompt_tokens_details.cached_tokens / response.usage.prompt_tokens
print(f"Cache hit rate: {cache_hit_rate:.1%}")

The prompt structure rules

OpenAI caches from the beginning of the prompt up to the first token that doesn't match a previous request. So anything variable placed early kills all downstream caching. Common mistakes:

# BAD — timestamp in system prompt kills caching on every call
system_prompt = f"Current time: {datetime.now()}. You are a helpful agent..."

# BAD — user ID early in system prompt
system_prompt = f"You are helping user {user_id}. Follow these instructions..."

# GOOD — stable content first, user context in the user message
system_prompt = "You are a helpful agent. Follow these instructions..."
messages = [
    {"role": "system", "content": system_prompt},   # Stable, cached
    {"role": "user", "content": f"[User {user_id} at {datetime.now()}] {actual_message}"}
]

OpenAI Responses API caching

For the Responses API (GPT-5.6 reasoning workflow), caching works the same way but you interact with it through the input parameter:

response = client.responses.create(
    model="gpt-5-6",
    input=[
        {"role": "system", "content": LARGE_STABLE_SYSTEM_PROMPT},
        *conversation_history,
        {"role": "user", "content": user_message}
    ]
)
print(response.usage.input_tokens_details.cached_tokens)

Same rules apply: stable content first.

Setting up context caching on Gemini

Gemini's context caching is more explicit than the other two — you create a stored CachedContent object and reference it. This works well for large stable contexts (long documents, big codebases, extensive tool schemas) but is overkill for typical chat prompts.

Creating a cached context

from google import genai
from google.genai import types

client = genai.Client()

# Step 1: create the CachedContent (one-time, or per session)
cache = client.caches.create(
    model="gemini-3.5-pro",
    config=types.CreateCachedContentConfig(
        display_name="acme_customer_support_v3",   # For your own tracking
        system_instruction=LARGE_SYSTEM_PROMPT,
        contents=[
            types.Content(
                role="user",
                parts=[types.Part.from_text(text=FEW_SHOT_EXAMPLES_50K_TOKENS)]
            )
        ],
        ttl="3600s"   # 1 hour
    )
)
print(f"Cache ID: {cache.name}")
# projects/YOUR_PROJECT/locations/us-central1/cachedContents/abc123...

Using the cached context in requests

def chat_with_cache(user_message, cache_name):
    response = client.models.generate_content(
        model="gemini-3.5-pro",
        contents=[
            types.Content(
                role="user",
                parts=[types.Part.from_text(text=user_message)]
            )
        ],
        config=types.GenerateContentConfig(
            cached_content=cache_name   # Reference the cache
        )
    )
    print(response.usage_metadata)
    # cached_content_token_count: 51284  (billed at 25% of normal rate)
    # prompt_token_count: 51334          (total, cached + fresh)
    return response

Cache lifecycle management

Unlike Claude and OpenAI, Gemini caches are stored resources you're paying to keep alive. Manage them:

# List active caches
for cache in client.caches.list():
    print(f"{cache.display_name}: {cache.name} — expires {cache.expire_time}")

# Extend TTL of an existing cache
client.caches.update(
    name=cache.name,
    config=types.UpdateCachedContentConfig(ttl="7200s")
)

# Delete when done
client.caches.delete(name=cache.name)

When Gemini caching pays off

The 32,768-token minimum plus the storage fee mean Gemini caching only makes sense for large, stable contexts reused many times. Typical winners:

• Long documents your agent references repeatedly (contracts, codebases, manuals)
• Large few-shot example sets you use across many requests
• Fixed knowledge base content in RAG-style workflows

For typical chat prompts under 32K tokens, Gemini caching doesn't apply — you'd just pay full rate on every call. That's OK for those workloads; use Claude or OpenAI if caching economics matter and your prompts are shorter.

Measuring your cache hit rate

Turning caching on isn't enough. Instrument it so you know whether it's actually working. Every LLM call should emit cache metrics that let you see the hit rate over time, per feature, per user segment.

from opentelemetry import metrics
meter = metrics.get_meter("ai.cache")

cache_reads = meter.create_counter("ai.cache.read_tokens", unit="tokens")
cache_writes = meter.create_counter("ai.cache.write_tokens", unit="tokens")
fresh_tokens = meter.create_counter("ai.cache.fresh_tokens", unit="tokens")

def emit_cache_metrics(response, provider, feature):
    if provider == "anthropic":
        reads = response.usage.cache_read_input_tokens or 0
        writes = response.usage.cache_creation_input_tokens or 0
        fresh = response.usage.input_tokens
    elif provider == "openai":
        reads = response.usage.prompt_tokens_details.cached_tokens or 0
        writes = 0   # OpenAI doesn't distinguish writes
        fresh = response.usage.prompt_tokens - reads
    elif provider == "gemini":
        reads = response.usage_metadata.cached_content_token_count or 0
        writes = 0
        fresh = response.usage_metadata.prompt_token_count - reads

    tags = {"provider": provider, "feature": feature}
    cache_reads.add(reads, tags)
    cache_writes.add(writes, tags)
    fresh_tokens.add(fresh, tags)

# Query in Grafana / Datadog:
# hit_rate = rate(ai_cache_read_tokens) / (rate(ai_cache_read_tokens) + rate(ai_cache_fresh_tokens))

Watch for these signals in your dashboard:

• Hit rate below 60% on cacheable prompts: your cache breakpoint is probably wrong or content is drifting. Chapter 6 of the Observability Playbook covers dashboard design in depth.
• Sudden hit-rate drop: usually a prompt version change moved the breakpoint. Correlate with your prompt deploy log.
• Rising cache write cost with flat cache reads: you're writing caches that never get read. Either TTL is too short or traffic pattern doesn't cluster requests together.

The five common pitfalls (and fixes)

The mistakes that eat caching's benefits are consistent across teams. Watch for these five, in rough order of frequency.

1. Variable content in the prefix

The single biggest killer. A timestamp, user ID, session token, or random UUID early in your prompt means every request gets a fresh cache write, never a read. Audit your system prompts for anything that changes per request.

# Anti-pattern audit — search your system prompt for any of:
# - datetime.now(), time.time(), uuid.uuid4()
# - f-strings with user_id, session_id, tenant_id
# - Environment values that could change (deployment version tags)
# - Any variable interpolation before the first cache breakpoint

2. Cache breakpoint too early

On Claude, if you put cache_control after just 500 tokens of system prompt but your tool definitions add another 3,000 tokens, you're caching 500 tokens and paying full rate for the other 3,000. Move the breakpoint after everything stable.

3. TTL mismatch with traffic pattern

Default 5-minute TTL requires requests to cluster within 5 minutes. If your traffic is bursty (a request every 20 minutes), you'll pay for cache writes that never get read. Either extend TTL (Claude 1-hour option, Gemini custom TTL) or accept that caching won't help this pattern.

4. Cache invalidation from message history

On Claude, cached content must appear before the cache breakpoint. If you put cache_control on your system prompt but insert a message before it, the cache misses. Message history goes in messages, which is always after the system prefix.

5. Cross-tenant cache pollution

If you have a multi-tenant product and different tenants have different system prompts, ensure each tenant's requests hit their own cache. On OpenAI (automatic caching), this happens naturally as long as tenants have different prefixes. On Claude, verify your prompt-building code doesn't accidentally mix tenants' content into the same request. See Chapter 10 of the Security Playbook for full multi-tenant isolation patterns.

Cost math: a worked example

Concrete example. Team running a customer support agent on Claude Sonnet 4.6, ~1 million requests/month. Each request has:

• System prompt + tools: 6,000 tokens (stable across all requests)
• Recent conversation history: ~2,000 tokens (variable)
• User message: ~500 tokens (variable)
• Output: ~400 tokens

Without caching

Every request pays full input rate on 8,500 tokens:

Claude Sonnet 4.6: $3/M input tokens, $15/M output tokens

Monthly input tokens: 1M requests × 8,500 tokens = 8.5B tokens
Monthly output tokens: 1M requests × 400 tokens = 400M tokens

Input cost:  8,500,000,000 × ($3 / 1,000,000) = $25,500/mo
Output cost:   400,000,000 × ($15 / 1,000,000) = $6,000/mo
Total:                                             $31,500/mo

With caching (5-minute TTL, ~85% hit rate assumed)

The 6,000-token stable prefix is cached. Assume 85% hit rate (good but realistic for chat workloads with clustered traffic):

Cached input rate:      $0.30/M (10% of normal $3/M)
Cache write rate:       $3.75/M (1.25× of normal $3/M)
Fresh input rate:       $3/M

Cache reads: 6,000 × 1M × 0.85 = 5.1B tokens @ $0.30/M   = $1,530
Cache writes: 6,000 × 1M × 0.15 = 900M tokens @ $3.75/M   = $3,375
Fresh input: 2,500 × 1M         = 2.5B tokens @ $3/M      = $7,500
Output: 400M tokens             @ $15/M                    = $6,000

Total: $18,405/mo (vs $31,500/mo without caching)
Savings: $13,095/mo (~42%)

Break-even analysis

Cache writes cost 1.25× normal. Cache reads cost 0.10× normal. Break-even on writing a cache requires the prefix to be read at least:

1.25 (write cost) - 1.0 (would have paid normal) = 0.25 (extra cost)
1.0 (normal rate) - 0.10 (cache read rate)       = 0.90 (savings per read)

Break-even reads: 0.25 / 0.90 = ~0.28 reads

Meaning: even 1 subsequent hit within the TTL window pays back the cache write. Caching is essentially free upside for any prefix reused at all, provided you're above the minimum size (1,024 tokens on Sonnet, 2,048 on Haiku).

When prompt caching is the wrong answer

Caching isn't universally right. Skip it — or use it carefully — in these cases:

• Very small prompts. Below the minimum cacheable size (1,024 tokens on most models), caching does nothing. Don't add breakpoints on tiny prompts.
• Extremely low-volume workloads. If you make one request per hour, the 5-minute cache always expires between calls. Cache writes without reads = pure loss. Either extend TTL or skip.
• Every-request-unique prompts. If your prompt genuinely varies significantly across requests (e.g., per-document analysis with unique documents), caching provides little benefit. Focus on other cost levers.
• Prompts you can't stabilize. If business logic requires timestamps or per-request identifiers early in the prompt for correctness, you can't cache — accept the cost, or refactor to move variables later in the prompt.
• PII in the cached prefix. Anthropic and OpenAI both isolate caches per organization, but for regulatory workloads (HIPAA, financial data), verify with the provider that your cache boundaries meet your compliance requirements before caching PII-adjacent content.

For the majority of production workloads — chat products, agents, RAG, coding assistants — caching pays off substantially. The teams that don't use it are typically leaving 30-60% of input token cost on the table.

Frequently asked questions

Common questions from teams setting up prompt caching for the first time.