Three things broke when I enabled GPT-6 caching—the fourth cut my bill 70%

Explicit cache breakpoints promised to slash repeat-call costs. I turned them on in production and watched the hit rate, headers, and bill—here's what happened.

We’ve been paying full price to send the same system prompt ten thousand times a day. Everyone has. You know it, I know it, the finance person who keeps asking why the “AI line item” doubled knows it. So when OpenAI shipped explicit cache breakpoints for GPT-6, I did the thing you’re supposed to do before writing about it: I turned it on in a real app and watched it misbehave.

The app is boring on purpose — a support-answer service. Big stable preamble: a system prompt, a style guide, six few-shot examples, and a chunk of retrieved policy docs. Roughly 8,000 tokens of context that never changes across a conversation, followed by a ~200-token user turn that changes every time. This is the textbook case for caching. If it doesn’t pay off here, it doesn’t pay off anywhere.

What a breakpoint actually is

Prompt caching, at its core, is a prefix cache. The model hashes your input from the start and reuses the computed state up to the point where two requests stop matching. The old, automatic behaviour worked — but you had no say in where the boundary sat, and no visibility into whether you got a hit.

A cache breakpoint is you drawing that line by hand. You mark the end of the stable region, and everything before the mark becomes a cacheable segment. The bytes after it — the volatile user turn — stay cheap to recompute. The whole game is putting the marker at the exact seam between “never changes” and “changes every call.”

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

from openai import OpenAI
​
client = OpenAI()
​
STABLE_CONTEXT = build_system_prompt() + fewshot_examples() + retrieved_docs()
# ~8,000 tokens, byte-identical across every request in a conversation
​
def ask(user_turn: str):
    return client.responses.with_raw_response.create(
        model="gpt-6",
        input=[
            {
                "role": "system",
                "content": STABLE_CONTEXT,
                # mark the end of the cacheable prefix here
                "cache_control": {"breakpoint": True},
            },
            {"role": "user", "content": user_turn},
        ],
    )

Use with_raw_response. You want the headers, and the plain call throws them away. That single method call is the difference between “I think it’s caching” and “I can prove it.”

The first thing that broke: a hit rate of zero

Five identical follow-up questions. I expected four warm reads. I got five cold ones. Every call billed the full 8,200 input tokens. The diagnostics said so plainly:

1
2
3
4
5
6
7
8

resp = ask("How do I request a refund after 30 days?")
​
print(resp.headers["openai-cache-status"])        # -> "miss"
print(resp.headers["openai-cache-hit-tokens"])    # -> "0"
​
usage = resp.parse().usage
print(usage.input_tokens)                          # 8203
print(usage.input_tokens_details.cached_tokens)    # 0

The reason was stupid and it will get you too. My retrieved_docs() call ran fresh on every request, and the retriever returned chunks in a slightly different order each time — same content, different byte sequence. A prefix cache does not care that the meaning is identical. One flipped chunk near the front of the segment and the hash diverges from that byte onward. Everything after the divergence is a miss.

Fix: sort the retrieved chunks deterministically and build STABLE_CONTEXT once per conversation, not once per turn. The moment the prefix was byte-stable, the hits showed up.

The second thing that broke: the breakpoint was one token too late

Now I was getting hits, but the cached-token count was lower than the stable region. Turns out I’d appended a timestamp to the system prompt — “current date: 2026-09-23T14:07:11Z” — for freshness. It sat inside the cached segment, before the breakpoint. Every second, the prefix changed. The cache warmed for exactly one request and then evicted itself into the sea.

Move anything with a heartbeat — timestamps, request IDs, per-user tokens, session nonces — after the breakpoint. The rule that fixed everything: the breakpoint goes at the last byte you can guarantee is identical across calls, and not one byte further. Truncate to date granularity if you need the date in context and can tolerate it going stale for a day.

The third thing that broke: streaming ate my headers

The production path streams tokens. When I flipped stream=True, my header-parsing code returned None and I briefly convinced myself caching didn’t work with streaming at all. It does. The cache is applied before generation starts; streaming only changes how you receive output. What changed was where the diagnostics live — the usage block arrives in the terminal event of the stream, not in a response object you can poke afterward.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

with client.responses.stream(
    model="gpt-6",
    input=[
        {"role": "system", "content": STABLE_CONTEXT,
         "cache_control": {"breakpoint": True}},
        {"role": "user", "content": user_turn},
    ],
) as stream:
    for event in stream:
        if event.type == "response.output_text.delta":
            print(event.delta, end="")
​
    final = stream.get_final_response()
    print(final.usage.input_tokens_details.cached_tokens)  # now populated

The fourth thing — the one that paid the rent

Once the prefix was clean and the breakpoint sat in the right place, the five-call run looked like this:

• Call 1 (cold): 8,203 input tokens, 0 cached. First token at ~910 ms.
• Calls 2–5 (warm): 8,203 input tokens, 8,000 cached, ~203 billed at full rate. First token at ~380 ms.

Now the money, and I’m going to show the arithmetic in dollars instead of waving at a percentage. GPT-6 lists input at $1.25 per million tokens and cached input at $0.125 per million tokens — a flat 90% discount, cached tokens billed at one-tenth of the standard rate. Read your own pricing page before you trust mine, because that ratio is the entire argument; it’s the one number that decides whether any of this ceremony is worth it.

1
2
3
4
5
6
7
8
9
10
11
12
13

INPUT_RATE  = 1.25   # $ per million input tokens (GPT-6 list price)
CACHED_RATE = 0.125  # $ per million cached input tokens - 90% off, 1/10 the input rate
​
def call_cost(cached_tokens, uncached_input_tokens):
    return (uncached_input_tokens / 1_000_000 * INPUT_RATE
            + cached_tokens        / 1_000_000 * CACHED_RATE)
​
cold = call_cost(cached_tokens=0,    uncached_input_tokens=8203)   # $0.010254
warm = call_cost(cached_tokens=8000, uncached_input_tokens=203)    # $0.001254
​
no_cache = 5 * cold            # $0.051269
cached   = cold + 4 * warm     # $0.015269
# savings across five calls: (no_cache - cached) / no_cache -> ~70%

Per call that’s a rounding error — a cold call is about a cent, a warm one an eighth of that. The point was never one call. Run the five and you go from $0.051269 to $0.015269, a 70% cut. Push the conversation longer and the single cold call amortizes away; the ceiling is ~88%, the point where every turn is a warm read against a fat prefix. For readers on a different rate tier, the formula is all that matters: warm / cold is your floor, and here it’s $0.001254 / $0.010254 ≈ 0.12.

Scale it and the shape is the whole story: a service running a million five-turn conversations a day against this prefix goes from roughly $51,000 to $15,000 in daily input spend — call it a million dollars a month — for the work of moving a timestamp and sorting some chunks. The latency delta, 910 ms down to 380 ms to first token, is the part users actually feel.

Where it quietly does nothing

Two limits matter. The cached prefix has to clear a minimum size before anything gets stored — small prompts never cache, so don’t expect savings on a terse one-liner. And the cache has a TTL measured in minutes of inactivity; a warm prefix goes cold if nobody hits it, and busy prefixes get extended. There’s also a ceiling on how much prefix is retained, so a genuinely enormous stable block may only partially warm. The diagnostics tell you the truth — check cached_tokens against your segment size rather than assuming.

And the honest counsel: don’t reach for this if your prompts are single-shot, if the “stable” context isn’t actually stable, or if the volatile part sits near the front of your input. A cache that never hits isn’t free — it’s the same bill plus the false comfort of thinking you optimised something. I’ve watched us relearn prefix caching under a new name roughly every eighteen months. The name changed again. The rule didn’t: keep the front of your prompt boring, keep the volatile bits at the back, and read the headers instead of trusting the marketing.

If you do one thing before shipping: log cached_tokens on every request in staging for a day. The graph will show you exactly which of your “stable” prompts is lying to you.