Skip to content

Quickstart

pip install "cachemaxxing[anthropic]"   # or [openai], [gemini], [litellm]
uv add cachemaxxing                      # zero runtime dependencies; extras just pull in a provider SDK

Your chat, as you already keep it

Keep your conversation as the usual OpenAI-style list of {"role": ..., "content": ...} messages. Pass it to Prompt().messages(...), then render it for your provider:

from openai import OpenAI
from cachemaxxing import Prompt, usage

client = OpenAI()
chat = [{"role": "system", "content": SYSTEM}]  # your long, fixed instructions

chat.append({"role": "user", "content": "How do I reset my router?"})
p = Prompt("support").messages(chat)
response = client.chat.completions.create(**p.openai_chat("gpt-6-luna"))
chat.append({"role": "assistant", "content": response.choices[0].message.content})

print(f"{usage(response).hit_rate:.0%} of the input came from cache")

The same p renders for every provider: p.openai(...) for the Responses API, p.anthropic(...) and p.gemini(...). Your list is never changed; markers go only on copies. Anthropic and Gemini don't take a system message inside the conversation, so messages() moves it to where each provider expects it.

examples/chat.py runs three turns on any provider. Real output, OpenAI:

$ uv run --env-file .env python examples/chat.py openai
How do I reset my router?             6,283 input tokens,      0 from cache (0%)
And how do I update its firmware?     6,347 input tokens,  6,280 from cache (99%)
Thanks! Is that safe?                 6,433 input tokens,  6,344 from cache (99%)

Where the caching happens

Part of the request Cached? Anthropic OpenAI GPT-5.6+ Gemini
Tools Yes: rarely change cache_control on the last tool covered by the system prompt's breakpoint (tools come first) implicit
System prompt Yes: the biggest saving cache_control on its last block prompt_cache_breakpoint on its last part implicit
Conversation so far Yes: the next turn reads it back cache_control on the latest user message a breakpoint on the latest user message and on the one before implicit
Per-request context (.retrieved(): RAG chunks, current time) No: it changes on every call placed after the markers placed after the markers placed last

Gemini caches repeated prefixes on its own, so nothing is marked. An explicit cache is used only when gemini_cache() finds that it pays.

What gets sent

For a short chat, with a system prompt, a question, an answer and a follow-up:

{"model": "gpt-6-luna",
 "messages": [
   {"role": "system", "content": [{"type": "text", "text": "You are Acme's support bot. ...",
                                   "prompt_cache_breakpoint": {"mode": "explicit"}}]},
   {"role": "user", "content": [{"type": "text", "text": "How do I reset my router?",
                                 "prompt_cache_breakpoint": {"mode": "explicit"}}]},
   {"role": "assistant", "content": "Hold the reset button for 10 seconds."},
   {"role": "user", "content": [{"type": "text", "text": "And the firmware?",
                                 "prompt_cache_breakpoint": {"mode": "explicit"}}]}],
 "prompt_cache_options": {"mode": "explicit", "ttl": "30m"}}
{"model": "claude-haiku-5-5",
 "system": [{"type": "text", "text": "You are Acme's support bot. ...", "cache_control": {"type": "ephemeral"}}],
 "messages": [
   {"role": "user", "content": "How do I reset my router?"},
   {"role": "assistant", "content": "Hold the reset button for 10 seconds."},
   {"role": "user", "content": [{"type": "text", "text": "And the firmware?",
                                 "cache_control": {"type": "ephemeral"}}]}]}
{"model": "gemini-3.8-flash",
 "contents": [
   {"role": "user", "parts": [{"text": "How do I reset my router?"}]},
   {"role": "model", "parts": [{"text": "Hold the reset button for 10 seconds."}]},
   {"role": "user", "parts": [{"text": "And the firmware?"}]}],
 "config": {"system_instruction": {"parts": [{"text": "You are Acme's support bot. ..."}]}}}

Using LangChain's SystemMessage, HumanMessage and AIMessage? See LangChain and LangGraph.

Tools, RAG and per-request context: build the prompt from parts

When a request has more than a system prompt and a chat, set each part on its own. Segments go from most to least stable, whatever order you call them in: tools, system, context, examples, history, retrieved, user.

from cachemaxxing import Prompt

p = Prompt(call_site="support.answer")
p.tools(TOOLS)                 # sorted by name; your provider's tool format
p.system(SYSTEM)               # frozen instructions
p.context(DOCS)                # static reference material (dicts become canonical JSON)
p.history(messages)            # earlier turns, append-only, never modified
p.retrieved([chunks, f"Time: {now}"])  # per-request context: after the history
p.user(question)

Render it for your provider

resp = client.messages.create(max_tokens=1024, **p.anthropic("claude-haiku-5-5"))
resp = client.responses.create(**p.openai("gpt-6-luna"))
# or: client.chat.completions.create(**p.openai_chat("gpt-6-luna"))
resp = client.models.generate_content(**p.gemini("gemini-3.8-flash"))
resp = litellm.completion(max_tokens=1024, **p.litellm("anthropic/claude-haiku-5-5"))

Each adapter returns the full kwargs, model included, with the provider's cache markers at the stability boundaries.

Measure

from cachemaxxing import usage

u = usage(resp)
print(u.input, u.cache_read, u.cache_write, u.output, f"{u.hit_rate:.0%}", u.est_cost)

usage() accepts the SDK response object or its JSON, for every provider and LiteLLM.

Runnable example

uv run --env-file .env python examples/quickstart.py anthropic   # or openai, gemini, litellm
"""Quickstart: ask two questions over the same long system prompt; the 2nd call should read the cached prefix.

uv run --env-file .env python examples/quickstart.py {anthropic|openai|gemini|litellm}"""

import sys
from pathlib import Path
from typing import Any

from cachemaxxing import Prompt, usage

DATA = Path(__file__).parents[1] / "benchmarks" / "data"
SYSTEM = (DATA / "policy.md").read_text() + "\n\n" + (DATA / "docs.md").read_text()  # > 4,096 tokens


def ask(provider: str, question: str) -> Any:
    p = Prompt(call_site="quickstart").system(SYSTEM).user(question)
    if provider == "anthropic":
        import anthropic

        return anthropic.Anthropic().messages.create(max_tokens=64, **p.anthropic("claude-haiku-5-5"))
    if provider == "openai":
        import openai

        return openai.OpenAI().responses.create(max_output_tokens=64, **p.openai("gpt-6-luna"))
    if provider == "gemini":
        from google import genai

        client = genai.Client()  # keep a reference: a temporary Client closes its connection when collected
        kw = p.gemini("gemini-3.8-flash")
        kw["config"]["max_output_tokens"] = 64
        return client.models.generate_content(**kw)
    import litellm  # pip install "cachemaxxing[litellm]"

    return litellm.completion(max_tokens=64, **p.litellm("gemini/gemini-3.8-flash"))


if __name__ == "__main__":
    provider = sys.argv[1] if len(sys.argv) > 1 else "gemini"
    for q in ("How do I reset the router?", "And how do I update its firmware?"):
        u = usage(ask(provider, q))
        print(f"{provider}: input={u.input} cache_read={u.cache_read} hit={u.hit_rate:.0%} est=${u.est_cost or 0:.6f}")

Next: the tutorial finds and fixes a real cache miss, step by step. To log your calls and find misses in production, see Diagnosing.