Prompt caching for agents: measure the hit before claiming the savings

Quick answer

A repeated prompt is not proof of a cache hit. Read the usage fields returned by the provider, compare matched and unmatched runs, and apply the current price of that model to each token category.

Agent prompts often repeat a long instruction prefix and change only the task at the end. That's a good candidate for caching. But changing an early instruction, inserting a timestamp near the top, or switching the model can change what is reusable. Don't estimate savings from prompt length alone.

What to log on every request

Keep a request id, model, prompt-template version, input tokens, output tokens, and cache usage fields. OpenAI's Responses API exposes cached input in usage.input_tokens_details.cached_tokens for supported requests. Anthropic reports cache_creation_input_tokens and cache_read_input_tokens. The field names and caching rules are provider-specific, so store the raw usage response alongside your normalized record.

Official references: OpenAI prompt caching and Anthropic prompt caching.

A small experiment beats a big estimate

1. Choose one repeatable workflow and freeze its model and prompt-template version.
2. Run it with a stable instruction prefix and record the provider's cache fields.
3. Change only the task-specific suffix, run it again, and inspect the fields.
4. Compare total request cost and latency against the same workflow before the prompt change.

If the provider reports no cached tokens, call it a miss. If it reports cached tokens, use the model's current cached-input rate for those tokens, its standard input rate for the rest, and its output rate for generated tokens. Do not apply a blanket percentage discount to the whole request.

Keep the prefix stable, not stale

Put reusable instructions before volatile material. Move timestamps, current task details, and short-lived retrieval results later in the prompt where your application allows it. Still update safety and product instructions when they need to change. A cache optimization is not a reason to preserve wrong context.

FAQ

Does every provider cache the same way?

No. Check the provider's current documentation and usage fields for the model and API you're using.

Can caching fix a bloated prompt?

It can reduce the price of some reused input. It does not make irrelevant context useful or remove the need to validate output quality.

Which metric matters most?

Cost per accepted task. Cache hits are a useful diagnostic, not the final outcome.