OpenAI Boosts GPT-6 Prompt Caching for Faster, Cheaper Agents

Overview

OpenAI has released a significant upgrade to its prompt caching system for the GPT‑6 model family, aiming to reduce latency and lower costs for developers building persistent AI agents. The update promises higher cache hit rates by default and introduces new diagnostic tools to help engineers optimize their integrations.

The improvements are designed specifically for applications that run complex, long-running tasks. By efficiently reusing shared context across multiple API requests, OpenAI is enabling agents to handle extensive operations—such as refactoring entire codebases or generating researched documents—more economically than before.

“OpenAI’s prompt caching plays a critical role in helping GitHub Copilot deliver fast, efficient experiences at scale. Over the past several months, we’ve reduced by more than 50% the share of prompt tokens requiring fresh processing across billions of requests.” — Mario Rodriguez, Chief Product Officer, GitHub

How Prompt Caching Works

Persistent agents typically make a series of API requests that build upon one another. These requests often carry forward identical instructions, tool definitions, and contextual information from earlier turns. OpenAI’s caching system identifies this shared context and stores it, allowing the model to reuse computation rather than reprocessing the same tokens for every single request.

With the GPT‑6 family, the underlying engine now delivers higher cache hit rates by default. A key feature of this update is the expansion of the discount window; eligible shared prefixes are now reused within a 30-minute window. Developers can receive discounts of up to 90% on cached input tokens, significantly reducing the cost of inference for applications that maintain long sessions.

This architecture allows applications to scale efficiently. Instead of paying full price for every token sent in a request, developers only pay full price for the new or changed portions of the prompt. This economic model is fundamental to the viability of agents that run for hours rather than seconds.

New Monitoring and Diagnostic Tools

To help developers maximize these benefits, OpenAI is introducing two primary tools focused on visibility and optimization.

Prompt Caching Dashboard

The new Prompt Caching Dashboard provides a comprehensive view of how an application’s input is being served. Key features include:

Hit rate tracking over time: Monitor performance trends to identify when cache efficiency drops.
Input composition charts: Compare cached versus uncached tokens visually to understand what part of the prompt is being reused.

These views allow teams to spot anomalies quickly and evaluate how specific code changes impact caching performance.

Cache Miss Diagnostics

When an unexpected cache miss occurs, the prompt caching diagnostics tool helps engineers understand the root cause. By comparing a request with a recent response, developers can identify specific changes that prevented reuse, such as alterations to:

The model version.
Tool definitions or schemas.
Request settings.
Input content.

The tool provides an estimated number of affected tokens, helping teams assess the scale of the impact and decide where to optimize their integration.

{
  "prompt_cache_diagnostics": {
    "type": "cache_miss",
    "reason": "tools_changed",
    "comparison_reusable_tokens": 5629,
    "cache_missed_tokens": 5629
  }
}

Optimizing Application Performance

OpenAI has also updated its guidance to help developers fine-tune their caching strategies. The refreshed prompt caching guide explains how to implement explicit cache breakpoints and manage context retention.

Explicit Cache Breakpoints

Developers can now choose exactly which parts of a prompt to cache using explicit cache breakpoints. This allows for granular control over what is considered static versus dynamic. Stable context, such as system instructions, can be cached while frequently changing content remains at the end of the prompt to avoid invalidating the cache.

Adjusting Reasoning Effort

A significant flexibility improvement on GPT‑6 models allows developers to change reasoning effort between responses without breaking the cache.

Previously, altering the reasoning level might have invalidated shared prefixes. Now, developers can raise effort for a difficult sub-task or lower it for a routine follow-up by appending a configuration_update while leaving the request-level reasoning effort unchanged. This preserves reusable context while adapting to the computational needs of the specific step.

Managing Tool Changes

As agent capabilities evolve, tool definitions often change. To preserve cache hits during these updates, OpenAI recommends keeping tool definitions, schemas, and their ordering stable.

Techniques to maintain cache integrity include:

Using allowed_tools to make only relevant tools callable for a specific request.
Setting tool_choice to none when no tools are needed, rather than removing the tool definitions entirely.
Using new developer messages to append updated instructions at the end of the context, overriding older ones without shifting the static prefix.

Cache Prewarming

For applications with known initial contexts, prewarming allows teams to prepare data ahead of time. An application can prewarm shared instructions, tool definitions, or reference material during startup, before the user submits their first query. This moves processing time out of the user’s wait period, resulting in faster time-to-first-response.

Industry Impact and Case Studies

Early adopters of the GPT‑6 caching improvements have reported substantial gains in efficiency and cost reduction.

GitHub Copilot

GitHub has integrated these caching improvements deeply into its infrastructure. Mario Rodriguez noted that the company reduced the share of prompt tokens requiring fresh processing by more than 50% relative to previous baselines. This has resulted in a more efficient inference stack and noticeably faster responses for millions of developers using Copilot.

Manus

Manus, a company building autonomous agents, worked directly with OpenAI’s engineering team to refine cache breakpoint placement. By combining explicit and automatic caching and analyzing real request data, they identified and fixed unexpected cache misses. Within less than a week, their cache hit rate rose from approximately 85% to consistently above 90%, further lowering inference costs in production.

Strawberry Browser

Arian Hanifi, CTO of Strawberry Browser, highlighted the economic viability of forking conversations for background tasks. By using alerts to detect caching breaks and Codex agents to diagnose root causes, they improved hit rates by a few percentage points, leading to a 20% reduction in costs. Explicit breakpoints allow them to cache stable context while keeping dynamic content flexible.

Wordsmith

Eugene Mikhantyev, an AI Engineer at Wordsmith, reported that moving session agents to explicit cache breakpoints raised evaluation hit rates from 83% to 91% in under a week. The optimizations resulted in cache writes falling by roughly two-thirds and inference costs dropping by 36% for the same workload.

Getting Started

Developers can begin optimizing their GPT‑6 integrations immediately using the new tools and guidelines. The recommended steps include monitoring cache hit rates in the new dashboard, investigating any unexpected misses with the diagnostics tool, and following the updated prompt caching guide to adjust breakpoint placements. Teams can also use Codex to review existing integrations and identify opportunities for further efficiency gains.