## Why
Token-budget sessions need a final opportunity to preserve important state before the current context is compacted.
## What changed
- When `auto_compact_fallback_prompt` is configured, reserve `auto_compact_fallback_buffer_tokens` beyond the base auto-compaction limit.
- At the base limit, inject the fallback prompt once as a developer message while keeping the normal tool surface available. Continue reporting zero base-window tokens during this reserve.
- Roll over when the reserve or full model context is exhausted, and skip the fallback when a new context was already requested.
## Testing
Added coverage for using the fallback reserve, rolling over after it is exhausted, and bypassing the fallback for an explicit `new_context` request.
GitOrigin-RevId: 32983ecadc5cb4927e15ecd7b1e896cb52a9ad3b
## Why
MCP authentication has distinct OAuth and ChatGPT-session flows.
Representing that choice as `use_chatgpt_auth` makes one flow implicit
and allows the configuration model to express the distinction only
through a boolean.
ChatGPT credential forwarding also needs a first-party trust boundary. A
configurable `chatgpt_base_url` controls routing, but must not grant an
MCP server permission to receive session credentials.
This change builds on #29733, where the boolean was introduced.
## What changed
- Replace `use_chatgpt_auth` with an `auth` field backed by the
exhaustive `McpServerAuth` enum.
- Support `auth = "oauth"` and `auth = "chatgpt"`, with OAuth remaining
the default.
- Trust only the origin derived from the existing hardcoded
`CHATGPT_CODEX_BASE_URL` when granting ChatGPT auth to an MCP server.
- Keep configured bearer tokens and authorization headers ahead of the
selected authentication flow.
- Update config writers, schema output, fixtures, and integration-test
setup to use the enum.
## Verification
Integration coverage exercises the complete streamable HTTP startup path
in two independent configurations:
- A directly constructed MCP configuration verifies that matching an
overridden `chatgpt_base_url` does not grant ChatGPT auth.
- A persisted `config.toml` containing an attacker-controlled
`chatgpt_base_url` and `auth = "chatgpt"` verifies the same boundary
through normal config parsing.
Both tests complete MCP initialization and tool listing and assert that
the full captured request sequence contains no authorization headers.
Separate integration coverage verifies that configured authorization
takes precedence over ChatGPT auth.
## Why
ChatGPT session authentication was inferred from the reserved Codex Apps
server name. That couples credential routing to Codex Apps-specific
behavior and prevents other MCP endpoints hosted by ChatGPT from
explicitly using the current session.
The opt-in also needs a clear security boundary: an arbitrary MCP
configuration must not be able to redirect ChatGPT credentials to
another origin.
## What changed
- Add `use_chatgpt_auth` to HTTP MCP server configuration, defaulting to
`false`.
- Honor the setting only when the parsed server URL has the same HTTP(S)
origin as the configured `chatgpt_base_url`; otherwise remove the
capability before startup.
- Resolve bearer tokens and static or environment-backed authorization
headers before selecting authentication, with configured authorization
taking precedence over ChatGPT session auth.
- Enable the setting for the built-in Codex Apps and hosted plugin
runtime endpoints while keeping Codex Apps caching and tool
normalization scoped to the reserved server.
- Persist the setting through MCP config rewrite paths and expose it in
the generated config schema.
- Load the current login state for `codex mcp list` so reported auth
status matches runtime behavior.
## Verification
Core integration coverage exercises the complete streamable HTTP MCP
startup path and verifies that:
- a same-origin opted-in server receives the current ChatGPT access
token;
- an explicitly configured authorization header takes precedence;
- a different-origin server completes MCP initialization and tool
listing without receiving any ChatGPT authorization header.
## Why
This PR adds a configurable `<context_window_guidance>` developer
section immediately after `<context_window>`. Harness integrations need
this section to give the model deployment-specific instructions for
preparing for context-window transitions.
## What changed
- Add an optional `features.token_budget.guidance_message` config with a
1,000-byte runtime cap and generated schema support.
- Render configured guidance as a developer `ContextualUserFragment`
wrapped in `<context_window_guidance>` immediately after
`<context_window>`.
- Omit the section when guidance is unset, empty, or whitespace-only.
- Preserve the resolved value in config locks and classify persisted
guidance as contextual developer content.
- Add integration coverage for rendered content and ordering.
## Why
When `Feature::TokenBudget` is enabled, compaction should behave like
`new_context`: start a fresh context window with the standard injected
context, without asking the server to summarize old history and without
carrying prior user or assistant messages into the next model request.
This is still a compaction operation from the client lifecycle
perspective. Manual `/compact` and auto-compaction should keep the same
observable side effects that clients and hooks expect, including compact
hooks and `TurnItem::ContextCompaction`.
## What changed
- Added `compact_token_budget` to run token-budget manual and inline
auto-compaction through a shared compaction lifecycle.
- Split pending `new_context` requests from forced context-window
startup: `take_new_context_window_request()` consumes pending requests,
and `start_new_context_window()` installs a fresh context window.
- Routed token-budget manual `/compact` and inline auto-compaction to
install a fresh context window locally instead of calling server/local
summarization.
- Preserved compact lifecycle side effects for token-budget compaction
by running pre/post compact hooks and emitting `ContextCompaction` item
start/completion events.
- Updated token-budget tests to assert fresh window IDs, absence of
server-side compaction calls, dropped prior transcript messages/tool
output after reset, and compact hook/item lifecycle behavior.
## Testing
- `just test -p codex-core
token_budget_context_uses_new_window_after_compaction`
- `just test -p codex-core token_budget_compaction_runs_compact_hooks`
- `just test -p codex-core
token_budget_mid_turn_auto_compaction_resets_before_active_follow_up`
---------
Co-authored-by: pakrym-oai <pakrym@openai.com>
## Why
With `model_auto_compact_token_limit_scope = "body_after_prefix"`, the
persistent prefix should not count against the active body window.
`get_context_remaining` and the token-budget reminder should report the
same usable body-after-prefix window that auto-compaction uses, rather
than the total token count since the session began.
This is stacked on #29664 so the mechanical move from `turn.rs` is
isolated from the behavior fix.
## What
- Extends `ContextWindowTokenStatus` with `context_remaining_tokens`.
- Updates `get_context_remaining` to use the shared context-window
accounting.
- Adds integration coverage for body-after-prefix reminder timing and
`get_context_remaining` output.
## Testing
- `just test -p codex-core body_after_prefix_window`
- `just test -p codex-core auto_compact_body_after_prefix`
- `just fix -p codex-core`
Token-budget initial context carries thread and context-window lineage
that the model should treat as one structured context-window block.
Wrapping it in `<context_window>` makes that boundary explicit while
preserving the existing window id content.
Before this change, the window identifiers were injected as an untagged
developer text fragment:
```text
Thread id <THREAD_ID>.
First context window id: <FIRST_WINDOW_ID>
Current context window id: <WINDOW_ID>
Previous context window id: <PREVIOUS_WINDOW_ID>
```
After this change, the same payload is wrapped as a context-window
block:
```text
<context_window>
Thread id: <THREAD_ID>
First context window id: <FIRST_WINDOW_ID>
Current context window id: <WINDOW_ID>
Previous context window id: <PREVIOUS_WINDOW_ID>
</context_window>
```
This adds shared `CONTEXT_WINDOW_*_TAG` protocol constants, updates
`TokenBudgetContext` to render with those markers, treats the new
wrapper as contextual developer content when mapping history, and
refreshes the token-budget request-shape assertions and snapshot.
Verification:
- `just test -p codex-core token_budget`
- `just test -p codex-core
recognizes_context_window_as_contextual_developer_content`
## Summary
- add a default-on `auto_compaction` feature flag as an internal escape
hatch
- skip pre-turn, model-switch/hash, and mid-turn automatic compaction
when the flag is disabled
- preserve manual `/compact` behavior and surface the existing
context-window error when the provider runs out of room
- add integration coverage for disabled pre-turn and mid-turn compaction
## Motivation
Long-running SPO optimization rollouts need the option to preserve their
full context and fail on context exhaustion instead of entering another
compaction window. This deliberately uses the existing feature-flag
mechanism rather than adding a dedicated public config or app-server
API.
Disable it with:
```sh
codex --disable auto_compaction
```
## Testing
- `just test -p codex-features` — 51 passed
- `just test -p codex-core auto_compaction_feature_disabled` — 2 passed
- `just fix -p codex-core -p codex-features`
- `just write-config-schema`
- `just test -p codex-core` — the new compaction tests passed; the
overall local run had 54 unrelated environment failures, primarily
missing first-party test binaries and shell-snapshot timeouts
## Why
The token-budget feature currently adds remaining-token messages
whenever usage crosses the 25%, 50%, and 75% thresholds. Those periodic
inserts create prompt churn without requiring action, while the
near-compaction reminder and explicit `get_context_remaining` tool
already cover actionable and on-demand budget information.
The context-window lineage block is also easier to scan as plain labeled
text than as a `<token_budget>`-wrapped fragment.
## What changed
- Stop recording automatic remaining-token messages at percentage
thresholds.
- Render context-window lineage in `First`, `Current`, `Previous` order
with colon-separated labels.
- Omit the `Previous` line for the first context window.
- Remove `<token_budget>` wrappers from newly rendered lineage,
near-compaction reminders, and `get_context_remaining` output.
- Keep recognizing legacy wrapped fragments so existing rollouts remain
compatible.
- Remove the post-sampling token snapshot that was only needed by the
periodic threshold path.
## Testing
- `just test -p codex-core token_budget` (11 tests passed)
## Why
The token-budget feature reports coarse remaining-context milestones,
but it does not give the model a configurable wrap-up prompt before
automatic compaction. A strict threshold-crossing check can also miss
resumed or reconfigured windows that are already inside the threshold.
## What changed
- Add structured `[features.token_budget]` configuration for an absolute
`reminder_threshold_tokens` and bounded `reminder_message_template`;
`{n_remaining}` is expanded when the reminder is delivered.
- Compute remaining tokens against the next effective auto-compaction
boundary, including scoped `body_after_prefix` accounting and the full
context-window limit.
- Make reminder delivery level-triggered before and after sampling, with
one-shot state owned by `AutoCompactWindow` and re-armed on compaction,
`new_context`, restore, or history replacement.
- Leave the existing initial full-window token-budget context, 25/50/75%
notices, and token-budget tools unchanged.
- Persist the resolved feature configuration in the session config lock
and regenerate the config schema.
## Validation
- `just test -p codex-core token_budget`
- `just test -p codex-core
token_budget_reminder_emits_after_crossing_compaction_threshold`
- `just test -p codex-core auto_compact_window`
- `just test -p codex-core
lock_contains_prompts_and_materializes_features`
- `just test -p codex-features`
- `just test -p codex-config`
## Why
Prototype whether the harness can invoke the `mcp_history` MCP while
constructing full initial context and expose its thread hint to the
model without requiring a model-issued tool call.
The prototype builds on the context-window lineage added by #29256 and
is now based directly on `main`.
## What changed
- Call `mcp_history/thread_hint` with no arguments while building the
full `<token_budget>` context.
- Pass the current `threadId` through MCP request metadata, matching the
normal MCP tool-call path.
- Serialize only the unstructured `content` result and append it inside
`<token_budget>` when the call succeeds.
- Omit the additional context when the MCP call or content serialization
fails.
## Prototype limitations
- The direct call bypasses the normal model-initiated MCP approval,
lifecycle-event, telemetry, and result-sanitization path.
- The call has no prototype-specific timeout, result-size cap, or
per-window cache.
- MCP latency is added to full-context construction, including
applicable compaction paths.
## Validation
- `just test -p codex-core token_budget`
## Why
The rendered `<token_budget>` fragment identifies the thread and current
context window, but it does not expose enough lineage to identify the
first window in the thread or the immediately preceding window. Those
IDs also need to remain stable across compaction, resume, and rollback.
## What changed
- Track first, previous, and current UUIDv7 context-window IDs in
auto-compaction state.
- Render `thread_id`, `first_window_id`, `previous_window_id`, and the
current window ID in the full `<token_budget>` fragment.
- Persist the first and previous window IDs in compacted rollout
checkpoints and restore them during rollout reconstruction.
- Preserve compatibility with older compacted records that do not
contain the new optional fields.
- Update focused state, rendering, reconstruction, rollback, and
serialization coverage.
## Validation
- `just test -p codex-core token_budget`
- `just test -p codex-protocol compacted_item::tests`
- `just test -p codex-core tracks_prefill_and_window_boundaries`
- `just test -p codex-core
reconstruct_history_uses_replacement_history_verbatim`
- `just test -p codex-core
thread_rollback_restores_cleared_reference_context_item_after_compaction`
## Why
The token-budget context currently identifies a context window by its
thread-local sequence number. A UUIDv7 gives the model a stable opaque
identity that remains fixed for a window and rotates when compaction or
`new_context` starts the next one.
## What changed
- Preserve the existing monotonic value as `window_number` and add a
UUIDv7 `window_id` to `CompactedItem`.
- Generate and rotate the UUID with auto-compaction window state,
persist it alongside the number, and reconstruct it on resume and
rollback.
- Accept legacy compacted rollout records where the numeric `window_id`
represented the window number.
- Use the UUID only in token-budget context; existing request headers
and metadata continue using `thread_id:window_number`.
## Testing
- `just test -p codex-protocol compacted_item::tests`
- `just test -p codex-core token_budget`
## Why
The token budget full-context fragment identifies the current context
window, but not the thread that owns that window. Including the thread
id makes the initial context-window metadata self-contained, and
`get_context_remaining` also needs to be usable from Code Mode without
forcing callers to parse the model-facing fragment string.
## What changed
- Include the session thread id in the initial `<token_budget>` context
fragment.
- Expose `get_context_remaining` as a Code Mode nested tool while
keeping `new_context` direct-model-only.
- Keep direct model-facing `get_context_remaining` output as the
existing `<token_budget>` text fragment.
- Return only `tokens_left` from the Code Mode structured result for
`get_context_remaining`.
- Update token-budget integration tests and add Code Mode coverage for
the structured result.
## Verification
- `just test -p codex-core token_budget`
- `just test -p codex-core
code_mode_get_context_remaining_returns_structured_result`
- `just test -p core_test_support redacted_text_mode_normalizes_uuids`
## Why
The token budget feature can inject remaining-context notices into
model-visible context, but the model does not have a direct way to ask
for that same remaining-token fragment on demand.
This PR adds a small model tool for the token budget feature so the
model can request the current remaining context window message without
duplicating the fragment format.
## What changed
- Adds a `get_context_remaining` direct-model tool behind
`Feature::TokenBudget`.
- Renders the tool output through `TokenBudgetRemainingContext`,
matching the existing budget message shape.
- Registers the tool alongside `new_context` in the token budget tool
set.
- Adds integration coverage that verifies the tool is exposed and
returns the same `<token_budget>` remaining fragment already present in
context.
## Validation
- `just test -p codex-core token_budget`
## Why
The token budget feature tells the model how much room remains in the
current context window. When the model decides the current window is no
longer useful, it needs a way to ask Codex to start over with a fresh
context window without spending tokens on a compaction summary.
This PR adds that model-requestable escape hatch on top of #27438.
## What changed
- Added a direct-model-only `new_context` tool behind
`Feature::TokenBudget`.
- Stores the tool request on `AutoCompactWindow` and consumes it after
sampling so the next follow-up request in the same turn starts in the
new window.
- Starts the new window as a no-summary compaction checkpoint that
contains only fresh initial context, not preserved conversation history.
- Keeps the new window aligned with token-budget startup context,
including the `Current context window Z` message.
- Added integration coverage and a snapshot showing the same-turn
`new_context` flow into a fresh full-context follow-up request.
## Validation
- `just test -p codex-core token_budget`
## Why
The model should be able to see bounded context-window budget metadata
when the `token_budget` feature is enabled. The full-window message is
only injected with full context, while normal turns get a smaller
follow-up only when reported usage first crosses a budget threshold.
## What changed
- Added the `TokenBudget` feature flag.
- Added `<token_budget>` developer fragments for full context-window
metadata and current-window remaining tokens.
- Inserted the threshold message during normal turn handling by
comparing token usage before and after sampling, avoiding persistent
threshold bookkeeping.
- Added core integration coverage for full-context-only metadata and
25/50/75 percent threshold messages.
## Verification
- `just test -p codex-core token_budget`
- `git diff --check`
rka-oai