jif 816fa702ec Account for token usage from incomplete responses (#53670) ...

## Why

Terminal `response.incomplete` errors discard reported token usage. Counting that usage as a new context baseline can also double-count input and trigger unnecessary compaction on the next turn.

## What changed

- Carry the response ID and valid reported usage with incomplete-response errors.
- Record usage for sampling and foreground compaction in cumulative totals, rollout records, budgets, and token-usage contributors without replacing the context baseline. Preserve the original terminal error, including when accounting exhausts the budget or cancellation races the error.
- Charge incomplete background compaction requests to the rollout budget without publishing foreground token usage.
- Ignore missing or malformed usage rather than estimating it.

## Testing

Extend coverage for SSE and WebSocket incomplete responses, rollout usage records, compaction accounting, deferred token-count events, and token-usage contributors. Add regressions for unnecessary compaction on the next turn and background compaction publishing foreground usage.

GitOrigin-RevId: bf1ee6df3912052ab7c19e1c6af8ad3dc4b07675

2026-10-13 13:36:24 +00:00

..

agent

Keep spawned children provisional until the spawn result returns (#53654)

2026-10-13 12:42:53 +00:00

apps

Make Apps guidance react to MCP availability (#30226)

2026-07-06 14:29:19 -07:00

config

Allow hiding native tools while retaining the multi-agent V2 runtime (#53617)

2026-10-13 09:06:39 +00:00

context

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

context_manager

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

exec_policy

Remove the patched zsh shell execution backend (#52160)

2026-10-08 16:19:09 +00:00

guardian

Record truncation metrics for accepted Guardian review inputs (#53628)

2026-10-13 10:21:42 +00:00

mcp_tool_call

Enable parallel execution for read-only tools (#52245)

2026-10-08 19:29:47 +00:00

plugins

Attribute command and plugin analytics to the invoking model (#45445)

2026-09-14 15:30:22 +00:00

realtime_conversation

Expose and persist turn lineage across the app server (#51415)

2026-10-06 16:56:39 +00:00

realtime_history

Record realtime conversation history in Core (#41924)

2026-08-31 22:38:23 +00:00

sandboxing

Run sandbox integrity checks before commands and filesystem operations (#51843)

2026-10-07 21:52:43 +00:00

session

Account for token usage from incomplete responses (#53670)

2026-10-13 13:36:24 +00:00

state

Add infrastructure for speculative remote V2 compaction (#53614)

2026-10-13 08:44:23 +00:00

tasks

Account for token usage from incomplete responses (#53670)

2026-10-13 13:36:24 +00:00

thread_manager

Preserve agent state when unloading idle multi-agent v2 children (#52034)

2026-10-08 09:39:33 +00:00

tools

Extract shared multi-agent v2 thread spawning (#53660)

2026-10-13 12:59:15 +00:00

unified_exec

Preserve remote exec output until it is collected (#53622)

2026-10-13 09:34:17 +00:00

utils

Count serialized JSON bytes without allocating strings (#40604)

2026-08-25 13:10:24 +00:00

agent_communication.rs

Preserve queued agent mail across session eviction (#50087)

2026-10-01 20:17:34 +00:00

agent_message_board.rs

Add explicit local board connections with fixed runtime access (#53400)

2026-10-12 14:44:22 +00:00

agents_md_manager.rs

Roll back stale Guardian reviews before reusing reviewer history (#51683)

2026-10-07 13:05:06 +00:00

agents_md_tests.rs

Bind capability roots to environment selections (#51493)

2026-10-06 22:39:32 +00:00

agents_md.rs

Reject paths in project documentation fallback filenames (#45865)

2026-09-16 04:59:23 +00:00

apply_patch_tests.rs

Fix macOS system-alias matching in patch permission checks (#47879)

2026-09-24 16:36:57 +00:00

apply_patch.rs

Fix macOS system-alias matching in patch permission checks (#47879)

2026-09-24 16:36:57 +00:00

attestation.rs

…

client_common_tests.rs

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

client_common.rs

Send incremental tool catalog updates in Responses Lite (#50540)

2026-10-03 05:59:07 +00:00

client_tests.rs

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

client_tool_metadata_tests.rs

Send base instructions as Responses input messages (#51156)

2026-10-05 18:52:51 +00:00

client_tool_metadata.rs

Bound tool-call observations to the outgoing Responses message budget (#47957)

2026-09-24 22:54:29 +00:00

client.rs

Prepare compaction helpers for background requests (#53606)

2026-10-13 08:13:34 +00:00

code_mode_host_tests.rs

Verify reset notification during code-mode host recovery (#52952)

2026-10-10 18:26:18 +00:00

code_mode_host.rs

Add opt-in gRPC over stdio for the code-mode host (#52723)

2026-10-09 23:26:34 +00:00

codex_delegate_tests.rs

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

codex_delegate.rs

Restore saved thread permissions during configuration loading (#53419)

2026-10-12 16:09:41 +00:00

codex_thread.rs

Remove per-content source attribution metadata (#52329)

2026-10-08 23:57:36 +00:00

command_canonicalization_tests.rs

…

command_canonicalization.rs

…

compact_model_fallback.rs

Treat incomplete responses as terminal errors (#53451)

2026-10-12 18:53:49 +00:00

compact_remote_history_tests.rs

Remove the unused legacy remote compaction implementation (#44273)

2026-09-09 18:40:53 +00:00

compact_remote_history.rs

Prepare compaction helpers for background requests (#53606)

2026-10-13 08:13:34 +00:00

compact_remote_v2_attempt.rs

Reuse background results for remote V2 compaction (#53620)

2026-10-13 09:18:45 +00:00

compact_remote_v2_image_budget_tests.rs

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

compact_remote_v2_images.rs

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

compact_remote_v2_installation.rs

Reuse background results for remote V2 compaction (#53620)

2026-10-13 09:18:45 +00:00

compact_remote_v2.rs

Account for token usage from incomplete responses (#53670)

2026-10-13 13:36:24 +00:00

compact_retained_tool_outputs.rs

Retain client-marked tool outputs across compaction (#52937)

2026-10-10 17:30:25 +00:00

compact_tests.rs

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

compact_token_budget.rs

Install full context in compaction replacement history (#51117)

2026-10-05 16:10:27 +00:00

compact.rs

Account for token usage from incomplete responses (#53670)

2026-10-13 13:36:24 +00:00

connectors_tests.rs

Honor app link settings for MCP tool approvals (#42056)

2026-09-01 13:43:27 +00:00

connectors.rs

Allow exec-server to proxy permitted private IPs upstream (#48568)

2026-09-26 23:17:41 +00:00

current_time.rs

[codex] route sleep through time providers (#29973)

2026-06-24 22:17:43 -07:00

cyber_access_program.rs

Forward per-turn Cyber access programs to Guardian (#52689)

2026-10-09 20:14:04 +00:00

elicitation_tests.rs

elicitations: Move to shared ElicitationService (#30627)

2026-07-06 11:20:32 -07:00

elicitation.rs

elicitations: Move to shared ElicitationService (#30627)

2026-07-06 11:20:32 -07:00

environment_selection_snapshot_tests.rs

Bind capability roots to environment selections (#51493)

2026-10-06 22:39:32 +00:00

environment_selection.rs

Refresh persisted capability roots from owner-provided configuration (#52676)

2026-10-09 19:06:34 +00:00

event_mapping_tests.rs

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

event_mapping.rs

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

exec_env_tests.rs

Make apply_patch preserve line endings unconditionally (#51203)

2026-10-05 23:52:06 +00:00

exec_env.rs

Expose originating tool call metadata in command environments (#52119)

2026-10-08 13:45:19 +00:00

exec_policy_tests.rs

Remove selected core test cases (#42066)

2026-09-01 14:43:27 +00:00

exec_policy_windows_tests.rs

Retire the untrusted approval policy (#39630)

2026-08-20 07:03:02 +00:00

exec_policy.rs

Remove the patched zsh shell execution backend (#52160)

2026-10-08 16:19:09 +00:00

exec_tests.rs

Run sandbox integrity checks before commands and filesystem operations (#51843)

2026-10-07 21:52:43 +00:00

exec.rs

Run sandbox integrity checks before commands and filesystem operations (#51843)

2026-10-07 21:52:43 +00:00

feedback_config_tests.rs

Add configuration and feature diagnostics to report metadata (#46501)

2026-09-18 23:07:36 +00:00

feedback_config.rs

Add configuration and feature diagnostics to report metadata (#46501)

2026-09-18 23:07:36 +00:00

function_tool.rs

Complete dynamic tool lifecycles on cancellation (#51556)

2026-10-07 03:07:51 +00:00

git_info_tests.rs

Use descriptor-safe helpers for executable test fixtures (#50018)

2026-10-01 15:57:03 +00:00

guardian_review.rs

Add opt-in conversation history retrieval to Guardian reviews (#49036)

2026-09-28 17:29:52 +00:00

hook_mcp_executor.rs

Remove per-content source attribution metadata (#52329)

2026-10-08 23:57:36 +00:00

hook_runtime.rs

Remove per-content source attribution metadata (#52329)

2026-10-08 23:57:36 +00:00

image_preparation_tests.rs

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

image_preparation.rs

Remove per-content source attribution metadata (#52329)

2026-10-08 23:57:36 +00:00

installation_id.rs

…

lib.rs

Extract shared multi-agent v2 thread spawning (#53660)

2026-10-13 12:59:15 +00:00

mcp_openai_file.rs

Bind permission grants to the originating turn (#49880)

2026-10-01 05:33:36 +00:00

mcp_skill_dependencies.rs

Add configurable default plugin activation policy (#52220)

2026-10-08 18:42:58 +00:00

mcp_tool_approval_templates.rs

Organize bundled Rust resources under asset directories (#41477)

2026-08-29 06:05:28 +00:00

mcp_tool_call_tests.rs

Add configurable default plugin activation policy (#52220)

2026-10-08 18:42:58 +00:00

mcp_tool_call.rs

Expose a verified sandbox launcher to local MCP servers (#52182)

2026-10-08 17:25:09 +00:00

mcp_tool_exposure_test.rs

Reuse MCP handlers across sampling steps (#37273)

2026-08-06 13:37:07 +00:00

mcp_tool_exposure.rs

Reuse MCP handler metadata for cache comparisons (#51755)

2026-10-07 16:35:05 +00:00

mcp.rs

Add configurable default plugin activation policy (#52220)

2026-10-08 18:42:58 +00:00

memory_usage.rs

Tag memory usage telemetry with the memory version (#45960)

2026-09-16 15:17:41 +00:00

mention_syntax.rs

…

model_request_tests.rs

Preserve WebSocket continuations when steering a turn (#48508)

2026-09-26 18:36:55 +00:00

model_request.rs

Add extension hooks for model requests and response streams (#47679)

2026-09-23 22:10:04 +00:00

network_policy_decision_tests.rs

fix: attribut network requests to the exact exec on linux (#29697)

2026-07-06 14:51:34 -07:00

network_policy_decision.rs

…

original_image_detail.rs

…

otel_init.rs

Honor custom OTLP metrics exporters when analytics is disabled (#52278)

2026-10-08 21:18:24 +00:00

prompt_debug.rs

Preserve tool declaration mode across resumed context windows (#51480)

2026-10-06 21:28:13 +00:00

realtime_context_tests.rs

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

realtime_context.rs

Add experimental app-server project APIs (#38940)

2026-08-17 04:01:37 +00:00

realtime_conversation_tests.rs

Add completion-aware realtime attachment and session-scoped detach (#51539)

2026-10-07 01:28:59 +00:00

realtime_conversation.rs

Expand realtime v3 voice support (#52363)

2026-10-09 02:04:35 +00:00

realtime_history_tests.rs

Record resolved model and reasoning effort in sub-agent activity (#51463)

2026-10-06 19:29:40 +00:00

realtime_history.rs

Record realtime conversation history in Core (#41924)

2026-08-31 22:38:23 +00:00

realtime_prompt.rs

[codex] Consolidate shared prompts in codex-prompts (#25151)

2026-06-01 18:45:07 +00:00

responses_headers.rs

Route Guardian requests through /responses with identifying headers (#45736)

2026-09-15 17:09:11 +00:00

responses_metadata.rs

Track history initialization in Responses turn metadata (#52325)

2026-10-08 23:34:16 +00:00

responses_retry_tests.rs

Move content-filter guidance into the shared Responses retry handler (#49130)

2026-09-29 02:05:56 +00:00

responses_retry.rs

Honor server retry advice across Responses retries and fallback (#49441)

2026-09-30 02:12:42 +00:00

rollout_budget.rs

Expose a backend-independent agent control contract (#46547)

2026-09-19 01:03:55 +00:00

rollout.rs

Honor the configured SQLite home across state consumers (#34994)

2026-07-23 19:19:35 +00:00

safety_tests.rs

Fix macOS system-alias matching in patch permission checks (#47879)

2026-09-24 16:36:57 +00:00

safety.rs

Fix macOS system-alias matching in patch permission checks (#47879)

2026-09-24 16:36:57 +00:00

sandbox_tags_tests.rs

Report configured sandbox policy consistently (#41933)

2026-08-31 23:29:20 +00:00

sandbox_tags.rs

Enable MXC selection through Windows sandbox configuration (#46271)

2026-09-17 18:20:28 +00:00

session_prefix_tests.rs

Notify parent agents when Guardian stops a subagent (#49312)

2026-09-29 16:53:11 +00:00

session_prefix.rs

Notify parent agents when Guardian stops a subagent (#49312)

2026-09-29 16:53:11 +00:00

session_rollout_init_error.rs

Preserve sandbox errors during session initialization (#40381)

2026-08-24 10:56:54 +00:00

shell_snapshot_sandbox_tests.rs

Preserve sandbox overrides across execution requests (#51786)

2026-10-07 17:59:22 +00:00

shell_snapshot_sandbox.rs

Run sandbox integrity checks before commands and filesystem operations (#51843)

2026-10-07 21:52:43 +00:00

shell_snapshot_tests.rs

Add opt-in credential masking for proxied sandboxed sessions (#52302)

2026-10-08 22:27:00 +00:00

shell_snapshot.rs

Expose originating tool call metadata in command environments (#52119)

2026-10-08 13:45:19 +00:00

shell_tests.rs

Restore executor tool paths after login shell startup (#49467)

2026-09-30 03:22:55 +00:00

shell.rs

Restore executor tool paths after login shell startup (#49467)

2026-09-30 03:22:55 +00:00

skills.rs

Use PathUri for skill identity and path matching (#51482)

2026-10-06 21:29:35 +00:00

spawn.rs

Remove the patched zsh shell execution backend (#52160)

2026-10-08 16:19:09 +00:00

state_db_bridge.rs

…

stream_events_utils_tests.rs

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

stream_events_utils.rs

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

test_support.rs

Default local threads to paginated history (#47900)

2026-09-24 18:07:37 +00:00

thread_manager_tests.rs

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

thread_manager.rs

Restore saved thread permissions during configuration loading (#53419)

2026-10-12 16:09:41 +00:00

thread_rollout_truncation_tests.rs

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

thread_rollout_truncation.rs

Remove partial-history subagent forks (#51329)

2026-10-06 10:39:16 +00:00

thread_spawn.rs

Extract shared multi-agent v2 thread spawning (#53660)

2026-10-13 12:59:15 +00:00

thread_startup_metadata.rs

Remove initial messages from session configuration events (#50360)

2026-10-02 16:55:41 +00:00

turn_diff_tracker_tests.rs

Render turn diffs for foreign environment paths (#34654)

2026-07-22 02:19:45 +00:00

turn_diff_tracker.rs

Render turn diffs for foreign environment paths (#34654)

2026-07-22 02:19:45 +00:00

turn_extension_data.rs

Capture host-supplied extension data for each turn (#49846)

2026-10-01 03:31:48 +00:00

turn_metadata_tests.rs

Track history initialization in Responses turn metadata (#52325)

2026-10-08 23:34:16 +00:00

turn_metadata.rs

Preserve review targets and scope misalignment continuation metadata (#51217)

2026-10-06 01:26:11 +00:00

turn_timing_tests.rs

Add opt-in output token replay for OpenAI requests (#52742)

2026-10-10 01:25:00 +00:00

turn_timing.rs

Track inference tool changes in turn analytics (#50964)

2026-10-04 21:19:05 +00:00

user_shell_command_tests.rs

…

user_shell_command.rs

Snapshot resolved settings for each model step (#40651)

2026-08-25 17:41:22 +00:00

util_tests.rs

Trim TUI legacy core helper usage (#22695)

2026-05-14 16:54:59 -07:00

util.rs

Move retry backoff into codex-async-utils (#46330)

2026-09-18 00:25:37 +00:00

web_search.rs

Remove unused Rust helpers (#34345)

2026-07-20 13:45:29 +00:00

windows_sandbox_read_grants_tests.rs

windows-sandbox: pass workspace roots to runner (#24108)

2026-05-28 15:26:55 -07:00

windows_sandbox_read_grants.rs

windows-sandbox: pass workspace roots to runner (#24108)

2026-05-28 15:26:55 -07:00

windows_sandbox_tests.rs

Always use private desktops for legacy Windows sandboxes (#46554)

2026-09-19 01:21:32 +00:00

windows_sandbox.rs

Log Windows sandbox availability and backend selection (#51974)

2026-10-08 06:31:56 +00:00

windows_system_config.rs

Add telemetry for the Windows system config namespace (#44284)

2026-09-09 19:31:12 +00:00

worktree_trust_tests.rs

Add portable project trust lookup APIs (#47620)

2026-09-23 18:40:12 +00:00