Commit Graph
98 Commits
Author SHA1 Message Date
Anthony Tafoya 20b4ff2f35 Enrich V2 capability discovery with resolved plugin and skill metadata (#53763)
## What changed

- Return qualified skill names such as `team:deploy` and parsed `dependencies` and `policy` instead of a separate namespace and raw metadata file. Share `agents/openai.yaml` parsing and validation between skill loaders and executor discovery.
- Include each standalone skill's search-root ID and path in `discoveryRoot`, while accepting responses from older executors that omit it.
- Add plugin `displayName`, falling back to the manifest name or installation directory.
- Materialize inline MCP declarations in `mcpConfig` with the manifest as their source path, and report incomplete discovery when declared MCP or Apps configuration files have no returned contents.

## Testing

Add coverage for JSON round trips, omitted `discoveryRoot`, optional and invalid skill metadata, qualified names, display-name fallbacks, and inline MCP configuration. Verify standalone search roots survive cached discovery and sandboxed RPC responses.

GitOrigin-RevId: 4124972444ad7c2fb8c6418acbf9a589d797fad2
2026-10-13 20:40:38 +00:00
Anthony Tafoya cf0ed4c600 Acquire and cache opt-in executor capability inventories via V2 (#53736)
## What changed

- Add the disabled-by-default `executor_capability_discovery_v2` feature to acquire `capabilities/discoverV2` inventories from supporting remote executors using each selected environment's working directory and sandbox.
- Share snapshots with MCP and world-state contributors and skill providers without activating discovered plugins or publishing discovered skills to the model.
- Cache inventories by environment instance, executor session, working directory, and sandbox. Coalesce concurrent requests, refresh the cache at turn start, retry transient failures, and refresh MCP projections after recovery.
- Resolve the system skills root in the executor and pass it explicitly to capability location discovery.

## Testing

Add coverage for feature gating, unsupported executors, inert inventories, cache reuse and eviction, turn refreshes, failure recovery, executor replacement, and system skill locations.

GitOrigin-RevId: 1bdca28fa7f17622b27e2713ab3dcc0c13426889
2026-10-13 18:15:50 +00:00
aaronl-openai 7efd184c24 Add observers for initial exec-server connection attempts (#52724)
## What changed

Expose `Environment::observe_connection_attempts` and `ConnectionAttemptOutcome` to report elapsed time and success, failure, or cancellation for initial connections. Measure from the end of provisioning through client installation, including connection retries.

The first registered observer wins. Past attempts and reconnections are omitted.

## Testing

Extend tests to assert failure followed by success when startup is retried, and a single cancellation notification when an environment is replaced during startup.

GitOrigin-RevId: 88300f38dcf9428b824c5ff36aa1b40d7d466b6f
2026-10-09 23:28:38 +00:00
iceweasel-oai c2eb1f42a0 Add a sandbox integrity runner with outcome and timing metrics (#51842)
## What changed

Export `run_integrity_checks` from `exec-server` to check whether the effective filesystem policy permits writing containment dependencies on Linux, macOS, and Windows. Run preparation and checks on a blocking worker, limit waiting to five seconds, and keep results observational without affecting execution.

Skip mismatched sandbox backends, full-disk-write policies, and `EscalatedSandboxWithRestrictions` overrides. Continue checking ordinary configured policies with root write access and deny restrictions.

Record check outcomes, preparation time, and total duration with bounded metric tags that omit filesystem paths.

## Testing

Add coverage for findings, clean results, missing dependencies, preparation errors, eligibility rules, and timeouts. Verify that timed-out workers emit no late results and that metrics contain no filesystem paths.

GitOrigin-RevId: a234f4a2f629f9bb090da622cea352c99456cb00
2026-10-07 21:52:20 +00:00
iceweasel-oai b5c37c35fd Add a macOS Seatbelt backend for sandbox integrity (#51841)
## What changed

Enable the exec server's `sandbox_integrity` module on macOS and select the Seatbelt backend. Inventory the Codex executable and the system Seatbelt executable as critical dependencies.

Preserve the filesystem sandbox policy unchanged: Seatbelt handles deny globs as native patterns, so no snapshot scan is needed.

GitOrigin-RevId: 78299dd61e168f5ce34454c8b3bb887e67c5f342
2026-10-07 21:51:13 +00:00
iceweasel-oai d9960e12bb Add a bubblewrap backend for sandbox integrity checks (#51840)
## What changed

Add Linux dependency discovery and policy preparation to `exec-server`'s sandbox integrity module. Inventory bundled and system `bwrap` candidates and the deny-glob scanner using the launcher's location and the command's `PATH` and working directory.

Resolve `:tmpdir` and tilde-prefixed deny globs using the command's `TMPDIR` and `HOME`, with account home-directory fallback. Reuse Linux sandbox helpers to expand deny globs in the command's explicit environment.

## Testing

Add Linux tests covering symlinked launchers, relative and empty `PATH` entries, and empty, relative, and tilde-prefixed `TMPDIR` values alongside home-relative deny globs.

GitOrigin-RevId: fe404fa4be77ece677b22d52d77a7234dbf0cffe
2026-10-07 21:50:48 +00:00
iceweasel-oai 3342ee8c07 Add MXC sandbox integrity dependency and policy preparation (#51831)
## What changed

Add Windows MXC backend helpers to inventory the Codex executable, sandbox
launcher, and available `rg` scanner when deny globs are configured. Prepare
filesystem policies for inspection by reusing MXC temporary-directory and volume
resolution and replacing deny globs with concrete path snapshots.

Allow Windows deny-glob resolution to run its scanner with an explicit command
environment and working directory, including an empty environment. The integrity
helpers are not yet invoked by an execution runner.

## Testing

Add a backend test verifying that deny-glob snapshots replace the original
patterns, including empty snapshots, and affect `FileContentsChecker` results.

GitOrigin-RevId: 409c527663d230d76d65be3bf0a37959f694d77e
2026-10-07 21:16:25 +00:00
jif 1a913493dc Clean up process groups for failed shell snapshot captures (#49782)
## Why

Failed shell snapshot captures could leave processes launched by shell startup files running after the capture ended.

## What changed

Run each Unix shell snapshot capture in its own process group and kill that group on failure, timeout, or cancellation. Keep the capture leader unreaped until output and exit status are validated so its process group ID cannot be reused during cleanup. Preserve startup helpers after a successful capture.

## Testing

Add regression coverage for startup child cleanup after nonzero exits, invalid output, cancellation, and timeout, alongside fallback and retry behavior. Verify that successful captures preserve delayed startup output and background services and reuse the cached snapshot.

GitOrigin-RevId: a56ef5d8615325952bdc8bb78727de1559e7e9c7
2026-10-01 00:48:08 +00:00
Adam Perry @ OpenAI 875bf9209b Define exec-server protocol types for streamed file writes (#49778)
## What changed

- Add `read` and `replace` modes to `fs/open`, defaulting omitted modes to `read` for existing callers.
- Define `fs/writeBlock` request and response types for offset-based writes, with a maximum decoded chunk size of 1 MiB.
- Add the `fileWriteStreaming` capability, defaulting to `false`. Keep writable streams unsupported in the server and explicitly reject replacement opens.

## Testing

Add a protocol test verifying that legacy `fs/open` requests without a mode deserialize as read-only opens.

GitOrigin-RevId: 4d3997717e821cd1f62df12a03c8eec36332a66d
2026-10-01 00:43:55 +00:00
Adam Perry @ OpenAI 3b16b5a5b0 Rename exec-server file handle management identifiers (#49702)
## What changed

Rename `file_read` to `file_handle` and `FileReadHandleManager` to `FileHandleManager`. Update the filesystem handler field, handle ID validation helper, and related constants to use file handle terminology. File operations, limits, and error messages remain unchanged.

GitOrigin-RevId: 25ad81a917386da23fd42ecff79bd47a482ce430
2026-09-30 18:37:18 +00:00
Charlie Marsh f2b2e5b2a9 Make exec-server file reads cancellable between chunks (#49696)
## What changed

Run direct `read_file` operations in a blocking task with bounded read chunks and cancellation checks. Dropping the async caller cancels further reads between chunks. Preserve the 512 MiB size limit, symlink-following options, and regular-file validation.

## Testing

Add regression coverage for 4 MiB reads and oversized-file rejection with both symlink-following options, cancellation after the first chunk, and FIFO rejection without waiting for a writer.

GitOrigin-RevId: 19f2816de15a03bd89b463de6b002367e6a692a9
2026-09-30 18:18:03 +00:00
open-matt b8d5e3f12e Allow exec-server to proxy permitted private IPs upstream (#48568)
## Why

Private IP destinations always bypassed inherited upstream proxies, preventing their use for private networks reachable through an upstream VPN proxy.

## What changed

- Add `codex exec-server --proxy-private-ips-via-upstream`, also configurable with `CODEX_EXEC_SERVER_PROXY_PRIVATE_IPS_VIA_UPSTREAM=true`. The setting defaults to disabled.
- Allow permitted RFC 1918, carrier-grade NAT, and IPv6 unique-local destinations to use an applicable upstream proxy. Loopback and link-local destinations retain direct routing, and destination access policy still applies.
- Keep connections direct when no valid upstream proxy applies or `allow_upstream_proxy=false`. Errors after selecting an upstream proxy do not trigger a direct retry.
- Rename `ExecServerRuntimePaths` to `ExecServerRuntimeOptions` and carry the routing setting from executor startup into the managed network proxy.

## Testing

Add routing coverage for private address ranges, special-use addresses, and public targets with the option enabled and disabled. Verify that HTTP and CONNECT requests still enforce destination allowlists and denylists, and update the CLI help snapshot.

GitOrigin-RevId: b7c9cc7da0e0f545694a6521b74c9b36b7b92769
2026-09-26 23:17:41 +00:00
jif c98e263fb5 Replay exec-server shell snapshots through unnamed files (#48078)
## Why

Large shell snapshots need a replay path that does not add their state to the child process environment.

## What changed

- Keep captured state cached in memory and give each Bash or Zsh launch an independent, read-only file descriptor to source. Unlink the file before writing shell state, create it under the sandbox-protected daemon directory, and close the carrier descriptor before restoring functions.
- Probe descriptor-path access inside the capture sandbox. Retain environment replay for `sh` and when descriptor paths are unavailable; use normal shell startup if preparing the replay file fails.
- Group options and aliases during capture so sourcing preserves their parsing behavior.

## Testing

Add coverage for concurrent replays with 480 KiB payloads, pipe and TTY execution, stdin preservation, descriptor cleanup, sandbox protection, blocked descriptor-path fallback, and alias/option parsing equivalence.

GitOrigin-RevId: 8103d5c37ce4d1885628a4efba3551549f12bea8
2026-09-25 12:23:15 +00:00
Anthony Tafoya ab7439231c Add executor capability discovery V2 infrastructure (#47683)
## What changed

- Define `capabilities/discoverV2` request and inventory types and advertise `capabilityDiscoveryV2` in executor metadata.
- Prewarm installed plugin and global skill locations at server startup, with nonfatal scan failures.
- Add sandbox-scoped discovery caching with file watcher invalidation and request-time reloads when watching is unavailable.
- Batch metadata reads through the filesystem sandbox helper and bound discovery responses to 4 MiB and 2,048 capabilities.
- Share plugin identity and installed-version selection in `codex-core-plugin-common`, and fall back to `.app.json` when a plugin does not declare an apps configuration.

The V2 discovery manager is not yet wired into RPC request dispatch.

## Testing

Add coverage for inventory metadata, sandbox cache isolation, watcher invalidation, startup scan failures, response limits, and compatibility with older executor metadata.

GitOrigin-RevId: a223a5bc15fdf251023c0c71f468fd92c3b2d847
2026-09-23 22:20:34 +00:00
Adam Perry @ OpenAI 6fa3bcaae8 Add exec-server RPC timing and process startup tracing (#47680)
## Why

Detached work can keep request spans open beyond response enqueueing. Record explicit phase timings to distinguish request handling from the full span lifetime.

## What changed

- Record server dispatch and response enqueue offsets from request receipt, plus a response enqueue event for HTTP responses that bypass the dispatcher.
- Emit client request enqueue and response receipt events, including success or server-error outcomes and reader-to-caller delay. Carry completion timestamps to the awaiting task so receipt events retain its tracing context.
- Add spans for process startup, sandbox preparation, shell snapshot preparation and capture, and process spawning. Record process identifiers only when nonempty and at most 64 bytes long.

GitOrigin-RevId: 237db898b3284fe8a6a36cb0055b3c7991d8e8a4
2026-09-23 22:10:28 +00:00
mtsui-oai c847294c60 Classify retryable exec-server preparation errors by type (#47638)
## Why

Transient connection and registry failures can reach remote filesystem callers as generic `io::ErrorKind::Other` errors. Callers need a typed signal to identify failures that read-only preparation can retry.

## What changed

- Add `ExecServerError::is_retryable_preparation_error()` to classify transient registry, connection, handshake, timeout, and session-attachment failures, including errors wrapped in `ConnectionAttempt`.
- Map these failures to `io::ErrorKind::BrokenPipe` in the remote filesystem adapter.
- Keep authentication, TLS, and malformed protocol failures outside the retryable classification; error text alone does not trigger it.

## Testing

Add a regression test showing that a wrapped registry service-unavailable error maps to `BrokenPipe`, while a protocol error containing transport-closed text remains `Other`.

GitOrigin-RevId: c3e1bbab8f977a0b421eba5d6d3071ada72c09ff
2026-09-23 19:37:16 +00:00
viyatb-oai 54487a5b61 Bound inbound exec-server requests across client transports (#47362)
## What changed

- Enforce an 8 KiB encoded-message limit for requests, unrecognized notifications, and malformed messages received by exec-server clients over stdio, WebSocket, relay, and Noise relay transports.
- Allow process output, exit, close, and HTTP request body notifications up to 2 MiB. Keep responses and errors subject to existing transport limits, and preserve executor-side request limits.
- Read stdio stderr in chunks of at most 8 KiB.
- Drop outgoing server-request `tracestate` values larger than 512 bytes while preserving `traceparent`.

## Testing

Add regression coverage for oversized requests across transports, malformed messages, requests wrapped in `RawValue`, fragmented Noise messages, large responses, streamed notifications, and trace-context preservation.

GitOrigin-RevId: 3fb87b6dc334296985ef40983d7276acfe9b9796
2026-09-22 21:07:20 +00:00
Adam Perry @ OpenAI 3d3ae4965a Add filesystem accessors bound to environment permissions (#46268)
## What changed

Add `EnvironmentAccess` and `FileSystemEnvironmentAccessor` to expose filesystem operations with a captured sandbox configuration, without allowing consumers to extract the filesystem or select another sandbox. Include a text-reading helper and an explicit unrestricted constructor.

Provide opaque cache keys that compare filesystem identity and captured permissions without keeping the filesystem alive. Allow opened read streams to outlive the accessor. Export the new APIs through `codex-exec-server` and add `Environment::filesystem_ref()` for borrowing the shared filesystem.

## Testing

Add local and remote coverage for text reads through `EnvironmentAccess`, streams surviving accessor disposal, and cache keys distinguishing changed permissions or a replacement filesystem.

GitOrigin-RevId: 4f6787ed9ba0fb94adea2f31716c4d3132fed07d
2026-09-17 17:59:19 +00:00
iceweasel-oai d4e11a9b97 Separate executor sandbox selection from Windows sandbox levels (#45730)
## Why

MXC is a sandbox implementation, not a restricted-token sandbox level. Executor requests need to represent that choice separately from `WindowsSandboxLevel`.

## What changed

- Introduce `WindowsSandboxSelection` for executor sandbox contexts and remove `Mxc` from `WindowsSandboxLevel`.
- Preserve the `windowsSandboxLevel` wire field and its serialized values for compatibility.
- Share sandbox selection between executor process launches and filesystem helpers, and use the new selection in capability discovery and skill reads.
- Disable Windows sandbox selection for executor paths that do not use Windows path conventions.

## Testing

Extend coverage for MXC wire serialization, Windows skill-read sandbox checks, and capability discovery with distinct permissions. Exercise remote filesystem write restrictions with both restricted-token and MXC sandboxes, including rejection when native MXC is unavailable.

GitOrigin-RevId: 266211377bcb138a0dc75861e9ff2225fa37a53d
2026-09-15 16:40:06 +00:00
bkotsopoulos 574a36ff99 Add client-side exec-server RPC attempt metrics (#42883)
## What changed

- Record `exec_server_client_requests_total` for every client RPC call attempt,
  labeled by protocol method.
- Count attempts before local admission so rejected, timed-out, cancelled, and
  transport-failed calls are included, while notifications and responses are not.
- Export the counter to configured OpenTelemetry collectors while excluding it
  from the built-in Statsig metrics set.

## Testing

- Cover every RPC call entry point, successful responses, local and transport
  failures, timeouts, cancellation, notifications, and disabled metrics.
- Verify that the OTLP HTTP exporter retains the counter.

GitOrigin-RevId: d266e840d5871aa2c34bed8ce44f3345e2b4650f
2026-09-04 23:06:00 +00:00
ostepanian b3f5e45cc1 Add direct SigV4 transport to exec-server (#42781)
## Why

Allow remote exec servers to connect directly to AWS-hosted registries that
authenticate registry requests and WebSocket handshakes with AWS SigV4.

## What changed

- Add `--remote-transport direct` with SigV4 profile, region, and service
  options while keeping Noise as the default transport.
- Register the `direct_jsonrpc_v1` transport and carry plain exec-server
  JSON-RPC messages over the authenticated WebSocket.
- Reuse direct registrations across transient disconnects, refresh them after
  a `409 Conflict`, and require TLS for non-loopback endpoints.

## Testing

- Cover CLI validation and SigV4 request signing.
- Exercise direct registration, handshake retry behavior, JSON-RPC
  interoperability, and process recovery after reconnecting.

GitOrigin-RevId: 0755df330ba3abe5db0a516fdaa49338d9bbe2d2
2026-09-04 14:49:46 +00:00
ostepanian 801ca0d0d1 Support trusted headers for remote exec WebSockets (#42606)
## What changed

- Add `RemoteEnvironmentOptions` and `upsert_environment_with_options` so embedding hosts can attach trusted HTTP headers to remote exec-server WebSocket handshakes.
- Preserve the headers across session reconnects while redacting them from debug output.
- Reject invalid, duplicate, and WebSocket-controlled headers, and require `wss://` for non-loopback destinations.
- Connect header-bearing loopback WebSockets directly, with DNS results restricted to loopback addresses.

## Testing

- Cover header validation and redaction, delivery on initial connections and reconnects, and loopback destination enforcement.

GitOrigin-RevId: f2e2b0456c8e33c13fd8ab4ee040a8b5c15e885d
2026-09-03 20:01:07 +00:00
viyatb-oai f53c91be2c Add attributed exec process lifecycle telemetry (#42373)
## Why

Exec-server process and network-policy events need stable attribution to the
launching tool call and executor without recording process payloads.

## What changed

- Add optional `ExecMetadata` to `ExecParams` and propagate the thread and tool
  call IDs from unified exec.
- Emit bounded OpenTelemetry events for process start, spawn failure, sandbox
  denial, and exit. Correlate them with the launch trace and registry-issued
  executor identity while excluding arguments, paths, environment values,
  output, and error text.
- Preserve launch attribution across long-running processes and reconnects, and
  attach the same metadata to network-policy audit events.
- Keep the protocol backward compatible when metadata is omitted, and prevent
  invalid trace headers from inheriting an unrelated active span.

## Testing

Add coverage for metadata serialization and propagation, lifecycle event
fields, trace relationships, reconnect behavior, spawn failures, sandbox
denials, and network-policy attribution.

GitOrigin-RevId: 7aa480a7289c73cb95e2c124c35500bb6f0d5084
2026-09-02 21:58:58 +00:00
jif d0cc662b8c Cache shell snapshots in the exec server (#39756)
## What changed

- Add the `shellSnapshotV2` executor capability and an optional shell snapshot request to `ExecParams`.
- Capture and restore Unix shell state and profile exports from an in-memory, attachment-scoped cache for `bash`, `zsh`, and `sh`.
- Apply environment policies, runtime `PATH` entries, sandbox context, and live managed-proxy settings when preparing restored commands.
- Bound snapshot size, capture time, scope length, and cache capacity, and fall back to the original command when capture fails.

## Testing

- Cover local, remote, TTY, sandboxed, and supported-shell execution, plus environment filtering, proxy handling, in-memory reuse, and capture failure fallback.

GitOrigin-RevId: 624f747972c249c88c6f10f42cf0af97b75b5541
2026-08-20 17:39:06 +00:00
pakrym-oai e3e5ad2847 Harden unsandboxed patch filesystem access (#39659)
## Why

An `apply_patch` path can be replaced with a symlink after verification, allowing an unsandboxed patch operation to reach a different file than the one that was approved.

## What changed

- Add `follow_symlinks` options to executor filesystem reads, writes, metadata lookups, directory creation, and removal, including the corresponding `followSymlinks` protocol fields.
- Implement no-follow filesystem operations on Unix and Windows that reject links in any path component and restrict file access to regular files.
- Run `apply_patch` with symlink traversal disabled when an otherwise-required sandbox is bypassed, while retaining the existing follow-symlink default for standalone callers.

## Testing

- Cover leaf and ancestor symlinks across patch add, update, delete, and move operations, including a path swap after verification.
- Exercise local and remote no-follow filesystem behavior, concurrent directory creation, special-file rejection, and Windows reparse points.

GitOrigin-RevId: 43fd479084891493ce13564fbd894b98f329c6dd
2026-08-20 08:10:08 +00:00
pakrym-oai 77e6889601 Add exec-server forwarding mode (#39249)
## What changed

- Add `codex exec-server forward --connect ws://HOST:PORT` to register an
  existing WebSocket exec-server as a remote environment.
- Open a separate destination WebSocket for each authenticated Noise relay
  stream and forward complete payloads unchanged in both directions.
- Preserve large messages through WebSocket fragmentation and coordinate
  disconnects, close acknowledgements, and pending output delivery.

## Testing

- Cover CLI validation, opaque payload forwarding, disconnect handling, and
  destination close behavior.
- Verify commands and large file transfers through the complete relay and
  forwarding path.

GitOrigin-RevId: 6e2dccff05b9e3571865611f4d432b0e2b4cdbac
2026-08-18 18:15:21 +00:00
jif b5ea64a203 Add a symlink-safe reader for sensitive files (#39200)
## What changed

- Export `read_sensitive_file_to_string` from `codex-exec-server`.
- Require the opened path to be a regular disk file and avoid following its
  final symlink component on Unix or reparse point on Windows.
- Read valid UTF-8 file contents asynchronously and return I/O errors for
  unsupported inputs.

## Testing

Add tests covering regular files, directories, and symlinks.

GitOrigin-RevId: 68809e94c0d3719e5685c064f9610a0455ffd8d7
2026-08-18 13:36:16 +00:00
Anton Panasenko 89e297729e Prevent Noise auth tokens from reaching child processes (#38941)
## Why

`CODEX_EXEC_SERVER_NOISE_AUTH_TOKEN` is an execution-server credential and
must not be exposed to model-reachable commands or command hooks, including
when shell environment policy explicitly sets a case variant of the name.

## What changed

- Add `CODEX_EXEC_SERVER_NOISE_AUTH_TOKEN` to the shared list of
  non-inheritable environment variables.
- Keep the environment variable constant in `codex-protocol` so the exec
  server and environment scrubber use the same definition.
- Verify case-insensitive removal after policy overrides for shell commands
  and command hooks.

GitOrigin-RevId: 759b224b6a7fb4f56f7b1a6d94ffbb0b188c658f
2026-08-17 04:17:45 +00:00
jif c30a3e49c9 Support sandboxed file streaming in exec-server (#38356)
## Why

Streaming reads previously rejected requests that used a platform filesystem
sandbox.

## What changed

- Open streamed files in the sandbox helper and return the open file to the
  exec-server by passing a file descriptor on Unix or duplicating a file handle
  on Windows.
- Advertise support through the `sandboxedFileStreaming` environment capability.
- Preserve close-on-exec behavior for transferred descriptors, including the
  required inherited-descriptor cleanup on macOS.

## Testing

- Cover bounded sandboxed streams, continued reads after path replacement, and
  rejection of symlink escapes outside readable roots.

GitOrigin-RevId: 677b2444b74e834b78b87a8554bc119c1c6e08b2
2026-08-13 11:11:26 +00:00
sayan-oai 95c7265e84 Add executor-local config reads to the exec server (#37408)
## What changed

- Add the `environmentConfig/read` RPC for selecting literal TOML paths from executor-local config and requirements layers.
- Return layer precedence, cloud insertion points, source and base-directory metadata, and executor home and hostname context without normalizing path-bearing values.
- Expose the operation through both remote clients and local `Environment` instances, with invalid selectors reported as invalid parameters.

## Testing

- Cover projected remote config reads and rejection of empty selectors.

GitOrigin-RevId: fee15bf833de6bcb2058d405ed5b1d5d928d7218
2026-08-07 08:32:30 +00:00
TAFOYA-OAI f8ac8fa6c6 Consolidate deferred environment provisioning APIs (#37340)
## What changed

- Remove the separate deferred registration handle, direct readiness publisher,
  and Noise environment upsert API from `EnvironmentManager`.
- Use `report_environment_provisioning_status` and
  `materialize_pending_noise_environment` as the provisioning flow while keeping
  ordinary environments isolated from provisioning reports.
- Cover readiness updates, invalid reports, duplicate materialization, and
  conflicts with ordinary environments in the deferred environment tests.

GitOrigin-RevId: d14207c0abb0636d331be4f875fec6e53f3d2fe6
2026-08-07 00:32:59 +00:00
Adam Perry @ OpenAI eeae88d8a6 Add opt-in concurrent exec-server request dispatch (#36987)
## Why

Sequential dispatch lets a long-running request block unrelated health checks
and cleanup on the same connection.

## What changed

- Add `--concurrent-requests <COUNT>` for local and remote exec-server
  connections, while retaining sequential dispatch when the option is omitted
  or set to `1`.
- Preserve handshake ordering before enabling concurrent dispatch.
- Reserve separate capacity for status, signal, terminate, and close requests so
  they remain responsive when ordinary request capacity is saturated.
- Drain queued client responses during disconnect and cancel outstanding
  request tasks during connection shutdown.

## Testing

- Cover CLI parsing and concurrency-limit validation.
- Verify default sequential behavior, pipelined handshake ordering, concurrent
  request progress, control-request responsiveness, and disconnect handling.

GitOrigin-RevId: 48e4b092e318204ed635543f01f9ee0e7df095fc
2026-08-04 22:28:15 +00:00
Alexi Christakis 166658a34a Tie remote exec servers to their parent stdin (#35843)
## What changed

- Add `--exit-on-stdin-close` and the `CODEX_EXEC_SERVER_EXIT_ON_STDIN_CLOSE` environment variable as opt-in controls for remote exec servers.
- Gracefully drain active sessions and processes when the parent closes stdin, then flush telemetry before exiting.
- Remove the parent-lifetime environment variable from child process environments.

## Testing

- Cover parent disconnects after signal-listener failures.
- Exercise remote shutdown end to end, including child termination and final telemetry metrics.
- Verify that explicitly disabling the environment variable preserves local exec-server behavior.

GitOrigin-RevId: 63063bc097b54684c370bd545cd32d17c4e55d90
2026-07-28 23:27:46 +00:00
Celia Chen 09241ae4db Decouple exec-server HTTP from reqwest types (#35059)
## What changed

- Rename `ReqwestHttpClient` to `RouteAwareHttpClient` to reflect that delegated HTTP uses Codex's shared route-aware transport.
- Use `codex_http_client` response and error types plus transport-neutral `http` and `url` types, removing the exec server's direct `reqwest` dependency.

## Testing

- Cover fragment stripping and Unicode hostname normalization for delegated HTTP requests.

GitOrigin-RevId: 8b0fc60a76004feb57198bfb4afb1371c9ceb1bd
2026-07-24 00:53:05 +00:00
Celia Chen 265cd2e100 Initialize execution environments with the final HTTP policy (#34995)
## Why

The TUI must inspect the default execution environment before loading its final
configuration. Initializing the environment manager at that point can give
startup services the bootstrap HTTP policy instead of the effective policy after
managed requirements are applied.

## What changed

- Split environment discovery from manager construction so callers can inspect
  the default environment without starting remote connections.
- Build the environment manager after final configuration loading and pass its
  resolved `HttpClientFactory` through all construction paths.
- Add shared test support for managers that use the legacy default HTTP policy.

## Testing

- Cover connection-free environment discovery and explicit HTTP policy
  propagation.
- Verify TUI startup services use the final managed `respect_system_proxy` value.

GitOrigin-RevId: 928fa31e6b4bcfbe1a121cade2f351427fdfa0f4
2026-07-23 19:24:41 +00:00
viyatb-oai 32f4687b8c Enable exec-server network policy callbacks (#34770)
## What changed

- Let `exec-server` issue JSON-RPC requests to its client and correlate responses, with bounded concurrency, timeouts, and connection cleanup.
- When `request_policy_decisions` is enabled, forward proxy policy requests with the process ID and network destination, then apply the client's allow, deny, or ask decision.
- Fail closed for invalid inputs, unavailable or malformed responses, timeouts, process shutdown, and disconnects. Keep callbacks active while inherited process streams keep the proxy alive.

## Testing

- Cover response correlation, request limits, timeouts, disconnect cleanup, validation boundaries, decision forwarding, and process lifecycle behavior.

GitOrigin-RevId: 60e658102c50036589ca9de844760a8bffaa53b0
2026-07-22 15:22:21 +00:00
viyatb-oai d838ea0f64 Add exec-server network policy callback types (#34620)
## What changed

- Define the `network/policyRequest` RPC payloads for associating a network request with a process and returning an `allow`, `deny`, or `ask` decision.
- Cover HTTP, HTTPS CONNECT, and SOCKS5 TCP/UDP requests, with shared size limits for callback fields.
- Add the backward-compatible `requestPolicyDecisions` remote proxy option so executors can opt in to sending policy decisions to clients.
- Re-export the new protocol types from `codex-exec-server`.

## Testing

- Verify the request and decision JSON shapes.
- Verify that the proxy opt-in is omitted and defaults to `false` for legacy configurations, while an enabled value round-trips.

GitOrigin-RevId: 65b0a0723d1abb6e4f504a41d3a0dd41f5f21bf7
2026-07-21 21:54:31 +00:00
jif 4f1992732c Preserve custom arg0 for sandboxed exec-server processes (#34497)
## Why

Sandbox wrappers replaced the process launch command and did not carry an
`ExecParams.arg0` override through to the inner process.

## What changed

- Route sandboxed Unix launches with a custom `arg0` through a helper mode that
  re-execs the requested program with the override.
- Expose the helper executable to the filesystem sandbox and dispatch its mode
  from Codex and exec-server test binaries.

## Testing

Add coverage for the prepared sandbox command and an end-to-end remote process
that verifies both the custom `arg0` and filesystem restrictions.

GitOrigin-RevId: c9f8eef3906d184e670184c2eeed250d5895a9ca
2026-07-21 08:59:41 +00:00
viyatb-oai 9f6c29e281 Launch managed network proxies on remote executors (#33906)
## Why

Remote executions need managed-network proxy listeners in the executor so their
loopback proxy addresses are reachable by the launched process.

## What changed

- Add a capability-gated exec-server protocol field for executor-local proxy
  launch configuration, including network policy, audit metadata, and execution
  attribution.
- Start the proxy while preparing a remote process, replace inherited proxy
  environment variables with its local addresses, and derive the sandbox
  context from its listeners.
- Keep the proxy alive until inherited output streams close, then shut it down.
- Reject unsupported remote settings such as MITM and credential injection.

## Testing

- Cover configuration round trips and rejection of unsupported settings.
- Verify executor-local startup, blocked-domain enforcement, protocol
  compatibility, and proxy lifetime through process closure.

GitOrigin-RevId: c984f54e3e600aa9ebcbf8cf4574046e2c199d11
2026-07-17 21:20:14 +00:00
Bryan Ashley 08e30a2e4e Add batched executor capability discovery (#33852)
## Why

Selected capability roots can contribute plugins, MCP servers, connectors, and
skills. Discovering each contribution separately requires repeated access to the
executor filesystem.

## What changed

- Add the `capabilityRoots/discoverV1` exec-server RPC to scan selected roots and
  materialize recognized plugin manifests, configuration files, skill
  instructions, and skill metadata in one bounded request.
- Add the opt-in `executor_capability_discovery` feature, with a thread-scoped
  cache and per-step snapshot shared by MCP and skill discovery.
- Parse MCP, connector, and skill contributions from the materialized snapshot,
  including serving cached skill instructions without another filesystem read.

## Testing

- Cover discovery limits, manifest precedence, root-local failures, cache reuse,
  plugin contributions, and parity with the existing environment skill loader.

GitOrigin-RevId: f98fd2321cafb58c596db02da1f83c09d8eb375d
2026-07-17 15:45:48 +00:00
jif 79177c3e20 Propagate deferred environment capability roots to MCP (#33427)
## What changed

- Let deferred environments provide selected capability roots with their ready signal.
- Validate that those roots have unique, non-empty IDs, belong to the registering environment, and stay within the root limit.
- Include roots from ready turn environments when resolving MCP contributions, and refresh the MCP runtime when the selected root set changes.
- Expose the exact ready root set to MCP contributors so executor plugins become available with their environment.

## Testing

- Cover ready-root propagation, validation failures, replacement isolation, reconnection, and MCP plugin availability refresh.

GitOrigin-RevId: ec3498aab1164824025094e96a9b1063b7b731ad
2026-07-15 21:07:43 +00:00
sayan-oai 3afbd8dd45 Report selected environment connection transitions (#33251)
## What changed

- Track connected and disconnected states across initial remote exec-server connections and reconnection attempts.
- Emit experimental `thread/environment/connected` and `thread/environment/disconnected` app-server notifications for each thread selecting the environment. Each payload identifies the thread and environment; current state is not replayed when a thread starts.
- Stop forwarding connection events when an environment selection is removed or replaced.

## Testing

- Cover connection, disconnection, reconnection, shared-environment notifications, and replacement of a selected environment.

GitOrigin-RevId: 5dd767372363c4a2a8319fc16164be117d5bd20c
2026-07-15 05:21:13 +00:00
TAFOYA-OAI 32cd5d4eab Defer Noise environment connections until registration (#33166)
## What changed

- Replace pending WebSocket URL registration with deferred Noise environment
  registration that gates connection attempts on an explicit readiness signal.
- Reuse the Noise rendezvous transport after readiness so reconnects request a
  fresh connection bundle.
- Preserve terminal errors for failed or dropped registrations and keep late
  completion isolated from replacement environments.

## Testing

- Add coverage for readiness gating, registration failure and replacement,
  eager Noise connections, and reconnection through a fresh rendezvous bundle.

GitOrigin-RevId: 83e23fa03a02e3b2bdf1a83fe26d7ac461f55cf3
2026-07-14 21:07:27 +00:00
Adam Perry @ OpenAI 75470c3e2f Add exec-server environment status checks (#32899)
## What changed

- Add the initialized `environment/status` RPC, which reports `ready` when the exec server can handle requests.
- Expose environment IDs and `ready`, `pending`, or `disconnected` status through `EnvironmentManager` and `Environment`.
- Keep status checks non-mutating: they do not start or recover lazy remote environments, and probe only an existing connection.

## Testing

- Cover the status RPC over WebSocket and the in-process request processor.
- Verify that checking an unstarted stdio environment leaves it pending and that failed connections report as disconnected.

GitOrigin-RevId: 22febeb6a3457849292128a8991c6400c22b3fd8
2026-07-13 23:43:15 +00:00
pakrym-oai 4472698728 Support pending remote environment registration (#32231)
## Why

Remote environment provisioning can finish after a thread starts, before an
exec-server WebSocket URL is available.

## What changed

- Add `EnvironmentManager::register_pending_environment` and a one-shot
  `PendingEnvironmentRegistration` handle that resolves to either a validated
  WebSocket URL or a terminal provisioning error.
- Let lazy remote exec-server clients wait for that result, while preserving
  reconnection behavior after a successful registration.
- Keep replacement registrations isolated so completing an older handle does
  not resolve the current environment with the same ID.

## Testing

Add coverage for successful connection and reconnection, provisioning and
dropped-registration failures, invalid URLs, replacement isolation, and the
deferred-executor startup flow.

GitOrigin-RevId: 5c05be2b72291b77a1f71176d7075b1ad63332a5
2026-07-10 18:08:35 +00:00
jif 13ba8058f2 Resolve selected capability roots without starting executors (#31581)
## Why

A thread can select skill roots that live in an executor environment.
`skills/list` needs a passive snapshot of the roots that are usable now:
it must not start an executor, wait for recovery, or reconnect a failed
environment.

The initial implementation checked the immutable first startup result.
After a successful connection later entered recovery or failed, that
result still looked successful. A read-only catalog request could then
wait for recovery or trigger a new connection while reading the
filesystem.

## What

- inspect readiness from the current exec-server connection state
- return roots only while their environment can serve a request
immediately
- omit environments that have not started, are connecting, or are
recovering
- return warnings for missing environments and terminal connection
failures
- add a fail-fast filesystem view that never starts, waits for, or
reconnects an environment
- expose the passive selected-root snapshot through `CodexThread`

## Behavior

- Local and currently connected environments are ready.
- Starting and recovering environments are omitted without a warning so
callers can retry later.
- Missing and terminally failed environments are omitted with a warning.
- A disconnect between readiness inspection and filesystem access fails
promptly instead of crossing into the normal recovery path.
- Normal model-turn and execution paths keep their existing reconnect
behavior.

## Design

The recovery policy is private to the exec-server client. Callers choose
the explicit fail-fast filesystem method; the existing client and
filesystem APIs remain reconnecting. This keeps the passive contract at
the transport boundary instead of plumbing timeout or retry flags
through the skills stack.

## Coverage

- a lazy stdio environment stays unstarted during passive inspection
- missing and terminally failed environments surface warnings
- a real websocket disconnect proves current readiness drops, a
previously acquired fail-fast filesystem handle returns promptly, and
readiness returns after recovery

## Scope

This PR only provides passive readiness and fail-fast filesystem
primitives. It does not add app-server API fields or notifications.

## Stack

- #31582 uses these primitives for experimental thread-scoped
`skills/list`.
- #30228 adds targeted invalidation notifications.
2026-07-09 11:17:05 +01:00
richardopenai 042e61726d [codex] bound Rendezvous WebSocket liveness (#30643)
## Summary

- require a Pong within 60 seconds for established Noise Rendezvous
WebSockets on both the harness and executor
- bound steady-state WebSocket writes and harness event delivery so
backpressure cannot mask the deadline
- classify executor disconnects with bounded reasons and feed them into
the existing reconnect metric and structured log
- cover silent peers, responsive peers, continuous non-Pong traffic, and
local application backpressure

## Why

The existing periodic Pings did not track Pongs, so a half-open or
blackholed connection could remain stuck until the operating system's
TCP timeout. This adds the smallest explicit liveness contract without
new spans, RTT histograms, feature flags, or TCP diagnostics.

## Testing

- `just test -p codex-exec-server` on devbox `richard-6` — 300 passed, 2
skipped
- `just fix -p codex-exec-server`
- `just fmt`
- independent correctness, performance/security, and YAGNI reviews — no
findings
2026-07-01 14:15:34 -07:00
richardopenai 3b22498f69 [codex] Observe remote exec-server lifecycle (#27470)
## Summary

- Record bounded duration and outcome metrics for remote environment
registration and Noise rendezvous connection attempts.
- Count reconnects by bounded reason: disconnect, connection failure, or
rejected registration.
- Trace registration at the owning client boundary without exporting raw
environment or registration identifiers.
- Replace the stale pre-Noise WebSocket observability design with the
current remote transport model.

## Stack

Review and land this stack in order:

1. #27466 — trace exec-server JSON-RPC requests
2. #27467 — record bounded connection, request, and process lifecycle
metrics
3. #27470 — observe remote registration and Noise rendezvous lifecycle
**(this PR)**

## Validation

- `just test -p codex-exec-server --lib` (149 passed)
- `just test -p codex-cli --test exec_server` (4 passed)
- `just argument-comment-lint`
- `just bazel-lock-check`
- `just fix -p codex-exec-server -p codex-cli`
- `just fmt`
2026-06-25 13:42:40 -07:00
richardopenai 2dec46e30a [codex] Record exec-server lifecycle metrics (#27467)
## Summary

- Record bounded connection, request, and process lifecycle metrics.
- Report active gauges from callbacks on every collection, including
delta exports.
- Serialize active-count updates so concurrent starts and finishes
cannot publish stale values.
- Serialize process exit, explicit termination, and shutdown through the
process registry so exactly one completion result wins.
- Keep the implementation small with single-owner RAII guards and one
real OTLP/HTTP integration test using the existing `wiremock`
dependency.

## Root cause

Process exit and session shutdown previously used cloned completion
state. That avoided duplicate emission, but it duplicated lifecycle
ownership and made the ordering harder to reason about. The process
registry mutex already defines the lifecycle ordering, so the final
implementation stores the metric guard and termination flag directly on
the process entry. Whichever path claims the entry first owns the
completion result.

Production metric export uses delta temporality. Event-only synchronous
gauge recordings disappear after the next collection when no count
changes, so active counts now use observable callbacks that report
current state on every collection.

The cleanup also removes the constant `result="accepted"` connection
tag, redundant route and response assertions, a custom HTTP collector,
and fallback initialization machinery that did not add behavior.

## Stack

Review and land this stack in order:

1. #27466 — trace exec-server JSON-RPC requests
2. #27467 — record bounded connection, request, and process lifecycle
metrics **(this PR)**
3. #27470 — observe remote registration and Noise rendezvous lifecycle

## Validation

- `just test -p codex-exec-server --lib` (158 passed)
- `just test -p codex-cli --test exec_server` (3 passed)
- `just test -p codex-otel
observable_gauge_is_collected_on_every_delta_snapshot` (1 passed)
- `CARGO_BUILD_JOBS=1 just fix -p codex-otel -p codex-exec-server`
- `just fmt`
- `git diff --check`
2026-06-25 11:02:11 -07:00
jif 8f02973d25 Persist selected capability roots and resolve availability per model step (#29856)
## Why

`selectedCapabilityRoots` is durable thread intent: “use this capability
root from environment `worker`.”

The important product assumption is:

> One environment ID always names the same logical executor and stable
contents.

`worker` does not silently change from executor A to an unrelated
executor B. The process-local connection handle for `worker` can still
be replaced while Codex is running, though, for example when
`environment/add` registers a fresh handle for the same logical
environment.

The thread should persist only the stable selection. Each model step
should pair that selection with the exact ready handle captured for that
step.

## The boundary

```text
persisted thread intent
  plugin@1 -> environment "worker"
                |
                | capture the current step
                v
model-step view
  unavailable, or
  plugin@1 + worker's exact captured ready handle
```

The environment ID is the stable identity and cache key. The
`Arc<Environment>` is only a process-local handle retained so consumers
of one model step use the same captured environment. It is never
persisted and it does not imply different environment contents.

## What changes

### Persist the stable selection

Selected roots are written into `SessionMeta` and restored with the
thread. Forked subagents inherit the same selections, including
bounded-history forks.

Only stable data is persisted: root ID, environment ID, and root path.

### Capture readiness together with the exact handle

The environment snapshot records:

```rust
environment_id -> Some(Arc<Environment>) // ready in this step
environment_id -> None                   // still starting in this step
```

This prevents readiness and execution from coming from different
registry snapshots.

For example:

```text
step snapshot: worker -> handle A, ready
environment/add: worker -> fresh handle B for the same logical environment
current step: plugin@1 still uses captured handle A
```

Without carrying handle A in the snapshot, the resolver could combine “A
was ready” with handle B and treat B as ready before it had finished
starting.

This does not change cache invalidation. Stable capability metadata
remains identified by environment ID and capability root. Replacing a
process-local handle under the same stable environment ID does not
invalidate or rediscover that metadata.

### Resolve availability per model step

- A ready captured environment produces resolved roots using its
captured handle.
- A starting, missing, or failed environment is omitted from that step.
- A selected lazy environment that is outside the turn's captured
environment set is asked to start, and a later step can observe it as
ready.
- No capability files are scanned here.

Transient transport disconnects remain the remote client's reconnect
concern. This PR models initial attachment/readiness; it does not add
live socket-connectivity state.

## Example

```text
thread selection: plugin@1 -> environment "worker"

step 1: worker is starting -> plugin@1 unavailable
step 2: worker is ready    -> plugin@1 resolves through worker's captured handle
step 3: fresh local handle -> current step remains pinned; a later step captures its own view
```

Temporary unavailability does not discard the durable selection. Later
PRs can retain stable metadata caches while projecting only currently
available capabilities into model-visible World State.

## Compatibility

The app-server request shape does not change. Older rollouts without
`selected_capability_roots` deserialize to an empty list.

## Stack

1. **This PR:** persist stable selected roots and resolve them through
an exact model-step handle.
2. #29960: cache stable skill metadata and project available skills into
World State.
3. #29946: cache stable plugin declarations and manage the separate live
MCP runtime.
2026-06-25 17:49:43 +00:00