mirror of
https://github.com/openai/codex.git
synced 2026-11-13 14:46:14 +00:00
Read compressed rollouts and materialize before append (#25087)
## Why Local rollout compression needs a cold `.jsonl.zst` representation without letting compressed physical paths leak into append-mode writers. The unsafe case is resume or metadata update code successfully reading a compressed rollout and then appending raw JSONL bytes to the zstd file. This PR folds the former #25088 materialization slice into the read-support PR so the reader changes and append-safety invariant land together. ## What Changed - Teach rollout readers, discovery, listing, search, and ID lookup to understand compressed `.jsonl.zst` rollouts. - Keep `.jsonl` as the logical/stored rollout path while allowing read paths to open either plain or compressed storage. - Materialize compressed rollouts back to plain `.jsonl` before append-mode writes, including resume and direct metadata append paths. - Preserve compressed-file permissions when materializing back to plain JSONL. - Refresh thread-store resolved rollout paths after compatibility metadata writes so reconciliation follows the materialized file. - Avoid treating transient compression temp files as real rollout lookup results. ## Remaining Stack #25089 remains the separate worker PR. It is based directly on this PR and stays behind the disabled `local_thread_store_compression` feature flag. The worker still has a broader coordination question: a resume or metadata update can race with background compression while a plain file is being replaced by `.jsonl.zst`. This PR handles the read and materialize-before-append primitives; it does not make the worker production-ready. ## Validation - `just test -p codex-rollout` - `just test -p codex-thread-store` - `just fix -p codex-rollout` - `just fix -p codex-thread-store` - `just bazel-lock-check`
This commit is contained in:
1 parent
f27bbbd49c
commit
a8a6071279
16 files changed
+924
-131
No files matched your search
@@ -69,13 +69,14 @@ pub(super) async fn update_thread_metadata(
|
||||
if live_writer::rollout_path(store, thread_id).await.is_ok() {
|
||||
live_writer::persist_thread(store, thread_id).await?;
|
||||
}
|
||||
let resolved_rollout_path =
|
||||
let mut resolved_rollout_path =
|
||||
resolve_rollout_path(store, thread_id, params.include_archived).await?;
|
||||
let name = patch.name;
|
||||
let git_info = patch.git_info;
|
||||
if let Some(memory_mode) = patch.memory_mode {
|
||||
apply_thread_memory_mode(resolved_rollout_path.path.as_path(), thread_id, memory_mode)
|
||||
.await?;
|
||||
refresh_resolved_rollout_path(&mut resolved_rollout_path).await;
|
||||
}
|
||||
|
||||
let state_db_ctx = store.state_db().await;
|
||||
@@ -143,6 +144,7 @@ pub(super) async fn update_thread_metadata(
|
||||
memory_mode.as_deref(),
|
||||
)
|
||||
.await?;
|
||||
refresh_resolved_rollout_path(&mut resolved_rollout_path).await;
|
||||
apply_thread_git_info(store, thread_id, sha, branch, origin_url).await?;
|
||||
}
|
||||
|
||||
@@ -173,6 +175,12 @@ pub(super) async fn update_thread_metadata(
|
||||
Ok(thread)
|
||||
}
|
||||
|
||||
async fn refresh_resolved_rollout_path(resolved: &mut ResolvedRolloutPath) {
|
||||
if let Some(path) = codex_rollout::existing_rollout_path(resolved.path.as_path()).await {
|
||||
resolved.path = path;
|
||||
}
|
||||
}
|
||||
|
||||
async fn apply_metadata_update(
|
||||
store: &LocalThreadStore,
|
||||
thread_id: ThreadId,
|
||||
|
||||
Reference in new issue
Block a user