mirror of
https://github.com/openai/codex.git
synced 2026-11-22 15:15:18 +00:00
Read compressed rollouts and materialize before append (#25087)
## Why Local rollout compression needs a cold `.jsonl.zst` representation without letting compressed physical paths leak into append-mode writers. The unsafe case is resume or metadata update code successfully reading a compressed rollout and then appending raw JSONL bytes to the zstd file. This PR folds the former #25088 materialization slice into the read-support PR so the reader changes and append-safety invariant land together. ## What Changed - Teach rollout readers, discovery, listing, search, and ID lookup to understand compressed `.jsonl.zst` rollouts. - Keep `.jsonl` as the logical/stored rollout path while allowing read paths to open either plain or compressed storage. - Materialize compressed rollouts back to plain `.jsonl` before append-mode writes, including resume and direct metadata append paths. - Preserve compressed-file permissions when materializing back to plain JSONL. - Refresh thread-store resolved rollout paths after compatibility metadata writes so reconciliation follows the materialized file. - Avoid treating transient compression temp files as real rollout lookup results. ## Remaining Stack #25089 remains the separate worker PR. It is based directly on this PR and stays behind the disabled `local_thread_store_compression` feature flag. The worker still has a broader coordination question: a resume or metadata update can race with background compression while a plain file is being replaced by `.jsonl.zst`. This PR handles the read and materialize-before-append primitives; it does not make the worker production-ready. ## Validation - `just test -p codex-rollout` - `just test -p codex-thread-store` - `just fix -p codex-rollout` - `just fix -p codex-thread-store` - `just bazel-lock-check`
This commit is contained in:
1 parent
f27bbbd49c
commit
a8a6071279
16 files changed
+924
-131
No files matched your search
@@ -14,11 +14,11 @@ use crate::cwd_prompt::CwdPromptOutcome;
|
||||
use crate::cwd_prompt::CwdSelection;
|
||||
use crate::tui::Tui;
|
||||
use codex_protocol::ThreadId;
|
||||
use codex_rollout::open_rollout_line_reader;
|
||||
use codex_state::StateRuntime;
|
||||
use codex_utils_path as path_utils;
|
||||
use serde::Deserialize;
|
||||
use serde_json::Value;
|
||||
use tokio::io::AsyncBufReadExt;
|
||||
|
||||
#[derive(Default)]
|
||||
struct RolloutResumeState {
|
||||
@@ -142,13 +142,11 @@ pub(crate) fn cwds_differ(current_cwd: &Path, session_cwd: &Path) -> bool {
|
||||
}
|
||||
|
||||
async fn read_rollout_resume_state(path: &Path) -> io::Result<RolloutResumeState> {
|
||||
let file = tokio::fs::File::open(path).await?;
|
||||
let reader = tokio::io::BufReader::new(file);
|
||||
let mut lines = reader.lines();
|
||||
let mut reader = open_rollout_line_reader(path).await?;
|
||||
let mut state = RolloutResumeState::default();
|
||||
let mut saw_record = false;
|
||||
|
||||
while let Some(line) = lines.next_line().await? {
|
||||
while let Some(line) = reader.next_line().await? {
|
||||
let trimmed = line.trim();
|
||||
if trimmed.is_empty() {
|
||||
continue;
|
||||
|
||||
Reference in new issue
Block a user