mirror of https://github.com/openai/codex.git synced 2026-10-19 16:17:49 +00:00

Files

T

History

jif 47db419304 Batch thread history materialization into SQLite (#55244 )

## Why

History catch-up collected the entire durable rollout prefix before applying it in one SQLite transaction. Large rollouts therefore required buffering all projection steps and committing them together.

## What changed

Stream projection batches through a bounded queue and commit each batch with its checkpoint. Flush between complete records after reaching 1 MiB of source data or 256 projection steps, keeping ordinal gaps with their following records and allowing oversized records to remain intact.

Wait for batches before opening SQLite transactions so an idle reader does not hold the writer. Preserve committed progress across later failures or cancellation, allowing retries to resume from the last checkpoint.

## Testing

Add coverage for reader and writer failures under backpressure, atomic batch rollback, retries without replaying committed records, and unrelated writes while projection is paused. Verify that producer disconnection and consumer cancellation retain committed progress.

GitOrigin-RevId: 06e79fa4cf273de9abd95faf7eed370f18cedc33

2026-10-19 10:48:13 +00:00

src

Batch thread history materialization into SQLite (#55244 )

2026-10-19 10:48:13 +00:00

BUILD.bazel

ThreadStore interface (#17659 )

2026-04-14 13:51:00 -07:00

Cargo.toml

Avoid building a host SQLite driver for SQLx macros (#47084 )

2026-09-21 18:30:31 +00:00

README.md

Unify thread metadata updates above store (#22236 )

2026-05-13 00:28:15 +00:00

README.md

Thread Store

codex-thread-store is the storage boundary for Codex threads. It defines the ThreadStore trait plus local and in-memory implementations. Other storage implementations may live outside this repository.

Responsibilities

ThreadStore::append_items is the raw canonical history append API. It does not infer metadata from item contents.
ThreadStore::update_thread_metadata is the only thread metadata write API. It accepts a single literal metadata patch shape, regardless of whether the caller is applying a user/API mutation or facts derived above the store from appended history.
LiveThread is the preferred API for active session persistence. It owns a per-thread metadata sync helper, applies the rollout persistence policy, appends canonical history, and then sends metadata patches through ThreadStore::update_thread_metadata.
ThreadManager routes metadata mutations for loaded and cold threads through one entrypoint. Loaded threads use their LiveThread; cold threads go directly to the store.
LocalThreadStore persists history through codex-rollout JSONL files and persists queryable metadata through the SQLite state database when available. Local explicit metadata mutations also maintain JSONL/name-index compatibility so reading old or SQLite-less local storage keeps working.
RolloutRecorder is the local JSONL writer. It writes already-canonical items for ThreadStore::append_items; it no longer decides metadata updates for live thread-store appends.
core/session creates or resumes LiveThread handles and does not need to know whether persistence is backed by local files or another store.

Direction

New metadata observation semantics should live above ThreadStore. Stores persist explicit metadata fields, but raw history appends remain history-only.