Move memory state to a dedicated SQLite DB (#24591)

## Summary

Generated memory rows and their stage-one/stage-two job state currently
live in `state_5.sqlite` alongside thread metadata. That makes memory
cleanup and regeneration share the main state schema even though those
rows are memory-pipeline data and can be rebuilt independently from the
durable thread records.

This PR moves the memory-owned tables into a dedicated
`memories_1.sqlite` runtime database while keeping thread metadata in
`state_5.sqlite`.

## Changes

- Adds a separate memories DB runtime, migrator, path helpers, telemetry
kind, and Bazel compile data for `state/memory_migrations`.
- Introduces `MemoryStore` behind `StateRuntime::memories()` and moves
memory table/job operations onto that store.
- Drops the old memory tables from the state DB and recreates their
schema in `state/memory_migrations/0001_memories.sql`.
- Updates memory startup, citation usage tracking, rollout pollution
handling, `debug clear-memories`, and app-server `memory/reset` to
operate through the memories DB.
- Preserves cross-DB behavior by hydrating thread metadata from the
state DB when selecting visible memory outputs and checking stage-one
staleness.

## Verification

- Added/updated `codex-state` tests for deleted-thread memory visibility
and already-polluted phase-two enqueue behavior.
- Updated `debug clear-memories`, app-server `memory/reset`, and
memories startup tests to seed and assert memory rows through
`memories_1.sqlite`.
This commit is contained in:
jif-oai authored and GitHub committed 2026-05-26 20:07:25 +02:00
1 parent 823381e867
commit aad59a0916
22 files changed
+1014 -280

No files matched your search

@@ -1513,9 +1513,13 @@ impl ThreadRequestProcessor {
.clone()
.ok_or_else(|| internal_error("sqlite state db unavailable for memory reset"))?;
state_db.clear_memory_data().await.map_err(|err| {
internal_error(format!("failed to clear memory rows in state db: {err}"))
})?;
state_db
.memories()
.clear_memory_data()
.await
.map_err(|err| {
internal_error(format!("failed to clear memory rows in memories db: {err}"))
})?;
clear_memory_roots_contents(&self.config.codex_home)
.await