feat: mem v2 - PR2 (#11365)

# Memories migration plan (simplified global workflow)

## Target behavior

- One shared memory root only: `~/.codex/memories/`.
- No per-cwd memory buckets, no cwd hash handling.
- Phase 1 candidate rules:
- Not currently being processed unless the job lease is stale.
- Rollout updated within the max-age window (currently 30 days).
- Rollout idle for at least 12 hours (new constant).
- Global cap: at most 64 stage-1 jobs in `running` state at any time
(new invariant).
- Stage-1 model output shape (new):
- `rollout_slug` (accepted but ignored for now).
- `rollout_summary`.
- `raw_memory`.
- Phase-1 artifacts written under the shared root:
- `rollout_summaries/<thread_id>.md` for each rollout summary.
- `raw_memories.md` containing appended/merged raw memory paragraphs.
- Phase 2 runs one consolidation agent for the shared `memories/`
directory.
- Phase-2 lock is DB-backed with 1 hour lease and heartbeat/expiry.

## Current code map

- Core startup pipeline: `core/src/memories/startup/mod.rs`.
- Stage-1 request+parse: `core/src/memories/startup/extract.rs`,
`core/src/memories/stage_one.rs`, templates in
`core/templates/memories/`.
- File materialization: `core/src/memories/storage.rs`,
`core/src/memories/layout.rs`.
- Scope routing (cwd/user): `core/src/memories/scope.rs`,
`core/src/memories/startup/mod.rs`.
- DB job lifecycle and scope queueing: `state/src/runtime/memory.rs`.

## PR plan

## PR 1: Correct phase-1 selection invariants (no behavior-breaking
layout changes yet)

- Add `PHASE_ONE_MIN_ROLLOUT_IDLE_HOURS: i64 = 12` in
`core/src/memories/mod.rs`.
- Thread this into `state::claim_stage1_jobs_for_startup(...)`.
- Enforce idle-time filter in DB selection logic (not only in-memory
filtering after `scan_limit`) so eligible threads are not starved by
very recent threads.
- Enforce global running cap of 64 at claim time in DB logic:
- Count fresh `memory_stage1` running jobs.
- Only allow new claims while count < cap.
- Keep stale-lease takeover behavior intact.
- Add/adjust tests in `state/src/runtime.rs`:
- Idle filter inclusion/exclusion around 12h boundary.
- Global running-cap guarantee.
- Existing stale/fresh ownership behavior still passes.

Acceptance criteria:
- Startup never creates more than 64 fresh `memory_stage1` running jobs.
- Threads updated <12h ago are skipped.
- Threads older than 30d are skipped.

## PR 2: Stage-1 output contract + storage artifacts
(forward-compatible)

- Update parser/types to accept the new structured output while keeping
backward compatibility:
- Add `rollout_slug` (optional for now).
- Add `rollout_summary`.
- Keep alias support for legacy `summary` and `rawMemory` until prompt
swap completes.
- Update stage-1 schema generator in `core/src/memories/stage_one.rs` to
include the new keys.
- Update prompt templates:
- `core/templates/memories/stage_one_system.md`.
- `core/templates/memories/stage_one_input.md`.
- Replace storage model in `core/src/memories/storage.rs`:
- Introduce `rollout_summaries/` directory writer (`<thread_id>.md`
files).
- Introduce `raw_memories.md` aggregator writer from DB rows.
- Keep deterministic rebuild behavior from DB outputs so files can
always be regenerated.
- Update consolidation prompt template to reference `rollout_summaries/`
+ `raw_memories.md` inputs.

Acceptance criteria:
- Stage-1 accepts both old and new output keys during migration.
- Phase-1 artifacts are generated in new format from DB state.
- No dependence on per-thread files in `raw_memories/`.

## PR 3: Remove per-cwd memories and move to one global memory root

- Simplify layout in `core/src/memories/layout.rs`:
- Single root: `codex_home/memories`.
- Remove cwd-hash bucket helpers and normalization logic used only for
memory pathing.
- Remove scope branching from startup phase-2 dispatch path:
- No cwd/user mapping in `core/src/memories/startup/mod.rs`.
- One target root for consolidation.
- In `state/src/runtime/memory.rs`, stop enqueueing/handling cwd
consolidation scope.
- Keep one logical consolidation scope/job key (global/user) to avoid a
risky schema rewrite in same PR.
- Add one-time migration helper (core side) to preserve current shared
memory output:
- If `~/.codex/memories/user/memory` exists and new root is empty,
move/copy contents into `~/.codex/memories`.
- Leave old hashed cwd buckets untouched for now (safe/no-destructive
migration).

Acceptance criteria:
- New runs only read/write `~/.codex/memories`.
- No new cwd-scoped consolidation jobs are enqueued.
- Existing user-shared memory content is preserved.

## PR 4: Phase-2 global lock simplification and cleanup

- Replace multi-scope dispatch with a single global consolidation claim
path:
- Either reuse jobs table with one fixed key, or add a tiny dedicated
lock helper; keep 1h lease.
- Ensure at most one consolidation agent can run at once.
- Keep heartbeat + stale lock recovery semantics in
`core/src/memories/startup/watch.rs`.
- Remove dead scope code and legacy constants no longer used.
- Update tests:
- One-agent-at-a-time behavior.
- Lock expiry allows takeover after stale lease.

Acceptance criteria:
- Exactly one phase-2 consolidation agent can be active cluster-wide
(per local DB).
- Stale lock recovers automatically.

## PR 5: Final cleanup and docs

- Remove legacy artifacts and references:
- `raw_memories/` and `memory_summary.md` assumptions from
prompts/comments/tests.
- Scope constants for cwd memory pathing in core/state if fully unused.
- Update docs under `docs/` for memory workflow and directory layout.
- Add a brief operator note for rollout: compatibility window for old
stage-1 JSON keys and when to remove aliases.

Acceptance criteria:
- Code and docs reflect only the simplified global workflow.
- No stale references to per-cwd memory buckets.

## Notes on sequencing

- PR 1 is safest first because it improves correctness without changing
external artifact layout.
- PR 2 keeps parser compatibility so prompt deployment can happen
independently.
- PR 3 and PR 4 split filesystem/scope simplification from locking
simplification to reduce blast radius.
- PR 5 is intentionally cleanup-only.

codex-rs/core/src/memories/tests.rs

@@ -2,13 +2,13 @@ use super::rollout::StageOneResponseItemKinds;
 use super::rollout::StageOneRolloutFilter;
 use super::rollout::serialize_filtered_rollout_response_items;
 use super::stage_one::parse_stage_one_output;
-use super::storage::rebuild_memory_summary_from_memories;
-use super::storage::sync_raw_memories_from_memories;
+use super::storage::rebuild_raw_memories_file_from_memories;
+use super::storage::sync_rollout_summaries_from_memories;
 use super::storage::wipe_consolidation_outputs;
 use crate::memories::layout::ensure_layout;
 use crate::memories::layout::memory_root_for_cwd;
-use crate::memories::layout::memory_summary_file;
-use crate::memories::layout::raw_memories_dir;
+use crate::memories::layout::raw_memories_file;
+use crate::memories::layout::rollout_summaries_dir;
 use chrono::TimeZone;
 use chrono::Utc;
 use codex_protocol::ThreadId;
@@ -65,10 +65,20 @@ fn memory_root_encoding_avoids_component_collisions() {
 
 #[test]
 fn parse_stage_one_output_accepts_fenced_json() {
-    let raw = "```json\n{\"rawMemory\":\"abc\",\"summary\":\"short\"}\n```";
+    let raw = "```json\n{\"raw_memory\":\"abc\",\"rollout_summary\":\"short\",\"rollout_slug\":\"slug\"}\n```";
     let parsed = parse_stage_one_output(raw).expect("parsed");
     assert!(parsed.raw_memory.contains("abc"));
-    assert_eq!(parsed.summary, "short");
+    assert_eq!(parsed.rollout_summary, "short");
+    assert_eq!(parsed.rollout_slug, Some("slug".to_string()));
+}
+
+#[test]
+fn parse_stage_one_output_accepts_legacy_keys() {
+    let raw = r#"{"rawMemory":"abc","summary":"short"}"#;
+    let parsed = parse_stage_one_output(raw).expect("parsed");
+    assert!(parsed.raw_memory.contains("abc"));
+    assert_eq!(parsed.rollout_summary, "short");
+    assert_eq!(parsed.rollout_slug, None);
 }
 
 #[test]
@@ -172,15 +182,15 @@ fn serialize_filtered_rollout_response_items_filters_by_response_item_kind() {
 }
 
 #[tokio::test]
-async fn prune_and_rebuild_summary_keeps_latest_memories_only() {
+async fn sync_rollout_summaries_and_raw_memories_file_keeps_latest_memories_only() {
     let dir = tempdir().expect("tempdir");
     let root = dir.path().join("memory");
     ensure_layout(&root).await.expect("ensure layout");
 
     let keep_id = ThreadId::default().to_string();
     let drop_id = ThreadId::default().to_string();
-    let keep_path = raw_memories_dir(&root).join(format!("{keep_id}.md"));
-    let drop_path = raw_memories_dir(&root).join(format!("{drop_id}.md"));
+    let keep_path = rollout_summaries_dir(&root).join(format!("{keep_id}.md"));
+    let drop_path = rollout_summaries_dir(&root).join(format!("{drop_id}.md"));
     tokio::fs::write(&keep_path, "keep")
         .await
         .expect("write keep");
@@ -196,21 +206,21 @@ async fn prune_and_rebuild_summary_keeps_latest_memories_only() {
         generated_at: Utc.timestamp_opt(101, 0).single().expect("timestamp"),
     }];
 
-    sync_raw_memories_from_memories(&root, &memories)
+    sync_rollout_summaries_from_memories(&root, &memories)
         .await
-        .expect("sync raw memories");
-    rebuild_memory_summary_from_memories(&root, &memories)
+        .expect("sync rollout summaries");
+    rebuild_raw_memories_file_from_memories(&root, &memories)
         .await
-        .expect("rebuild memory summary");
+        .expect("rebuild raw memories");
 
     assert!(keep_path.is_file());
     assert!(!drop_path.exists());
 
-    let summary = tokio::fs::read_to_string(memory_summary_file(&root))
+    let raw_memories = tokio::fs::read_to_string(raw_memories_file(&root))
         .await
-        .expect("read summary");
-    assert!(summary.contains("short summary"));
-    assert!(summary.contains(&keep_id));
+        .expect("read raw memories");
+    assert!(raw_memories.contains("raw memory"));
+    assert!(raw_memories.contains(&keep_id));
 }
 
 #[tokio::test]