mirror of
https://github.com/openai/codex.git
synced 2026-11-19 23:05:23 +00:00
[codex] exec-server: stream files in chunks (#28354)
## Why `fs/readFile` buffers the entire file in one response, which makes large remote reads expensive and prevents callers from applying backpressure. We need an opt-in streaming path with bounded block sizes while preserving the existing single-call API for small and sandboxed reads. ## What changed - Add `ExecServerClient::stream`, returning a named `FileReadStream` that implements `futures::Stream` and yields immutable 1 MiB byte blocks. - Add internal `fs/open`, `fs/readBlock`, and `fs/close` RPCs. `fs/readBlock` accepts an explicit offset and length. - Keep unsandboxed files open between block reads, cap open handles per connection, and clean them up on EOF, error, stream drop, explicit close, or connection shutdown. - Reject platform-sandboxed streaming opens instead of turning the one-shot sandbox helper into a persistent server. Existing `fs/readFile` behavior is unchanged. ## Testing - `just test -p codex-exec-server` - Integration coverage for 1 MiB chunking, exact block-boundary EOF, sandbox rejection, and continued reads from the opened file after path replacement. - Handle-manager coverage for non-sequential offsets, variable block lengths, the 128-handle limit, and capacity release after close.
This commit is contained in:
1 parent
76135cbe7e
commit
a4711b88dd
25 files changed
+1229
-36
No files matched your search
@@ -3,6 +3,7 @@ use codex_file_system::CopyOptions;
|
||||
use codex_file_system::CreateDirectoryOptions;
|
||||
use codex_file_system::ExecutorFileSystemFuture;
|
||||
use codex_file_system::FileMetadata;
|
||||
use codex_file_system::FileSystemReadStream;
|
||||
use codex_file_system::FileSystemSandboxContext;
|
||||
use codex_file_system::ReadDirectoryEntry;
|
||||
use codex_file_system::RemoveOptions;
|
||||
@@ -36,6 +37,19 @@ impl ExecutorFileSystem for TestFileSystem {
|
||||
})
|
||||
}
|
||||
|
||||
fn read_file_stream<'a>(
|
||||
&'a self,
|
||||
_path: &'a PathUri,
|
||||
_sandbox: Option<&'a FileSystemSandboxContext>,
|
||||
) -> ExecutorFileSystemFuture<'a, FileSystemReadStream> {
|
||||
Box::pin(async {
|
||||
Err(std::io::Error::new(
|
||||
std::io::ErrorKind::Unsupported,
|
||||
"test filesystem does not support streaming reads",
|
||||
))
|
||||
})
|
||||
}
|
||||
|
||||
fn write_file<'a>(
|
||||
&'a self,
|
||||
_path: &'a PathUri,
|
||||
|
||||
Reference in new issue
Block a user