mirror of
https://github.com/openai/codex.git
synced 2026-11-17 22:59:25 +00:00
Send processed microphone audio over RTP in voice-host (#43090)
## Why `voice-host` previously drained captured audio locally without sending it to the peer. Connect capture to the outgoing media track while preserving mute boundaries and limiting stale audio. ## What changed - Resample capture and render references to 48 kHz with Rubato, apply Sonora echo cancellation, noise suppression, and gain control, and encode capture as 20 ms mono Opus packets. - Attach an Opus track to the WebRTC peer and advance RTP timestamps across mute gaps without introducing sequence-number gaps. - Prioritize queued controls and shutdown between sends. Clear pending audio and processing history on microphone mute transitions, reject delayed pre-unmute buffers, and reset partial history after capture gaps. - Size bounded capture and render queues for device rates, validate callback timing, enforce a 500 ms processing deadline, and time out stalled sends after 100 ms. Receive/decode and TUI integration remain unimplemented. The bundled Opus encoder build requires CMake and a C compiler. ## Testing Add synthetic-audio tests for Opus decoding at 44.1 and 48 kHz input rates, echo reduction, capture gaps, stale audio, queue capacity, and control priority. A loopback WebRTC test verifies RTP delivery, decoded signal, mute isolation, and timestamp and sequence continuity across mute gaps. GitOrigin-RevId: 38b8438577cfadb8124f8aeb426dffbab4845964
This commit is contained in:
1 parent
0683368584
commit
7dc7c7a756
20 files changed
+1526
-67
No files matched your search
@@ -44,8 +44,14 @@ stale at one second. The queue's sample capacity must span more than that servic
|
||||
interval.
|
||||
Processing lag can still overflow a queue. Before devices open, the worker blocks
|
||||
on commands instead of polling.
|
||||
`devicesOpened` confirms device opening only: capture/reference are drained locally
|
||||
until the following media stage connects encoding, decoding and the peer.
|
||||
`devicesOpened` confirms device opening only. Capture now uses Rubato resampling,
|
||||
Sonora echo/noise/gain processing and 20 ms Opus encoding before sending RTP.
|
||||
Mute resets retained capture history and rejects delayed pre-unmute buffers.
|
||||
The receive/decode pipeline and TUI connection remain subsequent stages.
|
||||
|
||||
The capture encoder uses `opus 0.4.0` and its bundled `opusic-sys` build, which
|
||||
requires CMake and a C compiler. This is separate from the runtime's decoder Opus
|
||||
copy; final symbol binding and packaging validation must cover both copies.
|
||||
|
||||
Cargo Linux builds require ALSA development inputs discoverable by pkg-config
|
||||
(for example `libasound2-dev` on Debian/Ubuntu). Bazel uses the declared `alsa_lib`
|
||||
|
||||
Reference in new issue
Block a user