mirror of
https://github.com/openai/codex.git
synced 2026-11-13 22:47:20 +00:00
Send processed microphone audio over RTP in voice-host (#43090)
## Why `voice-host` previously drained captured audio locally without sending it to the peer. Connect capture to the outgoing media track while preserving mute boundaries and limiting stale audio. ## What changed - Resample capture and render references to 48 kHz with Rubato, apply Sonora echo cancellation, noise suppression, and gain control, and encode capture as 20 ms mono Opus packets. - Attach an Opus track to the WebRTC peer and advance RTP timestamps across mute gaps without introducing sequence-number gaps. - Prioritize queued controls and shutdown between sends. Clear pending audio and processing history on microphone mute transitions, reject delayed pre-unmute buffers, and reset partial history after capture gaps. - Size bounded capture and render queues for device rates, validate callback timing, enforce a 500 ms processing deadline, and time out stalled sends after 100 ms. Receive/decode and TUI integration remain unimplemented. The bundled Opus encoder build requires CMake and a C compiler. ## Testing Add synthetic-audio tests for Opus decoding at 44.1 and 48 kHz input rates, echo reduction, capture gaps, stale audio, queue capacity, and control priority. A loopback WebRTC test verifies RTP delivery, decoded signal, mute isolation, and timestamp and sequence continuity across mute gaps. GitOrigin-RevId: 38b8438577cfadb8124f8aeb426dffbab4845964
This commit is contained in:
1 parent
0683368584
commit
7dc7c7a756
20 files changed
+1526
-67
No files matched your search
@@ -27,6 +27,7 @@ use tokio::sync::Notify;
|
||||
use tokio::sync::mpsc;
|
||||
use tokio::time::timeout;
|
||||
use webrtc::data_channel::DataChannel;
|
||||
use webrtc::peer_connection::MediaEngine;
|
||||
use webrtc::peer_connection::PeerConnection;
|
||||
use webrtc::peer_connection::PeerConnectionBuilder;
|
||||
use webrtc::peer_connection::PeerConnectionEventHandler;
|
||||
@@ -307,7 +308,10 @@ async fn installed_client_negotiates_and_closes_over_udp_and_tcp() -> Result<()>
|
||||
let gathered = Arc::new(Notify::new());
|
||||
let mut settings = SettingEngine::default();
|
||||
settings.set_lite(/*lite*/ true);
|
||||
let mut media = MediaEngine::default();
|
||||
media.register_default_codecs()?;
|
||||
let builder = PeerConnectionBuilder::new()
|
||||
.with_media_engine(media)
|
||||
.with_setting_engine(settings)
|
||||
.with_handler(Arc::new(RemoteEvents {
|
||||
channels: sender,
|
||||
|
||||
Reference in new issue
Block a user