Preserve audio across history and tool outputs (#34385)

## Why

Audio attachments could be dropped from tool outputs and reconstructed thread history, while inline base64 payload size could overstate their model context cost.

## What changed

- Preserve remote and local audio attachments in legacy user-message events and thread history.
- Keep audio from function and custom tool outputs when the model supports it and the output fits the truncation budget; otherwise replace it with an omission marker.
- Estimate inline audio cost from its duration, with a serialized-size fallback for malformed or unsupported data URLs, so compaction and truncation use model-visible cost instead of raw base64 size.
- Replace audio in messages and tool outputs with an explanatory placeholder for models without audio input support.

## Testing

Add coverage for history round trips, dynamic tool responses, modality filtering, duration-based context estimates, output-budget truncation, and remote compaction.

GitOrigin-RevId: 5ed4628850a668f8383b7cd73db71b144cc02cd1
This commit is contained in:
nhamidi-oai authored and copyberry committed 2026-07-20 17:36:44 +00:00
1 parent 692a0fb7e5
commit 6f785632b0
20 files changed
+975 -38

No files matched your search

+21
View File
@@ -287,6 +287,27 @@ mod tests {
assert_eq!(metadata.title, "");
}
#[test]
fn event_msg_audio_only_user_message_sets_audio_placeholder_preview() {
let mut metadata = metadata_for_test();
let item = RolloutItem::EventMsg(EventMsg::UserMessage(UserMessageEvent {
client_id: None,
message: String::new(),
images: None,
local_images: vec![],
audio: Some(vec!["https://example.com/audio.mp3".to_string()]),
local_audio: vec![],
text_elements: vec![],
..Default::default()
}));
apply_rollout_item(&mut metadata, &item, "test-provider");
assert_eq!(metadata.first_user_message.as_deref(), Some("[Audio]"));
assert_eq!(metadata.preview.as_deref(), Some("[Audio]"));
assert_eq!(metadata.title, "");
}
#[test]
fn event_msg_blank_user_message_without_images_keeps_first_user_message_empty() {
let mut metadata = metadata_for_test();