Support image references by file ID in inputs and tool outputs (#45794)

## What changed

- Accept `fileId` alongside the existing `url` form for app-server image inputs, and forward file references to the Responses API as `file_id`. Update generated schemas and client types.
- Preserve file references, image detail hints, and mixed inline/file image ordering through user-message events, thread history, and rollout migration. Retain file images when truncating tool output.
- Pass file references through image preparation without resolving them, while keeping resize-notice numbering correct. Omit them from unsupported TUI display and Guardian image context.
- Reject image-edit requests whose recent-image window includes a file reference, preventing selection of an older inline image instead.

## Testing

Add coverage for serialization, request and rollout preservation, mixed-image history ordering, incomplete ordering metadata, tool-output truncation, and rejection of unsupported image-edit selections.

GitOrigin-RevId: 6ca20a8577155cc934b720803c3b7b3bffdf972a
This commit is contained in:
Krish Chainani authored and copyberry committed 2026-09-15 21:19:08 +00:00
1 parent c51cb968e4
commit 7b8b17b97a
95 files changed
+2434 -324

No files matched your search

@@ -1771,6 +1771,18 @@
],
"title": "ImageUrlFunctionCallOutputContentItem",
"type": "object"
},
{
"properties": {
"file_id": {
"type": "string"
}
},
"required": [
"file_id"
],
"title": "FileIdFunctionCallOutputContentItem",
"type": "object"
}
],
"properties": {
@@ -6469,6 +6481,32 @@
"type": "object"
},
{
"anyOf": [
{
"properties": {
"url": {
"type": "string"
}
},
"required": [
"url"
],
"title": "UrlUserInput",
"type": "object"
},
{
"properties": {
"fileId": {
"type": "string"
}
},
"required": [
"fileId"
],
"title": "FileIdUserInput",
"type": "object"
}
],
"properties": {
"detail": {
"anyOf": [
@@ -6487,14 +6525,10 @@
],
"title": "ImageUserInputType",
"type": "string"
},
"url": {
"type": "string"
}
},
"required": [
"type",
"url"
"type"
],
"title": "ImageUserInput",
"type": "object"