mirror of
https://github.com/Wei-Shaw/sub2api.git
synced 2026-10-27 16:07:02 +00:00
HuntercodeT
60756c0ca4
The passthrough streaming loop buffers leading events in `pendingLines` until
the first visible output, so a client receives **zero bytes** — not even the
HTTP response headers, since gin's ResponseWriter only commits them on the
first write — for as long as the upstream takes to produce its first visible
event.
The Forward path already recognises this exact hazard and solves it with a
downstream keepalive:
// Track downstream writes separately from upstream reads: pre-output
// failover can buffer response.created / response.in_progress, so
// keepalive must be based on downstream idle time.
The passthrough path has the same buffering but never starts a keepalive.
Reasoning models routinely think for several minutes before emitting a visible
event, so intermediate proxies close the connection on idle timeout.
Production evidence from one deployment (12h window, gpt-5.6 family):
* 44 `/v1/responses` requests produced their first visible output only after
600–900 s, each carrying 30k–50k output tokens — the upstream had finished
the work.
* Every one of them was killed by an intermediate nginx `proxy_read_timeout`
(600 s) and returned 504. The user received nothing.
* Cutting early does not even save resources: the code deliberately keeps
draining the upstream for usage after the client disconnects.
Fix: start the existing keepalive helper for the passthrough stream as well.
* `StartOpenAICompactSSEKeepalive` keeps its compact-marker gate; the gate-free
body is extracted as `startOpenAISSEKeepalive` for callers that already know
they are in an SSE context.
* The passthrough loop starts it when `gateway.stream_keepalive_interval > 0`
and stops it immediately before its first real write, so the ResponseWriter
is never shared.
Pre-output failover is unaffected: keepalive bytes are already excluded by
`OpenAICompactKeepaliveAdjustedWrittenSize`, which `openAIStreamClientOutputStarted`
on this very path already consults (#3887). No pendingLines and no
account-specific headers are ever emitted by a keepalive.
Behaviour is unchanged when `gateway.stream_keepalive_interval` is 0.