Paused (schedulable=false) active accounts were excluded from the
background token refresh candidates, so their stored access_token
silently expired. The admin usage-window quota probe uses the stored
token verbatim (no token provider / no refresh), so the probe then gets
HTTP 401 UNAUTHENTICATED and the dashboard reports a false "needs
re-auth" for a perfectly healthy account, while Test Connection and live
traffic (which refresh on demand via the token provider) keep working.
Pause is not permanent rejection: error accounts are already excluded by
the status = 'active' filter, and accounts whose refresh genuinely fails
are rate-limited by the retry-cooldown exclusion (10-minute retry-
exhausted cooldown). Refresh success only persists credentials and never
touches schedulable, so a paused account cannot be silently un-paused.
Drop the schedulable filter and update the SQL-filter test, the
integration pagination test, and the service-level candidate fake.
Real Ollama Cloud 429s are driven by ollama.com usage windows, not by
response headers that OpenAI/Anthropic/CN rate-limit parsing understands.
Handle them separately: apply a never-shortening temporary cooldown first,
then schedule an async probe that learns the true usage-window reset point
with a controlled, coalesced per-api_key refresh.
- ratelimit_service: route real Ollama Cloud usage accounts to a dedicated
429 path before the CN-provider branch.
- ollama_cloud_usage: add a background probe coordinator (started in Start,
waited by Stop) that fans out model-429 probes to one controlled refresh
per api_key group, reusing the recent group snapshot for coalesced members.
- repository/account_repo: add SetRateLimitedIfUnchanged, a write-back CAS
counterpart to ClearRateLimitIfObserved, so an async reset never overwrites
a newer 429, an admin clear, a re-armed generation, or a key/state change
observed by another writer between read and write.
- wire: inject OllamaCloudUsageService into RateLimitService.
Tests: unit tests for service and repository pass; targeted race tests pass;
build and vet clean; git diff --check clean. The CAS integration test compiles
but is not run (no Docker available).
Resolve three conflicts from main's #6680 (supplementUnmappedOpenAIModels):
1. gateway_handler.go — keep PR's allowlist refactoring (ModelAllowlistEnabled,
writeAllowlistedModelsList, modelListingSource); main's OpenAI branch in
writeModelsList was auto-merged.
2. group_models_list.go (modify/delete) — delete file per PR intent; relocate
supplementUnmappedOpenAIModels to group_model_allowlist.go where gateway_service
and openai_codex_models_service can still call it.
3. openai_codex_models_service.go — combine main's supplementUnmappedOpenAIModels
wrapping with PR's ModelAllowlistEnabled() guard.
Also fix two auto-merged test files that still referenced the old
GroupModelsListConfig / ModelsListConfig types.
The redeem failure counter used INCR followed by Expire(24h) on every
attempt, so each failed attempt refreshed the key TTL and the window
never actually closed for a user who kept retrying within 24h.
- Count attempts in a fixed 10-minute window via a small Lua script
(INCR, then PEXPIRE only when the key is created or has no TTL), which
is atomic and also repairs keys that lost their TTL.
- Move the counter to the redeem:ratelimit:v2: key space so pre-existing
24h counters do not seed the new window.
- Rename the service threshold to redeemMaxFailedAttempts (30) to reflect
that it bounds attempts per window, not per hour.
Public redeem behavior is unchanged apart from the window semantics:
the check fails open when the counter read fails, and a failed
increment never masks the underlying domain error.
Long-running LLM inferences (such as high-thinking Gemini models) may experience
silent TCP disconnections across intermediate proxies, NAT gateways, or firewalls
when no data frames are emitted for tens of seconds to minutes.
Go's default http2.Transport sets ReadIdleTimeout=0 (disabling active PING checks).
This commit introduces HTTPUpstreamProfileLongStream with:
- ReadIdleTimeout = 10s
- PingTimeout = 5s
When a pooled H2 connection stays idle for 10s during long streams, active PING frames
are dispatched. Dead connections failing to respond within 5s are proactively evicted,
preventing client generation requests from hanging on dead sockets or failing with EOF/502.
Signed-off-by: WUJI-Labs <zoujincool@gmail.com>
Wesley Liddick