Real Ollama Cloud 429s are driven by ollama.com usage windows, not by
response headers that OpenAI/Anthropic/CN rate-limit parsing understands.
Handle them separately: apply a never-shortening temporary cooldown first,
then schedule an async probe that learns the true usage-window reset point
with a controlled, coalesced per-api_key refresh.
- ratelimit_service: route real Ollama Cloud usage accounts to a dedicated
429 path before the CN-provider branch.
- ollama_cloud_usage: add a background probe coordinator (started in Start,
waited by Stop) that fans out model-429 probes to one controlled refresh
per api_key group, reusing the recent group snapshot for coalesced members.
- repository/account_repo: add SetRateLimitedIfUnchanged, a write-back CAS
counterpart to ClearRateLimitIfObserved, so an async reset never overwrites
a newer 429, an admin clear, a re-armed generation, or a key/state change
observed by another writer between read and write.
- wire: inject OllamaCloudUsageService into RateLimitService.
Tests: unit tests for service and repository pass; targeted race tests pass;
build and vet clean; git diff --check clean. The CAS integration test compiles
but is not run (no Docker available).
Skip OpenAI messages-dispatch gpt-5.x defaults for OpenCode groups and
composite OpenCode targets. Filter unpriced claude-* billing candidates,
estimate count_tokens locally, sanitize body session IDs before setting
X-OpenCode-Session, and treat Zen 402 as recoverable payg cooldown.
Stop rewriting the inbound body before protocol routing so account
model_mapping applies once and usage_logs.model stays the client name.
Keep the internal platform ID opencode_go. Product surface is OpenCode
with account_mode zen|go, model-aware routing across Chat Completions,
Responses, and Anthropic Messages, and X-OpenCode-Session derived from
prompt_cache_key / metadata.user_id so prompt cache can hit.
Unmatched models fall back to /v1/chat/completions. Related to #6742.
Recognize GoUsageLimitError responses and parse both single and composite reset durations from their human-readable messages. Keep API-key accounts rate-limited until the parsed reset instead of falling back to the short default cooldown.
OpenAI OAuth (ChatGPT) accounts deterministically reject plan-gated
models with 400 "The 'X' model is not supported when using Codex with
a ChatGPT account". Account selection had no capability filtering for
this, so the scheduler kept picking the same account for the same model
forever; every attempt burned an upstream call and surfaced to clients
as a retryable 502, sustaining client retry storms.
Treat this 400 like upstream model-not-found: mark the (account, model)
pair via SetModelRateLimit (30min cooldown) so
IsSchedulableForModelWithContext skips the account for that model
during selection, and return true so the in-flight request fails over
to another account through the existing UpstreamFailoverError path.
Anthropic OAuth/setup-token accounts now expose a Fable-only 7d window in
429 responses (anthropic-ratelimit-unified-7d_oi-*). Previously, a 429
triggered solely by this window fell through to the legacy path and marked
the whole account rate limited (picking the sooner 5h/7d reset) even though
5h/7d were still "allowed".
Backend:
- HandleUpstreamError: when only 7d_oi is rejected/exceeded, set a
model-level rate limit (scope "claude-fable-5") and skip account-level
SetRateLimited / session-window rewrite / temp-unsched entirely.
Real 5h/7d exhaustion keeps the exact legacy account-level behavior.
- modelRateLimitKeysForRequest: append the Fable family scope for any
mapped Anthropic model containing "fable", so claude-fable-5[1m] and
other variants are excluded from scheduling together.
- Passive sampling: collect 7d_oi utilization/reset into Extra
(passive_usage_7d_oi_*), including from the 429 response itself so the
usage bar doesn't freeze at a stale value during the limit window;
cleared on 5h window reset like the existing 7d keys.
- Usage API: new seven_day_fable window in UsageInfo. GetPassiveUsage
builds it from passive samples; active queries parse the upstream
seven_day_overage_included field (claim name from the
representative-claim header) and fall back to passive data when the
upstream usage API doesn't return it, so the active-query button keeps
the bar populated. seven_day_sonnet handling is preserved unchanged.
Frontend:
- /admin/accounts usage cell renders a "7d F" (amber) progress bar next to
the existing 5h / 7d / 7d S bars for Anthropic OAuth/setup-token accounts.
The 401 handler in RateLimitService.HandleUpstreamError set
account.Credentials["expires_at"] = time.Now() and then persisted the
full credentials map via persistAccountCredentials, which routes through
accountRepository.UpdateCredentials -> ent SetCredentials and replaces
the entire JSONB column.
The account passed to the handler is the request-start snapshot taken
by the gateway at SelectAccount time. When another worker has just
rotated refresh_token via oauth_refresh_api.RefreshIfNeeded, the
snapshot still holds the old refresh_token; writing the full snapshot
back rolls refresh_token in the DB back to the stale value.
The next refresh cycle then calls the upstream with the stale token,
receives invalid_grant, and tryRecoverFromRefreshRace re-reads the DB
only to find currentRT == usedRT (because the 401 handler just poisoned
the DB), returns false, and the account is incorrectly disabled.
Drop the credentials write. InvalidateToken + SetTempUnschedulable is
sufficient: the account is held out of scheduling during the cooldown,
and after the cooldown the next request goes through token_provider's
NeedsRefresh check, which routes through the locked, DB-re-reading
RefreshIfNeeded path.
The "force background refresh by setting expires_at = now" semantic is
intentionally dropped. token_refresh_service will naturally pick the
account up when the real expires_at enters the refresh window, and if
the real expires_at has already passed by the time the account becomes
schedulable again, token_provider's NeedsRefresh returns true and
RefreshIfNeeded fires synchronously on the next request.
When an Anthropic API key's credit balance is depleted, the upstream
returns HTTP 400 with message containing "credit balance". Previously,
the 400 handler only checked for "organization has been disabled",
so credit-exhausted accounts kept being scheduled — every request
returned the same error.
Treat this case identically to 402 (Payment Required): call
handleAuthError → SetError to stop scheduling the account until
an admin manually recovers it after topping up credits.
Closes#1586
- ratelimit_service: detect non-standard OpenAI 401 format and permanently disable account
- account_test_service: mark account error on 401 during connection test
Made-with: Cursor
alfadb