OpenAI OAuth (ChatGPT) accounts deterministically reject plan-gated
models with 400 "The 'X' model is not supported when using Codex with
a ChatGPT account". Account selection had no capability filtering for
this, so the scheduler kept picking the same account for the same model
forever; every attempt burned an upstream call and surfaced to clients
as a retryable 502, sustaining client retry storms.
Treat this 400 like upstream model-not-found: mark the (account, model)
pair via SetModelRateLimit (30min cooldown) so
IsSchedulableForModelWithContext skips the account for that model
during selection, and return true so the in-flight request fails over
to another account through the existing UpstreamFailoverError path.
Anthropic OAuth/setup-token accounts now expose a Fable-only 7d window in
429 responses (anthropic-ratelimit-unified-7d_oi-*). Previously, a 429
triggered solely by this window fell through to the legacy path and marked
the whole account rate limited (picking the sooner 5h/7d reset) even though
5h/7d were still "allowed".
Backend:
- HandleUpstreamError: when only 7d_oi is rejected/exceeded, set a
model-level rate limit (scope "claude-fable-5") and skip account-level
SetRateLimited / session-window rewrite / temp-unsched entirely.
Real 5h/7d exhaustion keeps the exact legacy account-level behavior.
- modelRateLimitKeysForRequest: append the Fable family scope for any
mapped Anthropic model containing "fable", so claude-fable-5[1m] and
other variants are excluded from scheduling together.
- Passive sampling: collect 7d_oi utilization/reset into Extra
(passive_usage_7d_oi_*), including from the 429 response itself so the
usage bar doesn't freeze at a stale value during the limit window;
cleared on 5h window reset like the existing 7d keys.
- Usage API: new seven_day_fable window in UsageInfo. GetPassiveUsage
builds it from passive samples; active queries parse the upstream
seven_day_overage_included field (claim name from the
representative-claim header) and fall back to passive data when the
upstream usage API doesn't return it, so the active-query button keeps
the bar populated. seven_day_sonnet handling is preserved unchanged.
Frontend:
- /admin/accounts usage cell renders a "7d F" (amber) progress bar next to
the existing 5h / 7d / 7d S bars for Anthropic OAuth/setup-token accounts.
The 401 handler in RateLimitService.HandleUpstreamError set
account.Credentials["expires_at"] = time.Now() and then persisted the
full credentials map via persistAccountCredentials, which routes through
accountRepository.UpdateCredentials -> ent SetCredentials and replaces
the entire JSONB column.
The account passed to the handler is the request-start snapshot taken
by the gateway at SelectAccount time. When another worker has just
rotated refresh_token via oauth_refresh_api.RefreshIfNeeded, the
snapshot still holds the old refresh_token; writing the full snapshot
back rolls refresh_token in the DB back to the stale value.
The next refresh cycle then calls the upstream with the stale token,
receives invalid_grant, and tryRecoverFromRefreshRace re-reads the DB
only to find currentRT == usedRT (because the 401 handler just poisoned
the DB), returns false, and the account is incorrectly disabled.
Drop the credentials write. InvalidateToken + SetTempUnschedulable is
sufficient: the account is held out of scheduling during the cooldown,
and after the cooldown the next request goes through token_provider's
NeedsRefresh check, which routes through the locked, DB-re-reading
RefreshIfNeeded path.
The "force background refresh by setting expires_at = now" semantic is
intentionally dropped. token_refresh_service will naturally pick the
account up when the real expires_at enters the refresh window, and if
the real expires_at has already passed by the time the account becomes
schedulable again, token_provider's NeedsRefresh returns true and
RefreshIfNeeded fires synchronously on the next request.
When an Anthropic API key's credit balance is depleted, the upstream
returns HTTP 400 with message containing "credit balance". Previously,
the 400 handler only checked for "organization has been disabled",
so credit-exhausted accounts kept being scheduled — every request
returned the same error.
Treat this case identically to 402 (Payment Required): call
handleAuthError → SetError to stop scheduling the account until
an admin manually recovers it after topping up credits.
Closes#1586
- ratelimit_service: detect non-standard OpenAI 401 format and permanently disable account
- account_test_service: mark account error on 401 during connection test
Made-with: Cursor
Move 529 overload cooldown configuration from config file to admin
settings UI. Adds an enable/disable toggle and configurable cooldown
duration (1-120 min) under /admin/settings gateway tab, stored as
JSON in the settings table.
When disabled, 529 errors are logged but accounts are no longer
paused from scheduling. Falls back to config file value when DB
is unreachable or settingService is nil.
- Add compile-time interface assertion for sessionWindowMockRepo
- Fix flaky fallback test by capturing time.Now() before calling UpdateSessionWindow
- Replace stale hardcoded timestamps with dynamic future values
- Add millisecond detection and bounds validation for reset header timestamp
- Use pause/resume pattern for interval in UsageProgressBar to avoid idle timers on large lists
- Fix gofmt comment alignment
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The 5h window reset time displayed for Setup Token accounts was inaccurate
because UpdateSessionWindow predicted the window end as "current hour + 5h"
instead of reading the actual `anthropic-ratelimit-unified-5h-reset` response
header. This caused the countdown to differ from the official Claude page.
Backend: parse the reset header (Unix timestamp) and use it as the real
window end, falling back to the hour-truncated prediction only when the
header is absent. Also correct stale predictions when a subsequent request
provides the real reset time.
Frontend: add a reactive 60s timer so the reset countdown in
UsageProgressBar ticks down in real-time instead of freezing at the
initial value.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace process-memory sync.Map + per-model runtime state with a single
"AICredits" key in model_rate_limits, making credits exhaustion fully
isomorphic with model-level rate limiting.
Scheduler: rate-limited accounts with overages enabled + credits available
are now scheduled instead of excluded.
Forwarding: when model is rate-limited + credits available, inject credits
proactively without waiting for a 429 round trip.
Storage: credits exhaustion stored as model_rate_limits["AICredits"] with
5h duration, reusing SetModelRateLimit/isRateLimitActiveForKey.
Frontend: show credits_active (yellow ⚡) when model rate-limited but
credits available, credits_exhausted (red) when AICredits key active.
Tests: add unit tests for shouldMarkCreditsExhausted, injectEnabledCreditTypes,
clearCreditsExhausted, and update existing overages tests.
The 403 detection PR changed the 401 handler condition from
`account.Type == AccountTypeOAuth` to
`account.Type == AccountTypeOAuth && account.Platform == PlatformOpenAI`,
which accidentally excluded Gemini OAuth from the temp-unschedulable path.
Fix: use `!= PlatformAntigravity` instead, preserving Gemini behavior
while correctly excluding Antigravity (whose 401 is handled by
applyErrorPolicy's temp_unschedulable_rules).
Update tests to reflect Antigravity's new 401 semantics:
- HandleUpstreamError: Antigravity OAuth 401 now uses SetError
- CheckErrorPolicy: Antigravity 401 second hit stays TempUnscheduled
- DB fallback: split into Gemini (escalates) and Antigravity (stays temp)
Backend:
- Detect and classify 403 responses into three types:
validation (account needs Google verification),
violation (terms of service / banned),
forbidden (generic 403)
- Extract verification/appeal URLs from 403 response body
(structured JSON parsing with regex fallback)
- Add needs_verify, is_banned, needs_reauth, error_code fields
to UsageInfo (omitempty for zero impact on other platforms)
- Handle 403 in request path: classify and permanently set account error
- Save validation_url in error_message for degraded path recovery
- Enrich usage with account error on both success and degraded paths
- Add singleflight dedup for usage requests with independent context
- Differentiate cache TTL: success/403 → 3min, errors → 1min
- Return degraded UsageInfo instead of HTTP 500 on quota fetch errors
Frontend:
- Display forbidden status badges with color coding (red for banned,
amber for needs verification, gray for generic)
- Show clickable verification/appeal URL links
- Display needs_reauth and degraded error states in usage cell
- Add Antigravity tier label badge next to platform type
Tests:
- Comprehensive unit tests for classifyForbiddenType (7 cases)
- Unit tests for extractValidationURL (8 cases including unicode escapes)
- Integration test for FetchQuota forbidden path
Wesley Liddick