- Inflight estimate now mirrors billableModelWithFallback: when the primary
billing model has no price, fall back to the channel-mapped model (also for
billing_model_source=requested) and to account-level mapped models (max over
schedulable accounts in the group, cached 30s).
- Composite groups also consider concrete account-mapped models (max) so the
alias' family fuzzy price cannot underestimate.
- Renew Lua refuses (and purges) members whose score <= now.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Address gaps found in adversarial verification of the in-flight balance
reservation:
- Early release: the reservation is now a ref-counted handle carried on the
request context. Billing tasks submitted via submit*UsageRecordTask take a
ref and return it only after RecordUsage has run; with reservation enabled
the balance cache is deducted synchronously inside the billing task, so the
reservation is released only after admission can see the deduction. Dropped
tasks return their ref immediately. The reservation is renewed every ttl/3
while the handler runs; after the handler returns it is no longer renewed,
so the post-handler hold is bounded by one TTL.
- Estimate: use the same billing model (channel mapping + BillingModelSource,
max over requested/mapped when ambiguous) and ModelPricingResolver (group
ModelPricing incl. wildcards, channel per_request/image/video/token
overrides), default/group/user rate + peak multiplier, plus image, video,
audio and search surcharge kinds. Unpriced requests fail open with a
throttled log; new billing.inflight_reservation.fail_closed_on_unpriced
(default false) rejects them instead.
- Wire images, embeddings, web/x search, alpha search, grok media generation,
grok realtime/voice, OpenAI-platform /v1/messages and ResponsesWebSocket
(session-level reservation with renewal).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Balance-mode admission only checked cached balance > threshold, so N
concurrent expensive requests could all pass and overdraw the user.
Add a Redis in-flight reservation: at admission estimate a conservative
per-request cost (input estimate + output price x max_tokens, times rate
multiplier) and atomically require cached_balance - inflight >= estimate
(first in-flight request always admitted). Reservations are released via
defer on every path and expire by TTL. Fails open on estimation / Redis
errors. Configurable via billing.inflight_reservation (enabled by default).
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
OpenAI candidate filtering reads the Redis metadata projection, but
filterSchedulerExtra dropped auto_reset_credit_enabled, the 5h/7d
reset-credit thresholds and codex_auto_reset_credit_state. The scheduler
therefore always saw automatic reset credits as disabled, so accounts
were paused at the ops quota auto-pause threshold even when a fresh
state showed an available reset credit. The "keep scheduling until the
reset-credit threshold" branch never took effect, and accounts with
credits stayed idle until their window reset naturally.
- Keep the four keys in the scheduler projection; the test round-trips
the real JSON payload and resolves the credit config from it.
- Once visible, every scheduling pass notifies the auto-reset worker for
paused credit-enabled accounts, and each notification reloads the
account. Add a 30s per-account cooldown for the scheduler hot path
only; other event-driven notifications are unchanged and the
per-minute scan remains the fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Extend the local plugin system's host<->plugin contract with generic,
capability-neutral facilities, so transport plugins can be built on top
without embedding plugin-specific logic in the host.
Host services (plugin dials a broker-provided HostService gRPC):
- per-plugin namespaced KV store (key/value size and TTL bounded)
- account directory: ListAccounts + ResolveOutboundIdentity, restricted to
the accounts a plugin's declared capability already covers
- InitHostServices lifecycle hook; plugins that don't implement it run
unchanged and are simply offered no host services
Read-only runtime status channel:
- HealthResponse gains an opaque, plugin-defined status_json produced
passively (no config apply, no upstream call, no side effects)
- GET /admin/plugins/:id/status (ungated) and the plugin.status UI-bridge
verb map to Health, so a status UI can poll live state without a step-up
prompt or a "test" side effect
- config.test no longer force-shows a host success toast; the plugin UI owns
result presentation, so plugins that read status do not spam toasts
Document the UI-bridge verb taxonomy (read vs mutate/test gating) and the
status contract under pkg/pluginapi.
Paused (schedulable=false) active accounts were excluded from the
background token refresh candidates, so their stored access_token
silently expired. The admin usage-window quota probe uses the stored
token verbatim (no token provider / no refresh), so the probe then gets
HTTP 401 UNAUTHENTICATED and the dashboard reports a false "needs
re-auth" for a perfectly healthy account, while Test Connection and live
traffic (which refresh on demand via the token provider) keep working.
Pause is not permanent rejection: error accounts are already excluded by
the status = 'active' filter, and accounts whose refresh genuinely fails
are rate-limited by the retry-cooldown exclusion (10-minute retry-
exhausted cooldown). Refresh success only persists credentials and never
touches schedulable, so a paused account cannot be silently un-paused.
Drop the schedulable filter and update the SQL-filter test, the
integration pagination test, and the service-level candidate fake.
Real Ollama Cloud 429s are driven by ollama.com usage windows, not by
response headers that OpenAI/Anthropic/CN rate-limit parsing understands.
Handle them separately: apply a never-shortening temporary cooldown first,
then schedule an async probe that learns the true usage-window reset point
with a controlled, coalesced per-api_key refresh.
- ratelimit_service: route real Ollama Cloud usage accounts to a dedicated
429 path before the CN-provider branch.
- ollama_cloud_usage: add a background probe coordinator (started in Start,
waited by Stop) that fans out model-429 probes to one controlled refresh
per api_key group, reusing the recent group snapshot for coalesced members.
- repository/account_repo: add SetRateLimitedIfUnchanged, a write-back CAS
counterpart to ClearRateLimitIfObserved, so an async reset never overwrites
a newer 429, an admin clear, a re-armed generation, or a key/state change
observed by another writer between read and write.
- wire: inject OllamaCloudUsageService into RateLimitService.
Tests: unit tests for service and repository pass; targeted race tests pass;
build and vet clean; git diff --check clean. The CAS integration test compiles
but is not run (no Docker available).
Wesley Liddick