userRepository.Delete used the r.client.Tx(ctx)+ErrTxStarted pattern,
which only reuses a transaction when r.client is itself a tx-bound
client. With the singleton base client (how Wire injects repos),
r.client.Tx() always starts a NEW independent transaction and ignores
any tx carried in the context.
As a result, AdminService.DeleteUser's transaction (added in #3047)
only covered the API-key deletions; the user soft-delete escaped into a
separate, immediately-committed transaction. If the outer commit failed
after the user tx had committed, the user was deleted while its API keys
stayed active, reintroducing the unrecoverable state of #3021.
Add a leading dbent.TxFromContext check (mirroring DeleteWithAudit) so
Delete joins an externally-provided transaction and lets the caller
commit/rollback. Extract the deletion body into deleteUser(ctx, exec,
id). All three original paths are preserved; callers passing a plain ctx
are unaffected.
Add user_repo_delete_atomicity_integration_test.go: builds repos with
the base client and a manual outer tx, asserting that delete-keys and
delete-user roll back together on abort and commit together on success.
Three periodic background jobs ran on every instance with no cross-instance
coordination, multiplying their cost (and side effects) by the replica count:
- DashboardAggregationService.runScheduledAggregation: N× heavy GROUP BY
aggregation queries every minute plus watermark write races.
- PaymentOrderExpiryService.runOnce: N× upstream payment-provider reconcile/
expiry API calls per pending order.
- SubscriptionExpiryService.sendExpiryReminders: N× full active-subscription
scans every minute and potential duplicate reminder emails.
Add a LeaderLockCache abstraction so only one instance runs each job per cycle:
- The interface lives in the service layer; the Redis-backed implementation
(SetNX + compare-and-delete release) lives in the repository layer, so the
service package keeps its depguard "must not import redis" boundary intact.
- tryAcquireSingletonLeaderLock prefers the cache and falls back to a Postgres
advisory lock when Redis errors, mirroring the Ops background services. When
neither backend is configured the job runs ungated, preserving single-instance
and test behavior (no self-lockout: the lock is released every cycle).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TTFT (first_token_ms) is only recorded for streaming requests, but the
ops dashboard weighted merged TTFT percentiles by success_count (all
successful requests, streaming + non-streaming). When non-streaming
traffic was present this diluted/skewed the merged TTFT figures shown
for longer (pre-aggregated) time ranges; the realtime path was exact.
Add a per-bucket ttft_sample_count (rows that actually recorded
first_token_ms) to ops_metrics_hourly / ops_metrics_daily and weight all
TTFT percentile merges by it instead of success_count:
- hourly/daily pre-agg upserts populate and propagate ttft_sample_count;
daily TTFT p50/p90/avg now weighted by ttft_sample_count.
- dashboard hourly-row merge and cross-segment combine weight TTFT by
the streaming sample count; queryUsageLatency returns it for raw
head/tail fragments.
duration stays weighted by success_count (recorded for every request);
p95/p99/max keep the conservative MAX merge (weight-independent).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Call redis.replicate_commands() at the top of every Lua script that
reads redis.call('TIME'), so the resulting writes (ZADD, ZREMRANGEBYSCORE,
SET, DEL, EXPIRE) replicate correctly on Redis 3.2-4.x.
This is a no-op on Redis 5.0+, where effects replication is the default.
The function remains a public API on Redis 7.0+ (deprecated but supported)
and is still the documented way to opt into effects replication on older
versions, so calling it unconditionally is safe and forward-compatible.
Affected scripts (8 in total, all already using redis.call('TIME') by
design to avoid client-side clock skew across multiple app instances):
- concurrency_cache.go: acquireScript, getCountScript,
cleanupExpiredSlotsScript
- session_limit_cache.go: registerSessionScript, refreshSessionScript,
getActiveSessionCountScript, isSessionActiveScript
- user_msg_queue_cache.go: releaseLockScript
Scripts that do not invoke non-deterministic commands are intentionally
left untouched.
- TopK initial filter now drops quota-paused accounts: fold the quota check
into isAccountRequestCompatible so session-hash, TopK pool, and per-candidate
rechecks all skip paused accounts. Previously the candidate pool was built
without the quota check, so paused accounts could fill TopK and leave the
scheduler returning "no available accounts" even with healthy ones available.
- Add per-account explicit disable flags auto_pause_5h_disabled /
auto_pause_7d_disabled with toggles in EditAccountModal. Without these,
leaving the account threshold blank silently falls back to the global default,
so admins could not exempt a single account once a global default existed.
Disable is per-window: an account can opt out of 5h auto-pause while still
honoring 7d. Schedule snapshot whitelist includes the new fields, i18n EN/ZH
updated, threshold-hint text revised to explain "blank = global default".
- Move quota auto-pause settings off the request hot path: replace the per-repo
TTL+singleflight sync DB read with a per-SettingService stale-while-revalidate
in-memory snapshot. Get is non-blocking (atomic.Pointer load + async refresh
on staleness); writes via UpdateOpsAdvancedSettings push directly into the
cache through an injected sink; wire warms the cache at startup. Adds Warm
(sync) for tests/init and SetOpenAIQuotaAutoPauseSettings (sink target).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- gate previous_response_id sticky path with quota auto-pause check at both
the snapshot and DB-recheck stages (previously bypassed, #1)
- skip pausing when the usage window already reset to avoid a stale stuck-pause;
carry codex_*_reset_at / reset_after_seconds / codex_usage_updated_at through
the scheduler snapshot whitelist (#2)
- remove the incomplete limit mode; percentage threshold only (#3)
- add global default 5h/7d threshold inputs to the Ops settings dialog with
validation and en/zh i18n (#4)
- downgrade account_auto_paused_by_quota log from Info to Debug; it fires
per-candidate on the scheduling hot path (#5)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
