diff --git a/AGENTS.md b/AGENTS.md index 8edfa58bd6..1a98b0f15e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,3 +1,19 @@ +# PR 默认语义(最高优先级) + +- 未特别说明时,文档或对话中的“PR”一律指**提交到上游仓库** `Wei-Shaw/sub2api:main` +- 如果只是合并回我们自己的仓库,必须明确表述为: + - “内部同步 PR” + - “合并回我们的 main” + - “fork 内部 PR” +- 禁止将“上游 PR”和“我们自己仓库内的同步 PR”混用为同一个概念 + +## 本地依赖联调 + +- 本地 `go-sora2api` 仓库固定路径:`C:\Users\16790\GolandProjects\go-sora2api` +- 需要联调 `go-sora2api` 时,优先使用 `backend/go.mod` 的 `replace` 指向该本地路径,而不是使用 `git submodule` +- 联调完成后,如需提交或部署,再切换为 fork 仓库的明确 tag 或 commit + +--- # Sub2API 开发说明 ## 版本管理策略 @@ -97,22 +113,40 @@ git push origin main ### 前置条件 -- 本地已配置 SSH 别名 `clicodeplus` 连接到生产服务器(运行服务) -- 本地已配置 SSH 别名 `us-asaki-root` 连接到构建服务器(拉取代码、构建镜像) +- 本地已配置 SSH 别名 `clicodeplus` 连接到生产服务器(运行服务 + 构建镜像) - 生产服务器部署目录:`/root/sub2api`(正式)、`/root/sub2api-beta`(测试)、`/root/sub2api-star`(Star) - 生产服务器使用 Docker Compose 部署 -- **镜像统一在构建服务器上构建**,避免生产服务器因编译占用 CPU/内存影响线上服务 +- **镜像在生产服务器本机构建**,使用资源限制的 `limited-builder` 构建器(3 核 CPU、4G 内存),避免构建占满服务器资源影响线上服务 ### 服务器角色说明 | 服务器 | SSH 别名 | 职责 | |--------|----------|------| -| 构建服务器 | `us-asaki-root` | 拉取代码、`docker build` 构建镜像 | -| 生产服务器 | `clicodeplus` | 加载镜像、运行服务、部署验证 | +| 生产服务器 | `clicodeplus` | 拉取代码、构建镜像、运行服务、部署验证 | | 数据库服务器 | `db-clicodeplus` | PostgreSQL 16 + Redis 7,所有环境共用 | > 数据库服务器运维手册:`db-clicodeplus:/root/README.md` +### 构建器说明 + +生产服务器上配置了资源限制的 Docker buildx 构建器 `limited-builder`,**所有构建操作必须使用此构建器**: + +- **构建器名称**:`limited-builder` +- **驱动**:`docker-container`(独立容器运行 BuildKit) +- **资源限制**:3 核 CPU、4G 内存(服务器共 6 核 8G,预留一半给线上服务) +- **容器名**:`buildx_buildkit_limited-builder0` + +```bash +# 构建命令格式(必须指定 --builder) +ssh clicodeplus "cd /root/sub2api && docker buildx build --builder limited-builder --no-cache --load -t sub2api:latest -f Dockerfile ." + +# 查看构建器状态 +ssh clicodeplus "docker buildx inspect limited-builder" + +# 如果构建器容器被意外删除,重新创建: +ssh clicodeplus "docker buildx create --name limited-builder --driver docker-container --driver-opt 'default-load=true' && docker buildx inspect --builder limited-builder --bootstrap && docker update --cpus=3 --memory=4g --memory-swap=4g buildx_buildkit_limited-builder0" +``` + ### 部署环境说明 | 环境 | 目录(生产服务器) | 端口 | 数据库 | Redis DB | 容器名 | @@ -193,51 +227,38 @@ git push origin release/custom-0.1.69 > **检查点**:如果有其他未提交的改动,应先 commit 并 push,确保 release 分支上的所有代码都已推送到远程。 -#### 1. 构建服务器拉取代码 +#### 1. 生产服务器拉取代码 ```bash # 拉取最新代码并切换分支 -ssh us-asaki-root "cd /root/sub2api && git fetch origin && git checkout -B release/custom-0.1.69 origin/release/custom-0.1.69" +ssh clicodeplus "cd /root/sub2api && git fetch fork && git checkout -B release/custom-0.1.69 fork/release/custom-0.1.69" # ⚠️ 验证版本号与步骤 0 一致 -ssh us-asaki-root "cat /root/sub2api/backend/cmd/server/VERSION" +ssh clicodeplus "cat /root/sub2api/backend/cmd/server/VERSION" ``` -> **首次使用构建服务器?** 需要先初始化仓库,参见下方「构建服务器首次初始化」章节。 - -#### 2. 构建服务器构建镜像 +#### 2. 生产服务器构建镜像(使用 limited-builder) ```bash -ssh us-asaki-root "cd /root/sub2api && docker build --no-cache -t sub2api:latest -f Dockerfile ." +ssh clicodeplus "cd /root/sub2api && docker buildx build --builder limited-builder --no-cache --load -t sub2api:latest -f Dockerfile ." # ⚠️ 必须看到构建成功输出,如果失败需要先排查问题 ``` > **常见构建问题**: -> - `buildx` 版本过旧导致 API 版本不兼容 → 更新 buildx:`curl -fsSL "https://github.com/docker/buildx/releases/latest/download/buildx-$(curl -fsSL https://api.github.com/repos/docker/buildx/releases/latest | grep tag_name | cut -d'"' -f4).linux-amd64" -o ~/.docker/cli-plugins/docker-buildx && chmod +x ~/.docker/cli-plugins/docker-buildx` +> - 构建器未启动 → `docker buildx inspect --builder limited-builder --bootstrap` > - 磁盘空间不足 → `docker system prune -f` 清理无用镜像 +> - 构建器被删除 → 参见上方「构建器说明」重新创建 -#### 3. 传输镜像到生产服务器并加载 +#### 3. 更新镜像标签并重启 ```bash -# 导出镜像 → 通过管道传输 → 生产服务器加载 -ssh us-asaki-root "docker save sub2api:latest" | ssh clicodeplus "docker load" - -# ⚠️ 必须看到 "Loaded image: sub2api:latest" 输出 -``` - -#### 4. 生产服务器同步代码、更新标签并重启 - -```bash -# 同步代码(用于版本号确认和 deploy 配置) -ssh clicodeplus "cd /root/sub2api && git fetch fork && git checkout -B release/custom-0.1.69 fork/release/custom-0.1.69" - # 更新镜像标签并重启 ssh clicodeplus "docker tag sub2api:latest weishaw/sub2api:latest" ssh clicodeplus "cd /root/sub2api/deploy && docker compose up -d --force-recreate sub2api" ``` -#### 5. 验证部署 +#### 4. 验证部署 ```bash # 查看启动日志 @@ -252,33 +273,6 @@ ssh clicodeplus "docker ps | grep sub2api" --- -### 构建服务器首次初始化 - -首次使用 `us-asaki-root` 作为构建服务器时,需要执行以下一次性操作: - -```bash -ssh us-asaki-root - -# 1) 克隆仓库 -cd /root -git clone https://github.com/touwaeriol/sub2api.git sub2api -cd sub2api - -# 2) 验证 Docker 和 buildx 版本 -docker version -docker buildx version -# 如果 buildx 版本过旧(< v0.14),执行更新: -# LATEST=$(curl -fsSL https://api.github.com/repos/docker/buildx/releases/latest | grep tag_name | cut -d'"' -f4) -# curl -fsSL "https://github.com/docker/buildx/releases/download/${LATEST}/buildx-${LATEST}.linux-amd64" -o ~/.docker/cli-plugins/docker-buildx -# chmod +x ~/.docker/cli-plugins/docker-buildx - -# 3) 验证构建能力 -docker build --no-cache -t sub2api:test -f Dockerfile . -docker rmi sub2api:test -``` - ---- - ## Beta 并行部署(不影响现网) 目标:在同一台服务器上并行启动一个 beta 实例(例如端口 `8084`),**严禁改动/重启**现网实例(默认目录 `/root/sub2api`)。 @@ -302,21 +296,14 @@ ssh clicodeplus "docker ps --format 'table {{.Names}}\t{{.Image}}\t{{.Ports}}' | ### 首次部署步骤 -> **构建服务器说明**:正式和 beta 共用构建服务器上的 `/root/sub2api` 仓库,通过不同的镜像标签区分(`sub2api:latest` 用于正式,`sub2api:beta` 用于测试)。 +> **构建说明**:正式和 beta 通过不同的镜像标签区分(`sub2api:latest` 用于正式,`sub2api:beta` 用于测试),均在生产服务器本机使用 `limited-builder` 构建。 ```bash -# 1) 构建服务器构建 beta 镜像(共用 /root/sub2api 仓库,切到目标分支后打 beta 标签) -ssh us-asaki-root "cd /root/sub2api && git fetch origin && git checkout -B release/custom-0.1.71 origin/release/custom-0.1.71" -ssh us-asaki-root "cd /root/sub2api && docker build --no-cache -t sub2api:beta -f Dockerfile ." +# 1) 在生产服务器上拉取代码并构建 beta 镜像 +ssh clicodeplus "cd /root/sub2api-beta && git fetch --all --tags && git checkout -f release/custom-0.1.71 && git reset --hard origin/release/custom-0.1.71" +ssh clicodeplus "cd /root/sub2api-beta && docker buildx build --builder limited-builder --no-cache --load -t sub2api:beta -f Dockerfile ." -# ⚠️ 构建完成后如需恢复正式分支: -# ssh us-asaki-root "cd /root/sub2api && git checkout release/custom-<正式版本>" - -# 2) 传输镜像到生产服务器 -ssh us-asaki-root "docker save sub2api:beta" | ssh clicodeplus "docker load" -# ⚠️ 必须看到 "Loaded image: sub2api:beta" 输出 - -# 3) 在生产服务器上准备 beta 环境 +# 2) 在生产服务器上准备 beta 环境 ssh clicodeplus # 克隆代码(仅用于 deploy 配置和版本号确认,不在此构建) @@ -336,7 +323,7 @@ perl -pi -e 's/^SERVER_PORT=.*/SERVER_PORT=8084/' ./.env perl -pi -e 's/^POSTGRES_USER=.*/POSTGRES_USER=beta/' ./.env perl -pi -e 's/^POSTGRES_DB=.*/POSTGRES_DB=beta/' ./.env -# 5) 写 compose override(避免与现网容器名冲突,镜像使用构建服务器传输的 sub2api:beta,Redis 使用外部服务) +# 5) 写 compose override(避免与现网容器名冲突,镜像使用本机构建的 sub2api:beta,Redis 使用外部服务) cat > docker-compose.override.yml <<'YAML' services: sub2api: @@ -371,22 +358,15 @@ docker logs sub2api-beta --tail 50 注意:需要数据库侧已存在 `beta` 用户与 `beta` 数据库,并授予权限;否则容器会启动失败并不断重启。 -### 更新 beta(构建服务器构建 + 传输 + 仅重启 beta 容器) +### 更新 beta(本机构建 + 仅重启 beta 容器) ```bash -# 1) 构建服务器拉取代码并构建镜像(共用 /root/sub2api 仓库) -ssh us-asaki-root "cd /root/sub2api && git fetch origin && git checkout -B release/custom-0.1.71 origin/release/custom-0.1.71" -ssh us-asaki-root "cd /root/sub2api && docker build --no-cache -t sub2api:beta -f Dockerfile ." +# 1) 生产服务器拉取代码并构建镜像 +ssh clicodeplus "cd /root/sub2api-beta && git fetch --all --tags && git checkout -f release/custom-0.1.71 && git reset --hard origin/release/custom-0.1.71" +ssh clicodeplus "cd /root/sub2api-beta && docker buildx build --builder limited-builder --no-cache --load -t sub2api:beta -f Dockerfile ." # ⚠️ 必须看到构建成功输出 -# 2) 传输镜像到生产服务器 -ssh us-asaki-root "docker save sub2api:beta" | ssh clicodeplus "docker load" -# ⚠️ 必须看到 "Loaded image: sub2api:beta" 输出 - -# 3) 生产服务器同步代码(用于版本号确认和 deploy 配置) -ssh clicodeplus "set -e; cd /root/sub2api-beta && git fetch --all --tags && git checkout -f release/custom-0.1.71 && git reset --hard origin/release/custom-0.1.71" - -# 4) 重启 beta 容器并验证 +# 2) 重启 beta 容器并验证 ssh clicodeplus "cd /root/sub2api-beta/deploy && docker compose -p sub2api-beta --env-file .env -f docker-compose.yml -f docker-compose.override.yml up -d --no-deps --force-recreate sub2api" ssh clicodeplus "sleep 5 && curl -fsS http://127.0.0.1:8084/health" ssh clicodeplus "cat /root/sub2api-beta/backend/cmd/server/VERSION" @@ -402,36 +382,7 @@ ssh clicodeplus "cd /root/sub2api-beta/deploy && docker compose -p sub2api-beta ## 服务器首次部署 -### 1. 构建服务器:克隆代码并配置远程仓库 - -```bash -ssh us-asaki-root -cd /root -git clone https://github.com/Wei-Shaw/sub2api.git -cd sub2api - -# 添加 fork 仓库 -git remote add fork https://github.com/touwaeriol/sub2api.git -``` - -### 2. 构建服务器:切换到定制分支并构建镜像 - -```bash -git fetch fork -git checkout -B release/custom-0.1.69 fork/release/custom-0.1.69 - -cd /root/sub2api -docker build -t sub2api:latest -f Dockerfile . -exit -``` - -### 3. 传输镜像到生产服务器 - -```bash -ssh us-asaki-root "docker save sub2api:latest" | ssh clicodeplus "docker load" -``` - -### 4. 生产服务器:克隆代码并配置环境 +### 1. 生产服务器:克隆代码并配置环境 ```bash ssh clicodeplus @@ -462,14 +413,24 @@ services: YAML ``` -### 5. 生产服务器:更新镜像标签并启动服务 +### 2. 生产服务器:创建构建器并构建镜像 ```bash +# 创建资源限制的构建器(首次执行一次即可) +docker buildx create --name limited-builder --driver docker-container --driver-opt "default-load=true" +docker buildx inspect --builder limited-builder --bootstrap +docker update --cpus=3 --memory=4g --memory-swap=4g buildx_buildkit_limited-builder0 + +# 构建镜像 +cd /root/sub2api +docker buildx build --builder limited-builder --no-cache --load -t sub2api:latest -f Dockerfile . + +# 更新镜像标签并启动 docker tag sub2api:latest weishaw/sub2api:latest cd /root/sub2api/deploy && docker compose up -d ``` -### 6. 验证部署 +### 3. 验证部署 ```bash # 查看应用日志 @@ -482,7 +443,7 @@ curl http://localhost:8080/health cat /root/sub2api/backend/cmd/server/VERSION ``` -### 7. 常用运维命令 +### 4. 常用运维命令 ```bash # 查看实时日志 @@ -503,56 +464,6 @@ docker stats sub2api --- -## 定制功能说明 - -当前定制分支包含以下功能(相对于官方版本): - -### UI/UX 定制 - -| 功能 | 说明 | -|------|------| -| 首页优化 | 面向用户的价值主张设计 | -| 移除 GitHub 链接 | 用户菜单中不显示 GitHub 导航 | -| 微信客服按钮 | 首页悬浮微信客服入口 | -| 限流时间精确显示 | 账号限流时间显示精确到秒 | - -### Antigravity 平台增强 - -| 功能 | 说明 | -|------|------| -| Scope 级别限流 | 按配额域(claude/gemini_text/gemini_image)独立限流,避免整个账号被锁定 | -| 模型级别限流 | 按具体模型(如 claude-opus-4-5)独立限流,更精细的限流控制 | -| 限流预检查 | 调度时预检查账号/模型限流状态,避免选中已限流账号 | -| 秒级冷却时间 | 支持 429 响应的秒级精确冷却时间 | -| 身份注入优化 | 模型身份信息注入 + 静默边界防止身份泄露 | -| thoughtSignature 修复 | Gemini 3 函数调用 400 错误修复 | -| max_tokens 自动修正 | 自动修正 max_tokens <= budget_tokens 导致的 400 错误 | - -### 调度算法优化 - -| 功能 | 说明 | -|------|------| -| 分层过滤选择 | 调度算法从全排序改为分层过滤,提升性能 | -| LRU 随机选择 | 相同 LRU 时间时随机选择,避免账号集中 | -| 限流等待阈值配置化 | 可配置的限流等待阈值 | - -### 运维增强 - -| 功能 | 说明 | -|------|------| -| Scope 限流统计 | 运维界面展示 Antigravity 账号 scope 级别限流统计 | -| 账号限流状态显示 | 账号列表显示 scope 和模型级别限流状态 | -| 清除限流按钮增强 | 有 scope/模型限流时也显示清除限流按钮 | - -### 其他修复 - -| 功能 | 说明 | -|------|------| -| .gitattributes | 确保迁移文件使用 LF 换行符(解决 Windows 下 SQL 摘要不一致) | -| 部署配置优化 | DATABASE_HOST 和 DATABASE_SSLMODE 可通过 .env 配置 | - ---- - ## Admin API 接口文档 ### ⚠️ API 操作流程规范 @@ -1003,7 +914,7 @@ curl -s "${BASE}/api/v1/admin/groups/all" -H "x-api-key: ${KEY}" ## 注意事项 -1. **前端必须打包进镜像**:使用 `docker build` 在构建服务器(`us-asaki-root`)上构建,Dockerfile 会自动编译前端并 embed 到后端二进制中,构建完成后通过 `docker save | docker load` 传输到生产服务器(`clicodeplus`) +1. **前端必须打包进镜像**:使用 `docker buildx build --builder limited-builder` 在生产服务器(`clicodeplus`)本机构建,Dockerfile 会自动编译前端并 embed 到后端二进制中 2. **镜像标签**:docker-compose.yml 使用 `weishaw/sub2api:latest`,本地构建后需要 `docker tag` 覆盖 @@ -1281,14 +1192,14 @@ PR 目标是上游官方仓库,**只包含通用功能改动**(bug fix、新 - 部署配置(`deploy/` 目录下的定制修改) **PR 流程**: -1. 从 `develop` 创建功能分支,只包含要提交给上游的改动 +1. 从我们的当前开发分支(如 `release/custom-0.1.93`)或对应功能分支创建 PR 分支,只包含要提交给上游的通用改动 2. 推送分支后,**等待 4 个 CI job 全部通过** 3. 确认通过后再创建 PR 4. 使用 `gh run list --repo touwaeriol/sub2api --branch ` 检查状态 -### 自有分支推送(develop / main) +### 自有分支推送(release/custom-X.Y.Z / 功能分支 / main) -推送到我们自己的 `develop` 或 `main` 分支时,包含所有改动(定制化 + 通用功能)。 +推送到我们自己的 `release/custom-X.Y.Z`、其他开发功能分支或 `main` 分支时,包含所有改动(定制化 + 通用功能)。 **推送前必须在本地执行全部 CI 检查**(不要等 GitHub Actions): diff --git a/CLAUDE.md b/CLAUDE.md index b634af05b3..1a98b0f15e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,3 +1,19 @@ +# PR 默认语义(最高优先级) + +- 未特别说明时,文档或对话中的“PR”一律指**提交到上游仓库** `Wei-Shaw/sub2api:main` +- 如果只是合并回我们自己的仓库,必须明确表述为: + - “内部同步 PR” + - “合并回我们的 main” + - “fork 内部 PR” +- 禁止将“上游 PR”和“我们自己仓库内的同步 PR”混用为同一个概念 + +## 本地依赖联调 + +- 本地 `go-sora2api` 仓库固定路径:`C:\Users\16790\GolandProjects\go-sora2api` +- 需要联调 `go-sora2api` 时,优先使用 `backend/go.mod` 的 `replace` 指向该本地路径,而不是使用 `git submodule` +- 联调完成后,如需提交或部署,再切换为 fork 仓库的明确 tag 或 commit + +--- # Sub2API 开发说明 ## 版本管理策略 @@ -448,56 +464,6 @@ docker stats sub2api --- -## 定制功能说明 - -当前定制分支包含以下功能(相对于官方版本): - -### UI/UX 定制 - -| 功能 | 说明 | -|------|------| -| 首页优化 | 面向用户的价值主张设计 | -| 移除 GitHub 链接 | 用户菜单中不显示 GitHub 导航 | -| 微信客服按钮 | 首页悬浮微信客服入口 | -| 限流时间精确显示 | 账号限流时间显示精确到秒 | - -### Antigravity 平台增强 - -| 功能 | 说明 | -|------|------| -| Scope 级别限流 | 按配额域(claude/gemini_text/gemini_image)独立限流,避免整个账号被锁定 | -| 模型级别限流 | 按具体模型(如 claude-opus-4-5)独立限流,更精细的限流控制 | -| 限流预检查 | 调度时预检查账号/模型限流状态,避免选中已限流账号 | -| 秒级冷却时间 | 支持 429 响应的秒级精确冷却时间 | -| 身份注入优化 | 模型身份信息注入 + 静默边界防止身份泄露 | -| thoughtSignature 修复 | Gemini 3 函数调用 400 错误修复 | -| max_tokens 自动修正 | 自动修正 max_tokens <= budget_tokens 导致的 400 错误 | - -### 调度算法优化 - -| 功能 | 说明 | -|------|------| -| 分层过滤选择 | 调度算法从全排序改为分层过滤,提升性能 | -| LRU 随机选择 | 相同 LRU 时间时随机选择,避免账号集中 | -| 限流等待阈值配置化 | 可配置的限流等待阈值 | - -### 运维增强 - -| 功能 | 说明 | -|------|------| -| Scope 限流统计 | 运维界面展示 Antigravity 账号 scope 级别限流统计 | -| 账号限流状态显示 | 账号列表显示 scope 和模型级别限流状态 | -| 清除限流按钮增强 | 有 scope/模型限流时也显示清除限流按钮 | - -### 其他修复 - -| 功能 | 说明 | -|------|------| -| .gitattributes | 确保迁移文件使用 LF 换行符(解决 Windows 下 SQL 摘要不一致) | -| 部署配置优化 | DATABASE_HOST 和 DATABASE_SSLMODE 可通过 .env 配置 | - ---- - ## Admin API 接口文档 ### ⚠️ API 操作流程规范 @@ -1226,14 +1192,14 @@ PR 目标是上游官方仓库,**只包含通用功能改动**(bug fix、新 - 部署配置(`deploy/` 目录下的定制修改) **PR 流程**: -1. 从 `develop` 创建功能分支,只包含要提交给上游的改动 +1. 从我们的当前开发分支(如 `release/custom-0.1.93`)或对应功能分支创建 PR 分支,只包含要提交给上游的通用改动 2. 推送分支后,**等待 4 个 CI job 全部通过** 3. 确认通过后再创建 PR 4. 使用 `gh run list --repo touwaeriol/sub2api --branch ` 检查状态 -### 自有分支推送(develop / main) +### 自有分支推送(release/custom-X.Y.Z / 功能分支 / main) -推送到我们自己的 `develop` 或 `main` 分支时,包含所有改动(定制化 + 通用功能)。 +推送到我们自己的 `release/custom-X.Y.Z`、其他开发功能分支或 `main` 分支时,包含所有改动(定制化 + 通用功能)。 **推送前必须在本地执行全部 CI 检查**(不要等 GitHub Actions): diff --git a/backend/cmd/server/wire_gen.go b/backend/cmd/server/wire_gen.go index 034c70ecde..df4aa9f179 100644 --- a/backend/cmd/server/wire_gen.go +++ b/backend/cmd/server/wire_gen.go @@ -140,7 +140,7 @@ func initializeApplication(buildInfo handler.BuildInfo) (*Application, error) { crsSyncService := service.NewCRSSyncService(accountRepository, proxyRepository, oAuthService, openAIOAuthService, geminiOAuthService, configConfig) sessionLimitCache := repository.ProvideSessionLimitCache(redisClient, configConfig) rpmCache := repository.NewRPMCache(redisClient) - accountHandler := admin.NewAccountHandler(adminService, oAuthService, openAIOAuthService, geminiOAuthService, antigravityOAuthService, rateLimitService, accountUsageService, accountTestService, concurrencyService, crsSyncService, sessionLimitCache, rpmCache, compositeTokenCacheInvalidator) + accountHandler := admin.NewAccountHandler(adminService, oAuthService, openAIOAuthService, geminiOAuthService, antigravityOAuthService, rateLimitService, accountUsageService, accountTestService, concurrencyService, crsSyncService, sessionLimitCache, rpmCache, compositeTokenCacheInvalidator, gatewayCache) adminAnnouncementHandler := admin.NewAnnouncementHandler(announcementService) dataManagementService := service.NewDataManagementService() dataManagementHandler := admin.NewDataManagementHandler(dataManagementService) @@ -169,11 +169,15 @@ func initializeApplication(buildInfo handler.BuildInfo) (*Application, error) { opsSystemLogSink := service.ProvideOpsSystemLogSink(opsRepository) opsService := service.NewOpsService(opsRepository, settingRepository, configConfig, accountRepository, userRepository, concurrencyService, gatewayService, openAIGatewayService, geminiMessagesCompatService, antigravityGatewayService, opsSystemLogSink) soraS3Storage := service.NewSoraS3Storage(settingService) - settingService.SetOnS3UpdateCallback(soraS3Storage.RefreshClient) + soraGDriveStorage := service.NewSoraGDriveStorage(settingService) + soraStorageRouter := service.NewSoraStorageRouter(settingService, soraS3Storage, soraGDriveStorage) + settingService.SetOnS3UpdateCallback(soraStorageRouter.RefreshAll) soraGenerationRepository := repository.NewSoraGenerationRepository(db) soraQuotaService := service.NewSoraQuotaService(userRepository, groupRepository, settingService) - soraGenerationService := service.NewSoraGenerationService(soraGenerationRepository, soraS3Storage, soraQuotaService) - settingHandler := admin.NewSettingHandler(settingService, emailService, turnstileService, opsService, soraS3Storage) + soraGenerationService := service.NewSoraGenerationService(soraGenerationRepository, soraStorageRouter, soraQuotaService) + settingHandler := admin.NewSettingHandler(settingService, emailService, turnstileService, opsService, soraS3Storage, soraGDriveStorage, soraGenerationService) + soraGDriveOAuthService := service.NewSoraGDriveOAuthService(settingService) + gdriveOAuthHandler := admin.NewGDriveOAuthHandler(settingService, soraGDriveOAuthService, soraGDriveStorage) opsHandler := admin.NewOpsHandler(opsService) updateCache := repository.NewUpdateCache(redisClient) gitHubReleaseClient := repository.ProvideGitHubReleaseClient(configConfig) @@ -199,7 +203,7 @@ func initializeApplication(buildInfo handler.BuildInfo) (*Application, error) { scheduledTestResultRepository := repository.NewScheduledTestResultRepository(db) scheduledTestService := service.ProvideScheduledTestService(scheduledTestPlanRepository, scheduledTestResultRepository) scheduledTestHandler := admin.NewScheduledTestHandler(scheduledTestService) - adminHandlers := handler.ProvideAdminHandlers(dashboardHandler, adminUserHandler, groupHandler, accountHandler, adminAnnouncementHandler, dataManagementHandler, oAuthHandler, openAIOAuthHandler, geminiOAuthHandler, antigravityOAuthHandler, proxyHandler, adminRedeemHandler, promoHandler, settingHandler, opsHandler, systemHandler, adminSubscriptionHandler, adminUsageHandler, userAttributeHandler, errorPassthroughHandler, adminAPIKeyHandler, scheduledTestHandler) + adminHandlers := handler.ProvideAdminHandlers(dashboardHandler, adminUserHandler, groupHandler, accountHandler, adminAnnouncementHandler, dataManagementHandler, oAuthHandler, openAIOAuthHandler, geminiOAuthHandler, antigravityOAuthHandler, gdriveOAuthHandler, proxyHandler, adminRedeemHandler, promoHandler, settingHandler, opsHandler, systemHandler, adminSubscriptionHandler, adminUsageHandler, userAttributeHandler, errorPassthroughHandler, adminAPIKeyHandler, scheduledTestHandler) usageRecordWorkerPool := service.NewUsageRecordWorkerPool(configConfig) userMsgQueueCache := repository.NewUserMsgQueueCache(redisClient) userMessageQueueService := service.ProvideUserMessageQueueService(userMsgQueueCache, rpmCache, configConfig) @@ -208,13 +212,18 @@ func initializeApplication(buildInfo handler.BuildInfo) (*Application, error) { soraSDKClient := service.ProvideSoraSDKClient(configConfig, httpUpstream, openAITokenProvider, accountRepository, soraAccountRepository) soraMediaStorage := service.ProvideSoraMediaStorage(configConfig) soraGatewayService := service.NewSoraGatewayService(soraSDKClient, rateLimitService, httpUpstream, configConfig) - soraClientHandler := handler.NewSoraClientHandler(soraGenerationService, soraQuotaService, soraS3Storage, soraGatewayService, gatewayService, soraMediaStorage, apiKeyService) + soraClientHandler := handler.NewSoraClientHandler(soraGenerationService, soraQuotaService, soraStorageRouter, soraGatewayService, gatewayService, soraMediaStorage, apiKeyService) + soraTaskRepository := repository.NewSoraTaskRepository(db) + soraTaskService := service.NewSoraTaskService(soraTaskRepository, accountRepository, soraSDKClient, httpUpstream) + soraTaskWorker := service.NewSoraTaskWorker(soraTaskService, accountRepository, soraStorageRouter, soraMediaStorage, 60*time.Second) + soraTaskWorker.Start() + soraVideosHandler := handler.NewSoraVideosHandler(soraTaskService, gatewayService, soraStorageRouter, soraMediaStorage, soraGatewayService) soraGatewayHandler := handler.NewSoraGatewayHandler(gatewayService, soraGatewayService, concurrencyService, billingCacheService, usageRecordWorkerPool, configConfig) handlerSettingHandler := handler.ProvideSettingHandler(settingService, buildInfo) totpHandler := handler.NewTotpHandler(totpService) idempotencyCoordinator := service.ProvideIdempotencyCoordinator(idempotencyRepository, configConfig) idempotencyCleanupService := service.ProvideIdempotencyCleanupService(idempotencyRepository, configConfig) - handlers := handler.ProvideHandlers(authHandler, userHandler, apiKeyHandler, usageHandler, redeemHandler, subscriptionHandler, announcementHandler, adminHandlers, gatewayHandler, openAIGatewayHandler, soraGatewayHandler, soraClientHandler, handlerSettingHandler, totpHandler, idempotencyCoordinator, idempotencyCleanupService) + handlers := handler.ProvideHandlers(authHandler, userHandler, apiKeyHandler, usageHandler, redeemHandler, subscriptionHandler, announcementHandler, adminHandlers, gatewayHandler, openAIGatewayHandler, soraGatewayHandler, soraClientHandler, soraVideosHandler, handlerSettingHandler, totpHandler, idempotencyCoordinator, idempotencyCleanupService) jwtAuthMiddleware := middleware.NewJWTAuthMiddleware(authService, userService) adminAuthMiddleware := middleware.NewAdminAuthMiddleware(authService, userService, settingService) apiKeyAuthMiddleware := middleware.NewAPIKeyAuthMiddleware(apiKeyService, subscriptionService, configConfig) @@ -230,7 +239,7 @@ func initializeApplication(buildInfo handler.BuildInfo) (*Application, error) { accountExpiryService := service.ProvideAccountExpiryService(accountRepository) subscriptionExpiryService := service.ProvideSubscriptionExpiryService(userSubscriptionRepository) scheduledTestRunnerService := service.ProvideScheduledTestRunnerService(scheduledTestPlanRepository, scheduledTestService, accountTestService, rateLimitService, configConfig) - v := provideCleanup(client, redisClient, opsMetricsCollector, opsAggregationService, opsAlertEvaluatorService, opsCleanupService, opsScheduledReportService, opsSystemLogSink, soraMediaCleanupService, schedulerSnapshotService, tokenRefreshService, accountExpiryService, subscriptionExpiryService, usageCleanupService, idempotencyCleanupService, pricingService, emailQueueService, billingCacheService, usageRecordWorkerPool, subscriptionService, oAuthService, openAIOAuthService, geminiOAuthService, antigravityOAuthService, openAIGatewayService, scheduledTestRunnerService) + v := provideCleanup(client, redisClient, opsMetricsCollector, opsAggregationService, opsAlertEvaluatorService, opsCleanupService, opsScheduledReportService, opsSystemLogSink, soraMediaCleanupService, schedulerSnapshotService, tokenRefreshService, accountExpiryService, subscriptionExpiryService, usageCleanupService, idempotencyCleanupService, pricingService, emailQueueService, billingCacheService, usageRecordWorkerPool, subscriptionService, oAuthService, openAIOAuthService, geminiOAuthService, antigravityOAuthService, openAIGatewayService, scheduledTestRunnerService, soraTaskWorker) application := &Application{ Server: httpServer, Cleanup: v, @@ -279,6 +288,7 @@ func provideCleanup( antigravityOAuth *service.AntigravityOAuthService, openAIGateway *service.OpenAIGatewayService, scheduledTestRunner *service.ScheduledTestRunnerService, + soraTaskWorker *service.SoraTaskWorker, ) func() { return func() { ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) @@ -414,6 +424,12 @@ func provideCleanup( } return nil }}, + {"SoraTaskWorker", func() error { + if soraTaskWorker != nil { + soraTaskWorker.Stop() + } + return nil + }}, } infraSteps := []cleanupStep{ diff --git a/backend/cmd/server/wire_gen_test.go b/backend/cmd/server/wire_gen_test.go index 8e203cb81c..cd50c4020e 100644 --- a/backend/cmd/server/wire_gen_test.go +++ b/backend/cmd/server/wire_gen_test.go @@ -75,6 +75,7 @@ func TestProvideCleanup_WithMinimalDependencies_NoPanic(t *testing.T) { antigravityOAuthSvc, nil, // openAIGateway nil, // scheduledTestRunner + nil, // soraTaskWorker ) require.NotPanics(t, func() { diff --git a/backend/go.mod b/backend/go.mod index 267fcf60a7..55cd5d428b 100644 --- a/backend/go.mod +++ b/backend/go.mod @@ -22,6 +22,8 @@ require ( github.com/imroc/req/v3 v3.57.0 github.com/lib/pq v1.10.9 github.com/patrickmn/go-cache v2.1.0+incompatible + github.com/pkoukk/tiktoken-go v0.1.8 + github.com/pkoukk/tiktoken-go-loader v0.0.2 github.com/pquerna/otp v1.5.0 github.com/redis/go-redis/v9 v9.17.2 github.com/refraction-networking/utls v1.8.2 @@ -37,8 +39,10 @@ require ( go.uber.org/zap v1.24.0 golang.org/x/crypto v0.48.0 golang.org/x/net v0.49.0 + golang.org/x/oauth2 v0.30.0 golang.org/x/sync v0.19.0 golang.org/x/term v0.40.0 + google.golang.org/api v0.153.0 gopkg.in/natefinch/lumberjack.v2 v2.2.1 gopkg.in/yaml.v3 v3.0.1 modernc.org/sqlite v1.44.3 @@ -46,6 +50,7 @@ require ( require ( ariga.io/atlas v0.32.1-0.20250325101103-175b25e1c1b9 // indirect + cloud.google.com/go/compute/metadata v0.7.0 // indirect dario.cat/mergo v1.0.2 // indirect github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect github.com/Microsoft/go-winio v0.6.2 // indirect @@ -106,8 +111,13 @@ require ( github.com/go-playground/universal-translator v0.18.1 // indirect github.com/go-playground/validator/v10 v10.14.0 // indirect github.com/goccy/go-json v0.10.2 // indirect + github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect + github.com/golang/protobuf v1.5.4 // indirect github.com/google/go-cmp v0.7.0 // indirect github.com/google/go-querystring v1.1.0 // indirect + github.com/google/s2a-go v0.1.7 // indirect + github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect + github.com/googleapis/gax-go/v2 v2.12.0 // indirect github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.3 // indirect github.com/hashicorp/hcl v1.0.0 // indirect github.com/hashicorp/hcl/v2 v2.18.1 // indirect @@ -138,8 +148,6 @@ require ( github.com/opencontainers/image-spec v1.1.1 // indirect github.com/pelletier/go-toml/v2 v2.2.2 // indirect github.com/pkg/errors v0.9.1 // indirect - github.com/pkoukk/tiktoken-go v0.1.8 // indirect - github.com/pkoukk/tiktoken-go-loader v0.0.2 // indirect github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect github.com/quic-go/qpack v0.6.0 // indirect @@ -165,11 +173,11 @@ require ( github.com/yusufpapurcu/wmi v1.2.4 // indirect github.com/zclconf/go-cty v1.14.4 // indirect github.com/zclconf/go-cty-yaml v1.1.0 // indirect + go.opencensus.io v0.24.0 // indirect go.opentelemetry.io/auto/sdk v1.1.0 // indirect go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect go.opentelemetry.io/otel v1.37.0 // indirect go.opentelemetry.io/otel/metric v1.37.0 // indirect - go.opentelemetry.io/otel/sdk v1.37.0 // indirect go.opentelemetry.io/otel/trace v1.37.0 // indirect go.uber.org/atomic v1.10.0 // indirect go.uber.org/automaxprocs v1.6.0 // indirect @@ -179,6 +187,8 @@ require ( golang.org/x/mod v0.32.0 // indirect golang.org/x/sys v0.41.0 // indirect golang.org/x/text v0.34.0 // indirect + google.golang.org/genproto/googleapis/api v0.0.0-20250929231259-57b25ae835d4 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20250929231259-57b25ae835d4 // indirect google.golang.org/grpc v1.75.1 // indirect google.golang.org/protobuf v1.36.10 // indirect gopkg.in/ini.v1 v1.67.0 // indirect diff --git a/backend/go.sum b/backend/go.sum index 965f74426d..612eebf179 100644 --- a/backend/go.sum +++ b/backend/go.sum @@ -1,5 +1,8 @@ ariga.io/atlas v0.32.1-0.20250325101103-175b25e1c1b9 h1:E0wvcUXTkgyN4wy4LGtNzMNGMytJN8afmIWXJVMi4cc= ariga.io/atlas v0.32.1-0.20250325101103-175b25e1c1b9/go.mod h1:Oe1xWPuu5q9LzyrWfbZmEZxFYeu4BHTyzfjeW2aZp/w= +cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= +cloud.google.com/go/compute/metadata v0.7.0 h1:PBWF+iiAerVNe8UCHxdOt6eHLVc3ydFeOCw78U8ytSU= +cloud.google.com/go/compute/metadata v0.7.0/go.mod h1:j5MvL9PprKL39t166CoB1uVHfQMs4tFQZZcKwksXUjo= dario.cat/mergo v1.0.2 h1:85+piFYR1tMbRrLcDwR18y4UKJ3aH1Tbzi24VRW1TK8= dario.cat/mergo v1.0.2/go.mod h1:E/hbnu0NxMFBjpMIE34DRGLWqDy0g5FuKDhCb31ngxA= entgo.io/ent v0.14.5 h1:Rj2WOYJtCkWyFo6a+5wB3EfBRP0rnx1fMk6gGA0UUe4= @@ -8,6 +11,7 @@ github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6 h1:He8af github.com/AdaLogics/go-fuzz-headers v0.0.0-20240806141605-e8a1dd7889d6/go.mod h1:8o94RPi1/7XTJvwPpRSzSUedZrtlirdB3r9Z20bi2f8= github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 h1:UQHMgLO+TxOElx5B5HZ4hJQsoJ/PvUvKRhJHDQXO8P8= github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E= +github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= github.com/DATA-DOG/go-sqlmock v1.5.2 h1:OcvFkGmslmlZibjAjaHm3L//6LiuBgolP7OputlJIzU= github.com/DATA-DOG/go-sqlmock v1.5.2/go.mod h1:88MAG/4G7SMwSE3CeA0ZKzrT5CiOU3OJ+JlNzwDqpNU= github.com/DouDOU-start/go-sora2api v1.1.0 h1:PxWiukK77StiHxEngOFwT1rKUn9oTAJJTl07wQUXwiU= @@ -89,15 +93,14 @@ github.com/bytedance/sonic v1.9.1 h1:6iJ6NqdoxCDr6mbY8h18oSO+cShGSMRGCEo7F2h0x8s github.com/bytedance/sonic v1.9.1/go.mod h1:i736AoUSYt75HyZLoJW9ERYxcy6eaN6h4BZXU064P/U= github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8= github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE= +github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= github.com/chenzhuoyu/base64x v0.0.0-20211019084208-fb5309c8db06/go.mod h1:DH46F32mSOjUmXrMHnKwZdA8wcEefY7UVqBKYGjpdQY= github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 h1:qSGYFH7+jGhDF8vLC+iwCD4WpbV1EBDSzWkJODFLams= github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311/go.mod h1:b583jCggY9gE99b6G5LEC39OIiVsWj+R97kbl5odCEk= -github.com/clipperhouse/stringish v0.1.1 h1:+NSqMOr3GR6k1FdRhhnXrLfztGzuG+VuFDfatpWHKCs= -github.com/clipperhouse/stringish v0.1.1/go.mod h1:v/WhFtE1q0ovMta2+m+UbpZ+2/HEXNWYXQgCt4hdOzA= -github.com/clipperhouse/uax29/v2 v2.5.0 h1:x7T0T4eTHDONxFJsL94uKNKPHrclyFI0lm7+w94cO8U= -github.com/clipperhouse/uax29/v2 v2.5.0/go.mod h1:Wn1g7MK6OoeDT0vL+Q0SQLDz/KpfsVRgg6W7ihQeh4g= +github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= +github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc= github.com/coder/websocket v1.8.14 h1:9L0p0iKiNOibykf283eHkKUHHrpG7f65OE3BhhO7v9g= github.com/coder/websocket v1.8.14/go.mod h1:NX3SzP+inril6yawo5CQXx8+fk145lPDC6pumgx0mVg= github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI= @@ -136,6 +139,10 @@ github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkp github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto= github.com/ebitengine/purego v0.8.4 h1:CF7LEKg5FFOsASUj0+QwaXf8Ht6TlFxg09+S9wz0omw= github.com/ebitengine/purego v0.8.4/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ= +github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= +github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= +github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98= +github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= github.com/fatih/color v1.18.0 h1:S8gINlzdQ840/4pfAwic/ZE0djQEH3wM94VfqLTZcOM= github.com/fatih/color v1.18.0/go.mod h1:4FelSpRwEGDpQ12mAdzqdOukCy4u8WUtOY6lkT/6HfU= github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg= @@ -173,7 +180,29 @@ github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU= github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I= github.com/golang-jwt/jwt/v5 v5.2.2 h1:Rl4B7itRWVtYIHFrSNd7vhTiz9UpLdi6gZhZ3wEeDy8= github.com/golang-jwt/jwt/v5 v5.2.2/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk= +github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= +github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da h1:oI5xCqsCo564l8iNU+DwB5epxmsaqB+rhGL0m5jtYqE= +github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= +github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= +github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= +github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= +github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= +github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= +github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8= +github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI= +github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= +github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= +github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= +github.com/google/go-cmp v0.5.3/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= @@ -182,11 +211,18 @@ github.com/google/go-querystring v1.1.0/go.mod h1:Kcdr2DB4koayq7X8pmAG4sNG59So17 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs= github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA= +github.com/google/s2a-go v0.1.7 h1:60BLSyTrOV4/haCDW4zb1guZItoSq8foHCXrAnjBo/o= +github.com/google/s2a-go v0.1.7/go.mod h1:50CgR4k1jNlWBu4UfS4AcfhVe1r6pdZPygJ3R8F0Qdw= github.com/google/subcommands v1.2.0/go.mod h1:ZjhPrFU+Olkh9WazFPsl27BQ4UPiG37m3yTrtFlrHVk= +github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/google/wire v0.7.0 h1:JxUKI6+CVBgCO2WToKy/nQk0sS+amI9z9EjVmdaocj4= github.com/google/wire v0.7.0/go.mod h1:n6YbUQD9cPKTnHXEBN2DXlOp/mVADhVErcMFb0v3J18= +github.com/googleapis/enterprise-certificate-proxy v0.3.2 h1:Vie5ybvEvT75RniqhfFxPRy3Bf7vr3h0cechB90XaQs= +github.com/googleapis/enterprise-certificate-proxy v0.3.2/go.mod h1:VLSiSSBs/ksPL8kq3OBOQ6WRI2QnaFynd1DCjZ62+V0= +github.com/googleapis/gax-go/v2 v2.12.0 h1:A+gCJKdRfqXkr+BIRGtZLibNXf0m1f9E4HG56etFpas= +github.com/googleapis/gax-go/v2 v2.12.0/go.mod h1:y+aIqrI5eb1YGMVJfuV3185Ts/D7qKpsEkdD5+I6QGU= github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg= github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= github.com/grpc-ecosystem/grpc-gateway/v2 v2.27.3 h1:NmZ1PKzSTQbuGHw9DGPFomqkkLWMC+vZCkfs+FHv1Vg= @@ -202,8 +238,6 @@ github.com/icholy/digest v1.1.0 h1:HfGg9Irj7i+IX1o1QAmPfIBNu/Q5A5Tu3n/MED9k9H4= github.com/icholy/digest v1.1.0/go.mod h1:QNrsSGQ5v7v9cReDI0+eyjsXGUoRSUZQHeQ5C4XLa0Y= github.com/imroc/req/v3 v3.57.0 h1:LMTUjNRUybUkTPn8oJDq8Kg3JRBOBTcnDhKu7mzupKI= github.com/imroc/req/v3 v3.57.0/go.mod h1:JL62ey1nvSLq81HORNcosvlf7SxZStONNqOprg0Pz00= -github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8= -github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw= github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM= github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg= github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo= @@ -239,8 +273,6 @@ github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovk github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM= github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= -github.com/mattn/go-runewidth v0.0.19 h1:v++JhqYnZuu5jSKrk9RbgF5v4CGUjqRfBm05byFGLdw= -github.com/mattn/go-runewidth v0.0.19/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs= github.com/mattn/go-sqlite3 v1.14.17 h1:mCRHCLDUBXgpKAqIKsaAaAsrAlbkeomtRFKXh2L6YIM= github.com/mattn/go-sqlite3 v1.14.17/go.mod h1:2eHXhiwb8IkHr+BDWZGa96P6+rkvnG63S2DGjv9HUNg= github.com/mdelapenya/tlscert v0.2.0 h1:7H81W6Z/4weDvZBNOfQte5GpIMo0lGYEeWbkGp5LJHI= @@ -274,8 +306,6 @@ github.com/morikuni/aec v1.0.0 h1:nP9CBfwrvYnBRgY6qfDQkygYDmYwOilePFkwzv4dU8A= github.com/morikuni/aec v1.0.0/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc= github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w= github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls= -github.com/olekukonko/tablewriter v0.0.5 h1:P2Ga83D34wi1o9J6Wh1mRuqd4mF/x/lgBS7N7AbDhec= -github.com/olekukonko/tablewriter v0.0.5/go.mod h1:hPp6KlRPjbx+hW8ykQs1w3UBbZlj6HuIJcUGPhkA7kY= github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U= github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM= github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040= @@ -299,6 +329,7 @@ github.com/pquerna/otp v1.5.0 h1:NMMR+WrmaqXU4EzdGJEE1aUUI0AMRzsp96fFFWNPwxs= github.com/pquerna/otp v1.5.0/go.mod h1:dkJfzwRKNiegxyNb54X/3fLwhCynbMspSyWKnvi1AEg= github.com/prashantv/gostub v1.1.0 h1:BTyx3RfQjRHnUWaGF9oQos79AlQ5k8WNktv7VGvVH4g= github.com/prashantv/gostub v1.1.0/go.mod h1:A5zLQHz7ieHGG7is6LLXLz7I8+3LZzsrV0P1IAHhP5U= +github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8= github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII= github.com/quic-go/quic-go v0.57.1 h1:25KAAR9QR8KZrCZRThWMKVAwGoiHIrNbT72ULHTuI10= @@ -331,8 +362,6 @@ github.com/spf13/afero v1.11.0 h1:WJQKhtpdm3v2IzqG8VMqrr6Rf3UYpEF239Jy9wNepM8= github.com/spf13/afero v1.11.0/go.mod h1:GH9Y3pIexgf1MTIWtNGyogA5MwRIDXGUr+hbWNoBjkY= github.com/spf13/cast v1.6.0 h1:GEiTHELF+vaR5dhz3VqZfFSzZjYbgeKDpBxQVS4GYJ0= github.com/spf13/cast v1.6.0/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo= -github.com/spf13/cobra v1.7.0 h1:hyqWnYt1ZQShIddO5kBpj3vu05/++x6tJ6dg8EC572I= -github.com/spf13/cobra v1.7.0/go.mod h1:uLxZILRyS/50WlhOIKD7W6V5bgeIt+4sICxh6uRMrb0= github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA= github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg= github.com/spf13/viper v1.18.2 h1:LUXCnvUvSM6FXAsj6nnfc8Q2tp1dIgUfY9Kc8GsSOiQ= @@ -389,6 +418,8 @@ github.com/zclconf/go-cty-yaml v1.1.0 h1:nP+jp0qPHv2IhUVqmQSzjvqAWcObN0KBkUl2rWB github.com/zclconf/go-cty-yaml v1.1.0/go.mod h1:9YLUH4g7lOhVWqUbctnVlZ5KLpg7JAprQNgxSZ1Gyxs= github.com/zeromicro/go-zero v1.9.4 h1:aRLFoISqAYijABtkbliQC5SsI5TbizJpQvoHc9xup8k= github.com/zeromicro/go-zero v1.9.4/go.mod h1:a17JOTch25SWxBcUgJZYps60hygK3pIYdw7nGwlcS38= +go.opencensus.io v0.24.0 h1:y73uSU6J157QMP2kn2r30vwW1A2W2WFwSCGnAVxeaD0= +go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo= go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA= go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A= go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 h1:jq9TW8u3so/bN+JPT166wjOI6/vQPF6Xe7nMNIltagk= @@ -403,6 +434,8 @@ go.opentelemetry.io/otel/metric v1.37.0 h1:mvwbQS5m0tbmqML4NqK+e3aDiO02vsf/Wgbsd go.opentelemetry.io/otel/metric v1.37.0/go.mod h1:04wGrZurHYKOc+RKeye86GwKiTb9FKm1WHtO+4EVr2E= go.opentelemetry.io/otel/sdk v1.37.0 h1:ItB0QUqnjesGRvNcmAcU0LyvkVyGJ2xftD29bWdDvKI= go.opentelemetry.io/otel/sdk v1.37.0/go.mod h1:VredYzxUvuo2q3WRcDnKDjbdvmO0sCzOvVAiY+yUkAg= +go.opentelemetry.io/otel/sdk/metric v1.37.0 h1:90lI228XrB9jCMuSdA0673aubgRobVZFhbjxHHspCPc= +go.opentelemetry.io/otel/sdk/metric v1.37.0/go.mod h1:cNen4ZWfiD37l5NhS+Keb5RXVWZWpRE+9WyVCpbo5ps= go.opentelemetry.io/otel/trace v1.37.0 h1:HLdcFNbRQBE2imdSEgm/kwqmQj1Or1l/7bW6mxVK7z4= go.opentelemetry.io/otel/trace v1.37.0/go.mod h1:TlgrlQ+PtQO5XFerSPUYG0JSgGyryXewPGyayAWSBS0= go.opentelemetry.io/proto/otlp v1.3.1 h1:TrMUixzpM0yuc/znrFTP9MMRh8trP93mkCiDVeXrui0= @@ -422,18 +455,40 @@ go.uber.org/zap v1.24.0/go.mod h1:2kMP+WWQ8aoFoedH3T2sq6iJ2yDWpHbP0f6MQbS9Gkg= golang.org/x/arch v0.0.0-20210923205945-b76863e36670/go.mod h1:5om86z9Hs0C8fWVUuoMHwpExlXzs5Tkyp9hOrfG7pp8= golang.org/x/arch v0.3.0 h1:02VY4/ZcO/gBOH6PUaoiptASxtXU10jazRCP865E97k= golang.org/x/arch v0.3.0/go.mod h1:5om86z9Hs0C8fWVUuoMHwpExlXzs5Tkyp9hOrfG7pp8= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts= golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos= +golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 h1:mgKeJMpvi0yx/sU5GsxQ7p6s2wtOnGAHZWCHUM4KGzY= golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546/go.mod h1:j/pmGrbnkbPtQfxEe5D0VQhZC6qKbfKifgD0oM7sR70= +golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= +golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= +golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= golang.org/x/mod v0.32.0 h1:9F4d3PHLljb6x//jOyokMv3eX+YDeepZSEo3mFJy93c= golang.org/x/mod v0.32.0/go.mod h1:SgipZ/3h2Ci89DlEtEXWUk/HteuRin+HHhN+WbNhguU= +golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= +golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= +golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU= golang.org/x/net v0.0.0-20211104170005-ce137452f963/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y= golang.org/x/net v0.49.0 h1:eeHFmOGUTtaaPSGNmjBKpbng9MulQsJURQUAfUwY++o= golang.org/x/net v0.49.0/go.mod h1:/ysNB2EvaqvesRkuLAyjI1ycPZlQHM3q01F02UY/MV8= +golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= +golang.org/x/oauth2 v0.30.0 h1:dnDm7JmhM45NNpd8FDDeLhK6FwqbOf4MLCM9zb1BOHI= +golang.org/x/oauth2 v0.30.0/go.mod h1:B++QgG3ZKulg6sRPGD/mqlHQs5rB3Ml9erfeDY7xKlU= +golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4= golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI= +golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -449,22 +504,50 @@ golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= golang.org/x/term v0.40.0 h1:36e4zGLqU4yhjlmxEaagx2KuYbJq3EwY8K943ZsHcvg= golang.org/x/term v0.40.0/go.mod h1:w2P8uVp06p2iyKKuvXIm7N/y0UCRt3UfJTfZ7oOpglM= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk= golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA= golang.org/x/time v0.12.0 h1:ScB/8o8olJvc+CQPWrK3fPZNfh7qgwCrY0zJmoEQLSE= golang.org/x/time v0.12.0/go.mod h1:CDIdPxbZBQxdj6cxyCIdrNogrJKMJ7pr37NYpMcMDSg= golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= +golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= +golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= golang.org/x/tools v0.41.0 h1:a9b8iMweWG+S0OBnlU36rzLp20z1Rp10w+IY2czHTQc= golang.org/x/tools v0.41.0/go.mod h1:XSY6eDqxVNiYgezAVqqCeihT4j1U2CCsqvH3WhQpnlg= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -google.golang.org/genproto v0.0.0-20231106174013-bbf56f31fb17 h1:wpZ8pe2x1Q3f2KyT5f8oP/fa9rHAKgFPr/HZdNuS+PQ= +gonum.org/v1/gonum v0.16.0 h1:5+ul4Swaf3ESvrOnidPp4GZbzf0mxVQpDCYUQE7OJfk= +gonum.org/v1/gonum v0.16.0/go.mod h1:fef3am4MQ93R2HHpKnLk4/Tbh/s0+wqD5nfa6Pnwy4E= +google.golang.org/api v0.153.0 h1:N1AwGhielyKFaUqH07/ZSIQR3uNPcV7NVw0vj+j4iR4= +google.golang.org/api v0.153.0/go.mod h1:3qNJX5eOmhiWYc67jRA/3GsDw97UFb5ivv7Y2PrriAY= +google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= +google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= +google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= +google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= +google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo= google.golang.org/genproto/googleapis/api v0.0.0-20250929231259-57b25ae835d4 h1:8XJ4pajGwOlasW+L13MnEGA8W4115jJySQtVfS2/IBU= google.golang.org/genproto/googleapis/api v0.0.0-20250929231259-57b25ae835d4/go.mod h1:NnuHhy+bxcg30o7FnVAZbXsPHUDQ9qKWAQKCD7VxFtk= google.golang.org/genproto/googleapis/rpc v0.0.0-20250929231259-57b25ae835d4 h1:i8QOKZfYg6AbGVZzUAY3LrNWCKF8O6zFisU9Wl9RER4= google.golang.org/genproto/googleapis/rpc v0.0.0-20250929231259-57b25ae835d4/go.mod h1:HSkG/KdJWusxU1F6CNrwNDjBMgisKxGnc5dAZfT0mjQ= +google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= +google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= +google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY= +google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= +google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv2fbc= google.golang.org/grpc v1.75.1 h1:/ODCNEuf9VghjgO3rqLcfg8fiOP0nSluljWFlDxELLI= google.golang.org/grpc v1.75.1/go.mod h1:JtPAzKiq4v1xcAB2hydNlWI2RnF85XXcV0mhKXr2ecQ= +google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= +google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= +google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= +google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= +google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= +google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU= +google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c= google.golang.org/protobuf v1.36.10 h1:AYd7cD/uASjIL6Q9LiTjz8JLcrh/88q5UObnmY3aOOE= google.golang.org/protobuf v1.36.10/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= @@ -479,6 +562,8 @@ gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= gotest.tools/v3 v3.5.2 h1:7koQfIKdy+I8UTetycgUqXWSDwpgv193Ka+qRsmBY8Q= gotest.tools/v3 v3.5.2/go.mod h1:LtdLGcnqToBH83WByAAi/wiwSFCArdFIUV/xxN4pcjA= +honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= +honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4= modernc.org/cc/v4 v4.27.1 h1:9W30zRlYrefrDV2JE2O8VDtJ1yPGownxciz5rrbQZis= modernc.org/cc/v4 v4.27.1/go.mod h1:uVtb5OGqUKpoLWhqwNQo/8LwvoiEBLvZXIQ/SmO6mL0= modernc.org/ccgo/v4 v4.30.1 h1:4r4U1J6Fhj98NKfSjnPUN7Ze2c6MnAdL0hWw6+LrJpc= diff --git a/backend/internal/handler/admin/account_data_handler_test.go b/backend/internal/handler/admin/account_data_handler_test.go index 285033a17d..e516b275f2 100644 --- a/backend/internal/handler/admin/account_data_handler_test.go +++ b/backend/internal/handler/admin/account_data_handler_test.go @@ -65,6 +65,7 @@ func setupAccountDataRouter() (*gin.Engine, *stubAdminService) { nil, nil, nil, + nil, ) router.GET("/api/v1/admin/accounts/data", h.ExportData) diff --git a/backend/internal/handler/admin/account_handler.go b/backend/internal/handler/admin/account_handler.go index 1f0d0b52e1..0b66d66ac0 100644 --- a/backend/internal/handler/admin/account_handler.go +++ b/backend/internal/handler/admin/account_handler.go @@ -9,6 +9,7 @@ import ( "errors" "fmt" "log" + "log/slog" "net/http" "strconv" "strings" @@ -57,6 +58,7 @@ type AccountHandler struct { sessionLimitCache service.SessionLimitCache rpmCache service.RPMCache tokenCacheInvalidator service.TokenCacheInvalidator + gatewayCache service.GatewayCache } // NewAccountHandler creates a new admin account handler @@ -74,6 +76,7 @@ func NewAccountHandler( sessionLimitCache service.SessionLimitCache, rpmCache service.RPMCache, tokenCacheInvalidator service.TokenCacheInvalidator, + gatewayCache service.GatewayCache, ) *AccountHandler { return &AccountHandler{ adminService: adminService, @@ -89,6 +92,7 @@ func NewAccountHandler( sessionLimitCache: sessionLimitCache, rpmCache: rpmCache, tokenCacheInvalidator: tokenCacheInvalidator, + gatewayCache: gatewayCache, } } @@ -206,6 +210,29 @@ func (h *AccountHandler) buildAccountResponseWithRuntime(ctx context.Context, ac } } + // 亲和客户端数据(启用亲和的账号始终返回 count,即使为 0) + if account.IsClientAffinityEnabled() { + if h.gatewayCache != nil && len(account.GroupIDs) > 0 { + accountGroups := map[int64][]int64{account.ID: account.GroupIDs} + if clients, err := h.gatewayCache.GetAccountAffinityClientsBatch(ctx, accountGroups, service.ClientAffinityTTL); err == nil { + if cl, ok := clients[account.ID]; ok && len(cl) > 0 { + count := int64(len(cl)) + item.AffinityClientCount = &count + item.AffinityClients = cl + } else { + zero := int64(0) + item.AffinityClientCount = &zero + } + } else { + zero := int64(0) + item.AffinityClientCount = &zero + } + } else { + zero := int64(0) + item.AffinityClientCount = &zero + } + } + return item } @@ -318,6 +345,21 @@ func (h *AccountHandler) List(c *gin.Context) { _ = g.Wait() } + // 获取亲和客户端数据(Redis Pipeline,低开销) + var affinityClients map[int64][]string + if h.gatewayCache != nil { + accountGroups := make(map[int64][]int64) + for i := range accounts { + acc := &accounts[i] + if acc.IsClientAffinityEnabled() && len(acc.GroupIDs) > 0 { + accountGroups[acc.ID] = acc.GroupIDs + } + } + if len(accountGroups) > 0 { + affinityClients, _ = h.gatewayCache.GetAccountAffinityClientsBatch(c.Request.Context(), accountGroups, service.ClientAffinityTTL) + } + } + // Build response with concurrency info result := make([]AccountWithConcurrency, len(accounts)) for i := range accounts { @@ -348,6 +390,18 @@ func (h *AccountHandler) List(c *gin.Context) { } } + // 注入亲和客户端数据到 DTO(启用亲和的账号始终返回 count,即使为 0) + if acc.IsClientAffinityEnabled() { + if clients, ok := affinityClients[acc.ID]; ok && len(clients) > 0 { + count := int64(len(clients)) + item.AffinityClientCount = &count + item.AffinityClients = clients + } else { + zero := int64(0) + item.AffinityClientCount = &zero + } + } + result[i] = item } @@ -568,6 +622,16 @@ func (h *AccountHandler) Update(c *gin.Context) { // base_rpm 输入校验:负值归零,超过 10000 截断 sanitizeExtraBaseRPM(req.Extra) + // 记录更新前的亲和状态,用于检测亲和关闭时清理 Redis 记录 + oldAffinityEnabled := false + var oldGroupIDs []int64 + if len(req.Extra) > 0 && h.gatewayCache != nil { + if oldAccount, err := h.adminService.GetAccount(c.Request.Context(), accountID); err == nil { + oldAffinityEnabled = oldAccount.IsClientAffinityEnabled() + oldGroupIDs = oldAccount.GroupIDs + } + } + // 确定是否跳过混合渠道检查 skipCheck := req.ConfirmMixedChannelRisk != nil && *req.ConfirmMixedChannelRisk @@ -604,6 +668,15 @@ func (h *AccountHandler) Update(c *gin.Context) { return } + // 亲和关闭时清理 Redis 中的亲和记录 + if oldAffinityEnabled && !account.IsClientAffinityEnabled() { + groupIDs := oldGroupIDs + if len(account.GroupIDs) > 0 { + groupIDs = mergeGroupIDs(oldGroupIDs, account.GroupIDs) + } + h.clearAccountAffinity(c.Request.Context(), accountID, groupIDs) + } + response.Success(c, h.buildAccountResponseWithRuntime(c.Request.Context(), account)) } @@ -1563,6 +1636,75 @@ func (h *AccountHandler) ResetQuota(c *gin.Context) { response.Success(c, h.buildAccountResponseWithRuntime(c.Request.Context(), account)) } +// GetAffinityClients returns the list of affinity clients for an account with last active timestamps. +// GET /api/v1/admin/accounts/:id/affinity-clients +func (h *AccountHandler) GetAffinityClients(c *gin.Context) { + accountID, err := strconv.ParseInt(c.Param("id"), 10, 64) + if err != nil { + response.BadRequest(c, "Invalid account ID") + return + } + + account, err := h.adminService.GetAccount(c.Request.Context(), accountID) + if err != nil { + response.ErrorFrom(c, err) + return + } + + if !account.IsClientAffinityEnabled() { + response.Success(c, []service.AffinityClient{}) + return + } + + if h.gatewayCache == nil || len(account.GroupIDs) == 0 { + response.Success(c, []service.AffinityClient{}) + return + } + + clients, err := h.gatewayCache.GetAccountAffinityClientsWithScores( + c.Request.Context(), accountID, account.GroupIDs, service.ClientAffinityTTL, + ) + if err != nil { + response.Success(c, []service.AffinityClient{}) + return + } + + response.Success(c, clients) +} + +// clearAccountAffinity 清除指定账号在所有分组的亲和记录。 +func (h *AccountHandler) clearAccountAffinity(ctx context.Context, accountID int64, groupIDs []int64) { + if h.gatewayCache == nil || len(groupIDs) == 0 { + return + } + if err := h.gatewayCache.ClearAccountAffinity(ctx, accountID, groupIDs); err != nil { + // 清理失败不影响主流程,记录日志即可 + slog.Warn("clear account affinity failed", + "account_id", accountID, + "error", err, + ) + } +} + +// mergeGroupIDs 合并两个 groupID 切片并去重。 +func mergeGroupIDs(a, b []int64) []int64 { + seen := make(map[int64]struct{}, len(a)+len(b)) + result := make([]int64, 0, len(a)+len(b)) + for _, id := range a { + if _, ok := seen[id]; !ok { + seen[id] = struct{}{} + result = append(result, id) + } + } + for _, id := range b { + if _, ok := seen[id]; !ok { + seen[id] = struct{}{} + result = append(result, id) + } + } + return result +} + // GetTempUnschedulable handles getting temporary unschedulable status // GET /api/v1/admin/accounts/:id/temp-unschedulable func (h *AccountHandler) GetTempUnschedulable(c *gin.Context) { diff --git a/backend/internal/handler/admin/account_handler_mixed_channel_test.go b/backend/internal/handler/admin/account_handler_mixed_channel_test.go index 5b81db2aad..5e45619064 100644 --- a/backend/internal/handler/admin/account_handler_mixed_channel_test.go +++ b/backend/internal/handler/admin/account_handler_mixed_channel_test.go @@ -15,7 +15,7 @@ import ( func setupAccountMixedChannelRouter(adminSvc *stubAdminService) *gin.Engine { gin.SetMode(gin.TestMode) router := gin.New() - accountHandler := NewAccountHandler(adminSvc, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil) + accountHandler := NewAccountHandler(adminSvc, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil) router.POST("/api/v1/admin/accounts/check-mixed-channel", accountHandler.CheckMixedChannel) router.POST("/api/v1/admin/accounts", accountHandler.Create) router.PUT("/api/v1/admin/accounts/:id", accountHandler.Update) diff --git a/backend/internal/handler/admin/account_handler_passthrough_test.go b/backend/internal/handler/admin/account_handler_passthrough_test.go index d86501c047..a11f2c5195 100644 --- a/backend/internal/handler/admin/account_handler_passthrough_test.go +++ b/backend/internal/handler/admin/account_handler_passthrough_test.go @@ -29,6 +29,7 @@ func TestAccountHandler_Create_AnthropicAPIKeyPassthroughExtraForwarded(t *testi nil, nil, nil, + nil, ) router := gin.New() diff --git a/backend/internal/handler/admin/batch_update_credentials_test.go b/backend/internal/handler/admin/batch_update_credentials_test.go index 0b1b669174..f2cd1e3a1c 100644 --- a/backend/internal/handler/admin/batch_update_credentials_test.go +++ b/backend/internal/handler/admin/batch_update_credentials_test.go @@ -36,7 +36,7 @@ func (f *failingAdminService) UpdateAccount(ctx context.Context, id int64, input func setupAccountHandlerWithService(adminSvc service.AdminService) (*gin.Engine, *AccountHandler) { gin.SetMode(gin.TestMode) router := gin.New() - handler := NewAccountHandler(adminSvc, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil) + handler := NewAccountHandler(adminSvc, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil) router.POST("/api/v1/admin/accounts/batch-update-credentials", handler.BatchUpdateCredentials) return router, handler } diff --git a/backend/internal/handler/admin/gdrive_oauth_handler.go b/backend/internal/handler/admin/gdrive_oauth_handler.go new file mode 100644 index 0000000000..bdf549066c --- /dev/null +++ b/backend/internal/handler/admin/gdrive_oauth_handler.go @@ -0,0 +1,148 @@ +package admin + +import ( + "log/slog" + "net/http" + + "github.com/Wei-Shaw/sub2api/internal/pkg/response" + "github.com/Wei-Shaw/sub2api/internal/service" + "github.com/gin-gonic/gin" +) + +// GDriveOAuthHandler 处理 Google Drive OAuth 授权流程。 +type GDriveOAuthHandler struct { + settingService *service.SettingService + gdriveOAuth *service.SoraGDriveOAuthService + gdriveStorage *service.SoraGDriveStorage +} + +// NewGDriveOAuthHandler 创建 GDrive OAuth Handler。 +func NewGDriveOAuthHandler(settingService *service.SettingService, gdriveOAuth *service.SoraGDriveOAuthService, gdriveStorage *service.SoraGDriveStorage) *GDriveOAuthHandler { + return &GDriveOAuthHandler{ + settingService: settingService, + gdriveOAuth: gdriveOAuth, + gdriveStorage: gdriveStorage, + } +} + +// StartOAuthRequest 启动 OAuth 授权请求。 +type StartOAuthRequest struct { + ClientID string `json:"client_id" binding:"required"` + ClientSecret string `json:"client_secret" binding:"required"` + RedirectURI string `json:"redirect_uri" binding:"required"` +} + +// StartOAuth 生成 Google OAuth 授权 URL。 +// POST /api/v1/admin/settings/sora-storage/gdrive-oauth/start +func (h *GDriveOAuthHandler) StartOAuth(c *gin.Context) { + if h.gdriveOAuth == nil { + response.Error(c, http.StatusInternalServerError, "GDrive OAuth service not initialized") + return + } + + var req StartOAuthRequest + if err := c.ShouldBindJSON(&req); err != nil { + response.BadRequest(c, "Invalid request: "+err.Error()) + return + } + + authURL, state, err := h.gdriveOAuth.GenerateAuthURL(req.ClientID, req.ClientSecret, req.RedirectURI) + if err != nil { + response.Error(c, http.StatusInternalServerError, "生成授权 URL 失败: "+err.Error()) + return + } + + response.Success(c, gin.H{ + "auth_url": authURL, + "state": state, + }) +} + +// OAuthCallbackRequest OAuth 回调请求。 +type OAuthCallbackRequest struct { + ClientID string `json:"client_id" binding:"required"` + ClientSecret string `json:"client_secret" binding:"required"` + RedirectURI string `json:"redirect_uri" binding:"required"` + Code string `json:"code" binding:"required"` + ProfileID string `json:"profile_id"` // 要保存到的 profile ID(可选) +} + +// OAuthCallback 用授权码换取 refresh_token 并保存到 profile。 +// POST /api/v1/admin/settings/sora-storage/gdrive-oauth/callback +func (h *GDriveOAuthHandler) OAuthCallback(c *gin.Context) { + if h.gdriveOAuth == nil { + response.Error(c, http.StatusInternalServerError, "GDrive OAuth service not initialized") + return + } + + var req OAuthCallbackRequest + if err := c.ShouldBindJSON(&req); err != nil { + response.BadRequest(c, "Invalid request: "+err.Error()) + return + } + + refreshToken, err := h.gdriveOAuth.ExchangeCode(c.Request.Context(), req.ClientID, req.ClientSecret, req.RedirectURI, req.Code) + if err != nil { + slog.Error("[GDriveOAuth] exchange failed", + "client_id_len", len(req.ClientID), + "client_secret_len", len(req.ClientSecret), + "redirect_uri", req.RedirectURI, + "code_len", len(req.Code), + "error", err, + ) + response.Error(c, http.StatusBadRequest, "换取 refresh_token 失败: "+err.Error()) + return + } + + // 如果指定了 profile_id,自动保存 refresh_token 到 profile + if req.ProfileID != "" { + profiles, err := h.settingService.ListSoraS3Profiles(c.Request.Context()) + if err == nil { + for _, p := range profiles.Items { + if p.ProfileID == req.ProfileID { + _, _ = h.settingService.UpdateSoraS3Profile(c.Request.Context(), req.ProfileID, &service.SoraS3Profile{ + Name: p.Name, + Provider: p.Provider, + AccessMode: p.AccessMode, + Enabled: p.Enabled, + Endpoint: p.Endpoint, + Region: p.Region, + Bucket: p.Bucket, + AccessKeyID: p.AccessKeyID, + Prefix: p.Prefix, + ForcePathStyle: p.ForcePathStyle, + CDNURL: p.CDNURL, + DefaultStorageQuotaBytes: p.DefaultStorageQuotaBytes, + AuthType: p.AuthType, + ClientID: p.ClientID, + FolderID: p.FolderID, + RefreshToken: refreshToken, + }) + break + } + } + } + } + + response.Success(c, gin.H{ + "refresh_token": refreshToken, + "message": "OAuth 授权成功", + }) +} + +// TestGDriveStorage 测试 GDrive 存储的完整上传→下载→删除流程。 +// POST /api/v1/admin/settings/sora-storage/gdrive-test +func (h *GDriveOAuthHandler) TestGDriveStorage(c *gin.Context) { + if h.gdriveStorage == nil { + response.Error(c, http.StatusInternalServerError, "GDrive storage not initialized") + return + } + + result, err := h.gdriveStorage.TestFullCycle(c.Request.Context()) + if err != nil { + response.Error(c, http.StatusBadRequest, "GDrive 测试失败: "+err.Error()) + return + } + + response.Success(c, result) +} diff --git a/backend/internal/handler/admin/setting_handler.go b/backend/internal/handler/admin/setting_handler.go index 8330868dfc..33dd066cf8 100644 --- a/backend/internal/handler/admin/setting_handler.go +++ b/backend/internal/handler/admin/setting_handler.go @@ -37,21 +37,25 @@ func generateMenuItemID() (string, error) { // SettingHandler 系统设置处理器 type SettingHandler struct { - settingService *service.SettingService - emailService *service.EmailService - turnstileService *service.TurnstileService - opsService *service.OpsService - soraS3Storage *service.SoraS3Storage + settingService *service.SettingService + emailService *service.EmailService + turnstileService *service.TurnstileService + opsService *service.OpsService + soraS3Storage *service.SoraS3Storage + soraGDriveStorage *service.SoraGDriveStorage + soraGenerationService *service.SoraGenerationService } // NewSettingHandler 创建系统设置处理器 -func NewSettingHandler(settingService *service.SettingService, emailService *service.EmailService, turnstileService *service.TurnstileService, opsService *service.OpsService, soraS3Storage *service.SoraS3Storage) *SettingHandler { +func NewSettingHandler(settingService *service.SettingService, emailService *service.EmailService, turnstileService *service.TurnstileService, opsService *service.OpsService, soraS3Storage *service.SoraS3Storage, soraGDriveStorage *service.SoraGDriveStorage, soraGenerationService *service.SoraGenerationService) *SettingHandler { return &SettingHandler{ - settingService: settingService, - emailService: emailService, - turnstileService: turnstileService, - opsService: opsService, - soraS3Storage: soraS3Storage, + settingService: settingService, + emailService: emailService, + turnstileService: turnstileService, + opsService: opsService, + soraS3Storage: soraS3Storage, + soraGDriveStorage: soraGDriveStorage, + soraGenerationService: soraGenerationService, } } @@ -993,6 +997,8 @@ func toSoraS3ProfileDTO(profile service.SoraS3Profile) dto.SoraS3Profile { ProfileID: profile.ProfileID, Name: profile.Name, IsActive: profile.IsActive, + Provider: profile.GetProvider(), + AccessMode: profile.AccessMode, Enabled: profile.Enabled, Endpoint: profile.Endpoint, Region: profile.Region, @@ -1004,6 +1010,13 @@ func toSoraS3ProfileDTO(profile service.SoraS3Profile) dto.SoraS3Profile { CDNURL: profile.CDNURL, DefaultStorageQuotaBytes: profile.DefaultStorageQuotaBytes, UpdatedAt: profile.UpdatedAt, + // Google Drive 专属 + AuthType: profile.AuthType, + ClientID: profile.ClientID, + ClientSecretConfigured: profile.ClientSecretConfigured, + RefreshTokenConfigured: profile.RefreshTokenConfigured, + ServiceAccountConfigured: profile.ServiceAccountConfigured, + FolderID: profile.FolderID, } } @@ -1083,6 +1096,8 @@ type CreateSoraS3ProfileRequest struct { ProfileID string `json:"profile_id"` Name string `json:"name"` SetActive bool `json:"set_active"` + Provider string `json:"provider"` // "s3" / "gdrive" + AccessMode string `json:"access_mode"` // "direct" / "proxy" Enabled bool `json:"enabled"` Endpoint string `json:"endpoint"` Region string `json:"region"` @@ -1093,10 +1108,19 @@ type CreateSoraS3ProfileRequest struct { ForcePathStyle bool `json:"force_path_style"` CDNURL string `json:"cdn_url"` DefaultStorageQuotaBytes int64 `json:"default_storage_quota_bytes"` + // Google Drive 专属 + AuthType string `json:"auth_type,omitempty"` + ClientID string `json:"client_id,omitempty"` + ClientSecret string `json:"client_secret,omitempty"` + RefreshToken string `json:"refresh_token,omitempty"` + ServiceAccountJSON string `json:"service_account_json,omitempty"` + FolderID string `json:"folder_id,omitempty"` } type UpdateSoraS3ProfileRequest struct { Name string `json:"name"` + Provider string `json:"provider"` + AccessMode string `json:"access_mode"` Enabled bool `json:"enabled"` Endpoint string `json:"endpoint"` Region string `json:"region"` @@ -1107,6 +1131,13 @@ type UpdateSoraS3ProfileRequest struct { ForcePathStyle bool `json:"force_path_style"` CDNURL string `json:"cdn_url"` DefaultStorageQuotaBytes int64 `json:"default_storage_quota_bytes"` + // Google Drive 专属 + AuthType string `json:"auth_type,omitempty"` + ClientID string `json:"client_id,omitempty"` + ClientSecret string `json:"client_secret,omitempty"` + RefreshToken string `json:"refresh_token,omitempty"` + ServiceAccountJSON string `json:"service_account_json,omitempty"` + FolderID string `json:"folder_id,omitempty"` } // CreateSoraS3Profile 创建 Sora S3 配置 @@ -1129,14 +1160,23 @@ func (h *SettingHandler) CreateSoraS3Profile(c *gin.Context) { response.BadRequest(c, "Profile ID is required") return } - if err := validateSoraS3RequiredWhenEnabled(req.Enabled, req.Endpoint, req.Bucket, req.AccessKeyID, req.SecretAccessKey, false); err != nil { - response.BadRequest(c, err.Error()) - return + // S3 专属字段验证:仅当 provider 为 s3(或未指定)时校验 + provider := req.Provider + if provider == "" { + provider = "s3" + } + if provider == "s3" { + if err := validateSoraS3RequiredWhenEnabled(req.Enabled, req.Endpoint, req.Bucket, req.AccessKeyID, req.SecretAccessKey, false); err != nil { + response.BadRequest(c, err.Error()) + return + } } created, err := h.settingService.CreateSoraS3Profile(c.Request.Context(), &service.SoraS3Profile{ ProfileID: req.ProfileID, Name: req.Name, + Provider: req.Provider, + AccessMode: req.AccessMode, Enabled: req.Enabled, Endpoint: req.Endpoint, Region: req.Region, @@ -1147,6 +1187,13 @@ func (h *SettingHandler) CreateSoraS3Profile(c *gin.Context) { ForcePathStyle: req.ForcePathStyle, CDNURL: req.CDNURL, DefaultStorageQuotaBytes: req.DefaultStorageQuotaBytes, + // Google Drive 专属 + AuthType: req.AuthType, + ClientID: req.ClientID, + ClientSecret: req.ClientSecret, + RefreshToken: req.RefreshToken, + ServiceAccountJSON: req.ServiceAccountJSON, + FolderID: req.FolderID, }, req.SetActive) if err != nil { response.ErrorFrom(c, err) @@ -1189,13 +1236,25 @@ func (h *SettingHandler) UpdateSoraS3Profile(c *gin.Context) { response.ErrorFrom(c, service.ErrSoraS3ProfileNotFound) return } - if err := validateSoraS3RequiredWhenEnabled(req.Enabled, req.Endpoint, req.Bucket, req.AccessKeyID, req.SecretAccessKey, existing.SecretAccessKeyConfigured); err != nil { - response.BadRequest(c, err.Error()) - return + // S3 专属字段验证 + provider := req.Provider + if provider == "" && existing != nil { + provider = existing.GetProvider() + } + if provider == "" { + provider = "s3" + } + if provider == "s3" { + if err := validateSoraS3RequiredWhenEnabled(req.Enabled, req.Endpoint, req.Bucket, req.AccessKeyID, req.SecretAccessKey, existing.SecretAccessKeyConfigured); err != nil { + response.BadRequest(c, err.Error()) + return + } } updated, updateErr := h.settingService.UpdateSoraS3Profile(c.Request.Context(), profileID, &service.SoraS3Profile{ Name: req.Name, + Provider: req.Provider, + AccessMode: req.AccessMode, Enabled: req.Enabled, Endpoint: req.Endpoint, Region: req.Region, @@ -1206,6 +1265,13 @@ func (h *SettingHandler) UpdateSoraS3Profile(c *gin.Context) { ForcePathStyle: req.ForcePathStyle, CDNURL: req.CDNURL, DefaultStorageQuotaBytes: req.DefaultStorageQuotaBytes, + // Google Drive 专属 + AuthType: req.AuthType, + ClientID: req.ClientID, + ClientSecret: req.ClientSecret, + RefreshToken: req.RefreshToken, + ServiceAccountJSON: req.ServiceAccountJSON, + FolderID: req.FolderID, }) if updateErr != nil { response.ErrorFrom(c, updateErr) @@ -1506,3 +1572,44 @@ func (h *SettingHandler) UpdateStreamTimeoutSettings(c *gin.Context) { ThresholdWindowMinutes: updatedSettings.ThresholdWindowMinutes, }) } + +// GetGDriveQuota 获取 Google Drive 配额信息。 +// GET /api/v1/admin/settings/sora-storage/gdrive-quota +func (h *SettingHandler) GetGDriveQuota(c *gin.Context) { + if h.soraGDriveStorage == nil { + response.Error(c, http.StatusServiceUnavailable, "GDrive storage not configured") + return + } + quota, err := h.soraGDriveStorage.GetQuotaInfo(c.Request.Context()) + if err != nil { + response.Error(c, http.StatusInternalServerError, fmt.Sprintf("failed to get GDrive quota: %v", err)) + return + } + response.Success(c, quota) +} + +// GetStorageVideoStats 获取各存储类型的视频统计信息。 +// GET /api/v1/admin/settings/sora-storage/video-stats +func (h *SettingHandler) GetStorageVideoStats(c *gin.Context) { + if h.soraGenerationService == nil { + response.Error(c, http.StatusServiceUnavailable, "generation service not configured") + return + } + + storageTypes := []string{service.SoraStorageTypeS3, service.SoraStorageTypeGDrive} + result := make(map[string]*service.StorageVideoStats, len(storageTypes)) + + for _, st := range storageTypes { + completed, inProgress, err := h.soraGenerationService.CountByStorageType(c.Request.Context(), st) + if err != nil { + log.Printf("[SettingHandler] CountByStorageType(%s) error: %v", st, err) + continue + } + result[st] = &service.StorageVideoStats{ + Completed: completed, + InProgress: inProgress, + } + } + + response.Success(c, result) +} diff --git a/backend/internal/handler/dto/mappers.go b/backend/internal/handler/dto/mappers.go index 402bf095ef..a330ace85e 100644 --- a/backend/internal/handler/dto/mappers.go +++ b/backend/internal/handler/dto/mappers.go @@ -265,6 +265,12 @@ func AccountFromServiceShallow(a *service.Account) *Account { } } + // 客户端亲和调度(Anthropic 和 Antigravity 账号) + if a.IsClientAffinityEnabled() { + enabled := true + out.ClientAffinityEnabled = &enabled + } + // 提取 API Key 账号配额限制(仅 apikey 类型有效) if a.Type == service.AccountTypeAPIKey { if limit := a.GetQuotaLimit(); limit > 0 { diff --git a/backend/internal/handler/dto/settings.go b/backend/internal/handler/dto/settings.go index 8a1bba5d2f..926ec95685 100644 --- a/backend/internal/handler/dto/settings.go +++ b/backend/internal/handler/dto/settings.go @@ -128,11 +128,13 @@ type SoraS3Settings struct { DefaultStorageQuotaBytes int64 `json:"default_storage_quota_bytes"` } -// SoraS3Profile Sora S3 存储配置项 DTO(响应用,不含敏感字段) +// SoraS3Profile Sora 存储配置项 DTO(响应用,不含敏感字段) type SoraS3Profile struct { ProfileID string `json:"profile_id"` Name string `json:"name"` IsActive bool `json:"is_active"` + Provider string `json:"provider"` // "s3" / "gdrive" + AccessMode string `json:"access_mode"` // "direct" / "proxy" Enabled bool `json:"enabled"` Endpoint string `json:"endpoint"` Region string `json:"region"` @@ -144,6 +146,14 @@ type SoraS3Profile struct { CDNURL string `json:"cdn_url"` DefaultStorageQuotaBytes int64 `json:"default_storage_quota_bytes"` UpdatedAt string `json:"updated_at"` + + // --- Google Drive 专属 --- + AuthType string `json:"auth_type,omitempty"` + ClientID string `json:"client_id,omitempty"` + ClientSecretConfigured bool `json:"client_secret_configured"` + RefreshTokenConfigured bool `json:"refresh_token_configured"` + ServiceAccountConfigured bool `json:"service_account_configured"` + FolderID string `json:"folder_id,omitempty"` } // ListSoraS3ProfilesResponse Sora S3 配置列表响应 diff --git a/backend/internal/handler/dto/types.go b/backend/internal/handler/dto/types.go index 7f1788a190..deba69204c 100644 --- a/backend/internal/handler/dto/types.go +++ b/backend/internal/handler/dto/types.go @@ -197,6 +197,14 @@ type Account struct { CacheTTLOverrideEnabled *bool `json:"cache_ttl_override_enabled,omitempty"` CacheTTLOverrideTarget *string `json:"cache_ttl_override_target,omitempty"` + // 客户端亲和调度(Anthropic 和 Antigravity 账号有效) + // 启用后新会话会优先调度到客户端之前使用过的账号 + ClientAffinityEnabled *bool `json:"client_affinity_enabled,omitempty"` + + // 亲和客户端数据(仅 admin 列表端点注入,不由 mapper 填充) + AffinityClientCount *int64 `json:"affinity_client_count,omitempty"` + AffinityClients []string `json:"affinity_clients,omitempty"` + // API Key 账号配额限制 QuotaLimit *float64 `json:"quota_limit,omitempty"` QuotaUsed *float64 `json:"quota_used,omitempty"` diff --git a/backend/internal/handler/handler.go b/backend/internal/handler/handler.go index 3f1d73ca31..37b3ab474b 100644 --- a/backend/internal/handler/handler.go +++ b/backend/internal/handler/handler.go @@ -28,6 +28,7 @@ type AdminHandlers struct { ErrorPassthrough *admin.ErrorPassthroughHandler APIKey *admin.AdminAPIKeyHandler ScheduledTest *admin.ScheduledTestHandler + GDriveOAuth *admin.GDriveOAuthHandler } // Handlers contains all HTTP handlers @@ -44,6 +45,7 @@ type Handlers struct { OpenAIGateway *OpenAIGatewayHandler SoraGateway *SoraGatewayHandler SoraClient *SoraClientHandler + SoraVideos *SoraVideosHandler Setting *SettingHandler Totp *TotpHandler } diff --git a/backend/internal/handler/sora_client_handler.go b/backend/internal/handler/sora_client_handler.go index 80acc83349..3cb85d18b6 100644 --- a/backend/internal/handler/sora_client_handler.go +++ b/backend/internal/handler/sora_client_handler.go @@ -31,7 +31,7 @@ const ( type SoraClientHandler struct { genService *service.SoraGenerationService quotaService *service.SoraQuotaService - s3Storage *service.SoraS3Storage + objectStorage service.SoraObjectStorage soraGatewayService *service.SoraGatewayService gatewayService *service.GatewayService mediaStorage *service.SoraMediaStorage @@ -48,7 +48,7 @@ type SoraClientHandler struct { func NewSoraClientHandler( genService *service.SoraGenerationService, quotaService *service.SoraQuotaService, - s3Storage *service.SoraS3Storage, + objectStorage service.SoraObjectStorage, soraGatewayService *service.SoraGatewayService, gatewayService *service.GatewayService, mediaStorage *service.SoraMediaStorage, @@ -57,7 +57,7 @@ func NewSoraClientHandler( return &SoraClientHandler{ genService: genService, quotaService: quotaService, - s3Storage: s3Storage, + objectStorage: objectStorage, soraGatewayService: soraGatewayService, gatewayService: gatewayService, mediaStorage: mediaStorage, @@ -291,11 +291,11 @@ func (h *SoraClientHandler) processGeneration(genID int64, userID int64, groupID return } - // 三层降级存储:S3 → 本地 → 上游临时 URL + // 三层降级存储:对象存储 → 本地 → 上游临时 URL storedURL, storedURLs, storageType, s3Keys, fileSize := h.storeMediaWithDegradation(ctx, userID, mediaType, mediaURL, mediaURLs) usageAdded := false - if (storageType == service.SoraStorageTypeS3 || storageType == service.SoraStorageTypeLocal) && fileSize > 0 && h.quotaService != nil { + if (service.IsObjectStorageType(storageType) || storageType == service.SoraStorageTypeLocal) && fileSize > 0 && h.quotaService != nil { if err := h.quotaService.AddUsage(ctx, userID, fileSize); err != nil { h.cleanupStoredMedia(ctx, storageType, s3Keys, storedURLs) var quotaErr *service.QuotaExceededError @@ -346,39 +346,41 @@ func (h *SoraClientHandler) storeMediaWithDegradation( urls = []string{mediaURL} } - // 第一层:尝试 S3 - if h.s3Storage != nil && h.s3Storage.Enabled(ctx) { + // 第一层:尝试对象存储(S3 / Google Drive) + if h.objectStorage != nil && h.objectStorage.Enabled(ctx) { keys := make([]string, 0, len(urls)) var totalSize int64 + var actualStorageType string allOK := true for _, u := range urls { - key, size, err := h.s3Storage.UploadFromURL(ctx, userID, u) + key, size, st, err := h.objectStorage.UploadFromURL(ctx, userID, u) if err != nil { - logger.LegacyPrintf("handler.sora_client", "[SoraClient] S3 上传失败 err=%v", err) + logger.LegacyPrintf("handler.sora_client", "[SoraClient] 对象存储上传失败 type=%s err=%v", h.objectStorage.StorageType(), err) allOK = false // 清理已上传的文件 if len(keys) > 0 { - _ = h.s3Storage.DeleteObjects(ctx, keys) + _ = h.objectStorage.DeleteObjects(ctx, keys) } break } keys = append(keys, key) totalSize += size + actualStorageType = st } if allOK && len(keys) > 0 { accessURLs := make([]string, 0, len(keys)) for _, key := range keys { - accessURL, err := h.s3Storage.GetAccessURL(ctx, key) + accessURL, err := h.objectStorage.GetAccessURL(ctx, key) if err != nil { - logger.LegacyPrintf("handler.sora_client", "[SoraClient] 生成 S3 访问 URL 失败 err=%v", err) - _ = h.s3Storage.DeleteObjects(ctx, keys) + logger.LegacyPrintf("handler.sora_client", "[SoraClient] 生成访问 URL 失败 type=%s err=%v", h.objectStorage.StorageType(), err) + _ = h.objectStorage.DeleteObjects(ctx, keys) allOK = false break } accessURLs = append(accessURLs, accessURL) } if allOK && len(accessURLs) > 0 { - return accessURLs[0], accessURLs, service.SoraStorageTypeS3, keys, totalSize + return accessURLs[0], accessURLs, actualStorageType, keys, totalSize } } } @@ -678,7 +680,7 @@ func (h *SoraClientHandler) SaveToStorage(c *gin.Context) { return } - if h.s3Storage == nil || !h.s3Storage.Enabled(c.Request.Context()) { + if h.objectStorage == nil || !h.objectStorage.Enabled(c.Request.Context()) { response.Error(c, http.StatusServiceUnavailable, "云存储未配置,请联系管理员") return } @@ -697,24 +699,24 @@ func (h *SoraClientHandler) SaveToStorage(c *gin.Context) { var totalSize int64 for _, sourceURL := range sourceURLs { - objectKey, fileSize, uploadErr := h.s3Storage.UploadFromURL(c.Request.Context(), userID, sourceURL) + objectKey, fileSize, _, uploadErr := h.objectStorage.UploadFromURL(c.Request.Context(), userID, sourceURL) if uploadErr != nil { if len(uploadedKeys) > 0 { - _ = h.s3Storage.DeleteObjects(c.Request.Context(), uploadedKeys) + _ = h.objectStorage.DeleteObjects(c.Request.Context(), uploadedKeys) } var upstreamErr *service.UpstreamDownloadError if errors.As(uploadErr, &upstreamErr) && (upstreamErr.StatusCode == http.StatusForbidden || upstreamErr.StatusCode == http.StatusNotFound) { response.Error(c, http.StatusGone, "媒体链接已过期,无法保存") return } - response.Error(c, http.StatusInternalServerError, "上传到 S3 失败: "+uploadErr.Error()) + response.Error(c, http.StatusInternalServerError, "上传到存储失败: "+uploadErr.Error()) return } - accessURL, err := h.s3Storage.GetAccessURL(c.Request.Context(), objectKey) + accessURL, err := h.objectStorage.GetAccessURL(c.Request.Context(), objectKey) if err != nil { uploadedKeys = append(uploadedKeys, objectKey) - _ = h.s3Storage.DeleteObjects(c.Request.Context(), uploadedKeys) - response.Error(c, http.StatusInternalServerError, "生成 S3 访问链接失败: "+err.Error()) + _ = h.objectStorage.DeleteObjects(c.Request.Context(), uploadedKeys) + response.Error(c, http.StatusInternalServerError, "生成访问链接失败: "+err.Error()) return } uploadedKeys = append(uploadedKeys, objectKey) @@ -725,7 +727,7 @@ func (h *SoraClientHandler) SaveToStorage(c *gin.Context) { usageAdded := false if totalSize > 0 && h.quotaService != nil { if err := h.quotaService.AddUsage(c.Request.Context(), userID, totalSize); err != nil { - _ = h.s3Storage.DeleteObjects(c.Request.Context(), uploadedKeys) + _ = h.objectStorage.DeleteObjects(c.Request.Context(), uploadedKeys) var quotaErr *service.QuotaExceededError if errors.As(err, "aErr) { response.Error(c, http.StatusTooManyRequests, "存储配额已满,请删除不需要的作品释放空间") @@ -742,11 +744,11 @@ func (h *SoraClientHandler) SaveToStorage(c *gin.Context) { id, accessURLs[0], accessURLs, - service.SoraStorageTypeS3, + h.objectStorage.StorageType(), uploadedKeys, totalSize, ); err != nil { - _ = h.s3Storage.DeleteObjects(c.Request.Context(), uploadedKeys) + _ = h.objectStorage.DeleteObjects(c.Request.Context(), uploadedKeys) if usageAdded && h.quotaService != nil { _ = h.quotaService.ReleaseUsage(c.Request.Context(), userID, totalSize) } @@ -755,7 +757,7 @@ func (h *SoraClientHandler) SaveToStorage(c *gin.Context) { } response.Success(c, gin.H{ - "message": "已保存到 S3", + "message": "已保存到云存储", "object_key": uploadedKeys[0], "object_keys": uploadedKeys, }) @@ -764,28 +766,30 @@ func (h *SoraClientHandler) SaveToStorage(c *gin.Context) { // GetStorageStatus 返回存储状态。 // GET /api/v1/sora/storage-status func (h *SoraClientHandler) GetStorageStatus(c *gin.Context) { - s3Enabled := h.s3Storage != nil && h.s3Storage.Enabled(c.Request.Context()) - s3Healthy := false - if s3Enabled { - s3Healthy = h.s3Storage.IsHealthy(c.Request.Context()) + objectStorageEnabled := h.objectStorage != nil && h.objectStorage.Enabled(c.Request.Context()) + objectStorageHealthy := false + storageType := "" + if objectStorageEnabled { + objectStorageHealthy = h.objectStorage.IsHealthy(c.Request.Context()) + storageType = h.objectStorage.StorageType() } localEnabled := h.mediaStorage != nil && h.mediaStorage.Enabled() response.Success(c, gin.H{ - "s3_enabled": s3Enabled, - "s3_healthy": s3Healthy, + "s3_enabled": objectStorageEnabled, // 保留字段名向后兼容 + "s3_healthy": objectStorageHealthy, + "storage_type": storageType, "local_enabled": localEnabled, }) } func (h *SoraClientHandler) cleanupStoredMedia(ctx context.Context, storageType string, s3Keys []string, localPaths []string) { - switch storageType { - case service.SoraStorageTypeS3: - if h.s3Storage != nil && len(s3Keys) > 0 { - if err := h.s3Storage.DeleteObjects(ctx, s3Keys); err != nil { - logger.LegacyPrintf("handler.sora_client", "[SoraClient] 清理 S3 文件失败 keys=%v err=%v", s3Keys, err) + if service.IsObjectStorageType(storageType) { + if h.objectStorage != nil && len(s3Keys) > 0 { + if err := h.objectStorage.DeleteObjects(ctx, s3Keys); err != nil { + logger.LegacyPrintf("handler.sora_client", "[SoraClient] 清理存储文件失败 type=%s keys=%v err=%v", storageType, s3Keys, err) } } - case service.SoraStorageTypeLocal: + } else if storageType == service.SoraStorageTypeLocal { if h.mediaStorage != nil && len(localPaths) > 0 { if err := h.mediaStorage.DeleteByRelativePaths(localPaths); err != nil { logger.LegacyPrintf("handler.sora_client", "[SoraClient] 清理本地文件失败 paths=%v err=%v", localPaths, err) diff --git a/backend/internal/handler/sora_client_handler_test.go b/backend/internal/handler/sora_client_handler_test.go index 30a761bdf6..87009882d6 100644 --- a/backend/internal/handler/sora_client_handler_test.go +++ b/backend/internal/handler/sora_client_handler_test.go @@ -124,6 +124,9 @@ func (r *stubSoraGenRepo) CountByUserAndStatus(_ context.Context, _ int64, _ []s } return r.countValue, nil } +func (r *stubSoraGenRepo) CountByStorageType(_ context.Context, _ string, _ []string) (int64, error) { + return 0, nil +} // ==================== 辅助函数 ==================== @@ -1641,7 +1644,7 @@ func TestStoreMediaWithDegradation_S3SuccessSingleURL(t *testing.T) { defer fakeS3.Close() s3Storage := newS3StorageForHandler(fakeS3.URL) - h := &SoraClientHandler{s3Storage: s3Storage} + h := &SoraClientHandler{objectStorage: s3Storage} storedURL, storedURLs, storageType, s3Keys, fileSize := h.storeMediaWithDegradation( context.Background(), 1, "video", sourceServer.URL+"/v.mp4", nil, @@ -1663,7 +1666,7 @@ func TestStoreMediaWithDegradation_S3SuccessMultiURL(t *testing.T) { defer fakeS3.Close() s3Storage := newS3StorageForHandler(fakeS3.URL) - h := &SoraClientHandler{s3Storage: s3Storage} + h := &SoraClientHandler{objectStorage: s3Storage} urls := []string{sourceServer.URL + "/a.mp4", sourceServer.URL + "/b.mp4"} storedURL, storedURLs, storageType, s3Keys, fileSize := h.storeMediaWithDegradation( @@ -1688,7 +1691,7 @@ func TestStoreMediaWithDegradation_S3DownloadFails(t *testing.T) { defer badSource.Close() s3Storage := newS3StorageForHandler(fakeS3.URL) - h := &SoraClientHandler{s3Storage: s3Storage} + h := &SoraClientHandler{objectStorage: s3Storage} _, _, storageType, _, _ := h.storeMediaWithDegradation( context.Background(), 1, "video", badSource.URL+"/missing.mp4", nil, @@ -1703,7 +1706,7 @@ func TestStoreMediaWithDegradation_S3FailsSingleURL(t *testing.T) { defer fakeS3.Close() s3Storage := newS3StorageForHandler(fakeS3.URL) - h := &SoraClientHandler{s3Storage: s3Storage} + h := &SoraClientHandler{objectStorage: s3Storage} _, _, storageType, s3Keys, _ := h.storeMediaWithDegradation( context.Background(), 1, "video", sourceServer.URL+"/v.mp4", nil, @@ -1720,7 +1723,7 @@ func TestStoreMediaWithDegradation_S3PartialFailureCleanup(t *testing.T) { defer fakeS3.Close() s3Storage := newS3StorageForHandler(fakeS3.URL) - h := &SoraClientHandler{s3Storage: s3Storage} + h := &SoraClientHandler{objectStorage: s3Storage} urls := []string{sourceServer.URL + "/a.mp4", sourceServer.URL + "/b.mp4"} _, _, storageType, s3Keys, _ := h.storeMediaWithDegradation( @@ -1804,8 +1807,8 @@ func TestStoreMediaWithDegradation_S3FailsFallbackToLocal(t *testing.T) { } mediaStorage := service.NewSoraMediaStorage(cfg) h := &SoraClientHandler{ - s3Storage: s3Storage, - mediaStorage: mediaStorage, + objectStorage: s3Storage, + mediaStorage: mediaStorage, } _, _, storageType, _, _ := h.storeMediaWithDegradation( @@ -1831,14 +1834,14 @@ func TestSaveToStorage_S3EnabledButUploadFails(t *testing.T) { } s3Storage := newS3StorageForHandler(fakeS3.URL) genService := service.NewSoraGenerationService(repo, nil, nil) - h := &SoraClientHandler{genService: genService, s3Storage: s3Storage} + h := &SoraClientHandler{genService: genService, objectStorage: s3Storage} c, rec := makeGinContext("POST", "/api/v1/sora/generations/1/save", "", 1) c.Params = gin.Params{{Key: "id", Value: "1"}} h.SaveToStorage(c) require.Equal(t, http.StatusInternalServerError, rec.Code) resp := parseResponse(t, rec) - require.Contains(t, resp["message"], "S3") + require.Contains(t, resp["message"], "上传到存储失败") } func TestSaveToStorage_UpstreamURLExpired(t *testing.T) { @@ -1857,7 +1860,7 @@ func TestSaveToStorage_UpstreamURLExpired(t *testing.T) { } s3Storage := newS3StorageForHandler(fakeS3.URL) genService := service.NewSoraGenerationService(repo, nil, nil) - h := &SoraClientHandler{genService: genService, s3Storage: s3Storage} + h := &SoraClientHandler{genService: genService, objectStorage: s3Storage} c, rec := makeGinContext("POST", "/api/v1/sora/generations/1/save", "", 1) c.Params = gin.Params{{Key: "id", Value: "1"}} @@ -1881,7 +1884,7 @@ func TestSaveToStorage_S3EnabledUploadSuccess(t *testing.T) { } s3Storage := newS3StorageForHandler(fakeS3.URL) genService := service.NewSoraGenerationService(repo, nil, nil) - h := &SoraClientHandler{genService: genService, s3Storage: s3Storage} + h := &SoraClientHandler{genService: genService, objectStorage: s3Storage} c, rec := makeGinContext("POST", "/api/v1/sora/generations/1/save", "", 1) c.Params = gin.Params{{Key: "id", Value: "1"}} @@ -1889,7 +1892,7 @@ func TestSaveToStorage_S3EnabledUploadSuccess(t *testing.T) { require.Equal(t, http.StatusOK, rec.Code) resp := parseResponse(t, rec) data := resp["data"].(map[string]any) - require.Contains(t, data["message"], "S3") + require.Contains(t, data["message"], "已保存到云存储") require.NotEmpty(t, data["object_key"]) // 验证记录已更新为 S3 存储 require.Equal(t, service.SoraStorageTypeS3, repo.gens[1].StorageType) @@ -1913,7 +1916,7 @@ func TestSaveToStorage_S3EnabledUploadSuccess_MultiMediaURLs(t *testing.T) { } s3Storage := newS3StorageForHandler(fakeS3.URL) genService := service.NewSoraGenerationService(repo, nil, nil) - h := &SoraClientHandler{genService: genService, s3Storage: s3Storage} + h := &SoraClientHandler{genService: genService, objectStorage: s3Storage} c, rec := makeGinContext("POST", "/api/v1/sora/generations/1/save", "", 1) c.Params = gin.Params{{Key: "id", Value: "1"}} @@ -1949,7 +1952,7 @@ func TestSaveToStorage_S3EnabledUploadSuccessWithQuota(t *testing.T) { SoraStorageUsedBytes: 0, } quotaService := service.NewSoraQuotaService(userRepo, nil, nil) - h := &SoraClientHandler{genService: genService, s3Storage: s3Storage, quotaService: quotaService} + h := &SoraClientHandler{genService: genService, objectStorage: s3Storage, quotaService: quotaService} c, rec := makeGinContext("POST", "/api/v1/sora/generations/1/save", "", 1) c.Params = gin.Params{{Key: "id", Value: "1"}} @@ -1975,7 +1978,7 @@ func TestSaveToStorage_S3UploadSuccessMarkCompletedFails(t *testing.T) { repo.updateErr = fmt.Errorf("db error") s3Storage := newS3StorageForHandler(fakeS3.URL) genService := service.NewSoraGenerationService(repo, nil, nil) - h := &SoraClientHandler{genService: genService, s3Storage: s3Storage} + h := &SoraClientHandler{genService: genService, objectStorage: s3Storage} c, rec := makeGinContext("POST", "/api/v1/sora/generations/1/save", "", 1) c.Params = gin.Params{{Key: "id", Value: "1"}} @@ -1991,7 +1994,7 @@ func TestGetStorageStatus_S3EnabledNotHealthy(t *testing.T) { defer fakeS3.Close() s3Storage := newS3StorageForHandler(fakeS3.URL) - h := &SoraClientHandler{s3Storage: s3Storage} + h := &SoraClientHandler{objectStorage: s3Storage} c, rec := makeGinContext("GET", "/api/v1/sora/storage-status", "", 0) h.GetStorageStatus(c) @@ -2007,7 +2010,7 @@ func TestGetStorageStatus_S3EnabledHealthy(t *testing.T) { defer fakeS3.Close() s3Storage := newS3StorageForHandler(fakeS3.URL) - h := &SoraClientHandler{s3Storage: s3Storage} + h := &SoraClientHandler{objectStorage: s3Storage} c, rec := makeGinContext("GET", "/api/v1/sora/storage-status", "", 0) h.GetStorageStatus(c) @@ -2447,7 +2450,7 @@ func TestProcessGeneration_FullSuccessWithS3(t *testing.T) { genService: genService, gatewayService: gatewayService, soraGatewayService: soraGatewayService, - s3Storage: s3Storage, + objectStorage: s3Storage, quotaService: quotaService, } @@ -2497,7 +2500,7 @@ func TestProcessGeneration_MarkCompletedFails(t *testing.T) { // ==================== cleanupStoredMedia 直接测试 ==================== func TestCleanupStoredMedia_S3Path(t *testing.T) { - // S3 清理路径:s3Storage 为 nil 时不 panic + // S3 清理路径:objectStorage 为 nil 时不 panic h := &SoraClientHandler{} // 不应 panic h.cleanupStoredMedia(context.Background(), service.SoraStorageTypeS3, []string{"key1"}, nil) @@ -2955,7 +2958,7 @@ func TestSaveToStorage_QuotaExceeded(t *testing.T) { SoraStorageUsedBytes: 10, } quotaService := service.NewSoraQuotaService(userRepo, nil, nil) - h := &SoraClientHandler{genService: genService, s3Storage: s3Storage, quotaService: quotaService} + h := &SoraClientHandler{genService: genService, objectStorage: s3Storage, quotaService: quotaService} c, rec := makeGinContext("POST", "/api/v1/sora/generations/1/save", "", 1) c.Params = gin.Params{{Key: "id", Value: "1"}} @@ -2983,7 +2986,7 @@ func TestSaveToStorage_QuotaNonQuotaError(t *testing.T) { // 用户不存在 → GetByID 失败 → AddUsage 返回普通 error userRepo := newStubUserRepoForHandler() quotaService := service.NewSoraQuotaService(userRepo, nil, nil) - h := &SoraClientHandler{genService: genService, s3Storage: s3Storage, quotaService: quotaService} + h := &SoraClientHandler{genService: genService, objectStorage: s3Storage, quotaService: quotaService} c, rec := makeGinContext("POST", "/api/v1/sora/generations/1/save", "", 1) c.Params = gin.Params{{Key: "id", Value: "1"}} @@ -3006,7 +3009,7 @@ func TestSaveToStorage_EmptyMediaURLs(t *testing.T) { } s3Storage := newS3StorageForHandler(fakeS3.URL) genService := service.NewSoraGenerationService(repo, nil, nil) - h := &SoraClientHandler{genService: genService, s3Storage: s3Storage} + h := &SoraClientHandler{genService: genService, objectStorage: s3Storage} c, rec := makeGinContext("POST", "/api/v1/sora/generations/1/save", "", 1) c.Params = gin.Params{{Key: "id", Value: "1"}} @@ -3033,7 +3036,7 @@ func TestSaveToStorage_MultiURL_SecondUploadFails(t *testing.T) { } s3Storage := newS3StorageForHandler(fakeS3.URL) genService := service.NewSoraGenerationService(repo, nil, nil) - h := &SoraClientHandler{genService: genService, s3Storage: s3Storage} + h := &SoraClientHandler{genService: genService, objectStorage: s3Storage} c, rec := makeGinContext("POST", "/api/v1/sora/generations/1/save", "", 1) c.Params = gin.Params{{Key: "id", Value: "1"}} @@ -3066,7 +3069,7 @@ func TestSaveToStorage_MarkCompletedFailsWithQuotaRollback(t *testing.T) { SoraStorageUsedBytes: 0, } quotaService := service.NewSoraQuotaService(userRepo, nil, nil) - h := &SoraClientHandler{genService: genService, s3Storage: s3Storage, quotaService: quotaService} + h := &SoraClientHandler{genService: genService, objectStorage: s3Storage, quotaService: quotaService} c, rec := makeGinContext("POST", "/api/v1/sora/generations/1/save", "", 1) c.Params = gin.Params{{Key: "id", Value: "1"}} @@ -3080,7 +3083,7 @@ func TestCleanupStoredMedia_WithS3Storage_ActualDelete(t *testing.T) { fakeS3 := newFakeS3Server("ok") defer fakeS3.Close() s3Storage := newS3StorageForHandler(fakeS3.URL) - h := &SoraClientHandler{s3Storage: s3Storage} + h := &SoraClientHandler{objectStorage: s3Storage} h.cleanupStoredMedia(context.Background(), service.SoraStorageTypeS3, []string{"key1", "key2"}, nil) } @@ -3089,7 +3092,7 @@ func TestCleanupStoredMedia_S3DeleteFails_LogOnly(t *testing.T) { fakeS3 := newFakeS3Server("fail") defer fakeS3.Close() s3Storage := newS3StorageForHandler(fakeS3.URL) - h := &SoraClientHandler{s3Storage: s3Storage} + h := &SoraClientHandler{objectStorage: s3Storage} h.cleanupStoredMedia(context.Background(), service.SoraStorageTypeS3, []string{"key1"}, nil) } diff --git a/backend/internal/handler/sora_videos_handler.go b/backend/internal/handler/sora_videos_handler.go new file mode 100644 index 0000000000..46edbd3624 --- /dev/null +++ b/backend/internal/handler/sora_videos_handler.go @@ -0,0 +1,391 @@ +package handler + +import ( + "encoding/json" + "errors" + "fmt" + "net/http" + "unicode/utf8" + + "github.com/Wei-Shaw/sub2api/internal/pkg/logger" + middleware2 "github.com/Wei-Shaw/sub2api/internal/server/middleware" + "github.com/Wei-Shaw/sub2api/internal/service" + + "github.com/gin-gonic/gin" +) + +// SoraVideosHandler handles Sora video/image async task API. +type SoraVideosHandler struct { + taskService *service.SoraTaskService + gatewayService *service.GatewayService + objectStorage service.SoraObjectStorage + mediaStorage *service.SoraMediaStorage + soraGatewayService *service.SoraGatewayService +} + +func NewSoraVideosHandler( + taskService *service.SoraTaskService, + gatewayService *service.GatewayService, + objectStorage service.SoraObjectStorage, + mediaStorage *service.SoraMediaStorage, + soraGatewayService *service.SoraGatewayService, +) *SoraVideosHandler { + if taskService == nil { + return nil + } + return &SoraVideosHandler{ + taskService: taskService, + gatewayService: gatewayService, + objectStorage: objectStorage, + mediaStorage: mediaStorage, + soraGatewayService: soraGatewayService, + } +} + +func (h *SoraVideosHandler) CreateVideo(c *gin.Context) { + apiKey, account, release, ok := h.selectAccount(c, "") + if !ok { + return + } + defer release() + + body, err := readBody(c) + if err != nil { + return + } + + var req service.CreateVideoRequest + if err := json.Unmarshal(body, &req); err != nil { + soraErrorResponse(c, http.StatusBadRequest, "invalid_request_error", "Invalid request body") + return + } + if req.Model == "" { + soraErrorResponse(c, http.StatusBadRequest, "parameter_missing", "model is required") + return + } + + task, err := h.taskService.CreateVideoTask(c.Request.Context(), apiKey.ID, account, &req, body) + if err != nil { + handleTaskCreateError(c, "CreateVideo", err) + return + } + + c.JSON(http.StatusOK, service.TaskToResponse(task)) +} + +func (h *SoraVideosHandler) GetVideo(c *gin.Context) { + apiKey, ok := h.getAPIKey(c) + if !ok { + return + } + + taskID := c.Param("id") + if taskID == "" { + soraErrorResponse(c, http.StatusBadRequest, "parameter_missing", "video_id is required") + return + } + + task, err := h.taskService.GetTask(c.Request.Context(), taskID, apiKey.ID) + if err != nil { + soraErrorResponse(c, http.StatusNotFound, "not_found", "Task not found") + return + } + + c.JSON(http.StatusOK, service.TaskToResponse(task)) +} + +func (h *SoraVideosHandler) RemixVideo(c *gin.Context) { + apiKey, ok := h.getAPIKey(c) + if !ok { + return + } + + taskID := c.Param("id") + if taskID == "" { + soraErrorResponse(c, http.StatusBadRequest, "parameter_missing", "video_id is required") + return + } + + body, err := readBody(c) + if err != nil { + return + } + + var req service.RemixRequest + if err := json.Unmarshal(body, &req); err != nil || req.Prompt == "" { + soraErrorResponse(c, http.StatusBadRequest, "parameter_missing", "prompt is required") + return + } + + originalTask, err := h.taskService.GetTask(c.Request.Context(), taskID, apiKey.ID) + if err != nil { + soraErrorResponse(c, http.StatusNotFound, "not_found", "Original video task not found") + return + } + if originalTask.Status != service.SoraTaskCompleted { + soraErrorResponse(c, http.StatusBadRequest, "invalid_request_error", "Original video must be completed before remix") + return + } + + remixTargetID := originalTask.ShareID + if remixTargetID == "" { + remixTargetID = originalTask.UpstreamTaskID + } + + account, err := h.selectAccountByID(c, originalTask.AccountID) + if err != nil { + soraErrorResponse(c, http.StatusServiceUnavailable, "server_error", "Failed to get account") + return + } + + videoReq := &service.CreateVideoRequest{ + Model: originalTask.Model, + Prompt: req.Prompt, + RemixTargetID: remixTargetID, + } + reqBody, _ := json.Marshal(videoReq) + + task, err := h.taskService.CreateVideoTask(c.Request.Context(), apiKey.ID, account, videoReq, reqBody) + if err != nil { + handleTaskCreateError(c, "RemixVideo", err) + return + } + + c.JSON(http.StatusOK, service.TaskToResponse(task)) +} + +// GetVideoContent returns video content based on storage configuration. +func (h *SoraVideosHandler) GetVideoContent(c *gin.Context) { + apiKey, ok := h.getAPIKey(c) + if !ok { + return + } + + taskID := c.Param("id") + if taskID == "" { + soraErrorResponse(c, http.StatusBadRequest, "parameter_missing", "video_id is required") + return + } + + task, err := h.taskService.GetTask(c.Request.Context(), taskID, apiKey.ID) + if err != nil { + soraErrorResponse(c, http.StatusNotFound, "not_found", "Task not found") + return + } + + switch task.Status { + case service.SoraTaskCompleted: + contentURL := h.resolveContentURL(c, task) + if contentURL == "" { + soraErrorResponse(c, http.StatusNotFound, "not_found", "Video URL not available") + return + } + c.Redirect(http.StatusFound, contentURL) + + case service.SoraTaskFailed: + c.JSON(http.StatusGone, gin.H{ + "id": task.ID, + "object": task.ObjectType, + "status": task.Status, + "error": gin.H{ + "message": task.ErrorMessage, + "type": task.ErrorType, + }, + }) + + default: + c.JSON(http.StatusAccepted, service.TaskToResponse(task)) + } +} + +func (h *SoraVideosHandler) resolveContentURL(c *gin.Context, task *service.SoraTask) string { + if task.StoredKey == "" { + return task.VideoURL + } + + switch task.StorageType { + case "s3", "gdrive": + if h.objectStorage != nil { + accessURL, err := h.objectStorage.GetAccessURL(c.Request.Context(), task.StoredKey) + if err != nil { + logger.LegacyPrintf("handler.sora_videos", + "[GetVideoContent] task=%s get access URL error: %v, fallback to upstream", task.ID, err) + return task.VideoURL + } + return accessURL + } + return task.VideoURL + + case "local": + return "/sora/media" + task.StoredKey + + default: + return task.VideoURL + } +} + +func (h *SoraVideosHandler) CreateImage(c *gin.Context) { + apiKey, account, release, ok := h.selectAccount(c, "") + if !ok { + return + } + defer release() + + body, err := readBody(c) + if err != nil { + return + } + + var req service.CreateImageRequest + if err := json.Unmarshal(body, &req); err != nil { + soraErrorResponse(c, http.StatusBadRequest, "invalid_request_error", "Invalid request body") + return + } + if req.Prompt == "" { + soraErrorResponse(c, http.StatusBadRequest, "parameter_missing", "prompt is required") + return + } + if req.Model == "" { + req.Model = inferImageModel(req.Size) + } + + task, err := h.taskService.CreateImageGeneration(c.Request.Context(), apiKey.ID, account, &req, body) + if err != nil { + handleTaskCreateError(c, "CreateImage", err) + return + } + + c.JSON(http.StatusOK, service.TaskToResponse(task)) +} + +func (h *SoraVideosHandler) EditImage(c *gin.Context) { + apiKey, account, release, ok := h.selectAccount(c, "") + if !ok { + return + } + defer release() + + body, err := readBody(c) + if err != nil { + return + } + + var req service.EditImageRequest + if err := json.Unmarshal(body, &req); err != nil { + soraErrorResponse(c, http.StatusBadRequest, "invalid_request_error", "Invalid request body") + return + } + if req.Image == "" { + soraErrorResponse(c, http.StatusBadRequest, "parameter_missing", "image is required") + return + } + if req.Prompt == "" { + soraErrorResponse(c, http.StatusBadRequest, "parameter_missing", "prompt is required") + return + } + if req.Model == "" { + req.Model = inferImageModel(req.Size) + } + + imageReq := &service.CreateImageRequest{ + Model: req.Model, + Prompt: req.Prompt, + Image: req.Image, + Size: req.Size, + ResponseFormat: req.ResponseFormat, + N: 1, + } + + task, err := h.taskService.CreateImageGeneration(c.Request.Context(), apiKey.ID, account, imageReq, body) + if err != nil { + handleTaskCreateError(c, "EditImage", err) + return + } + + c.JSON(http.StatusOK, service.TaskToResponse(task)) +} + +// ── Internal helpers ── + +func (h *SoraVideosHandler) getAPIKey(c *gin.Context) (*service.APIKey, bool) { + apiKey, ok := middleware2.GetAPIKeyFromContext(c) + if !ok { + soraErrorResponse(c, http.StatusUnauthorized, "authentication_error", "Invalid API key") + return nil, false + } + return apiKey, true +} + +func (h *SoraVideosHandler) selectAccount(c *gin.Context, model string) (*service.APIKey, *service.Account, func(), bool) { + apiKey, ok := h.getAPIKey(c) + if !ok { + return nil, nil, nil, false + } + + selection, err := h.gatewayService.SelectAccountWithLoadAwareness( + c.Request.Context(), apiKey.GroupID, "", model, nil, "", + ) + if err != nil { + soraErrorResponse(c, http.StatusServiceUnavailable, "server_error", "No available accounts") + return nil, nil, nil, false + } + + releaseFunc := func() {} + if selection.ReleaseFunc != nil { + releaseFunc = selection.ReleaseFunc + } + return apiKey, selection.Account, releaseFunc, true +} + +func (h *SoraVideosHandler) selectAccountByID(c *gin.Context, accountID int64) (*service.Account, error) { + return h.taskService.GetAccountByID(c.Request.Context(), accountID) +} + +func readBody(c *gin.Context) ([]byte, error) { + body, err := c.GetRawData() + if err != nil { + soraErrorResponse(c, http.StatusBadRequest, "invalid_request_error", "Failed to read request body") + return nil, err + } + if len(body) == 0 { + soraErrorResponse(c, http.StatusBadRequest, "invalid_request_error", "Request body is empty") + return nil, fmt.Errorf("empty body") + } + if !utf8.Valid(body) { + soraErrorResponse(c, http.StatusBadRequest, "invalid_request_error", "Request body must be valid UTF-8") + return nil, fmt.Errorf("invalid utf-8") + } + return body, nil +} + +func soraErrorResponse(c *gin.Context, status int, errType, message string) { + c.JSON(status, gin.H{ + "error": gin.H{ + "message": message, + "type": errType, + }, + }) +} + +// handleTaskCreateError writes the appropriate error response, transparently +// forwarding upstream HTTP status codes when available. +func handleTaskCreateError(c *gin.Context, logTag string, err error) { + logger.LegacyPrintf("handler.sora_videos", "[%s] error: %v", logTag, err) + var ue *service.SoraUpstreamError + if errors.As(err, &ue) { + c.Data(ue.StatusCode, "application/json", ue.Body) + return + } + soraErrorResponse(c, http.StatusInternalServerError, "server_error", "Failed to create task") +} + +func inferImageModel(size string) string { + switch size { + case "540x360": + return "gpt-image-landscape" + case "360x540": + return "gpt-image-portrait" + default: + return "gpt-image" + } +} diff --git a/backend/internal/handler/wire.go b/backend/internal/handler/wire.go index d1e12e0366..88c211887e 100644 --- a/backend/internal/handler/wire.go +++ b/backend/internal/handler/wire.go @@ -19,6 +19,7 @@ func ProvideAdminHandlers( openaiOAuthHandler *admin.OpenAIOAuthHandler, geminiOAuthHandler *admin.GeminiOAuthHandler, antigravityOAuthHandler *admin.AntigravityOAuthHandler, + gdriveOAuthHandler *admin.GDriveOAuthHandler, proxyHandler *admin.ProxyHandler, redeemHandler *admin.RedeemHandler, promoHandler *admin.PromoHandler, @@ -55,6 +56,7 @@ func ProvideAdminHandlers( ErrorPassthrough: errorPassthroughHandler, APIKey: apiKeyHandler, ScheduledTest: scheduledTestHandler, + GDriveOAuth: gdriveOAuthHandler, } } @@ -82,6 +84,7 @@ func ProvideHandlers( openaiGatewayHandler *OpenAIGatewayHandler, soraGatewayHandler *SoraGatewayHandler, soraClientHandler *SoraClientHandler, + soraVideosHandler *SoraVideosHandler, settingHandler *SettingHandler, totpHandler *TotpHandler, _ *service.IdempotencyCoordinator, @@ -100,6 +103,7 @@ func ProvideHandlers( OpenAIGateway: openaiGatewayHandler, SoraGateway: soraGatewayHandler, SoraClient: soraClientHandler, + SoraVideos: soraVideosHandler, Setting: settingHandler, Totp: totpHandler, } @@ -144,6 +148,7 @@ var ProviderSet = wire.NewSet( admin.NewErrorPassthroughHandler, admin.NewAdminAPIKeyHandler, admin.NewScheduledTestHandler, + admin.NewGDriveOAuthHandler, // AdminHandlers and Handlers constructors ProvideAdminHandlers, diff --git a/backend/internal/pkg/antigravity/client.go b/backend/internal/pkg/antigravity/client.go index d46bbc4541..1e63315bdd 100644 --- a/backend/internal/pkg/antigravity/client.go +++ b/backend/internal/pkg/antigravity/client.go @@ -19,6 +19,16 @@ import ( "github.com/Wei-Shaw/sub2api/internal/pkg/proxyutil" ) +// ForbiddenError 表示上游返回 403 Forbidden +type ForbiddenError struct { + StatusCode int + Body string +} + +func (e *ForbiddenError) Error() string { + return fmt.Sprintf("fetchAvailableModels 失败 (HTTP %d): %s", e.StatusCode, e.Body) +} + // NewAPIRequestWithURL 使用指定的 base URL 创建 Antigravity API 请求(v1internal 端点) func NewAPIRequestWithURL(ctx context.Context, baseURL, action, accessToken string, body []byte) (*http.Request, error) { // 构建 URL,流式请求添加 ?alt=sse 参数 @@ -514,7 +524,20 @@ type ModelQuotaInfo struct { // ModelInfo 模型信息 type ModelInfo struct { - QuotaInfo *ModelQuotaInfo `json:"quotaInfo,omitempty"` + QuotaInfo *ModelQuotaInfo `json:"quotaInfo,omitempty"` + DisplayName string `json:"displayName,omitempty"` + SupportsImages *bool `json:"supportsImages,omitempty"` + SupportsThinking *bool `json:"supportsThinking,omitempty"` + ThinkingBudget *int `json:"thinkingBudget,omitempty"` + Recommended *bool `json:"recommended,omitempty"` + MaxTokens *int `json:"maxTokens,omitempty"` + MaxOutputTokens *int `json:"maxOutputTokens,omitempty"` + SupportedMimeTypes map[string]bool `json:"supportedMimeTypes,omitempty"` +} + +// DeprecatedModelInfo 废弃模型转发信息 +type DeprecatedModelInfo struct { + NewModelID string `json:"newModelId"` } // FetchAvailableModelsRequest fetchAvailableModels 请求 @@ -524,7 +547,8 @@ type FetchAvailableModelsRequest struct { // FetchAvailableModelsResponse fetchAvailableModels 响应 type FetchAvailableModelsResponse struct { - Models map[string]ModelInfo `json:"models"` + Models map[string]ModelInfo `json:"models"` + DeprecatedModelIDs map[string]DeprecatedModelInfo `json:"deprecatedModelIds,omitempty"` } // FetchAvailableModels 获取可用模型和配额信息,返回解析后的结构体和原始 JSON @@ -573,6 +597,13 @@ func (c *Client) FetchAvailableModels(ctx context.Context, accessToken, projectI continue } + if resp.StatusCode == http.StatusForbidden { + return nil, nil, &ForbiddenError{ + StatusCode: resp.StatusCode, + Body: string(respBodyBytes), + } + } + if resp.StatusCode != http.StatusOK { return nil, nil, fmt.Errorf("fetchAvailableModels 失败 (HTTP %d): %s", resp.StatusCode, string(respBodyBytes)) } diff --git a/backend/internal/repository/gateway_cache.go b/backend/internal/repository/gateway_cache.go index 58291b6652..ec4bf40e3e 100644 --- a/backend/internal/repository/gateway_cache.go +++ b/backend/internal/repository/gateway_cache.go @@ -2,14 +2,42 @@ package repository import ( "context" + _ "embed" "fmt" + "strconv" "time" "github.com/Wei-Shaw/sub2api/internal/service" "github.com/redis/go-redis/v9" ) -const stickySessionPrefix = "sticky_session:" +const ( + stickySessionPrefix = "sticky_session:" + clientAffinityPrefix = "client_affinity:" + clientAffinityReversePrefix = "client_affinity_rev:" +) + +var ( + //go:embed lua/get_affinity.lua + getAffinityLua string + //go:embed lua/update_affinity.lua + updateAffinityLua string + //go:embed lua/get_affinity_count.lua + getAffinityCountLua string + //go:embed lua/get_affinity_clients.lua + getAffinityClientsLua string + //go:embed lua/get_affinity_clients_with_scores.lua + getAffinityClientsWithScoresLua string + //go:embed lua/clear_account_affinity.lua + clearAccountAffinityLua string + + getAffinityScript = redis.NewScript(getAffinityLua) + updateAffinityScript = redis.NewScript(updateAffinityLua) + getAffinityCountScript = redis.NewScript(getAffinityCountLua) + getAffinityClientsScript = redis.NewScript(getAffinityClientsLua) + getAffinityClientsWithScoresScript = redis.NewScript(getAffinityClientsWithScoresLua) + clearAccountAffinityScript = redis.NewScript(clearAccountAffinityLua) +) type gatewayCache struct { rdb *redis.Client @@ -19,6 +47,16 @@ func NewGatewayCache(rdb *redis.Client) service.GatewayCache { return &gatewayCache{rdb: rdb} } +// ensureScriptLoaded 确保 Lua 脚本已加载到 Redis 服务器的脚本缓存中。 +// Pipeline 中的 Script.Run 只发送 EVALSHA,如果 Redis 重启过导致脚本缓存丢失, +// EVALSHA 会返回 NOSCRIPT 错误。此方法提前加载脚本以避免该问题。 +func ensureScriptLoaded(ctx context.Context, rdb *redis.Client, script *redis.Script) { + exists, err := script.Exists(ctx, rdb).Result() + if err != nil || len(exists) == 0 || !exists[0] { + _ = script.Load(ctx, rdb).Err() + } +} + // buildSessionKey 构建 session key,包含 groupID 实现分组隔离 // 格式: sticky_session:{groupID}:{sessionHash} func buildSessionKey(groupID int64, sessionHash string) string { @@ -41,13 +79,218 @@ func (c *gatewayCache) RefreshSessionTTL(ctx context.Context, groupID int64, ses } // DeleteSessionAccountID 删除粘性会话与账号的绑定关系。 -// 当检测到绑定的账号不可用(如状态错误、禁用、不可调度等)时调用, -// 以便下次请求能够重新选择可用账号。 -// -// DeleteSessionAccountID removes the sticky session binding for the given session. -// Called when the bound account becomes unavailable (e.g., error status, disabled, -// or unschedulable), allowing subsequent requests to select a new available account. func (c *gatewayCache) DeleteSessionAccountID(ctx context.Context, groupID int64, sessionHash string) error { key := buildSessionKey(groupID, sessionHash) return c.rdb.Del(ctx, key).Err() } + +// buildAffinityKey 构建正向亲和 key(client → accounts) +// 格式: client_affinity:{groupID}:{clientID} +func buildAffinityKey(groupID int64, clientID string) string { + return fmt.Sprintf("%s%d:%s", clientAffinityPrefix, groupID, clientID) +} + +// buildAffinityReverseKey 构建反向亲和 key(account → clients) +// 格式: client_affinity_rev:{groupID}:{accountID} +func buildAffinityReverseKey(groupID int64, accountID int64) string { + return fmt.Sprintf("%s%d:%d", clientAffinityReversePrefix, groupID, accountID) +} + +func (c *gatewayCache) GetClientAffinityAccounts(ctx context.Context, groupID int64, clientID string, ttl time.Duration) ([]int64, error) { + key := buildAffinityKey(groupID, clientID) + now := time.Now().Unix() + expireThreshold := now - int64(ttl.Seconds()) + + result, err := getAffinityScript.Run(ctx, c.rdb, []string{key}, expireThreshold).StringSlice() + if err != nil { + if err == redis.Nil { + return nil, nil + } + return nil, err + } + + accountIDs := make([]int64, 0, len(result)) + for _, s := range result { + id, err := strconv.ParseInt(s, 10, 64) + if err != nil { + continue + } + accountIDs = append(accountIDs, id) + } + return accountIDs, nil +} + +func (c *gatewayCache) UpdateClientAffinity(ctx context.Context, groupID int64, clientID string, accountID int64, ttl time.Duration) error { + fwdKey := buildAffinityKey(groupID, clientID) + revKey := buildAffinityReverseKey(groupID, accountID) + now := time.Now().Unix() + ttlSeconds := int64(ttl.Seconds()) + expireThreshold := now - ttlSeconds + + return updateAffinityScript.Run(ctx, c.rdb, []string{fwdKey, revKey}, + now, ttlSeconds, accountID, expireThreshold, clientID, + ).Err() +} + +// GetAccountAffinityCountBatch 批量获取账号的亲和客户端数量(惰性清理过期成员) +func (c *gatewayCache) GetAccountAffinityCountBatch(ctx context.Context, groupID int64, accountIDs []int64, ttl time.Duration) (map[int64]int64, error) { + if len(accountIDs) == 0 { + return map[int64]int64{}, nil + } + + now := time.Now().Unix() + expireThreshold := now - int64(ttl.Seconds()) + + ensureScriptLoaded(ctx, c.rdb, getAffinityCountScript) + + pipe := c.rdb.Pipeline() + cmds := make([]*redis.Cmd, len(accountIDs)) + for i, accID := range accountIDs { + key := buildAffinityReverseKey(groupID, accID) + cmds[i] = getAffinityCountScript.Run(ctx, pipe, []string{key}, expireThreshold) + } + _, err := pipe.Exec(ctx) + if err != nil && err != redis.Nil { + return nil, err + } + + result := make(map[int64]int64, len(accountIDs)) + for i, accID := range accountIDs { + count, _ := cmds[i].Int64() + result[accID] = count + } + return result, nil +} + +// GetAccountAffinityClientsBatch 批量获取每个账号跨所有分组的亲和客户端列表(去重)。 +// accountGroups: map[accountID][]groupID,对每个 (groupID, accountID) 组合查询反向索引。 +func (c *gatewayCache) GetAccountAffinityClientsBatch(ctx context.Context, accountGroups map[int64][]int64, ttl time.Duration) (map[int64][]string, error) { + if len(accountGroups) == 0 { + return map[int64][]string{}, nil + } + + now := time.Now().Unix() + expireThreshold := now - int64(ttl.Seconds()) + + // 构建所有 (accountID, groupID) 组合的查询 + type queryItem struct { + accountID int64 + groupID int64 + } + var queries []queryItem + for accID, groupIDs := range accountGroups { + for _, gID := range groupIDs { + queries = append(queries, queryItem{accountID: accID, groupID: gID}) + } + } + + ensureScriptLoaded(ctx, c.rdb, getAffinityClientsScript) + + pipe := c.rdb.Pipeline() + cmds := make([]*redis.Cmd, len(queries)) + for i, q := range queries { + key := buildAffinityReverseKey(q.groupID, q.accountID) + cmds[i] = getAffinityClientsScript.Run(ctx, pipe, []string{key}, expireThreshold) + } + _, err := pipe.Exec(ctx) + if err != nil && err != redis.Nil { + return nil, err + } + + // 合并结果:同一个 accountID 跨多个 group 的 clientID 去重 + result := make(map[int64][]string, len(accountGroups)) + seen := make(map[int64]map[string]struct{}, len(accountGroups)) + for i, q := range queries { + clients, _ := cmds[i].StringSlice() + if len(clients) == 0 { + continue + } + if seen[q.accountID] == nil { + seen[q.accountID] = make(map[string]struct{}) + } + for _, clientID := range clients { + if _, exists := seen[q.accountID][clientID]; !exists { + seen[q.accountID][clientID] = struct{}{} + result[q.accountID] = append(result[q.accountID], clientID) + } + } + } + return result, nil +} + +// GetAccountAffinityClientsWithScores 获取单个账号跨所有分组的亲和客户端列表(含最后活跃时间戳,去重取最近)。 +func (c *gatewayCache) GetAccountAffinityClientsWithScores( + ctx context.Context, + accountID int64, + groupIDs []int64, + ttl time.Duration, +) ([]service.AffinityClient, error) { + if len(groupIDs) == 0 { + return nil, nil + } + + now := time.Now().Unix() + expireThreshold := now - int64(ttl.Seconds()) + + ensureScriptLoaded(ctx, c.rdb, getAffinityClientsWithScoresScript) + + pipe := c.rdb.Pipeline() + cmds := make([]*redis.Cmd, len(groupIDs)) + for i, gID := range groupIDs { + key := buildAffinityReverseKey(gID, accountID) + cmds[i] = getAffinityClientsWithScoresScript.Run(ctx, pipe, []string{key}, expireThreshold) + } + _, err := pipe.Exec(ctx) + if err != nil && err != redis.Nil { + return nil, err + } + + // 合并跨组结果,同一 clientID 取最近的 lastActive + seen := make(map[string]int64) // clientID → max timestamp + for _, cmd := range cmds { + vals, _ := cmd.StringSlice() + // vals 格式: [clientID1, score1, clientID2, score2, ...] + for j := 0; j+1 < len(vals); j += 2 { + clientID := vals[j] + ts, _ := strconv.ParseInt(vals[j+1], 10, 64) + if existing, ok := seen[clientID]; !ok || ts > existing { + seen[clientID] = ts + } + } + } + + result := make([]service.AffinityClient, 0, len(seen)) + for clientID, ts := range seen { + result = append(result, service.AffinityClient{ + ClientID: clientID, + LastActive: time.Unix(ts, 0), + }) + } + + // 按最后活跃时间降序排序 + service.SortAffinityClients(result) + + return result, nil +} + +// ClearAccountAffinity 清除指定账号在所有分组的亲和记录(正向+反向索引)。 +// 对每个 groupID 执行 Lua 脚本:读取反向索引获取所有客户端, +// 从每个客户端的正向索引中移除该账号,然后删除反向索引。 +func (c *gatewayCache) ClearAccountAffinity(ctx context.Context, accountID int64, groupIDs []int64) error { + if len(groupIDs) == 0 { + return nil + } + + ensureScriptLoaded(ctx, c.rdb, clearAccountAffinityScript) + + pipe := c.rdb.Pipeline() + for _, gID := range groupIDs { + revKey := buildAffinityReverseKey(gID, accountID) + clearAccountAffinityScript.Run(ctx, pipe, []string{revKey}, gID, accountID) + } + _, err := pipe.Exec(ctx) + if err != nil && err != redis.Nil { + return err + } + return nil +} diff --git a/backend/internal/repository/lua/clear_account_affinity.lua b/backend/internal/repository/lua/clear_account_affinity.lua new file mode 100644 index 0000000000..e125be1690 --- /dev/null +++ b/backend/internal/repository/lua/clear_account_affinity.lua @@ -0,0 +1,29 @@ +-- 清除单个账号在指定分组的所有亲和记录(正向+反向) +-- KEYS[1] = client_affinity_rev:{groupID}:{accountID} (反向索引) +-- ARGV[1] = groupID (用于构建正向 key) +-- ARGV[2] = accountID (正向索引中要移除的成员) +-- 返回: 清理的客户端数量 +local rev_key = KEYS[1] +local group_id = ARGV[1] +local account_id = ARGV[2] + +-- 获取反向索引中所有客户端 ID +local clients = redis.call('ZRANGE', rev_key, 0, -1) +if #clients == 0 then + return 0 +end + +-- 从每个客户端的正向索引中移除该账号 +for _, client_id in ipairs(clients) do + local fwd_key = 'client_affinity:' .. group_id .. ':' .. client_id + redis.call('ZREM', fwd_key, account_id) + -- 如果正向索引为空,删除 key + if redis.call('ZCARD', fwd_key) == 0 then + redis.call('DEL', fwd_key) + end +end + +-- 删除反向索引 +redis.call('DEL', rev_key) + +return #clients diff --git a/backend/internal/repository/lua/get_affinity.lua b/backend/internal/repository/lua/get_affinity.lua new file mode 100644 index 0000000000..7e3971dbea --- /dev/null +++ b/backend/internal/repository/lua/get_affinity.lua @@ -0,0 +1,5 @@ +-- 清理过期成员后返回亲和账号列表(按最近使用降序) +-- KEYS[1] = client_affinity:{groupID}:{clientID} +-- ARGV[1] = 过期阈值时间戳 (now - ttl) +redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', ARGV[1]) +return redis.call('ZREVRANGE', KEYS[1], 0, -1) diff --git a/backend/internal/repository/lua/get_affinity_clients.lua b/backend/internal/repository/lua/get_affinity_clients.lua new file mode 100644 index 0000000000..049b1c5b7c --- /dev/null +++ b/backend/internal/repository/lua/get_affinity_clients.lua @@ -0,0 +1,5 @@ +-- 清理过期成员后返回反向索引的 clientID 列表(按最近使用降序) +-- KEYS[1] = client_affinity_rev:{groupID}:{accountID} +-- ARGV[1] = 过期阈值时间戳 (now - ttl) +redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', ARGV[1]) +return redis.call('ZREVRANGE', KEYS[1], 0, -1) diff --git a/backend/internal/repository/lua/get_affinity_clients_with_scores.lua b/backend/internal/repository/lua/get_affinity_clients_with_scores.lua new file mode 100644 index 0000000000..8e70708efb --- /dev/null +++ b/backend/internal/repository/lua/get_affinity_clients_with_scores.lua @@ -0,0 +1,6 @@ +-- 清理过期成员后返回反向索引的 clientID 列表及其 score(最后活跃时间戳) +-- KEYS[1] = client_affinity_rev:{groupID}:{accountID} +-- ARGV[1] = 过期阈值时间戳 (now - ttl) +-- 返回: {clientID1, score1, clientID2, score2, ...}(按最近使用降序) +redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', ARGV[1]) +return redis.call('ZREVRANGEBYSCORE', KEYS[1], '+inf', '-inf', 'WITHSCORES') diff --git a/backend/internal/repository/lua/get_affinity_count.lua b/backend/internal/repository/lua/get_affinity_count.lua new file mode 100644 index 0000000000..4b00c22daa --- /dev/null +++ b/backend/internal/repository/lua/get_affinity_count.lua @@ -0,0 +1,5 @@ +-- 清理过期成员后返回反向索引的成员数量 +-- KEYS[1] = client_affinity_rev:{groupID}:{accountID} +-- ARGV[1] = 过期阈值时间戳 (now - ttl) +redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', ARGV[1]) +return redis.call('ZCARD', KEYS[1]) diff --git a/backend/internal/repository/lua/update_affinity.lua b/backend/internal/repository/lua/update_affinity.lua new file mode 100644 index 0000000000..27fe8cb217 --- /dev/null +++ b/backend/internal/repository/lua/update_affinity.lua @@ -0,0 +1,15 @@ +-- 原子双写正向+反向索引 +-- KEYS[1] = client_affinity:{groupID}:{clientID} (正向: client → accounts) +-- KEYS[2] = client_affinity_rev:{groupID}:{accountID} (反向: account → clients) +-- ARGV[1] = 当前时间戳 (score) +-- ARGV[2] = TTL 秒数 +-- ARGV[3] = accountID (正向索引的成员) +-- ARGV[4] = 过期阈值时间戳 (now - ttl) +-- ARGV[5] = clientID (反向索引的成员) +redis.call('ZREMRANGEBYSCORE', KEYS[1], '-inf', ARGV[4]) +redis.call('ZADD', KEYS[1], ARGV[1], ARGV[3]) +redis.call('EXPIRE', KEYS[1], ARGV[2]) +redis.call('ZREMRANGEBYSCORE', KEYS[2], '-inf', ARGV[4]) +redis.call('ZADD', KEYS[2], ARGV[1], ARGV[5]) +redis.call('EXPIRE', KEYS[2], ARGV[2]) +return 1 diff --git a/backend/internal/repository/sora_generation_repo.go b/backend/internal/repository/sora_generation_repo.go index aaf3cb2f54..7894cdb5e1 100644 --- a/backend/internal/repository/sora_generation_repo.go +++ b/backend/internal/repository/sora_generation_repo.go @@ -417,3 +417,22 @@ func (r *soraGenerationRepository) CountByUserAndStatus(ctx context.Context, use err := r.sql.QueryRowContext(ctx, query, args...).Scan(&count) return count, err } + +// CountByStorageType 按存储类型和状态统计生成记录数。 +func (r *soraGenerationRepository) CountByStorageType(ctx context.Context, storageType string, statuses []string) (int64, error) { + if len(statuses) == 0 { + return 0, nil + } + + placeholders := make([]string, len(statuses)) + args := []any{storageType} + for i, s := range statuses { + placeholders[i] = fmt.Sprintf("$%d", i+2) + args = append(args, s) + } + + var count int64 + query := fmt.Sprintf("SELECT COUNT(*) FROM sora_generations WHERE storage_type = $1 AND status IN (%s)", strings.Join(placeholders, ",")) + err := r.sql.QueryRowContext(ctx, query, args...).Scan(&count) + return count, err +} diff --git a/backend/internal/repository/sora_task_repo.go b/backend/internal/repository/sora_task_repo.go new file mode 100644 index 0000000000..d46d50fe3c --- /dev/null +++ b/backend/internal/repository/sora_task_repo.go @@ -0,0 +1,136 @@ +package repository + +import ( + "context" + "database/sql" + "encoding/json" + "unicode/utf8" + + "github.com/Wei-Shaw/sub2api/internal/service" +) + +type SoraTaskRepository struct { + db *sql.DB +} + +func NewSoraTaskRepository(sqlDB *sql.DB) service.SoraTaskRepository { + return &SoraTaskRepository{db: sqlDB} +} + +func (r *SoraTaskRepository) Create(ctx context.Context, task *service.SoraTask) error { + var charStr, reqStr *string + if task.CharacterInfo != nil { + b, _ := json.Marshal(task.CharacterInfo) + s := string(b) + charStr = &s + } + if len(task.RequestBody) > 0 && utf8.Valid(task.RequestBody) { + s := string(task.RequestBody) + reqStr = &s + } + + _, err := r.db.ExecContext(ctx, ` + INSERT INTO sora_tasks ( + id, account_id, api_key_id, upstream_task_id, object_type, + model, prompt, status, progress, video_url, stored_key, storage_type, + share_id, character_info, error_message, error_type, + request_body, seconds, size, created_at, completed_at + ) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21)`, + task.ID, task.AccountID, task.APIKeyID, task.UpstreamTaskID, task.ObjectType, + task.Model, task.Prompt, task.Status, task.Progress, task.VideoURL, + task.StoredKey, task.StorageType, + task.ShareID, charStr, task.ErrorMessage, task.ErrorType, + reqStr, task.Seconds, task.Size, task.CreatedAt, task.CompletedAt, + ) + return err +} + +func (r *SoraTaskRepository) GetByID(ctx context.Context, id string) (*service.SoraTask, error) { + row := r.db.QueryRowContext(ctx, `SELECT `+soraTaskColumns+` FROM sora_tasks WHERE id = $1`, id) + return scanTask(row) +} + +func (r *SoraTaskRepository) GetByIDAndAPIKey(ctx context.Context, id string, apiKeyID int64) (*service.SoraTask, error) { + row := r.db.QueryRowContext(ctx, + `SELECT `+soraTaskColumns+` FROM sora_tasks WHERE id = $1 AND api_key_id = $2`, + id, apiKeyID, + ) + return scanTask(row) +} + +func (r *SoraTaskRepository) Update(ctx context.Context, task *service.SoraTask) error { + var charStr *string + if task.CharacterInfo != nil { + b, _ := json.Marshal(task.CharacterInfo) + s := string(b) + charStr = &s + } + + _, err := r.db.ExecContext(ctx, ` + UPDATE sora_tasks SET + upstream_task_id = $2, status = $3, progress = $4, + video_url = $5, stored_key = $6, storage_type = $7, + share_id = $8, character_info = $9, + error_message = $10, error_type = $11, + completed_at = $12, seconds = $13, size = $14, object_type = $15 + WHERE id = $1`, + task.ID, task.UpstreamTaskID, task.Status, task.Progress, + task.VideoURL, task.StoredKey, task.StorageType, + task.ShareID, charStr, + task.ErrorMessage, task.ErrorType, + task.CompletedAt, task.Seconds, task.Size, task.ObjectType, + ) + return err +} + +func (r *SoraTaskRepository) ListPending(ctx context.Context) ([]*service.SoraTask, error) { + rows, err := r.db.QueryContext(ctx, + `SELECT `+soraTaskColumns+` FROM sora_tasks WHERE status IN ('queued', 'in_progress') ORDER BY created_at ASC LIMIT 200`, + ) + if err != nil { + return nil, err + } + defer func() { _ = rows.Close() }() + + var tasks []*service.SoraTask + for rows.Next() { + t, err := scanTaskFromRow(rows) + if err != nil { + return nil, err + } + tasks = append(tasks, t) + } + return tasks, rows.Err() +} + +const soraTaskColumns = `id, account_id, api_key_id, upstream_task_id, object_type, + model, prompt, status, progress, video_url, stored_key, storage_type, + share_id, character_info, error_message, error_type, + request_body, seconds, size, created_at, completed_at` + +func scanTask(s scannable) (*service.SoraTask, error) { + var t service.SoraTask + var charJSON, reqBody []byte + err := s.Scan( + &t.ID, &t.AccountID, &t.APIKeyID, &t.UpstreamTaskID, &t.ObjectType, + &t.Model, &t.Prompt, &t.Status, &t.Progress, &t.VideoURL, + &t.StoredKey, &t.StorageType, + &t.ShareID, &charJSON, &t.ErrorMessage, &t.ErrorType, + &reqBody, &t.Seconds, &t.Size, &t.CreatedAt, &t.CompletedAt, + ) + if err != nil { + return nil, err + } + if len(charJSON) > 0 { + var ch service.SoraCharacter + if json.Unmarshal(charJSON, &ch) == nil && ch.Username != "" { + t.CharacterInfo = &ch + } + } + t.RequestBody = reqBody + return &t, nil +} + +func scanTaskFromRow(rows *sql.Rows) (*service.SoraTask, error) { + return scanTask(rows) +} diff --git a/backend/internal/server/api_contract_test.go b/backend/internal/server/api_contract_test.go index 0b36bf6664..c132867f89 100644 --- a/backend/internal/server/api_contract_test.go +++ b/backend/internal/server/api_contract_test.go @@ -210,10 +210,9 @@ func TestAPIContracts(t *testing.T) { "sora_video_price_per_request": null, "sora_video_price_per_request_hd": null, "claude_code_only": false, - "allow_messages_dispatch": false, - "fallback_group_id": null, - "fallback_group_id_on_invalid_request": null, - "allow_messages_dispatch": false, + "allow_messages_dispatch": false, + "fallback_group_id": null, + "fallback_group_id_on_invalid_request": null, "created_at": "2025-01-02T03:04:05Z", "updated_at": "2025-01-02T03:04:05Z" } @@ -649,8 +648,8 @@ func newContractDeps(t *testing.T) *contractDeps { authHandler := handler.NewAuthHandler(cfg, nil, userService, settingService, nil, redeemService, nil) apiKeyHandler := handler.NewAPIKeyHandler(apiKeyService) usageHandler := handler.NewUsageHandler(usageService, apiKeyService) - adminSettingHandler := adminhandler.NewSettingHandler(settingService, nil, nil, nil, nil) - adminAccountHandler := adminhandler.NewAccountHandler(adminService, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil) + adminSettingHandler := adminhandler.NewSettingHandler(settingService, nil, nil, nil, nil, nil, nil) + adminAccountHandler := adminhandler.NewAccountHandler(adminService, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil, nil) jwtAuth := func(c *gin.Context) { c.Set(string(middleware.ContextKeyUser), middleware.AuthSubject{ diff --git a/backend/internal/server/routes/admin.go b/backend/internal/server/routes/admin.go index 46c2ccded7..6d26695d89 100644 --- a/backend/internal/server/routes/admin.go +++ b/backend/internal/server/routes/admin.go @@ -257,6 +257,7 @@ func registerAccountRoutes(admin *gin.RouterGroup, h *handler.Handlers) { accounts.POST("/today-stats/batch", h.Admin.Account.GetBatchTodayStats) accounts.POST("/:id/clear-rate-limit", h.Admin.Account.ClearRateLimit) accounts.POST("/:id/reset-quota", h.Admin.Account.ResetQuota) + accounts.GET("/:id/affinity-clients", h.Admin.Account.GetAffinityClients) accounts.GET("/:id/temp-unschedulable", h.Admin.Account.GetTempUnschedulable) accounts.DELETE("/:id/temp-unschedulable", h.Admin.Account.ClearTempUnschedulable) accounts.POST("/:id/schedulable", h.Admin.Account.SetSchedulable) @@ -404,7 +405,7 @@ func registerSettingsRoutes(admin *gin.RouterGroup, h *handler.Handlers) { // Beta 策略配置 adminSettings.GET("/beta-policy", h.Admin.Setting.GetBetaPolicySettings) adminSettings.PUT("/beta-policy", h.Admin.Setting.UpdateBetaPolicySettings) - // Sora S3 存储配置 + // Sora S3 存储配置(旧路由,保留兼容) adminSettings.GET("/sora-s3", h.Admin.Setting.GetSoraS3Settings) adminSettings.PUT("/sora-s3", h.Admin.Setting.UpdateSoraS3Settings) adminSettings.POST("/sora-s3/test", h.Admin.Setting.TestSoraS3Connection) @@ -413,6 +414,22 @@ func registerSettingsRoutes(admin *gin.RouterGroup, h *handler.Handlers) { adminSettings.PUT("/sora-s3/profiles/:profile_id", h.Admin.Setting.UpdateSoraS3Profile) adminSettings.DELETE("/sora-s3/profiles/:profile_id", h.Admin.Setting.DeleteSoraS3Profile) adminSettings.POST("/sora-s3/profiles/:profile_id/activate", h.Admin.Setting.SetActiveSoraS3Profile) + // Sora 统一存储配置(新路由,指向相同 handler) + adminSettings.GET("/sora-storage", h.Admin.Setting.GetSoraS3Settings) + adminSettings.PUT("/sora-storage", h.Admin.Setting.UpdateSoraS3Settings) + adminSettings.POST("/sora-storage/test", h.Admin.Setting.TestSoraS3Connection) + adminSettings.GET("/sora-storage/profiles", h.Admin.Setting.ListSoraS3Profiles) + adminSettings.POST("/sora-storage/profiles", h.Admin.Setting.CreateSoraS3Profile) + adminSettings.PUT("/sora-storage/profiles/:profile_id", h.Admin.Setting.UpdateSoraS3Profile) + adminSettings.DELETE("/sora-storage/profiles/:profile_id", h.Admin.Setting.DeleteSoraS3Profile) + adminSettings.POST("/sora-storage/profiles/:profile_id/activate", h.Admin.Setting.SetActiveSoraS3Profile) + // Google Drive OAuth + adminSettings.POST("/sora-storage/gdrive-oauth/start", h.Admin.GDriveOAuth.StartOAuth) + adminSettings.POST("/sora-storage/gdrive-oauth/callback", h.Admin.GDriveOAuth.OAuthCallback) + adminSettings.POST("/sora-storage/gdrive-test", h.Admin.GDriveOAuth.TestGDriveStorage) + // Sora 存储统计 + adminSettings.GET("/sora-storage/gdrive-quota", h.Admin.Setting.GetGDriveQuota) + adminSettings.GET("/sora-storage/video-stats", h.Admin.Setting.GetStorageVideoStats) } } diff --git a/backend/internal/server/routes/gateway.go b/backend/internal/server/routes/gateway.go index ea40f2f189..7dd089cbce 100644 --- a/backend/internal/server/routes/gateway.go +++ b/backend/internal/server/routes/gateway.go @@ -138,6 +138,16 @@ func RegisterGatewayRoutes( { soraV1.POST("/chat/completions", h.SoraGateway.ChatCompletions) soraV1.GET("/models", h.Gateway.Models) + + // Sora Videos/Images async task API + if h.SoraVideos != nil { + soraV1.POST("/videos", h.SoraVideos.CreateVideo) + soraV1.GET("/videos/:id", h.SoraVideos.GetVideo) + soraV1.POST("/videos/:id/remix", h.SoraVideos.RemixVideo) + soraV1.GET("/videos/:id/content", h.SoraVideos.GetVideoContent) + soraV1.POST("/images/generations", h.SoraVideos.CreateImage) + soraV1.POST("/images/edits", h.SoraVideos.EditImage) + } } // Sora 媒体代理(可选 API Key 验证) diff --git a/backend/internal/service/account.go b/backend/internal/service/account.go index 9d4f73d4b2..f4f0fbabfd 100644 --- a/backend/internal/service/account.go +++ b/backend/internal/service/account.go @@ -1168,6 +1168,90 @@ func (a *Account) IsSessionIDMaskingEnabled() bool { return false } +// IsClientAffinityEnabled 检查是否启用客户端亲和调度 +// 仅适用于 Anthropic 账号(OAuth/SetupToken/APIKey) +// 启用后,新会话会优先调度到之前使用过的账号 +func (a *Account) IsClientAffinityEnabled() bool { + if a.Platform != PlatformAnthropic { + return false + } + if a.Extra == nil { + return false + } + if v, ok := a.Extra["client_affinity_enabled"]; ok { + if enabled, ok := v.(bool); ok { + return enabled + } + } + return false +} + +// AffinityZone 表示账号的客户端亲和分区 +type AffinityZone int + +const ( + AffinityZoneGreen AffinityZone = iota // 绿区:允许绑定新客户端,优先调度 + AffinityZoneYellow // 黄区:允许绑定,仅在无绿区账号时降级调度 + AffinityZoneRed // 红区:禁止调度 +) + +// GetAffinityBase 获取亲和基础限制(绿区上限),0 表示未配置 +func (a *Account) GetAffinityBase() int { + if a.Extra == nil { + return 0 + } + if v, ok := a.Extra["affinity_base"]; ok { + return parseExtraInt(v) + } + return 0 +} + +// GetAffinityBuffer 获取亲和缓冲区大小(黄区范围) +// 返回 (value, configured): +// - (0, false): 未配置 → 无限黄区(超过 base 永远黄区,永不红区) +// - (0, true): 显式设为 0 → 无黄区,超过 base 直接红区 +// - (n, true): n > 0 → 黄区范围为 base+1 到 base+n +func (a *Account) GetAffinityBuffer() (int, bool) { + if a.Extra == nil { + return 0, false + } + v, ok := a.Extra["affinity_buffer"] + if !ok { + return 0, false + } + // 显式设为 null/nil → 视为未配置 + if v == nil { + return 0, false + } + return parseExtraInt(v), true +} + +// GetAffinityZone 根据当前绑定的客户端数量计算账号的亲和分区。 +// 未开启亲和或未配置 base 的账号永远返回绿区。 +func (a *Account) GetAffinityZone(clientCount int64) AffinityZone { + if !a.IsClientAffinityEnabled() { + return AffinityZoneGreen + } + base := a.GetAffinityBase() + if base <= 0 { + return AffinityZoneGreen + } + if clientCount <= int64(base) { + return AffinityZoneGreen + } + buffer, configured := a.GetAffinityBuffer() + if !configured { + return AffinityZoneYellow // 未配置 buffer → 无限黄区 + } + if buffer == 0 { + return AffinityZoneRed // buffer=0 → 无黄区,直接红区 + } + if clientCount <= int64(base+buffer) { + return AffinityZoneYellow + } + return AffinityZoneRed +} + // IsCacheTTLOverrideEnabled 检查是否启用缓存 TTL 强制替换 // 仅适用于 Anthropic OAuth/SetupToken 类型账号 // 启用后将所有 cache creation tokens 归入指定的 TTL 类型(5m 或 1h) diff --git a/backend/internal/service/account_usage_service.go b/backend/internal/service/account_usage_service.go index e42451339e..cf0c8fc0cf 100644 --- a/backend/internal/service/account_usage_service.go +++ b/backend/internal/service/account_usage_service.go @@ -6,6 +6,7 @@ import ( "encoding/json" "fmt" "log" + "log/slog" "math/rand/v2" "net/http" "strings" @@ -99,6 +100,7 @@ type antigravityUsageCache struct { const ( apiCacheTTL = 3 * time.Minute apiErrorCacheTTL = 1 * time.Minute // 负缓存 TTL:429 等错误缓存 1 分钟 + antigravityErrorTTL = 1 * time.Minute // Antigravity 错误缓存 TTL(可恢复错误) apiQueryMaxJitter = 800 * time.Millisecond // 用量查询最大随机延迟 windowStatsCacheTTL = 1 * time.Minute openAIProbeCacheTTL = 10 * time.Minute @@ -107,11 +109,12 @@ const ( // UsageCache 封装账户使用量相关的缓存 type UsageCache struct { - apiCache sync.Map // accountID -> *apiUsageCache - windowStatsCache sync.Map // accountID -> *windowStatsCache - antigravityCache sync.Map // accountID -> *antigravityUsageCache - apiFlight singleflight.Group // 防止同一账号的并发请求击穿缓存 - openAIProbeCache sync.Map // accountID -> time.Time + apiCache sync.Map // accountID -> *apiUsageCache + windowStatsCache sync.Map // accountID -> *windowStatsCache + antigravityCache sync.Map // accountID -> *antigravityUsageCache + apiFlight singleflight.Group // 防止同一账号的并发请求击穿缓存(Anthropic) + antigravityFlight singleflight.Group // 防止同一 Antigravity 账号的并发请求击穿缓存 + openAIProbeCache sync.Map // accountID -> time.Time } // NewUsageCache 创建 UsageCache 实例 @@ -148,6 +151,18 @@ type AntigravityModelQuota struct { ResetTime string `json:"reset_time"` // 重置时间 ISO8601 } +// AntigravityModelDetail Antigravity 单个模型的详细能力信息 +type AntigravityModelDetail struct { + DisplayName string `json:"display_name,omitempty"` + SupportsImages *bool `json:"supports_images,omitempty"` + SupportsThinking *bool `json:"supports_thinking,omitempty"` + ThinkingBudget *int `json:"thinking_budget,omitempty"` + Recommended *bool `json:"recommended,omitempty"` + MaxTokens *int `json:"max_tokens,omitempty"` + MaxOutputTokens *int `json:"max_output_tokens,omitempty"` + SupportedMimeTypes map[string]bool `json:"supported_mime_types,omitempty"` +} + // UsageInfo 账号使用量信息 type UsageInfo struct { UpdatedAt *time.Time `json:"updated_at,omitempty"` // 更新时间 @@ -163,6 +178,33 @@ type UsageInfo struct { // Antigravity 多模型配额 AntigravityQuota map[string]*AntigravityModelQuota `json:"antigravity_quota,omitempty"` + + // Antigravity 账号级信息 + SubscriptionTier string `json:"subscription_tier,omitempty"` // 归一化订阅等级: FREE/PRO/ULTRA/UNKNOWN + SubscriptionTierRaw string `json:"subscription_tier_raw,omitempty"` // 上游原始订阅等级名称 + + // Antigravity 模型详细能力信息(与 antigravity_quota 同 key) + AntigravityQuotaDetails map[string]*AntigravityModelDetail `json:"antigravity_quota_details,omitempty"` + + // Antigravity 废弃模型转发规则 (old_model_id -> new_model_id) + ModelForwardingRules map[string]string `json:"model_forwarding_rules,omitempty"` + + // Antigravity 账号是否被上游禁止 (HTTP 403) + IsForbidden bool `json:"is_forbidden,omitempty"` + ForbiddenReason string `json:"forbidden_reason,omitempty"` + ForbiddenType string `json:"forbidden_type,omitempty"` // "validation" / "violation" / "forbidden" + ValidationURL string `json:"validation_url,omitempty"` // 验证/申诉链接 + + // 状态标记(从 ForbiddenType / HTTP 错误码推导) + NeedsVerify bool `json:"needs_verify,omitempty"` // 需要人工验证(forbidden_type=validation) + IsBanned bool `json:"is_banned,omitempty"` // 账号被封(forbidden_type=violation) + NeedsReauth bool `json:"needs_reauth,omitempty"` // token 失效需重新授权(401) + + // 错误码(机器可读):forbidden / unauthenticated / rate_limited / network_error + ErrorCode string `json:"error_code,omitempty"` + + // 获取 usage 时的错误信息(降级返回,而非 500) + Error string `json:"error,omitempty"` } // ClaudeUsageResponse Anthropic API返回的usage结构 @@ -647,34 +689,154 @@ func (s *AccountUsageService) getAntigravityUsage(ctx context.Context, account * return &UsageInfo{UpdatedAt: &now}, nil } - // 1. 检查缓存(10 分钟) + // 1. 检查缓存 if cached, ok := s.cache.antigravityCache.Load(account.ID); ok { - if cache, ok := cached.(*antigravityUsageCache); ok && time.Since(cache.timestamp) < apiCacheTTL { - // 重新计算 RemainingSeconds - usage := cache.usageInfo - if usage.FiveHour != nil && usage.FiveHour.ResetsAt != nil { - usage.FiveHour.RemainingSeconds = int(time.Until(*usage.FiveHour.ResetsAt).Seconds()) + if cache, ok := cached.(*antigravityUsageCache); ok { + ttl := antigravityCacheTTL(cache.usageInfo) + if time.Since(cache.timestamp) < ttl { + usage := cache.usageInfo + if usage.FiveHour != nil && usage.FiveHour.ResetsAt != nil { + usage.FiveHour.RemainingSeconds = int(time.Until(*usage.FiveHour.ResetsAt).Seconds()) + } + return usage, nil } - return usage, nil } } - // 2. 获取代理 URL - proxyURL := s.antigravityQuotaFetcher.GetProxyURL(ctx, account) + // 2. singleflight 防止并发击穿 + flightKey := fmt.Sprintf("ag-usage:%d", account.ID) + result, flightErr, _ := s.cache.antigravityFlight.Do(flightKey, func() (any, error) { + // 再次检查缓存(等待期间可能已被填充) + if cached, ok := s.cache.antigravityCache.Load(account.ID); ok { + if cache, ok := cached.(*antigravityUsageCache); ok { + ttl := antigravityCacheTTL(cache.usageInfo) + if time.Since(cache.timestamp) < ttl { + usage := cache.usageInfo + // 重新计算 RemainingSeconds,避免返回过时的剩余秒数 + recalcAntigravityRemainingSeconds(usage) + return usage, nil + } + } + } - // 3. 调用 API 获取额度 - result, err := s.antigravityQuotaFetcher.FetchQuota(ctx, account, proxyURL) - if err != nil { - return nil, fmt.Errorf("fetch antigravity quota failed: %w", err) - } + // 使用独立 context,避免调用方 cancel 导致所有共享 flight 的请求失败 + fetchCtx, fetchCancel := context.WithTimeout(context.Background(), 30*time.Second) + defer fetchCancel() - // 4. 缓存结果 - s.cache.antigravityCache.Store(account.ID, &antigravityUsageCache{ - usageInfo: result.UsageInfo, - timestamp: time.Now(), + proxyURL := s.antigravityQuotaFetcher.GetProxyURL(fetchCtx, account) + fetchResult, err := s.antigravityQuotaFetcher.FetchQuota(fetchCtx, account, proxyURL) + if err != nil { + degraded := buildAntigravityDegradedUsage(err) + enrichUsageWithAccountError(degraded, account) + s.cache.antigravityCache.Store(account.ID, &antigravityUsageCache{ + usageInfo: degraded, + timestamp: time.Now(), + }) + return degraded, nil + } + + enrichUsageWithAccountError(fetchResult.UsageInfo, account) + s.cache.antigravityCache.Store(account.ID, &antigravityUsageCache{ + usageInfo: fetchResult.UsageInfo, + timestamp: time.Now(), + }) + return fetchResult.UsageInfo, nil }) - return result.UsageInfo, nil + if flightErr != nil { + return nil, flightErr + } + usage, ok := result.(*UsageInfo) + if !ok || usage == nil { + now := time.Now() + return &UsageInfo{UpdatedAt: &now}, nil + } + return usage, nil +} + +// recalcAntigravityRemainingSeconds 重新计算 Antigravity UsageInfo 中各窗口的 RemainingSeconds +// 用于从缓存取出时更新倒计时,避免返回过时的剩余秒数 +func recalcAntigravityRemainingSeconds(info *UsageInfo) { + if info == nil { + return + } + if info.FiveHour != nil && info.FiveHour.ResetsAt != nil { + remaining := int(time.Until(*info.FiveHour.ResetsAt).Seconds()) + if remaining < 0 { + remaining = 0 + } + info.FiveHour.RemainingSeconds = remaining + } +} + +// antigravityCacheTTL 根据 UsageInfo 内容决定缓存 TTL +// 403 forbidden 状态稳定,缓存与成功相同(3 分钟); +// 其他错误(401/网络)可能快速恢复,缓存 1 分钟。 +func antigravityCacheTTL(info *UsageInfo) time.Duration { + if info == nil { + return antigravityErrorTTL + } + if info.IsForbidden { + return apiCacheTTL // 封号/验证状态不会很快变 + } + if info.ErrorCode != "" || info.Error != "" { + return antigravityErrorTTL + } + return apiCacheTTL +} + +// buildAntigravityDegradedUsage 从 FetchQuota 错误构建降级 UsageInfo +func buildAntigravityDegradedUsage(err error) *UsageInfo { + now := time.Now() + errMsg := fmt.Sprintf("usage API error: %v", err) + slog.Warn("antigravity usage fetch failed, returning degraded response", "error", err) + + info := &UsageInfo{ + UpdatedAt: &now, + Error: errMsg, + } + + // 从错误信息推断 error_code 和状态标记 + // 错误格式来自 antigravity/client.go: "fetchAvailableModels 失败 (HTTP %d): ..." + errStr := err.Error() + switch { + case strings.Contains(errStr, "HTTP 401") || + strings.Contains(errStr, "UNAUTHENTICATED") || + strings.Contains(errStr, "invalid_grant"): + info.ErrorCode = errorCodeUnauthenticated + info.NeedsReauth = true + case strings.Contains(errStr, "HTTP 429"): + info.ErrorCode = errorCodeRateLimited + default: + info.ErrorCode = errorCodeNetworkError + } + + return info +} + +// enrichUsageWithAccountError 结合账号错误状态修正 UsageInfo +// 场景 1(成功路径):FetchAvailableModels 正常返回,但账号已因 403 被标记为 error, +// 需要在正常 usage 数据上附加 forbidden/validation 信息。 +// 场景 2(降级路径):被封号的账号 OAuth token 失效,FetchAvailableModels 返回 401, +// 降级逻辑设置了 needs_reauth,但账号实际是 403 封号/需验证,需覆盖为正确状态。 +func enrichUsageWithAccountError(info *UsageInfo, account *Account) { + if info == nil || account == nil || account.Status != StatusError { + return + } + msg := strings.ToLower(account.ErrorMessage) + if !strings.Contains(msg, "403") && !strings.Contains(msg, "forbidden") && + !strings.Contains(msg, "violation") && !strings.Contains(msg, "validation") { + return + } + fbType := classifyForbiddenType(account.ErrorMessage) + info.IsForbidden = true + info.ForbiddenType = fbType + info.ForbiddenReason = account.ErrorMessage + info.NeedsVerify = fbType == forbiddenTypeValidation + info.IsBanned = fbType == forbiddenTypeViolation + info.ValidationURL = extractValidationURL(account.ErrorMessage) + info.ErrorCode = errorCodeForbidden + info.NeedsReauth = false } // addWindowStats 为 usage 数据添加窗口期统计 diff --git a/backend/internal/service/antigravity_quota_fetcher.go b/backend/internal/service/antigravity_quota_fetcher.go index e950ec1d9d..f8990b1a21 100644 --- a/backend/internal/service/antigravity_quota_fetcher.go +++ b/backend/internal/service/antigravity_quota_fetcher.go @@ -2,12 +2,29 @@ package service import ( "context" + "encoding/json" + "errors" "fmt" + "log/slog" + "regexp" + "strings" "time" "github.com/Wei-Shaw/sub2api/internal/pkg/antigravity" ) +const ( + forbiddenTypeValidation = "validation" + forbiddenTypeViolation = "violation" + forbiddenTypeForbidden = "forbidden" + + // 机器可读的错误码 + errorCodeForbidden = "forbidden" + errorCodeUnauthenticated = "unauthenticated" + errorCodeRateLimited = "rate_limited" + errorCodeNetworkError = "network_error" +) + // AntigravityQuotaFetcher 从 Antigravity API 获取额度 type AntigravityQuotaFetcher struct { proxyRepo ProxyRepository @@ -40,11 +57,32 @@ func (f *AntigravityQuotaFetcher) FetchQuota(ctx context.Context, account *Accou // 调用 API 获取配额 modelsResp, modelsRaw, err := client.FetchAvailableModels(ctx, accessToken, projectID) if err != nil { + // 403 Forbidden: 不报错,返回 is_forbidden 标记 + var forbiddenErr *antigravity.ForbiddenError + if errors.As(err, &forbiddenErr) { + now := time.Now() + fbType := classifyForbiddenType(forbiddenErr.Body) + return &QuotaResult{ + UsageInfo: &UsageInfo{ + UpdatedAt: &now, + IsForbidden: true, + ForbiddenReason: forbiddenErr.Body, + ForbiddenType: fbType, + ValidationURL: extractValidationURL(forbiddenErr.Body), + NeedsVerify: fbType == forbiddenTypeValidation, + IsBanned: fbType == forbiddenTypeViolation, + ErrorCode: errorCodeForbidden, + }, + }, nil + } return nil, err } + // 调用 LoadCodeAssist 获取订阅等级(非关键路径,失败不影响主流程) + tierRaw, tierNormalized := f.fetchSubscriptionTier(ctx, client, accessToken) + // 转换为 UsageInfo - usageInfo := f.buildUsageInfo(modelsResp) + usageInfo := f.buildUsageInfo(modelsResp, tierRaw, tierNormalized) return &QuotaResult{ UsageInfo: usageInfo, @@ -52,15 +90,52 @@ func (f *AntigravityQuotaFetcher) FetchQuota(ctx context.Context, account *Accou }, nil } -// buildUsageInfo 将 API 响应转换为 UsageInfo -func (f *AntigravityQuotaFetcher) buildUsageInfo(modelsResp *antigravity.FetchAvailableModelsResponse) *UsageInfo { - now := time.Now() - info := &UsageInfo{ - UpdatedAt: &now, - AntigravityQuota: make(map[string]*AntigravityModelQuota), +// fetchSubscriptionTier 获取账号订阅等级,失败返回空字符串 +func (f *AntigravityQuotaFetcher) fetchSubscriptionTier(ctx context.Context, client *antigravity.Client, accessToken string) (raw, normalized string) { + loadResp, _, err := client.LoadCodeAssist(ctx, accessToken) + if err != nil { + slog.Warn("failed to fetch subscription tier", "error", err) + return "", "" + } + if loadResp == nil { + return "", "" } - // 遍历所有模型,填充 AntigravityQuota + raw = loadResp.GetTier() // 已有方法:paidTier > currentTier + normalized = normalizeTier(raw) + return raw, normalized +} + +// normalizeTier 将原始 tier 字符串归一化为 FREE/PRO/ULTRA/UNKNOWN +func normalizeTier(raw string) string { + if raw == "" { + return "" + } + lower := strings.ToLower(raw) + switch { + case strings.Contains(lower, "ultra"): + return "ULTRA" + case strings.Contains(lower, "pro"): + return "PRO" + case strings.Contains(lower, "free"): + return "FREE" + default: + return "UNKNOWN" + } +} + +// buildUsageInfo 将 API 响应转换为 UsageInfo +func (f *AntigravityQuotaFetcher) buildUsageInfo(modelsResp *antigravity.FetchAvailableModelsResponse, tierRaw, tierNormalized string) *UsageInfo { + now := time.Now() + info := &UsageInfo{ + UpdatedAt: &now, + AntigravityQuota: make(map[string]*AntigravityModelQuota), + AntigravityQuotaDetails: make(map[string]*AntigravityModelDetail), + SubscriptionTier: tierNormalized, + SubscriptionTierRaw: tierRaw, + } + + // 遍历所有模型,填充 AntigravityQuota 和 AntigravityQuotaDetails for modelName, modelInfo := range modelsResp.Models { if modelInfo.QuotaInfo == nil { continue @@ -73,6 +148,27 @@ func (f *AntigravityQuotaFetcher) buildUsageInfo(modelsResp *antigravity.FetchAv Utilization: utilization, ResetTime: modelInfo.QuotaInfo.ResetTime, } + + // 填充模型详细能力信息 + detail := &AntigravityModelDetail{ + DisplayName: modelInfo.DisplayName, + SupportsImages: modelInfo.SupportsImages, + SupportsThinking: modelInfo.SupportsThinking, + ThinkingBudget: modelInfo.ThinkingBudget, + Recommended: modelInfo.Recommended, + MaxTokens: modelInfo.MaxTokens, + MaxOutputTokens: modelInfo.MaxOutputTokens, + SupportedMimeTypes: modelInfo.SupportedMimeTypes, + } + info.AntigravityQuotaDetails[modelName] = detail + } + + // 废弃模型转发规则 + if len(modelsResp.DeprecatedModelIDs) > 0 { + info.ModelForwardingRules = make(map[string]string, len(modelsResp.DeprecatedModelIDs)) + for oldID, deprecated := range modelsResp.DeprecatedModelIDs { + info.ModelForwardingRules[oldID] = deprecated.NewModelID + } } // 同时设置 FiveHour 用于兼容展示(取主要模型) @@ -108,3 +204,58 @@ func (f *AntigravityQuotaFetcher) GetProxyURL(ctx context.Context, account *Acco } return proxy.URL() } + +// classifyForbiddenType 根据 403 响应体判断禁止类型 +func classifyForbiddenType(body string) string { + lower := strings.ToLower(body) + switch { + case strings.Contains(lower, "validation_required") || + strings.Contains(lower, "verify your account") || + strings.Contains(lower, "validation_url"): + return forbiddenTypeValidation + case strings.Contains(lower, "terms of service") || + strings.Contains(lower, "violation"): + return forbiddenTypeViolation + default: + return forbiddenTypeForbidden + } +} + +// urlPattern 用于从 403 响应体中提取 URL(降级方案) +var urlPattern = regexp.MustCompile(`https://[^\s"'\\]+`) + +// extractValidationURL 从 403 响应 JSON 中提取验证/申诉链接 +func extractValidationURL(body string) string { + // 1. 尝试结构化 JSON 提取: /error/details[*]/metadata/validation_url 或 appeal_url + var parsed struct { + Error struct { + Details []struct { + Metadata map[string]string `json:"metadata"` + } `json:"details"` + } `json:"error"` + } + if json.Unmarshal([]byte(body), &parsed) == nil { + for _, detail := range parsed.Error.Details { + if u := detail.Metadata["validation_url"]; u != "" { + return u + } + if u := detail.Metadata["appeal_url"]; u != "" { + return u + } + } + } + + // 2. 降级:正则匹配 URL + lower := strings.ToLower(body) + if !strings.Contains(lower, "validation") && + !strings.Contains(lower, "verify") && + !strings.Contains(lower, "appeal") { + return "" + } + // 先解码常见转义再匹配 + normalized := strings.ReplaceAll(body, `\u0026`, "&") + if m := urlPattern.FindString(normalized); m != "" { + return m + } + return "" +} diff --git a/backend/internal/service/antigravity_quota_fetcher_test.go b/backend/internal/service/antigravity_quota_fetcher_test.go new file mode 100644 index 0000000000..5ead8e605d --- /dev/null +++ b/backend/internal/service/antigravity_quota_fetcher_test.go @@ -0,0 +1,497 @@ +//go:build unit + +package service + +import ( + "errors" + "testing" + + "github.com/stretchr/testify/require" + + "github.com/Wei-Shaw/sub2api/internal/pkg/antigravity" +) + +// --------------------------------------------------------------------------- +// normalizeTier +// --------------------------------------------------------------------------- + +func TestNormalizeTier(t *testing.T) { + tests := []struct { + name string + raw string + expected string + }{ + {name: "empty string", raw: "", expected: ""}, + {name: "free-tier", raw: "free-tier", expected: "FREE"}, + {name: "g1-pro-tier", raw: "g1-pro-tier", expected: "PRO"}, + {name: "g1-ultra-tier", raw: "g1-ultra-tier", expected: "ULTRA"}, + {name: "unknown-something", raw: "unknown-something", expected: "UNKNOWN"}, + {name: "Google AI Pro contains pro keyword", raw: "Google AI Pro", expected: "PRO"}, + {name: "case insensitive FREE", raw: "FREE-TIER", expected: "FREE"}, + {name: "case insensitive Ultra", raw: "Ultra Plan", expected: "ULTRA"}, + {name: "arbitrary unrecognized string", raw: "enterprise-custom", expected: "UNKNOWN"}, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := normalizeTier(tt.raw) + require.Equal(t, tt.expected, got, "normalizeTier(%q)", tt.raw) + }) + } +} + +// --------------------------------------------------------------------------- +// buildUsageInfo +// --------------------------------------------------------------------------- + +func aqfBoolPtr(v bool) *bool { return &v } +func aqfIntPtr(v int) *int { return &v } + +func TestBuildUsageInfo_BasicModels(t *testing.T) { + fetcher := &AntigravityQuotaFetcher{} + + modelsResp := &antigravity.FetchAvailableModelsResponse{ + Models: map[string]antigravity.ModelInfo{ + "claude-sonnet-4-20250514": { + QuotaInfo: &antigravity.ModelQuotaInfo{ + RemainingFraction: 0.75, + ResetTime: "2026-03-08T12:00:00Z", + }, + DisplayName: "Claude Sonnet 4", + SupportsImages: aqfBoolPtr(true), + SupportsThinking: aqfBoolPtr(false), + ThinkingBudget: aqfIntPtr(0), + Recommended: aqfBoolPtr(true), + MaxTokens: aqfIntPtr(200000), + MaxOutputTokens: aqfIntPtr(16384), + SupportedMimeTypes: map[string]bool{ + "image/png": true, + "image/jpeg": true, + }, + }, + "gemini-2.5-pro": { + QuotaInfo: &antigravity.ModelQuotaInfo{ + RemainingFraction: 0.50, + ResetTime: "2026-03-08T15:00:00Z", + }, + DisplayName: "Gemini 2.5 Pro", + MaxTokens: aqfIntPtr(1000000), + MaxOutputTokens: aqfIntPtr(65536), + }, + }, + } + + info := fetcher.buildUsageInfo(modelsResp, "g1-pro-tier", "PRO") + + // 基本字段 + require.NotNil(t, info.UpdatedAt, "UpdatedAt should be set") + require.Equal(t, "PRO", info.SubscriptionTier) + require.Equal(t, "g1-pro-tier", info.SubscriptionTierRaw) + + // AntigravityQuota + require.Len(t, info.AntigravityQuota, 2) + + sonnetQuota := info.AntigravityQuota["claude-sonnet-4-20250514"] + require.NotNil(t, sonnetQuota) + require.Equal(t, 25, sonnetQuota.Utilization) // (1 - 0.75) * 100 = 25 + require.Equal(t, "2026-03-08T12:00:00Z", sonnetQuota.ResetTime) + + geminiQuota := info.AntigravityQuota["gemini-2.5-pro"] + require.NotNil(t, geminiQuota) + require.Equal(t, 50, geminiQuota.Utilization) // (1 - 0.50) * 100 = 50 + require.Equal(t, "2026-03-08T15:00:00Z", geminiQuota.ResetTime) + + // AntigravityQuotaDetails + require.Len(t, info.AntigravityQuotaDetails, 2) + + sonnetDetail := info.AntigravityQuotaDetails["claude-sonnet-4-20250514"] + require.NotNil(t, sonnetDetail) + require.Equal(t, "Claude Sonnet 4", sonnetDetail.DisplayName) + require.Equal(t, aqfBoolPtr(true), sonnetDetail.SupportsImages) + require.Equal(t, aqfBoolPtr(false), sonnetDetail.SupportsThinking) + require.Equal(t, aqfIntPtr(0), sonnetDetail.ThinkingBudget) + require.Equal(t, aqfBoolPtr(true), sonnetDetail.Recommended) + require.Equal(t, aqfIntPtr(200000), sonnetDetail.MaxTokens) + require.Equal(t, aqfIntPtr(16384), sonnetDetail.MaxOutputTokens) + require.Equal(t, map[string]bool{"image/png": true, "image/jpeg": true}, sonnetDetail.SupportedMimeTypes) + + geminiDetail := info.AntigravityQuotaDetails["gemini-2.5-pro"] + require.NotNil(t, geminiDetail) + require.Equal(t, "Gemini 2.5 Pro", geminiDetail.DisplayName) + require.Nil(t, geminiDetail.SupportsImages) + require.Nil(t, geminiDetail.SupportsThinking) + require.Equal(t, aqfIntPtr(1000000), geminiDetail.MaxTokens) + require.Equal(t, aqfIntPtr(65536), geminiDetail.MaxOutputTokens) +} + +func TestBuildUsageInfo_DeprecatedModels(t *testing.T) { + fetcher := &AntigravityQuotaFetcher{} + + modelsResp := &antigravity.FetchAvailableModelsResponse{ + Models: map[string]antigravity.ModelInfo{ + "claude-sonnet-4-20250514": { + QuotaInfo: &antigravity.ModelQuotaInfo{ + RemainingFraction: 1.0, + }, + }, + }, + DeprecatedModelIDs: map[string]antigravity.DeprecatedModelInfo{ + "claude-3-sonnet-20240229": {NewModelID: "claude-sonnet-4-20250514"}, + "claude-3-haiku-20240307": {NewModelID: "claude-haiku-3.5-latest"}, + }, + } + + info := fetcher.buildUsageInfo(modelsResp, "", "") + + require.Len(t, info.ModelForwardingRules, 2) + require.Equal(t, "claude-sonnet-4-20250514", info.ModelForwardingRules["claude-3-sonnet-20240229"]) + require.Equal(t, "claude-haiku-3.5-latest", info.ModelForwardingRules["claude-3-haiku-20240307"]) +} + +func TestBuildUsageInfo_NoDeprecatedModels(t *testing.T) { + fetcher := &AntigravityQuotaFetcher{} + + modelsResp := &antigravity.FetchAvailableModelsResponse{ + Models: map[string]antigravity.ModelInfo{ + "some-model": { + QuotaInfo: &antigravity.ModelQuotaInfo{RemainingFraction: 0.9}, + }, + }, + } + + info := fetcher.buildUsageInfo(modelsResp, "", "") + + require.Nil(t, info.ModelForwardingRules, "ModelForwardingRules should be nil when no deprecated models") +} + +func TestBuildUsageInfo_EmptyModels(t *testing.T) { + fetcher := &AntigravityQuotaFetcher{} + + modelsResp := &antigravity.FetchAvailableModelsResponse{ + Models: map[string]antigravity.ModelInfo{}, + } + + info := fetcher.buildUsageInfo(modelsResp, "", "") + + require.NotNil(t, info) + require.NotNil(t, info.AntigravityQuota) + require.Empty(t, info.AntigravityQuota) + require.NotNil(t, info.AntigravityQuotaDetails) + require.Empty(t, info.AntigravityQuotaDetails) + require.Nil(t, info.FiveHour, "FiveHour should be nil when no priority model exists") +} + +func TestBuildUsageInfo_ModelWithNilQuotaInfo(t *testing.T) { + fetcher := &AntigravityQuotaFetcher{} + + modelsResp := &antigravity.FetchAvailableModelsResponse{ + Models: map[string]antigravity.ModelInfo{ + "model-without-quota": { + DisplayName: "No Quota Model", + // QuotaInfo is nil + }, + }, + } + + info := fetcher.buildUsageInfo(modelsResp, "", "") + + require.NotNil(t, info) + require.Empty(t, info.AntigravityQuota, "models with nil QuotaInfo should be skipped") + require.Empty(t, info.AntigravityQuotaDetails, "models with nil QuotaInfo should be skipped from details too") +} + +func TestBuildUsageInfo_FiveHourPriorityOrder(t *testing.T) { + fetcher := &AntigravityQuotaFetcher{} + + // priorityModels = ["claude-sonnet-4-20250514", "claude-sonnet-4", "gemini-2.5-pro"] + // When the first priority model exists, it should be used for FiveHour + modelsResp := &antigravity.FetchAvailableModelsResponse{ + Models: map[string]antigravity.ModelInfo{ + "gemini-2.5-pro": { + QuotaInfo: &antigravity.ModelQuotaInfo{ + RemainingFraction: 0.40, + ResetTime: "2026-03-08T18:00:00Z", + }, + }, + "claude-sonnet-4-20250514": { + QuotaInfo: &antigravity.ModelQuotaInfo{ + RemainingFraction: 0.80, + ResetTime: "2026-03-08T12:00:00Z", + }, + }, + }, + } + + info := fetcher.buildUsageInfo(modelsResp, "", "") + + require.NotNil(t, info.FiveHour, "FiveHour should be set when a priority model exists") + // claude-sonnet-4-20250514 is first in priority list, so it should be used + expectedUtilization := (1.0 - 0.80) * 100 // 20 + require.InDelta(t, expectedUtilization, info.FiveHour.Utilization, 0.01) + require.NotNil(t, info.FiveHour.ResetsAt, "ResetsAt should be parsed from ResetTime") +} + +func TestBuildUsageInfo_FiveHourFallbackToClaude4(t *testing.T) { + fetcher := &AntigravityQuotaFetcher{} + + // Only claude-sonnet-4 exists (second in priority list), not claude-sonnet-4-20250514 + modelsResp := &antigravity.FetchAvailableModelsResponse{ + Models: map[string]antigravity.ModelInfo{ + "claude-sonnet-4": { + QuotaInfo: &antigravity.ModelQuotaInfo{ + RemainingFraction: 0.60, + ResetTime: "2026-03-08T14:00:00Z", + }, + }, + "gemini-2.5-pro": { + QuotaInfo: &antigravity.ModelQuotaInfo{ + RemainingFraction: 0.30, + }, + }, + }, + } + + info := fetcher.buildUsageInfo(modelsResp, "", "") + + require.NotNil(t, info.FiveHour) + expectedUtilization := (1.0 - 0.60) * 100 // 40 + require.InDelta(t, expectedUtilization, info.FiveHour.Utilization, 0.01) +} + +func TestBuildUsageInfo_FiveHourFallbackToGemini(t *testing.T) { + fetcher := &AntigravityQuotaFetcher{} + + // Only gemini-2.5-pro exists (third in priority list) + modelsResp := &antigravity.FetchAvailableModelsResponse{ + Models: map[string]antigravity.ModelInfo{ + "gemini-2.5-pro": { + QuotaInfo: &antigravity.ModelQuotaInfo{ + RemainingFraction: 0.30, + }, + }, + "other-model": { + QuotaInfo: &antigravity.ModelQuotaInfo{ + RemainingFraction: 0.90, + }, + }, + }, + } + + info := fetcher.buildUsageInfo(modelsResp, "", "") + + require.NotNil(t, info.FiveHour) + expectedUtilization := (1.0 - 0.30) * 100 // 70 + require.InDelta(t, expectedUtilization, info.FiveHour.Utilization, 0.01) +} + +func TestBuildUsageInfo_FiveHourNoPriorityModel(t *testing.T) { + fetcher := &AntigravityQuotaFetcher{} + + // None of the priority models exist + modelsResp := &antigravity.FetchAvailableModelsResponse{ + Models: map[string]antigravity.ModelInfo{ + "some-other-model": { + QuotaInfo: &antigravity.ModelQuotaInfo{ + RemainingFraction: 0.50, + }, + }, + }, + } + + info := fetcher.buildUsageInfo(modelsResp, "", "") + + require.Nil(t, info.FiveHour, "FiveHour should be nil when no priority model exists") +} + +func TestBuildUsageInfo_FiveHourWithEmptyResetTime(t *testing.T) { + fetcher := &AntigravityQuotaFetcher{} + + modelsResp := &antigravity.FetchAvailableModelsResponse{ + Models: map[string]antigravity.ModelInfo{ + "claude-sonnet-4-20250514": { + QuotaInfo: &antigravity.ModelQuotaInfo{ + RemainingFraction: 0.50, + ResetTime: "", // empty reset time + }, + }, + }, + } + + info := fetcher.buildUsageInfo(modelsResp, "", "") + + require.NotNil(t, info.FiveHour) + require.Nil(t, info.FiveHour.ResetsAt, "ResetsAt should be nil when ResetTime is empty") + require.Equal(t, 0, info.FiveHour.RemainingSeconds) +} + +func TestBuildUsageInfo_FullUtilization(t *testing.T) { + fetcher := &AntigravityQuotaFetcher{} + + modelsResp := &antigravity.FetchAvailableModelsResponse{ + Models: map[string]antigravity.ModelInfo{ + "claude-sonnet-4-20250514": { + QuotaInfo: &antigravity.ModelQuotaInfo{ + RemainingFraction: 0.0, // fully used + ResetTime: "2026-03-08T12:00:00Z", + }, + }, + }, + } + + info := fetcher.buildUsageInfo(modelsResp, "", "") + + quota := info.AntigravityQuota["claude-sonnet-4-20250514"] + require.NotNil(t, quota) + require.Equal(t, 100, quota.Utilization) +} + +func TestBuildUsageInfo_ZeroUtilization(t *testing.T) { + fetcher := &AntigravityQuotaFetcher{} + + modelsResp := &antigravity.FetchAvailableModelsResponse{ + Models: map[string]antigravity.ModelInfo{ + "claude-sonnet-4-20250514": { + QuotaInfo: &antigravity.ModelQuotaInfo{ + RemainingFraction: 1.0, // fully available + }, + }, + }, + } + + info := fetcher.buildUsageInfo(modelsResp, "", "") + + quota := info.AntigravityQuota["claude-sonnet-4-20250514"] + require.NotNil(t, quota) + require.Equal(t, 0, quota.Utilization) +} + +func TestFetchQuota_ForbiddenReturnsIsForbidden(t *testing.T) { + // 模拟 FetchQuota 遇到 403 时的行为: + // FetchAvailableModels 返回 ForbiddenError → FetchQuota 应返回 is_forbidden=true + forbiddenErr := &antigravity.ForbiddenError{ + StatusCode: 403, + Body: "Access denied", + } + + // 验证 ForbiddenError 满足 errors.As + var target *antigravity.ForbiddenError + require.True(t, errors.As(forbiddenErr, &target)) + require.Equal(t, 403, target.StatusCode) + require.Equal(t, "Access denied", target.Body) + require.Contains(t, forbiddenErr.Error(), "403") +} + +// --------------------------------------------------------------------------- +// classifyForbiddenType +// --------------------------------------------------------------------------- + +func TestClassifyForbiddenType(t *testing.T) { + tests := []struct { + name string + body string + expected string + }{ + { + name: "VALIDATION_REQUIRED keyword", + body: `{"error":{"message":"VALIDATION_REQUIRED"}}`, + expected: "validation", + }, + { + name: "verify your account", + body: `Please verify your account to continue`, + expected: "validation", + }, + { + name: "contains validation_url field", + body: `{"error":{"details":[{"metadata":{"validation_url":"https://..."}}]}}`, + expected: "validation", + }, + { + name: "terms of service violation", + body: `Your account has been suspended for Terms of Service violation`, + expected: "violation", + }, + { + name: "violation keyword", + body: `Account suspended due to policy violation`, + expected: "violation", + }, + { + name: "generic 403", + body: `Access denied`, + expected: "forbidden", + }, + { + name: "empty body", + body: "", + expected: "forbidden", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := classifyForbiddenType(tt.body) + require.Equal(t, tt.expected, got) + }) + } +} + +// --------------------------------------------------------------------------- +// extractValidationURL +// --------------------------------------------------------------------------- + +func TestExtractValidationURL(t *testing.T) { + tests := []struct { + name string + body string + expected string + }{ + { + name: "structured validation_url", + body: `{"error":{"details":[{"metadata":{"validation_url":"https://accounts.google.com/verify?token=abc"}}]}}`, + expected: "https://accounts.google.com/verify?token=abc", + }, + { + name: "structured appeal_url", + body: `{"error":{"details":[{"metadata":{"appeal_url":"https://support.google.com/appeal/123"}}]}}`, + expected: "https://support.google.com/appeal/123", + }, + { + name: "validation_url takes priority over appeal_url", + body: `{"error":{"details":[{"metadata":{"validation_url":"https://v.com","appeal_url":"https://a.com"}}]}}`, + expected: "https://v.com", + }, + { + name: "fallback regex with verify keyword", + body: `Please verify your account at https://accounts.google.com/verify`, + expected: "https://accounts.google.com/verify", + }, + { + name: "no URL in generic forbidden", + body: `Access denied`, + expected: "", + }, + { + name: "empty body", + body: "", + expected: "", + }, + { + name: "URL present but no validation keywords", + body: `Error at https://example.com/something`, + expected: "", + }, + { + name: "unicode escaped ampersand", + body: `validation required: https://accounts.google.com/verify?a=1\u0026b=2`, + expected: "https://accounts.google.com/verify?a=1&b=2", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := extractValidationURL(tt.body) + require.Equal(t, tt.expected, got) + }) + } +} diff --git a/backend/internal/service/antigravity_smart_retry_test.go b/backend/internal/service/antigravity_smart_retry_test.go index 432c80e59f..2999172612 100644 --- a/backend/internal/service/antigravity_smart_retry_test.go +++ b/backend/internal/service/antigravity_smart_retry_test.go @@ -9,6 +9,7 @@ import ( "net/http" "strings" "testing" + "time" "github.com/stretchr/testify/require" ) @@ -29,6 +30,24 @@ func (c *stubSmartRetryCache) DeleteSessionAccountID(_ context.Context, groupID c.deleteCalls = append(c.deleteCalls, deleteSessionCall{groupID: groupID, sessionHash: sessionHash}) return nil } +func (c *stubSmartRetryCache) GetClientAffinityAccounts(_ context.Context, _ int64, _ string, _ time.Duration) ([]int64, error) { + return nil, nil +} +func (c *stubSmartRetryCache) UpdateClientAffinity(_ context.Context, _ int64, _ string, _ int64, _ time.Duration) error { + return nil +} +func (c *stubSmartRetryCache) GetAccountAffinityCountBatch(_ context.Context, _ int64, _ []int64, _ time.Duration) (map[int64]int64, error) { + return map[int64]int64{}, nil +} +func (c *stubSmartRetryCache) GetAccountAffinityClientsBatch(_ context.Context, _ map[int64][]int64, _ time.Duration) (map[int64][]string, error) { + return map[int64][]string{}, nil +} +func (c *stubSmartRetryCache) GetAccountAffinityClientsWithScores(_ context.Context, _ int64, _ []int64, _ time.Duration) ([]AffinityClient, error) { + return nil, nil +} +func (c *stubSmartRetryCache) ClearAccountAffinity(_ context.Context, _ int64, _ []int64) error { + return nil +} // mockSmartRetryUpstream 用于 handleSmartRetry 测试的 mock upstream type mockSmartRetryUpstream struct { diff --git a/backend/internal/service/concurrency_service_test.go b/backend/internal/service/concurrency_service_test.go index 078ba0dc17..65f1c7c5f3 100644 --- a/backend/internal/service/concurrency_service_test.go +++ b/backend/internal/service/concurrency_service_test.go @@ -87,6 +87,7 @@ func (c *stubConcurrencyCacheForTest) GetAccountsLoadBatch(_ context.Context, _ func (c *stubConcurrencyCacheForTest) GetUsersLoadBatch(_ context.Context, _ []UserWithConcurrency) (map[int64]*UserLoadInfo, error) { return c.usersLoadBatch, c.usersLoadErr } + func (c *stubConcurrencyCacheForTest) CleanupExpiredAccountSlots(_ context.Context, _ int64) error { return c.cleanupErr } diff --git a/backend/internal/service/error_policy_test.go b/backend/internal/service/error_policy_test.go index dd9850bda8..b2055928c0 100644 --- a/backend/internal/service/error_policy_test.go +++ b/backend/internal/service/error_policy_test.go @@ -114,7 +114,7 @@ func TestCheckErrorPolicy(t *testing.T) { account: &Account{ ID: 15, Type: AccountTypeOAuth, - Platform: PlatformAntigravity, + Platform: PlatformGemini, // 非 Antigravity 平台才有 401 升级逻辑 TempUnschedulableReason: `{"status_code":401,"until_unix":1735689600}`, Credentials: map[string]any{ "temp_unschedulable_enabled": true, @@ -131,6 +131,28 @@ func TestCheckErrorPolicy(t *testing.T) { body: []byte(`unauthorized`), expected: ErrorPolicyNone, }, + { + name: "temp_unschedulable_401_antigravity_no_escalation", + account: &Account{ + ID: 16, + Type: AccountTypeOAuth, + Platform: PlatformAntigravity, // Antigravity 跳过 401 升级,由 rules 正常处理 + TempUnschedulableReason: `{"status_code":401,"until_unix":1735689600}`, + Credentials: map[string]any{ + "temp_unschedulable_enabled": true, + "temp_unschedulable_rules": []any{ + map[string]any{ + "error_code": float64(401), + "keywords": []any{"unauthorized"}, + "duration_minutes": float64(10), + }, + }, + }, + }, + statusCode: 401, + body: []byte(`unauthorized`), + expected: ErrorPolicyTempUnscheduled, // Antigravity 不升级,继续走规则匹配 + }, { name: "temp_unschedulable_body_miss_returns_none", account: &Account{ diff --git a/backend/internal/service/gateway_affinity_scheduling_test.go b/backend/internal/service/gateway_affinity_scheduling_test.go new file mode 100644 index 0000000000..0650081195 --- /dev/null +++ b/backend/internal/service/gateway_affinity_scheduling_test.go @@ -0,0 +1,800 @@ +//go:build unit + +package service + +import ( + "context" + "errors" + "sort" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +// --------------------------------------------------------------------------- +// Mock: GatewayCache for affinity tests +// --------------------------------------------------------------------------- + +// mockAffinityCache 为亲和调度测试提供可控的 GatewayCache mock。 +// 通过 getCountBatchFunc 可以自定义 GetAccountAffinityCountBatch 的行为。 +type mockAffinityCache struct { + getCountBatchFunc func(ctx context.Context, groupID int64, accountIDs []int64, ttl time.Duration) (map[int64]int64, error) + getCountBatchCalls int // 记录 GetAccountAffinityCountBatch 被调用次数 +} + +func (m *mockAffinityCache) GetSessionAccountID(_ context.Context, _ int64, _ string) (int64, error) { + return 0, errors.New("not found") +} +func (m *mockAffinityCache) SetSessionAccountID(_ context.Context, _ int64, _ string, _ int64, _ time.Duration) error { + return nil +} +func (m *mockAffinityCache) RefreshSessionTTL(_ context.Context, _ int64, _ string, _ time.Duration) error { + return nil +} +func (m *mockAffinityCache) DeleteSessionAccountID(_ context.Context, _ int64, _ string) error { + return nil +} +func (m *mockAffinityCache) GetClientAffinityAccounts(_ context.Context, _ int64, _ string, _ time.Duration) ([]int64, error) { + return nil, nil +} +func (m *mockAffinityCache) UpdateClientAffinity(_ context.Context, _ int64, _ string, _ int64, _ time.Duration) error { + return nil +} +func (m *mockAffinityCache) GetAccountAffinityCountBatch(ctx context.Context, groupID int64, accountIDs []int64, ttl time.Duration) (map[int64]int64, error) { + m.getCountBatchCalls++ + if m.getCountBatchFunc != nil { + return m.getCountBatchFunc(ctx, groupID, accountIDs, ttl) + } + return map[int64]int64{}, nil +} +func (m *mockAffinityCache) GetAccountAffinityClientsBatch(_ context.Context, _ map[int64][]int64, _ time.Duration) (map[int64][]string, error) { + return map[int64][]string{}, nil +} +func (m *mockAffinityCache) GetAccountAffinityClientsWithScores(_ context.Context, _ int64, _ []int64, _ time.Duration) ([]AffinityClient, error) { + return nil, nil +} +func (m *mockAffinityCache) ClearAccountAffinity(_ context.Context, _ int64, _ []int64) error { + return nil +} + +// --------------------------------------------------------------------------- +// Helper: 构造启用了客户端亲和的 Anthropic 账号 +// --------------------------------------------------------------------------- + +func newAffinityAccount(id int64, priority int, affinityEnabled bool) *Account { + acc := &Account{ + ID: id, + Platform: PlatformAnthropic, + Priority: priority, + Status: StatusActive, + } + if affinityEnabled { + acc.Extra = map[string]any{"client_affinity_enabled": true} + } + return acc +} + +func newAffinityAccountWithLoad(id int64, priority int, loadRate int, affinityCount int64, lastUsedAt *time.Time) accountWithLoad { + return accountWithLoad{ + account: newAffinityAccount(id, priority, true), + loadInfo: &AccountLoadInfo{AccountID: id, LoadRate: loadRate}, + affinityCount: affinityCount, + } +} + +// =========================================================================== +// 1. filterByMinAffinityCount 测试 +// =========================================================================== + +func TestAffinityFilterByMinAffinityCount(t *testing.T) { + t.Run("empty slice returns empty", func(t *testing.T) { + result := filterByMinAffinityCount(nil) + require.Empty(t, result) + }) + + t.Run("single element returned as-is", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 1}, loadInfo: &AccountLoadInfo{}, affinityCount: 5}, + } + result := filterByMinAffinityCount(accounts) + require.Len(t, result, 1) + require.Equal(t, int64(1), result[0].account.ID) + }) + + t.Run("all same affinityCount returns all", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 1}, loadInfo: &AccountLoadInfo{}, affinityCount: 3}, + {account: &Account{ID: 2}, loadInfo: &AccountLoadInfo{}, affinityCount: 3}, + {account: &Account{ID: 3}, loadInfo: &AccountLoadInfo{}, affinityCount: 3}, + } + result := filterByMinAffinityCount(accounts) + require.Len(t, result, 3) + }) + + t.Run("filters to min affinityCount only", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 1}, loadInfo: &AccountLoadInfo{}, affinityCount: 10}, + {account: &Account{ID: 2}, loadInfo: &AccountLoadInfo{}, affinityCount: 2}, + {account: &Account{ID: 3}, loadInfo: &AccountLoadInfo{}, affinityCount: 5}, + {account: &Account{ID: 4}, loadInfo: &AccountLoadInfo{}, affinityCount: 2}, + } + result := filterByMinAffinityCount(accounts) + require.Len(t, result, 2) + require.Equal(t, int64(2), result[0].account.ID) + require.Equal(t, int64(4), result[1].account.ID) + }) + + t.Run("zero affinityCount is smallest", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 1}, loadInfo: &AccountLoadInfo{}, affinityCount: 5}, + {account: &Account{ID: 2}, loadInfo: &AccountLoadInfo{}, affinityCount: 0}, + {account: &Account{ID: 3}, loadInfo: &AccountLoadInfo{}, affinityCount: 3}, + {account: &Account{ID: 4}, loadInfo: &AccountLoadInfo{}, affinityCount: 0}, + } + result := filterByMinAffinityCount(accounts) + require.Len(t, result, 2) + require.Equal(t, int64(2), result[0].account.ID) + require.Equal(t, int64(4), result[1].account.ID) + }) + + t.Run("preserves order within same affinityCount", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 5}, loadInfo: &AccountLoadInfo{}, affinityCount: 1}, + {account: &Account{ID: 3}, loadInfo: &AccountLoadInfo{}, affinityCount: 1}, + {account: &Account{ID: 7}, loadInfo: &AccountLoadInfo{}, affinityCount: 2}, + {account: &Account{ID: 1}, loadInfo: &AccountLoadInfo{}, affinityCount: 1}, + } + result := filterByMinAffinityCount(accounts) + require.Len(t, result, 3) + // 验证保持原始顺序 + require.Equal(t, int64(5), result[0].account.ID) + require.Equal(t, int64(3), result[1].account.ID) + require.Equal(t, int64(1), result[2].account.ID) + }) +} + +// =========================================================================== +// 2. populateAffinityCounts 测试 +// =========================================================================== + +func TestAffinityPopulateAffinityCounts(t *testing.T) { + ctx := context.Background() + + t.Run("nil cache does not panic", func(t *testing.T) { + svc := &GatewayService{cache: nil} + accounts := []accountWithLoad{ + {account: newAffinityAccount(1, 1, true), loadInfo: &AccountLoadInfo{}}, + } + // 不应 panic + svc.populateAffinityCounts(ctx, accounts, 0) + // affinityCount 保持零值 + require.Equal(t, int64(0), accounts[0].affinityCount) + }) + + t.Run("empty accounts returns immediately", func(t *testing.T) { + cache := &mockAffinityCache{} + svc := &GatewayService{cache: cache} + svc.populateAffinityCounts(ctx, nil, 0) + require.Equal(t, 0, cache.getCountBatchCalls, "should not call Redis for empty accounts") + }) + + t.Run("no affinity-enabled accounts skips Redis call", func(t *testing.T) { + cache := &mockAffinityCache{} + svc := &GatewayService{cache: cache} + accounts := []accountWithLoad{ + // Anthropic 但未启用亲和 + {account: newAffinityAccount(1, 1, false), loadInfo: &AccountLoadInfo{}}, + // 非 Anthropic 平台 + {account: &Account{ID: 2, Platform: PlatformOpenAI}, loadInfo: &AccountLoadInfo{}}, + } + svc.populateAffinityCounts(ctx, accounts, 0) + require.Equal(t, 0, cache.getCountBatchCalls, "should skip Redis when no affinity-enabled accounts") + }) + + t.Run("correctly populates affinityCount from Redis", func(t *testing.T) { + cache := &mockAffinityCache{ + getCountBatchFunc: func(_ context.Context, _ int64, accountIDs []int64, _ time.Duration) (map[int64]int64, error) { + result := map[int64]int64{} + for _, id := range accountIDs { + switch id { + case 1: + result[1] = 5 + case 2: + result[2] = 0 + case 3: + result[3] = 12 + } + } + return result, nil + }, + } + svc := &GatewayService{cache: cache} + + accounts := []accountWithLoad{ + {account: newAffinityAccount(1, 1, true), loadInfo: &AccountLoadInfo{}}, + {account: newAffinityAccount(2, 1, false), loadInfo: &AccountLoadInfo{}}, // 未启用,但仍在列表中 + {account: newAffinityAccount(3, 1, true), loadInfo: &AccountLoadInfo{}}, + } + + svc.populateAffinityCounts(ctx, accounts, 100) + + require.Equal(t, 1, cache.getCountBatchCalls, "should call Redis exactly once") + require.Equal(t, int64(5), accounts[0].affinityCount, "account 1 should have count 5") + require.Equal(t, int64(0), accounts[1].affinityCount, "account 2 should have count 0") + require.Equal(t, int64(12), accounts[2].affinityCount, "account 3 should have count 12") + }) + + t.Run("Redis error degrades gracefully with counts at 0", func(t *testing.T) { + cache := &mockAffinityCache{ + getCountBatchFunc: func(_ context.Context, _ int64, _ []int64, _ time.Duration) (map[int64]int64, error) { + return nil, errors.New("redis connection refused") + }, + } + svc := &GatewayService{cache: cache} + + accounts := []accountWithLoad{ + {account: newAffinityAccount(1, 1, true), loadInfo: &AccountLoadInfo{}}, + {account: newAffinityAccount(2, 1, true), loadInfo: &AccountLoadInfo{}}, + } + + svc.populateAffinityCounts(ctx, accounts, 0) + + require.Equal(t, 1, cache.getCountBatchCalls) + require.Equal(t, int64(0), accounts[0].affinityCount, "should remain 0 on error") + require.Equal(t, int64(0), accounts[1].affinityCount, "should remain 0 on error") + }) + + t.Run("partial Redis result fills only known accounts", func(t *testing.T) { + cache := &mockAffinityCache{ + getCountBatchFunc: func(_ context.Context, _ int64, _ []int64, _ time.Duration) (map[int64]int64, error) { + // 只返回部分账号的计数 + return map[int64]int64{1: 7}, nil + }, + } + svc := &GatewayService{cache: cache} + + accounts := []accountWithLoad{ + {account: newAffinityAccount(1, 1, true), loadInfo: &AccountLoadInfo{}}, + {account: newAffinityAccount(2, 1, true), loadInfo: &AccountLoadInfo{}}, + } + + svc.populateAffinityCounts(ctx, accounts, 0) + + require.Equal(t, int64(7), accounts[0].affinityCount) + require.Equal(t, int64(0), accounts[1].affinityCount, "missing account should default to 0") + }) + + t.Run("queries all account IDs regardless of affinity status", func(t *testing.T) { + // 验证:只要有至少一个 affinity-enabled 账号,就查询 ALL 账号的计数 + var queriedIDs []int64 + cache := &mockAffinityCache{ + getCountBatchFunc: func(_ context.Context, _ int64, accountIDs []int64, _ time.Duration) (map[int64]int64, error) { + queriedIDs = accountIDs + return map[int64]int64{}, nil + }, + } + svc := &GatewayService{cache: cache} + + accounts := []accountWithLoad{ + {account: newAffinityAccount(1, 1, true), loadInfo: &AccountLoadInfo{}}, + {account: newAffinityAccount(2, 1, false), loadInfo: &AccountLoadInfo{}}, + {account: &Account{ID: 3, Platform: PlatformOpenAI}, loadInfo: &AccountLoadInfo{}}, + } + + svc.populateAffinityCounts(ctx, accounts, 0) + + require.Equal(t, 1, cache.getCountBatchCalls) + require.Equal(t, []int64{1, 2, 3}, queriedIDs, "should query ALL account IDs, not just affinity-enabled ones") + }) +} + +// =========================================================================== +// 3. Layer 1 排序链测试(sort.SliceStable 中 affinityCount 的正确性) +// =========================================================================== + +func TestAffinityLayer1SortChain(t *testing.T) { + now := time.Now() + earlier := now.Add(-1 * time.Hour) + muchEarlier := now.Add(-2 * time.Hour) + + // 复现 Layer 1 的排序逻辑 + sortByLayer1 := func(accounts []accountWithLoad) { + sort.SliceStable(accounts, func(i, j int) bool { + a, b := accounts[i], accounts[j] + if a.account.Priority != b.account.Priority { + return a.account.Priority < b.account.Priority + } + if a.loadInfo.LoadRate != b.loadInfo.LoadRate { + return a.loadInfo.LoadRate < b.loadInfo.LoadRate + } + if a.affinityCount != b.affinityCount { + return a.affinityCount < b.affinityCount + } + switch { + case a.account.LastUsedAt == nil && b.account.LastUsedAt != nil: + return true + case a.account.LastUsedAt != nil && b.account.LastUsedAt == nil: + return false + case a.account.LastUsedAt == nil && b.account.LastUsedAt == nil: + return false + default: + return a.account.LastUsedAt.Before(*b.account.LastUsedAt) + } + }) + } + + t.Run("same priority same loadRate sorts by affinityCount asc", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 1, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 50}, affinityCount: 10}, + {account: &Account{ID: 2, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 50}, affinityCount: 2}, + {account: &Account{ID: 3, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 50}, affinityCount: 5}, + } + sortByLayer1(accounts) + require.Equal(t, int64(2), accounts[0].account.ID, "lowest affinityCount first") + require.Equal(t, int64(3), accounts[1].account.ID) + require.Equal(t, int64(1), accounts[2].account.ID, "highest affinityCount last") + }) + + t.Run("priority takes precedence over affinityCount", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 1, Priority: 2, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 50}, affinityCount: 0}, + {account: &Account{ID: 2, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 50}, affinityCount: 100}, + } + sortByLayer1(accounts) + require.Equal(t, int64(2), accounts[0].account.ID, "lower priority wins despite higher affinityCount") + }) + + t.Run("loadRate takes precedence over affinityCount", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 1, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 80}, affinityCount: 0}, + {account: &Account{ID: 2, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 20}, affinityCount: 100}, + } + sortByLayer1(accounts) + require.Equal(t, int64(2), accounts[0].account.ID, "lower loadRate wins despite higher affinityCount") + }) + + t.Run("affinityCount takes precedence over LRU", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 1, Priority: 1, LastUsedAt: &muchEarlier}, loadInfo: &AccountLoadInfo{LoadRate: 50}, affinityCount: 5}, + {account: &Account{ID: 2, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 50}, affinityCount: 1}, + } + sortByLayer1(accounts) + require.Equal(t, int64(2), accounts[0].account.ID, "lower affinityCount wins despite older LRU") + }) + + t.Run("same affinityCount falls through to LRU", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 1, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 50}, affinityCount: 3}, + {account: &Account{ID: 2, Priority: 1, LastUsedAt: &earlier}, loadInfo: &AccountLoadInfo{LoadRate: 50}, affinityCount: 3}, + {account: &Account{ID: 3, Priority: 1, LastUsedAt: &muchEarlier}, loadInfo: &AccountLoadInfo{LoadRate: 50}, affinityCount: 3}, + } + sortByLayer1(accounts) + require.Equal(t, int64(3), accounts[0].account.ID, "LRU: oldest used first") + require.Equal(t, int64(2), accounts[1].account.ID) + require.Equal(t, int64(1), accounts[2].account.ID, "LRU: most recently used last") + }) + + t.Run("full chain: priority > loadRate > affinityCount > LRU", func(t *testing.T) { + accounts := []accountWithLoad{ + // 优先级 2 - 不管其他维度如何,排在后面 + {account: &Account{ID: 10, Priority: 2, LastUsedAt: &muchEarlier}, loadInfo: &AccountLoadInfo{LoadRate: 0}, affinityCount: 0}, + // 优先级 1,负载 80% - 负载高 + {account: &Account{ID: 20, Priority: 1, LastUsedAt: &muchEarlier}, loadInfo: &AccountLoadInfo{LoadRate: 80}, affinityCount: 0}, + // 优先级 1,负载 20%,亲和 5 + {account: &Account{ID: 30, Priority: 1, LastUsedAt: &muchEarlier}, loadInfo: &AccountLoadInfo{LoadRate: 20}, affinityCount: 5}, + // 优先级 1,负载 20%,亲和 1,最近使用 + {account: &Account{ID: 40, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 20}, affinityCount: 1}, + // 优先级 1,负载 20%,亲和 1,更早使用(应排最前) + {account: &Account{ID: 50, Priority: 1, LastUsedAt: &earlier}, loadInfo: &AccountLoadInfo{LoadRate: 20}, affinityCount: 1}, + } + sortByLayer1(accounts) + // 期望排序:50 → 40 → 30 → 20 → 10 + require.Equal(t, int64(50), accounts[0].account.ID, "best: p1, lr20, ac1, LRU earlier") + require.Equal(t, int64(40), accounts[1].account.ID, "second: p1, lr20, ac1, LRU now") + require.Equal(t, int64(30), accounts[2].account.ID, "third: p1, lr20, ac5") + require.Equal(t, int64(20), accounts[3].account.ID, "fourth: p1, lr80") + require.Equal(t, int64(10), accounts[4].account.ID, "last: p2") + }) +} + +// =========================================================================== +// 4. Layer 2 分层过滤链完整性测试 +// =========================================================================== + +func TestAffinityLayer2FilterChain(t *testing.T) { + now := time.Now() + earlier := now.Add(-1 * time.Hour) + muchEarlier := now.Add(-2 * time.Hour) + + // 模拟 Layer 2 的完整过滤链:Priority → LoadRate → AffinityCount → LRU + applyLayer2 := func(accounts []accountWithLoad) *accountWithLoad { + candidates := filterByMinPriority(accounts) + candidates = filterByMinLoadRate(candidates) + candidates = filterByMinAffinityCount(candidates) + return selectByLRU(candidates, false) + } + + t.Run("priority different - affinityCount does not matter", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 1, Priority: 2, LastUsedAt: &muchEarlier}, loadInfo: &AccountLoadInfo{LoadRate: 0}, affinityCount: 0}, + {account: &Account{ID: 2, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 50}, affinityCount: 100}, + } + selected := applyLayer2(accounts) + require.NotNil(t, selected) + require.Equal(t, int64(2), selected.account.ID, "higher priority dimension overrides affinityCount") + }) + + t.Run("same priority same loadRate different affinityCount", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 1, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 30}, affinityCount: 10}, + {account: &Account{ID: 2, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 30}, affinityCount: 2}, + {account: &Account{ID: 3, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 30}, affinityCount: 5}, + } + selected := applyLayer2(accounts) + require.NotNil(t, selected) + require.Equal(t, int64(2), selected.account.ID, "lowest affinityCount wins") + }) + + t.Run("same priority same loadRate same affinityCount falls through to LRU", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 1, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 30}, affinityCount: 5}, + {account: &Account{ID: 2, Priority: 1, LastUsedAt: &earlier}, loadInfo: &AccountLoadInfo{LoadRate: 30}, affinityCount: 5}, + {account: &Account{ID: 3, Priority: 1, LastUsedAt: &muchEarlier}, loadInfo: &AccountLoadInfo{LoadRate: 30}, affinityCount: 5}, + } + selected := applyLayer2(accounts) + require.NotNil(t, selected) + require.Equal(t, int64(3), selected.account.ID, "LRU selects oldest") + }) + + t.Run("loadRate different overrides affinityCount", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 1, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 80}, affinityCount: 0}, + {account: &Account{ID: 2, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 10}, affinityCount: 100}, + } + selected := applyLayer2(accounts) + require.NotNil(t, selected) + require.Equal(t, int64(2), selected.account.ID, "lower loadRate wins over lower affinityCount") + }) + + t.Run("full chain integration: p → lr → ac → lru", func(t *testing.T) { + accounts := []accountWithLoad{ + // p=2 淘汰 + {account: &Account{ID: 1, Priority: 2, LastUsedAt: &muchEarlier}, loadInfo: &AccountLoadInfo{LoadRate: 0}, affinityCount: 0}, + // p=1, lr=50 淘汰 + {account: &Account{ID: 2, Priority: 1, LastUsedAt: &muchEarlier}, loadInfo: &AccountLoadInfo{LoadRate: 50}, affinityCount: 0}, + // p=1, lr=10, ac=8 淘汰 + {account: &Account{ID: 3, Priority: 1, LastUsedAt: &muchEarlier}, loadInfo: &AccountLoadInfo{LoadRate: 10}, affinityCount: 8}, + // p=1, lr=10, ac=2, lru=now 淘汰 + {account: &Account{ID: 4, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 10}, affinityCount: 2}, + // p=1, lr=10, ac=2, lru=muchEarlier → 胜出 + {account: &Account{ID: 5, Priority: 1, LastUsedAt: &muchEarlier}, loadInfo: &AccountLoadInfo{LoadRate: 10}, affinityCount: 2}, + } + selected := applyLayer2(accounts) + require.NotNil(t, selected) + require.Equal(t, int64(5), selected.account.ID, "full chain selects ID=5") + }) + + t.Run("empty input returns nil", func(t *testing.T) { + selected := applyLayer2(nil) + require.Nil(t, selected) + }) + + t.Run("single account always selected", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 42, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 50}, affinityCount: 100}, + } + selected := applyLayer2(accounts) + require.NotNil(t, selected) + require.Equal(t, int64(42), selected.account.ID) + }) + + t.Run("affinityCount zero preferred among same p and lr", func(t *testing.T) { + accounts := []accountWithLoad{ + {account: &Account{ID: 1, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 0}, affinityCount: 5}, + {account: &Account{ID: 2, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 0}, affinityCount: 0}, + {account: &Account{ID: 3, Priority: 1, LastUsedAt: &now}, loadInfo: &AccountLoadInfo{LoadRate: 0}, affinityCount: 3}, + } + selected := applyLayer2(accounts) + require.NotNil(t, selected) + require.Equal(t, int64(2), selected.account.ID, "zero affinityCount preferred") + }) +} + +// =========================================================================== +// 5. populateAffinityCounts + filterByMinAffinityCount 联合测试 +// =========================================================================== + +func TestAffinityPopulateAndFilterIntegration(t *testing.T) { + ctx := context.Background() + + t.Run("populate then filter selects least-loaded accounts", func(t *testing.T) { + cache := &mockAffinityCache{ + getCountBatchFunc: func(_ context.Context, _ int64, _ []int64, _ time.Duration) (map[int64]int64, error) { + return map[int64]int64{ + 1: 10, + 2: 3, + 3: 3, + 4: 7, + }, nil + }, + } + svc := &GatewayService{cache: cache} + + accounts := []accountWithLoad{ + {account: newAffinityAccount(1, 1, true), loadInfo: &AccountLoadInfo{}}, + {account: newAffinityAccount(2, 1, true), loadInfo: &AccountLoadInfo{}}, + {account: newAffinityAccount(3, 1, true), loadInfo: &AccountLoadInfo{}}, + {account: newAffinityAccount(4, 1, true), loadInfo: &AccountLoadInfo{}}, + } + + svc.populateAffinityCounts(ctx, accounts, 0) + result := filterByMinAffinityCount(accounts) + + require.Len(t, result, 2) + require.Equal(t, int64(2), result[0].account.ID) + require.Equal(t, int64(3), result[1].account.ID) + }) + + t.Run("Redis failure results in all accounts having 0 affinityCount", func(t *testing.T) { + cache := &mockAffinityCache{ + getCountBatchFunc: func(_ context.Context, _ int64, _ []int64, _ time.Duration) (map[int64]int64, error) { + return nil, errors.New("timeout") + }, + } + svc := &GatewayService{cache: cache} + + accounts := []accountWithLoad{ + {account: newAffinityAccount(1, 1, true), loadInfo: &AccountLoadInfo{}}, + {account: newAffinityAccount(2, 1, true), loadInfo: &AccountLoadInfo{}}, + } + + svc.populateAffinityCounts(ctx, accounts, 0) + result := filterByMinAffinityCount(accounts) + + // 全部为 0,全部返回 + require.Len(t, result, 2, "all accounts should pass filter when Redis fails (all have count 0)") + }) +} + +// =========================================================================== +// 6. IsClientAffinityEnabled 边界测试 +// =========================================================================== + +func TestAffinityIsClientAffinityEnabled(t *testing.T) { + t.Run("Anthropic with enabled flag", func(t *testing.T) { + acc := &Account{ + Platform: PlatformAnthropic, + Extra: map[string]any{"client_affinity_enabled": true}, + } + assert.True(t, acc.IsClientAffinityEnabled()) + }) + + t.Run("Anthropic with disabled flag", func(t *testing.T) { + acc := &Account{ + Platform: PlatformAnthropic, + Extra: map[string]any{"client_affinity_enabled": false}, + } + assert.False(t, acc.IsClientAffinityEnabled()) + }) + + t.Run("Anthropic with nil Extra", func(t *testing.T) { + acc := &Account{ + Platform: PlatformAnthropic, + Extra: nil, + } + assert.False(t, acc.IsClientAffinityEnabled()) + }) + + t.Run("Anthropic without the key", func(t *testing.T) { + acc := &Account{ + Platform: PlatformAnthropic, + Extra: map[string]any{"other_key": true}, + } + assert.False(t, acc.IsClientAffinityEnabled()) + }) + + t.Run("non-Anthropic platform always false", func(t *testing.T) { + platforms := []string{PlatformOpenAI, PlatformGemini, PlatformAntigravity} + for _, p := range platforms { + acc := &Account{ + Platform: p, + Extra: map[string]any{"client_affinity_enabled": true}, + } + assert.False(t, acc.IsClientAffinityEnabled(), "platform=%s should not support affinity", p) + } + }) + + t.Run("wrong type for enabled value", func(t *testing.T) { + acc := &Account{ + Platform: PlatformAnthropic, + Extra: map[string]any{"client_affinity_enabled": "true"}, // string 而非 bool + } + assert.False(t, acc.IsClientAffinityEnabled(), "string 'true' should not enable affinity") + }) +} + +// =========================================================================== +// GetAffinityZone 测试 +// =========================================================================== + +func TestGetAffinityZone(t *testing.T) { + makeAccount := func(enabled bool, base int, buffer any) *Account { + extra := map[string]any{"client_affinity_enabled": enabled} + if base > 0 { + extra["affinity_base"] = base + } + if buffer != nil { + extra["affinity_buffer"] = buffer + } + return &Account{ + Platform: PlatformAnthropic, + Extra: extra, + } + } + + t.Run("affinity disabled always green", func(t *testing.T) { + acc := makeAccount(false, 5, 3) + assert.Equal(t, AffinityZoneGreen, acc.GetAffinityZone(100)) + }) + + t.Run("no base configured always green", func(t *testing.T) { + acc := makeAccount(true, 0, nil) + assert.Equal(t, AffinityZoneGreen, acc.GetAffinityZone(100)) + }) + + t.Run("within base limit is green", func(t *testing.T) { + acc := makeAccount(true, 5, 3) + assert.Equal(t, AffinityZoneGreen, acc.GetAffinityZone(0)) + assert.Equal(t, AffinityZoneGreen, acc.GetAffinityZone(3)) + assert.Equal(t, AffinityZoneGreen, acc.GetAffinityZone(5)) + }) + + t.Run("no buffer configured infinite yellow", func(t *testing.T) { + acc := makeAccount(true, 5, nil) // buffer not set → infinite yellow + assert.Equal(t, AffinityZoneYellow, acc.GetAffinityZone(6)) + assert.Equal(t, AffinityZoneYellow, acc.GetAffinityZone(100)) + assert.Equal(t, AffinityZoneYellow, acc.GetAffinityZone(9999)) + }) + + t.Run("buffer zero no yellow zone", func(t *testing.T) { + acc := makeAccount(true, 5, 0) // buffer=0 → no yellow, direct red + assert.Equal(t, AffinityZoneGreen, acc.GetAffinityZone(5)) + assert.Equal(t, AffinityZoneRed, acc.GetAffinityZone(6)) + assert.Equal(t, AffinityZoneRed, acc.GetAffinityZone(100)) + }) + + t.Run("within buffer is yellow", func(t *testing.T) { + acc := makeAccount(true, 5, 3) + assert.Equal(t, AffinityZoneYellow, acc.GetAffinityZone(6)) + assert.Equal(t, AffinityZoneYellow, acc.GetAffinityZone(7)) + assert.Equal(t, AffinityZoneYellow, acc.GetAffinityZone(8)) // base(5)+buffer(3)=8 + }) + + t.Run("beyond buffer is red", func(t *testing.T) { + acc := makeAccount(true, 5, 3) + assert.Equal(t, AffinityZoneRed, acc.GetAffinityZone(9)) + assert.Equal(t, AffinityZoneRed, acc.GetAffinityZone(100)) + }) + + t.Run("boundary exactly at base", func(t *testing.T) { + acc := makeAccount(true, 10, 5) + assert.Equal(t, AffinityZoneGreen, acc.GetAffinityZone(10)) + assert.Equal(t, AffinityZoneYellow, acc.GetAffinityZone(11)) + }) + + t.Run("boundary exactly at base plus buffer", func(t *testing.T) { + acc := makeAccount(true, 10, 5) + assert.Equal(t, AffinityZoneYellow, acc.GetAffinityZone(15)) + assert.Equal(t, AffinityZoneRed, acc.GetAffinityZone(16)) + }) +} + +// =========================================================================== +// classifyByAffinityZone 测试 +// =========================================================================== + +func TestClassifyByAffinityZone(t *testing.T) { + makeAWL := func(id int64, base int, buffer any, count int64) accountWithLoad { + extra := map[string]any{"client_affinity_enabled": true} + if base > 0 { + extra["affinity_base"] = base + } + if buffer != nil { + extra["affinity_buffer"] = buffer + } + return accountWithLoad{ + account: &Account{ID: id, Platform: PlatformAnthropic, Extra: extra}, + loadInfo: &AccountLoadInfo{AccountID: id}, + affinityCount: count, + } + } + + t.Run("empty input returns empty", func(t *testing.T) { + result := classifyByAffinityZone(nil) + require.Empty(t, result) + }) + + t.Run("no zone config returns all", func(t *testing.T) { + // 没有账号配置 affinity_base → 原样返回 + accs := []accountWithLoad{ + {account: newAffinityAccount(1, 50, true), loadInfo: &AccountLoadInfo{AccountID: 1}}, + {account: newAffinityAccount(2, 50, true), loadInfo: &AccountLoadInfo{AccountID: 2}}, + } + result := classifyByAffinityZone(accs) + require.Len(t, result, 2) + }) + + t.Run("greens preferred over yellows", func(t *testing.T) { + accs := []accountWithLoad{ + makeAWL(1, 5, 3, 3), // green (3 ≤ 5) + makeAWL(2, 5, 3, 7), // yellow (5 < 7 ≤ 8) + makeAWL(3, 5, 3, 2), // green (2 ≤ 5) + } + result := classifyByAffinityZone(accs) + require.Len(t, result, 2) + + ids := []int64{result[0].account.ID, result[1].account.ID} + sort.Slice(ids, func(i, j int) bool { return ids[i] < ids[j] }) + assert.Equal(t, []int64{1, 3}, ids) + }) + + t.Run("reds excluded", func(t *testing.T) { + accs := []accountWithLoad{ + makeAWL(1, 5, 3, 10), // red (10 > 8) + makeAWL(2, 5, 3, 6), // yellow (5 < 6 ≤ 8) + makeAWL(3, 5, 3, 9), // red (9 > 8) + } + result := classifyByAffinityZone(accs) + require.Len(t, result, 1) + assert.Equal(t, int64(2), result[0].account.ID) + }) + + t.Run("all red returns empty", func(t *testing.T) { + accs := []accountWithLoad{ + makeAWL(1, 5, 0, 6), // buffer=0 → red (6 > 5) + makeAWL(2, 5, 0, 10), // buffer=0 → red (10 > 5) + } + result := classifyByAffinityZone(accs) + require.Empty(t, result) + }) + + t.Run("mixed with unconfigured accounts", func(t *testing.T) { + // 账号 1: 配置了 base=5,buffer=3 → green(3≤5) + // 账号 2: 未配置 base → 视为 green + // 账号 3: 配置了 base=5,buffer=3 → red(10>8) + accs := []accountWithLoad{ + makeAWL(1, 5, 3, 3), + {account: newAffinityAccount(2, 50, true), loadInfo: &AccountLoadInfo{AccountID: 2}, affinityCount: 20}, + makeAWL(3, 5, 3, 10), + } + result := classifyByAffinityZone(accs) + require.Len(t, result, 2) + + ids := []int64{result[0].account.ID, result[1].account.ID} + sort.Slice(ids, func(i, j int) bool { return ids[i] < ids[j] }) + assert.Equal(t, []int64{1, 2}, ids) + }) + + t.Run("infinite yellow never reds", func(t *testing.T) { + // buffer 未配置 → 无限黄区,永不红区 + accs := []accountWithLoad{ + makeAWL(1, 5, nil, 100), // yellow (100 > 5, no buffer → infinite yellow) + makeAWL(2, 5, nil, 3), // green (3 ≤ 5) + } + result := classifyByAffinityZone(accs) + // green 优先 + require.Len(t, result, 1) + assert.Equal(t, int64(2), result[0].account.ID) + }) + + t.Run("only yellows when no greens", func(t *testing.T) { + accs := []accountWithLoad{ + makeAWL(1, 5, nil, 10), // yellow + makeAWL(2, 5, nil, 20), // yellow + } + result := classifyByAffinityZone(accs) + require.Len(t, result, 2) + }) +} diff --git a/backend/internal/service/gateway_hotpath_optimization_test.go b/backend/internal/service/gateway_hotpath_optimization_test.go index 161c4ba4b1..0203c3311f 100644 --- a/backend/internal/service/gateway_hotpath_optimization_test.go +++ b/backend/internal/service/gateway_hotpath_optimization_test.go @@ -143,6 +143,24 @@ func (s *stickyGatewayCacheHotpathStub) RefreshSessionTTL(ctx context.Context, g func (s *stickyGatewayCacheHotpathStub) DeleteSessionAccountID(ctx context.Context, groupID int64, sessionHash string) error { return nil } +func (s *stickyGatewayCacheHotpathStub) GetClientAffinityAccounts(_ context.Context, _ int64, _ string, _ time.Duration) ([]int64, error) { + return nil, nil +} +func (s *stickyGatewayCacheHotpathStub) UpdateClientAffinity(_ context.Context, _ int64, _ string, _ int64, _ time.Duration) error { + return nil +} +func (s *stickyGatewayCacheHotpathStub) GetAccountAffinityCountBatch(_ context.Context, _ int64, _ []int64, _ time.Duration) (map[int64]int64, error) { + return map[int64]int64{}, nil +} +func (s *stickyGatewayCacheHotpathStub) GetAccountAffinityClientsBatch(_ context.Context, _ map[int64][]int64, _ time.Duration) (map[int64][]string, error) { + return map[int64][]string{}, nil +} +func (s *stickyGatewayCacheHotpathStub) GetAccountAffinityClientsWithScores(_ context.Context, _ int64, _ []int64, _ time.Duration) ([]AffinityClient, error) { + return nil, nil +} +func (s *stickyGatewayCacheHotpathStub) ClearAccountAffinity(_ context.Context, _ int64, _ []int64) error { + return nil +} func (s *modelsListAccountRepoStub) ListSchedulableByGroupID(ctx context.Context, groupID int64) ([]Account, error) { s.listByGroupCalls.Add(1) diff --git a/backend/internal/service/gateway_multiplatform_test.go b/backend/internal/service/gateway_multiplatform_test.go index f947a8ee86..2af568e342 100644 --- a/backend/internal/service/gateway_multiplatform_test.go +++ b/backend/internal/service/gateway_multiplatform_test.go @@ -235,6 +235,25 @@ func (m *mockGatewayCacheForPlatform) DeleteSessionAccountID(ctx context.Context return nil } +func (m *mockGatewayCacheForPlatform) GetClientAffinityAccounts(_ context.Context, _ int64, _ string, _ time.Duration) ([]int64, error) { + return nil, nil +} +func (m *mockGatewayCacheForPlatform) UpdateClientAffinity(_ context.Context, _ int64, _ string, _ int64, _ time.Duration) error { + return nil +} +func (m *mockGatewayCacheForPlatform) GetAccountAffinityCountBatch(_ context.Context, _ int64, _ []int64, _ time.Duration) (map[int64]int64, error) { + return map[int64]int64{}, nil +} +func (m *mockGatewayCacheForPlatform) GetAccountAffinityClientsBatch(_ context.Context, _ map[int64][]int64, _ time.Duration) (map[int64][]string, error) { + return map[int64][]string{}, nil +} +func (m *mockGatewayCacheForPlatform) GetAccountAffinityClientsWithScores(_ context.Context, _ int64, _ []int64, _ time.Duration) ([]AffinityClient, error) { + return nil, nil +} +func (m *mockGatewayCacheForPlatform) ClearAccountAffinity(_ context.Context, _ int64, _ []int64) error { + return nil +} + type mockGroupRepoForGateway struct { groups map[int64]*Group getByIDCalls int diff --git a/backend/internal/service/gateway_service.go b/backend/internal/service/gateway_service.go index 85a8e2b7d9..9fbbc57ebe 100644 --- a/backend/internal/service/gateway_service.go +++ b/backend/internal/service/gateway_service.go @@ -40,7 +40,8 @@ import ( const ( claudeAPIURL = "https://api.anthropic.com/v1/messages?beta=true" claudeAPICountTokensURL = "https://api.anthropic.com/v1/messages/count_tokens?beta=true" - stickySessionTTL = time.Hour // 粘性会话TTL + stickySessionTTL = time.Hour // 粘性会话TTL + ClientAffinityTTL = 24 * time.Hour // 客户端亲和TTL defaultMaxLineSize = 500 * 1024 * 1024 // Canonical Claude Code banner. Keep it EXACT (no trailing whitespace/newlines) // to match real Claude CLI traffic as closely as possible. When we need a visual @@ -68,8 +69,9 @@ type forceCacheBillingKeyType struct{} // accountWithLoad 账号与负载信息的组合,用于负载感知调度 type accountWithLoad struct { - account *Account - loadInfo *AccountLoadInfo + account *Account + loadInfo *AccountLoadInfo + affinityCount int64 // 亲和客户端数量(反向索引),越少越优先 } var ForceCacheBillingContextKey = forceCacheBillingKeyType{} @@ -304,6 +306,10 @@ var ( sessionIDRegex = regexp.MustCompile(`session_([a-f0-9-]{36})`) claudeCliUserAgentRe = regexp.MustCompile(`^claude-cli/\d+\.\d+\.\d+`) + // clientIDFromMetadataRegex 从 metadata.user_id 中提取客户端 ID(64位 hex) + // 格式: user_{64位hex}_account_... + clientIDFromMetadataRegex = regexp.MustCompile(`^user_([a-f0-9]{64})_account_`) + // claudeCodePromptPrefixes 用于检测 Claude Code 系统提示词的前缀列表 // 支持多种变体:标准版、Agent SDK 版、Explore Agent 版、Compact 版等 // 注意:前缀之间不应存在包含关系,否则会导致冗余匹配 @@ -364,6 +370,34 @@ type GatewayCache interface { // DeleteSessionAccountID 删除粘性会话绑定,用于账号不可用时主动清理 // Delete sticky session binding, used to proactively clean up when account becomes unavailable DeleteSessionAccountID(ctx context.Context, groupID int64, sessionHash string) error + + // GetClientAffinityAccounts 获取客户端亲和账号列表(按最近使用降序),同时清理过期成员 + GetClientAffinityAccounts(ctx context.Context, groupID int64, clientID string, ttl time.Duration) ([]int64, error) + // UpdateClientAffinity 添加/更新客户端亲和关系(更新 score 为当前时间戳,刷新 key TTL) + UpdateClientAffinity(ctx context.Context, groupID int64, clientID string, accountID int64, ttl time.Duration) error + // GetAccountAffinityCountBatch 批量获取账号的亲和客户端数量(惰性清理过期成员) + GetAccountAffinityCountBatch(ctx context.Context, groupID int64, accountIDs []int64, ttl time.Duration) (map[int64]int64, error) + // GetAccountAffinityClientsBatch 批量获取每个账号跨所有分组的亲和客户端列表(去重) + // accountGroups: map[accountID][]groupID + GetAccountAffinityClientsBatch(ctx context.Context, accountGroups map[int64][]int64, ttl time.Duration) (map[int64][]string, error) + // GetAccountAffinityClientsWithScores 获取单个账号跨所有分组的亲和客户端列表(含最后活跃时间) + GetAccountAffinityClientsWithScores(ctx context.Context, accountID int64, groupIDs []int64, ttl time.Duration) ([]AffinityClient, error) + // ClearAccountAffinity 清除指定账号在所有分组的亲和记录(正向+反向索引) + // 用于账号关闭客户端亲和时立即清理旧绑定 + ClearAccountAffinity(ctx context.Context, accountID int64, groupIDs []int64) error +} + +// AffinityClient 亲和客户端信息(含最后活跃时间) +type AffinityClient struct { + ClientID string `json:"client_id"` + LastActive time.Time `json:"last_active"` +} + +// SortAffinityClients 按最后活跃时间降序排序 +func SortAffinityClients(clients []AffinityClient) { + sort.Slice(clients, func(i, j int) bool { + return clients[i].LastActive.After(clients[j].LastActive) + }) } // derefGroupID safely dereferences *int64 to int64, returning 0 if nil @@ -434,6 +468,20 @@ func shouldClearStickySession(account *Account, requestedModel string) bool { return false } +// extractClientIDFromMetadata 从 metadata.user_id 中提取客户端 ID(64位 hex)。 +// 格式: user_{64位hex}_account_..._session_... +// 返回空字符串表示无法提取(非 Claude Code/Console 客户端)。 +func extractClientIDFromMetadata(metadataUserID string) string { + if metadataUserID == "" { + return "" + } + matches := clientIDFromMetadataRegex.FindStringSubmatch(metadataUserID) + if matches == nil { + return "" + } + return matches[1] +} + type AccountWaitPlan struct { AccountID int64 MaxConcurrency int @@ -1062,7 +1110,7 @@ func (s *GatewayService) SelectAccountForModelWithExclusions(ctx context.Context } // SelectAccountWithLoadAwareness selects account with load-awareness and wait plan. -// metadataUserID: 已废弃参数,会话限制现在统一使用 sessionHash +// metadataUserID: 用于客户端亲和调度,从中提取客户端 ID func (s *GatewayService) SelectAccountWithLoadAwareness(ctx context.Context, groupID *int64, sessionHash string, requestedModel string, excludedIDs map[int64]struct{}, metadataUserID string) (*AccountSelectionResult, error) { // 调试日志:记录调度入口参数 excludedIDsList := make([]int64, 0, len(excludedIDs)) @@ -1093,6 +1141,9 @@ func (s *GatewayService) SelectAccountWithLoadAwareness(ctx context.Context, gro } } + // 提取客户端 ID(用于客户端亲和调度) + affinityClientID := extractClientIDFromMetadata(metadataUserID) + if s.debugModelRoutingEnabled() && requestedModel != "" { groupPlatform := "" if group != nil { @@ -1356,7 +1407,10 @@ func (s *GatewayService) SelectAccountWithLoadAwareness(ctx context.Context, gro } if len(routingAvailable) > 0 { - // 排序:优先级 > 负载率 > 最后使用时间 + // 批量获取亲和客户端数量 + s.populateAffinityCounts(ctx, routingAvailable, derefGroupID(groupID)) + + // 排序:优先级 > 负载率 > 亲和客户端数 > 最后使用时间 sort.SliceStable(routingAvailable, func(i, j int) bool { a, b := routingAvailable[i], routingAvailable[j] if a.account.Priority != b.account.Priority { @@ -1365,6 +1419,9 @@ func (s *GatewayService) SelectAccountWithLoadAwareness(ctx context.Context, gro if a.loadInfo.LoadRate != b.loadInfo.LoadRate { return a.loadInfo.LoadRate < b.loadInfo.LoadRate } + if a.affinityCount != b.affinityCount { + return a.affinityCount < b.affinityCount + } switch { case a.account.LastUsedAt == nil && b.account.LastUsedAt != nil: return true @@ -1390,6 +1447,9 @@ func (s *GatewayService) SelectAccountWithLoadAwareness(ctx context.Context, gro if sessionHash != "" && s.cache != nil { _ = s.cache.SetSessionAccountID(ctx, derefGroupID(groupID), sessionHash, item.account.ID, stickySessionTTL) } + if affinityClientID != "" && s.cache != nil && item.account.IsClientAffinityEnabled() { + _ = s.cache.UpdateClientAffinity(ctx, derefGroupID(groupID), affinityClientID, item.account.ID, ClientAffinityTTL) + } if s.debugModelRoutingEnabled() { logger.LegacyPrintf("service.gateway", "[ModelRoutingDebug] routed select: group_id=%v model=%s session=%s account=%d", derefGroupID(groupID), requestedModel, shortSessionHash(sessionHash), item.account.ID) } @@ -1486,6 +1546,76 @@ func (s *GatewayService) SelectAccountWithLoadAwareness(ctx context.Context, gro } } + // ============ Layer 1.6: 客户端亲和(仅在粘性会话未命中时生效) ============ + if affinityClientID != "" && s.cache != nil && stickyAccountID <= 0 { + affinityAccountIDs, err := s.cache.GetClientAffinityAccounts(ctx, derefGroupID(groupID), affinityClientID, ClientAffinityTTL) + if err == nil && len(affinityAccountIDs) > 0 { + for _, affinityAccID := range affinityAccountIDs { + if isExcluded(affinityAccID) { + continue + } + account, ok := accountByID[affinityAccID] + if !ok || !s.isAccountSchedulableForSelection(account) { + continue + } + if !account.IsClientAffinityEnabled() { + continue + } + if !s.isAccountAllowedForPlatform(account, platform, useMixed) { + continue + } + if requestedModel != "" && !s.isModelSupportedByAccountWithContext(ctx, account, requestedModel) { + continue + } + if !s.isAccountSchedulableForModelSelection(ctx, account, requestedModel) { + continue + } + if !s.isAccountSchedulableForQuota(account) { + continue + } + if !s.isAccountSchedulableForWindowCost(ctx, account, false) { + continue + } + if !s.isAccountSchedulableForRPM(ctx, account, false) { + continue + } + // 亲和三区检查:红区账号不可通过亲和命中调度 + if account.GetAffinityBase() > 0 && s.cache != nil { + countMap, err := s.cache.GetAccountAffinityCountBatch(ctx, derefGroupID(groupID), []int64{affinityAccID}, ClientAffinityTTL) + if err == nil { + zone := account.GetAffinityZone(countMap[affinityAccID]) + if zone == AffinityZoneRed { + continue + } + } + } + + result, err := s.tryAcquireAccountSlot(ctx, affinityAccID, account.Concurrency) + if err == nil && result.Acquired { + if !s.checkAndRegisterSession(ctx, account, sessionHash) { + result.ReleaseFunc() + continue + } + // 亲和命中:更新亲和 score + 绑定粘性会话 + _ = s.cache.UpdateClientAffinity(ctx, derefGroupID(groupID), affinityClientID, affinityAccID, ClientAffinityTTL) + if sessionHash != "" { + _ = s.cache.SetSessionAccountID(ctx, derefGroupID(groupID), sessionHash, affinityAccID, stickySessionTTL) + } + slog.Debug("client_affinity_hit", + "group_id", derefGroupID(groupID), + "client_id", affinityClientID[:8]+"...", + "account_id", affinityAccID) + return &AccountSelectionResult{ + Account: account, + Acquired: true, + ReleaseFunc: result.ReleaseFunc, + }, nil + } + } + // 所有亲和账号不可用,继续到 Layer 2 + } + } + // ============ Layer 2: 负载感知选择 ============ candidates := make([]*Account, 0, len(accounts)) for i := range accounts { @@ -1538,6 +1668,9 @@ func (s *GatewayService) SelectAccountWithLoadAwareness(ctx context.Context, gro loadMap, err := s.concurrencyService.GetAccountsLoadBatch(ctx, accountLoads) if err != nil { if result, ok := s.tryAcquireByLegacyOrder(ctx, candidates, groupID, sessionHash, preferOAuth); ok { + if affinityClientID != "" && s.cache != nil && result.Account != nil && result.Account.IsClientAffinityEnabled() { + _ = s.cache.UpdateClientAffinity(ctx, derefGroupID(groupID), affinityClientID, result.Account.ID, ClientAffinityTTL) + } return result, nil } } else { @@ -1555,13 +1688,37 @@ func (s *GatewayService) SelectAccountWithLoadAwareness(ctx context.Context, gro } } - // 分层过滤选择:优先级 → 负载率 → LRU + // 批量获取亲和客户端数量(用于均衡分配新客户端) + s.populateAffinityCounts(ctx, available, derefGroupID(groupID)) + + // 分层过滤选择:优先级 → 亲和三区 → 负载率 → 亲和客户端数 → LRU for len(available) > 0 { // 1. 取优先级最小的集合 candidates := filterByMinPriority(available) - // 2. 取负载率最低的集合 + // 2. 按亲和三区过滤:绿区优先 → 黄区降级 → 红区移除(在同优先级内) + candidates = classifyByAffinityZone(candidates) + if len(candidates) == 0 { + // 当前优先级组全部在红区,移除后回退到下一优先级组 + minPri := available[0].account.Priority + for _, a := range available[1:] { + if a.account.Priority < minPri { + minPri = a.account.Priority + } + } + newAvailable := make([]accountWithLoad, 0, len(available)) + for _, a := range available { + if a.account.Priority != minPri { + newAvailable = append(newAvailable, a) + } + } + available = newAvailable + continue + } + // 3. 取负载率最低的集合 candidates = filterByMinLoadRate(candidates) - // 3. LRU 选择最久未用的账号 + // 3. 取亲和客户端数最少的集合 + candidates = filterByMinAffinityCount(candidates) + // 4. LRU 选择最久未用的账号 selected := selectByLRU(candidates, preferOAuth) if selected == nil { break @@ -1576,6 +1733,10 @@ func (s *GatewayService) SelectAccountWithLoadAwareness(ctx context.Context, gro if sessionHash != "" && s.cache != nil { _ = s.cache.SetSessionAccountID(ctx, derefGroupID(groupID), sessionHash, selected.account.ID, stickySessionTTL) } + // 更新客户端亲和关系 + if affinityClientID != "" && s.cache != nil && selected.account.IsClientAffinityEnabled() { + _ = s.cache.UpdateClientAffinity(ctx, derefGroupID(groupID), affinityClientID, selected.account.ID, ClientAffinityTTL) + } return &AccountSelectionResult{ Account: selected.account, Acquired: true, @@ -2333,6 +2494,36 @@ func (s *GatewayService) getSchedulableAccount(ctx context.Context, accountID in return s.accountRepo.GetByID(ctx, accountID) } +// populateAffinityCounts 批量获取账号的亲和客户端数量并填入 accountWithLoad 切片。 +// 仅当存在开启了客户端亲和的账号时才查询 Redis,否则跳过。 +func (s *GatewayService) populateAffinityCounts(ctx context.Context, accounts []accountWithLoad, groupID int64) { + if s.cache == nil || len(accounts) == 0 { + return + } + // 快速检查:是否有任何账号开启了亲和 + hasAffinity := false + for _, acc := range accounts { + if acc.account.IsClientAffinityEnabled() { + hasAffinity = true + break + } + } + if !hasAffinity { + return + } + accountIDs := make([]int64, len(accounts)) + for i, acc := range accounts { + accountIDs[i] = acc.account.ID + } + countMap, err := s.cache.GetAccountAffinityCountBatch(ctx, groupID, accountIDs, ClientAffinityTTL) + if err != nil { + return // 查询失败不影响调度,affinityCount 保持 0 + } + for i := range accounts { + accounts[i].affinityCount = countMap[accounts[i].account.ID] + } +} + // filterByMinPriority 过滤出优先级最小的账号集合 func filterByMinPriority(accounts []accountWithLoad) []accountWithLoad { if len(accounts) == 0 { @@ -2373,6 +2564,64 @@ func filterByMinLoadRate(accounts []accountWithLoad) []accountWithLoad { return result } +// filterByMinAffinityCount 过滤出亲和客户端数最少的账号集合 +func filterByMinAffinityCount(accounts []accountWithLoad) []accountWithLoad { + if len(accounts) == 0 { + return accounts + } + minCount := accounts[0].affinityCount + for _, acc := range accounts[1:] { + if acc.affinityCount < minCount { + minCount = acc.affinityCount + } + } + result := make([]accountWithLoad, 0, len(accounts)) + for _, acc := range accounts { + if acc.affinityCount == minCount { + result = append(result, acc) + } + } + return result +} + +// classifyByAffinityZone 按亲和分区对候选账号进行分类。 +// 返回值:仅绿区账号(有绿区时),否则返回黄区账号。红区账号被移除。 +// 如果没有任何账号开启了亲和三区配置(即 affinity_base <= 0),则原样返回所有账号。 +func classifyByAffinityZone(accounts []accountWithLoad) []accountWithLoad { + if len(accounts) == 0 { + return accounts + } + // 快速检查:是否有任何账号配置了 affinity_base + hasZoneConfig := false + for _, acc := range accounts { + if acc.account.IsClientAffinityEnabled() && acc.account.GetAffinityBase() > 0 { + hasZoneConfig = true + break + } + } + if !hasZoneConfig { + return accounts + } + + greens := make([]accountWithLoad, 0, len(accounts)) + yellows := make([]accountWithLoad, 0, len(accounts)) + for _, acc := range accounts { + zone := acc.account.GetAffinityZone(acc.affinityCount) + switch zone { + case AffinityZoneGreen: + greens = append(greens, acc) + case AffinityZoneYellow: + yellows = append(yellows, acc) + case AffinityZoneRed: + // 红区:移除,不参与调度 + } + } + if len(greens) > 0 { + return greens + } + return yellows +} + // selectByLRU 从集合中选择最久未用的账号 // 如果有多个账号具有相同的最小 LastUsedAt,则随机选择一个 func selectByLRU(accounts []accountWithLoad, preferOAuth bool) *accountWithLoad { @@ -3342,6 +3591,10 @@ func (s *GatewayService) isModelSupportedByAccount(account *Account, requestedMo if account.Platform == PlatformSora { return s.isSoraModelSupportedByAccount(account, requestedModel) } + // OpenAI 透传模式:仅替换认证,允许所有模型 + if account.Platform == PlatformOpenAI && account.IsOpenAIPassthroughEnabled() { + return true + } // OAuth/SetupToken 账号使用 Anthropic 标准映射(短ID → 长ID) if account.Platform == PlatformAnthropic && account.Type != AccountTypeAPIKey { requestedModel = claude.NormalizeModelID(requestedModel) diff --git a/backend/internal/service/gateway_service_affinity_test.go b/backend/internal/service/gateway_service_affinity_test.go new file mode 100644 index 0000000000..d3dc6f4bb0 --- /dev/null +++ b/backend/internal/service/gateway_service_affinity_test.go @@ -0,0 +1,122 @@ +//go:build unit + +// Package service 提供 API 网关核心服务。 +// 本文件包含 SortAffinityClients 函数的单元测试, +// 验证 AffinityClient 切片排序逻辑在各种输入条件下的正确行为。 +// +// This file contains unit tests for the SortAffinityClients function, +// verifying correct sorting behavior for AffinityClient slices under +// various input conditions including empty, single, sorted, reverse, +// and duplicate-timestamp scenarios. +package service + +import ( + "testing" + "time" + + "github.com/stretchr/testify/require" +) + +func TestSortAffinityClients_Empty(t *testing.T) { + var clients []AffinityClient + SortAffinityClients(clients) + require.Empty(t, clients) + + clients = []AffinityClient{} + SortAffinityClients(clients) + require.Empty(t, clients) +} + +func TestSortAffinityClients_SingleElement(t *testing.T) { + now := time.Now() + clients := []AffinityClient{ + {ClientID: "client-1", LastActive: now}, + } + SortAffinityClients(clients) + require.Len(t, clients, 1) + require.Equal(t, "client-1", clients[0].ClientID) + require.Equal(t, now, clients[0].LastActive) +} + +func TestSortAffinityClients_AlreadySorted(t *testing.T) { + now := time.Now() + clients := []AffinityClient{ + {ClientID: "newest", LastActive: now}, + {ClientID: "middle", LastActive: now.Add(-1 * time.Hour)}, + {ClientID: "oldest", LastActive: now.Add(-2 * time.Hour)}, + } + SortAffinityClients(clients) + + require.Equal(t, "newest", clients[0].ClientID) + require.Equal(t, "middle", clients[1].ClientID) + require.Equal(t, "oldest", clients[2].ClientID) +} + +func TestSortAffinityClients_ReverseOrder(t *testing.T) { + now := time.Now() + clients := []AffinityClient{ + {ClientID: "oldest", LastActive: now.Add(-2 * time.Hour)}, + {ClientID: "middle", LastActive: now.Add(-1 * time.Hour)}, + {ClientID: "newest", LastActive: now}, + } + SortAffinityClients(clients) + + require.Equal(t, "newest", clients[0].ClientID) + require.Equal(t, "middle", clients[1].ClientID) + require.Equal(t, "oldest", clients[2].ClientID) +} + +func TestSortAffinityClients_SameTimestamps(t *testing.T) { + now := time.Now() + clients := []AffinityClient{ + {ClientID: "c1", LastActive: now}, + {ClientID: "c2", LastActive: now}, + {ClientID: "c3", LastActive: now}, + } + SortAffinityClients(clients) + + // 所有时间戳相同时,排序结果应保持稳定(sort.Slice 不保证稳定性, + // 但只要结果是某种确定的顺序即可)。 + // 验证所有元素仍然存在且时间相同。 + require.Len(t, clients, 3) + ids := map[string]bool{} + for _, c := range clients { + ids[c.ClientID] = true + require.Equal(t, now, c.LastActive) + } + require.True(t, ids["c1"]) + require.True(t, ids["c2"]) + require.True(t, ids["c3"]) +} + +func TestSortAffinityClients_MixedOrder(t *testing.T) { + now := time.Now() + clients := []AffinityClient{ + {ClientID: "c3", LastActive: now.Add(-30 * time.Minute)}, + {ClientID: "c1", LastActive: now}, + {ClientID: "c5", LastActive: now.Add(-2 * time.Hour)}, + {ClientID: "c2", LastActive: now.Add(-10 * time.Minute)}, + {ClientID: "c4", LastActive: now.Add(-1 * time.Hour)}, + } + SortAffinityClients(clients) + + // 按 LastActive 降序排列 + require.Equal(t, "c1", clients[0].ClientID) // now + require.Equal(t, "c2", clients[1].ClientID) // -10m + require.Equal(t, "c3", clients[2].ClientID) // -30m + require.Equal(t, "c4", clients[3].ClientID) // -1h + require.Equal(t, "c5", clients[4].ClientID) // -2h +} + +func TestSortAffinityClients_SubSecondDifferences(t *testing.T) { + base := time.Now() + clients := []AffinityClient{ + {ClientID: "early", LastActive: base}, + {ClientID: "late", LastActive: base.Add(500 * time.Millisecond)}, + } + SortAffinityClients(clients) + + // 500ms 差异也应正确排序(更晚的在前) + require.Equal(t, "late", clients[0].ClientID) + require.Equal(t, "early", clients[1].ClientID) +} diff --git a/backend/internal/service/gemini_multiplatform_test.go b/backend/internal/service/gemini_multiplatform_test.go index b0b804eb00..7299261429 100644 --- a/backend/internal/service/gemini_multiplatform_test.go +++ b/backend/internal/service/gemini_multiplatform_test.go @@ -288,6 +288,25 @@ func (m *mockGatewayCacheForGemini) DeleteSessionAccountID(ctx context.Context, return nil } +func (m *mockGatewayCacheForGemini) GetClientAffinityAccounts(_ context.Context, _ int64, _ string, _ time.Duration) ([]int64, error) { + return nil, nil +} +func (m *mockGatewayCacheForGemini) UpdateClientAffinity(_ context.Context, _ int64, _ string, _ int64, _ time.Duration) error { + return nil +} +func (m *mockGatewayCacheForGemini) GetAccountAffinityCountBatch(_ context.Context, _ int64, _ []int64, _ time.Duration) (map[int64]int64, error) { + return map[int64]int64{}, nil +} +func (m *mockGatewayCacheForGemini) GetAccountAffinityClientsBatch(_ context.Context, _ map[int64][]int64, _ time.Duration) (map[int64][]string, error) { + return map[int64][]string{}, nil +} +func (m *mockGatewayCacheForGemini) GetAccountAffinityClientsWithScores(_ context.Context, _ int64, _ []int64, _ time.Duration) ([]AffinityClient, error) { + return nil, nil +} +func (m *mockGatewayCacheForGemini) ClearAccountAffinity(_ context.Context, _ int64, _ []int64) error { + return nil +} + // TestGeminiMessagesCompatService_SelectAccountForModelWithExclusions_GeminiPlatform 测试 Gemini 单平台选择 func TestGeminiMessagesCompatService_SelectAccountForModelWithExclusions_GeminiPlatform(t *testing.T) { ctx := context.Background() diff --git a/backend/internal/service/openai_account_scheduler.go b/backend/internal/service/openai_account_scheduler.go index 0fcf450ba4..b68c6216c2 100644 --- a/backend/internal/service/openai_account_scheduler.go +++ b/backend/internal/service/openai_account_scheduler.go @@ -323,7 +323,7 @@ func (s *defaultOpenAIAccountScheduler) selectBySessionHash( _ = s.service.deleteStickySessionAccountID(ctx, req.GroupID, sessionHash) return nil, nil } - if req.RequestedModel != "" && !account.IsModelSupported(req.RequestedModel) { + if req.RequestedModel != "" && !account.IsOpenAIPassthroughEnabled() && !account.IsModelSupported(req.RequestedModel) { return nil, nil } if !s.isAccountTransportCompatible(account, req.RequiredTransport) { @@ -582,7 +582,7 @@ func (s *defaultOpenAIAccountScheduler) selectByLoadBalance( if !account.IsSchedulable() || !account.IsOpenAI() { continue } - if req.RequestedModel != "" && !account.IsModelSupported(req.RequestedModel) { + if req.RequestedModel != "" && !account.IsOpenAIPassthroughEnabled() && !account.IsModelSupported(req.RequestedModel) { continue } if !s.isAccountTransportCompatible(account, req.RequiredTransport) { diff --git a/backend/internal/service/openai_gateway_service.go b/backend/internal/service/openai_gateway_service.go index 023e4ed426..b38057438e 100644 --- a/backend/internal/service/openai_gateway_service.go +++ b/backend/internal/service/openai_gateway_service.go @@ -1363,7 +1363,7 @@ func (s *OpenAIGatewayService) SelectAccountWithLoadAwareness(ctx context.Contex if !acc.IsSchedulable() { continue } - if requestedModel != "" && !acc.IsModelSupported(requestedModel) { + if requestedModel != "" && !acc.IsOpenAIPassthroughEnabled() && !acc.IsModelSupported(requestedModel) { continue } candidates = append(candidates, acc) @@ -1524,7 +1524,7 @@ func (s *OpenAIGatewayService) resolveFreshSchedulableOpenAIAccount(ctx context. if !fresh.IsSchedulable() || !fresh.IsOpenAI() { return nil } - if requestedModel != "" && !fresh.IsModelSupported(requestedModel) { + if requestedModel != "" && !fresh.IsOpenAIPassthroughEnabled() && !fresh.IsModelSupported(requestedModel) { return nil } return fresh diff --git a/backend/internal/service/openai_gateway_service_test.go b/backend/internal/service/openai_gateway_service_test.go index 43e2f39d14..ffa37d63fd 100644 --- a/backend/internal/service/openai_gateway_service_test.go +++ b/backend/internal/service/openai_gateway_service_test.go @@ -282,6 +282,25 @@ func (c *stubGatewayCache) DeleteSessionAccountID(ctx context.Context, groupID i return nil } +func (c *stubGatewayCache) GetClientAffinityAccounts(_ context.Context, _ int64, _ string, _ time.Duration) ([]int64, error) { + return nil, nil +} +func (c *stubGatewayCache) UpdateClientAffinity(_ context.Context, _ int64, _ string, _ int64, _ time.Duration) error { + return nil +} +func (c *stubGatewayCache) GetAccountAffinityCountBatch(_ context.Context, _ int64, _ []int64, _ time.Duration) (map[int64]int64, error) { + return map[int64]int64{}, nil +} +func (c *stubGatewayCache) GetAccountAffinityClientsBatch(_ context.Context, _ map[int64][]int64, _ time.Duration) (map[int64][]string, error) { + return map[int64][]string{}, nil +} +func (c *stubGatewayCache) GetAccountAffinityClientsWithScores(_ context.Context, _ int64, _ []int64, _ time.Duration) ([]AffinityClient, error) { + return nil, nil +} +func (c *stubGatewayCache) ClearAccountAffinity(_ context.Context, _ int64, _ []int64) error { + return nil +} + func TestOpenAISelectAccountWithLoadAwareness_FiltersUnschedulable(t *testing.T) { now := time.Now() resetAt := now.Add(10 * time.Minute) diff --git a/backend/internal/service/openai_ws_state_store_test.go b/backend/internal/service/openai_ws_state_store_test.go index 235d42331d..40cce0dd47 100644 --- a/backend/internal/service/openai_ws_state_store_test.go +++ b/backend/internal/service/openai_ws_state_store_test.go @@ -193,6 +193,25 @@ func (c *openAIWSStateStoreTimeoutProbeCache) DeleteSessionAccountID(ctx context return nil } +func (c *openAIWSStateStoreTimeoutProbeCache) GetClientAffinityAccounts(_ context.Context, _ int64, _ string, _ time.Duration) ([]int64, error) { + return nil, nil +} +func (c *openAIWSStateStoreTimeoutProbeCache) UpdateClientAffinity(_ context.Context, _ int64, _ string, _ int64, _ time.Duration) error { + return nil +} +func (c *openAIWSStateStoreTimeoutProbeCache) GetAccountAffinityCountBatch(_ context.Context, _ int64, _ []int64, _ time.Duration) (map[int64]int64, error) { + return map[int64]int64{}, nil +} +func (c *openAIWSStateStoreTimeoutProbeCache) GetAccountAffinityClientsBatch(_ context.Context, _ map[int64][]int64, _ time.Duration) (map[int64][]string, error) { + return map[int64][]string{}, nil +} +func (c *openAIWSStateStoreTimeoutProbeCache) GetAccountAffinityClientsWithScores(_ context.Context, _ int64, _ []int64, _ time.Duration) ([]AffinityClient, error) { + return nil, nil +} +func (c *openAIWSStateStoreTimeoutProbeCache) ClearAccountAffinity(_ context.Context, _ int64, _ []int64) error { + return nil +} + func TestOpenAIWSStateStore_RedisOpsUseShortTimeout(t *testing.T) { probe := &openAIWSStateStoreTimeoutProbeCache{} store := NewOpenAIWSStateStore(probe) diff --git a/backend/internal/service/ops_system_log_sink_test.go b/backend/internal/service/ops_system_log_sink_test.go index 12a2ec0c7d..137ee33c72 100644 --- a/backend/internal/service/ops_system_log_sink_test.go +++ b/backend/internal/service/ops_system_log_sink_test.go @@ -183,6 +183,15 @@ func TestOpsSystemLogSink_StartStopAndFlushSuccess(t *testing.T) { if strings.TrimSpace(item.Message) == "" { t.Fatalf("message should not be empty") } + // writtenCount is incremented after BatchInsertSystemLogsFn returns, + // so poll briefly to avoid a race between the done signal and the atomic add. + deadline := time.Now().Add(time.Second) + for time.Now().Before(deadline) { + if sink.Health().WrittenCount > 0 { + break + } + time.Sleep(time.Millisecond) + } health := sink.Health() if health.WrittenCount == 0 { t.Fatalf("written_count should be >0") diff --git a/backend/internal/service/ratelimit_service.go b/backend/internal/service/ratelimit_service.go index 5df2d639ea..4345e66c2c 100644 --- a/backend/internal/service/ratelimit_service.go +++ b/backend/internal/service/ratelimit_service.go @@ -149,8 +149,9 @@ func (s *RateLimitService) HandleUpstreamError(ctx context.Context, account *Acc } // 其他 400 错误(如参数问题)不处理,不禁用账号 case 401: - // 对所有 OAuth 账号在 401 错误时调用缓存失效并强制下次刷新 - if account.Type == AccountTypeOAuth { + // OpenAI OAuth 账号在 401 错误时临时不可调度(保持 active 让刷新服务恢复); + // 其他平台 OAuth 账号保持原有 SetError 行为(Antigravity 主流程不走此路径)。 + if account.Type == AccountTypeOAuth && account.Platform == PlatformOpenAI { // 1. 失效缓存 if s.tokenCacheInvalidator != nil { if err := s.tokenCacheInvalidator.InvalidateToken(ctx, account); err != nil { @@ -199,11 +200,6 @@ func (s *RateLimitService) HandleUpstreamError(ctx context.Context, account *Acc s.handleAuthError(ctx, account, msg) shouldDisable = true case 403: - // 禁止访问:停止调度,记录错误 - msg := "Access forbidden (403): account may be suspended or lack permissions" - if upstreamMsg != "" { - msg = "Access forbidden (403): " + upstreamMsg - } logger.LegacyPrintf( "service.ratelimit", "[HandleUpstreamErrorRaw] account_id=%d platform=%s type=%s status=403 request_id=%s cf_ray=%s upstream_msg=%s raw_body=%s", @@ -215,8 +211,7 @@ func (s *RateLimitService) HandleUpstreamError(ctx context.Context, account *Acc upstreamMsg, truncateForLog(responseBody, 1024), ) - s.handleAuthError(ctx, account, msg) - shouldDisable = true + shouldDisable = s.handle403(ctx, account, upstreamMsg, responseBody) case 429: s.handle429(ctx, account, headers, responseBody) shouldDisable = false @@ -621,6 +616,62 @@ func (s *RateLimitService) handleAuthError(ctx context.Context, account *Account slog.Warn("account_disabled_auth_error", "account_id", account.ID, "error", errorMsg) } +// handle403 处理 403 Forbidden 错误 +// Antigravity 平台区分 validation/violation/generic 三种类型,均 SetError 永久禁用; +// 其他平台保持原有 SetError 行为。 +func (s *RateLimitService) handle403(ctx context.Context, account *Account, upstreamMsg string, responseBody []byte) (shouldDisable bool) { + if account.Platform == PlatformAntigravity { + return s.handleAntigravity403(ctx, account, upstreamMsg, responseBody) + } + // 非 Antigravity 平台:保持原有行为 + msg := "Access forbidden (403): account may be suspended or lack permissions" + if upstreamMsg != "" { + msg = "Access forbidden (403): " + upstreamMsg + } + s.handleAuthError(ctx, account, msg) + return true +} + +// handleAntigravity403 处理 Antigravity 平台的 403 错误 +// validation(需要验证)→ 永久 SetError(需人工去 Google 验证后恢复) +// violation(违规封号)→ 永久 SetError(需人工处理) +// generic(通用禁止)→ 永久 SetError +func (s *RateLimitService) handleAntigravity403(ctx context.Context, account *Account, upstreamMsg string, responseBody []byte) (shouldDisable bool) { + fbType := classifyForbiddenType(string(responseBody)) + + switch fbType { + case forbiddenTypeValidation: + // VALIDATION_REQUIRED: 永久禁用,需人工去 Google 验证后手动恢复 + msg := "Validation required (403): account needs Google verification" + if upstreamMsg != "" { + msg = "Validation required (403): " + upstreamMsg + } + if validationURL := extractValidationURL(string(responseBody)); validationURL != "" { + msg += " | validation_url: " + validationURL + } + s.handleAuthError(ctx, account, msg) + return true + + case forbiddenTypeViolation: + // 违规封号: 永久禁用,需人工处理 + msg := "Account violation (403): terms of service violation" + if upstreamMsg != "" { + msg = "Account violation (403): " + upstreamMsg + } + s.handleAuthError(ctx, account, msg) + return true + + default: + // 通用 403: 保持原有行为 + msg := "Access forbidden (403): account may be suspended or lack permissions" + if upstreamMsg != "" { + msg = "Access forbidden (403): " + upstreamMsg + } + s.handleAuthError(ctx, account, msg) + return true + } +} + // handleCustomErrorCode 处理自定义错误码,停止账号调度 func (s *RateLimitService) handleCustomErrorCode(ctx context.Context, account *Account, statusCode int, errorMsg string) { msg := "Custom error code " + strconv.Itoa(statusCode) + ": " + errorMsg @@ -1213,7 +1264,8 @@ func (s *RateLimitService) tryTempUnschedulable(ctx context.Context, account *Ac } // 401 首次命中可临时不可调度(给 token 刷新窗口); // 若历史上已因 401 进入过临时不可调度,则本次应升级为 error(返回 false 交由默认错误逻辑处理)。 - if statusCode == http.StatusUnauthorized { + // Antigravity 跳过:其 401 由 applyErrorPolicy 的 temp_unschedulable_rules 自行控制,无需升级逻辑。 + if statusCode == http.StatusUnauthorized && account.Platform != PlatformAntigravity { reason := account.TempUnschedulableReason // 缓存可能没有 reason,从 DB 回退读取 if reason == "" { diff --git a/backend/internal/service/ratelimit_service_401_db_fallback_test.go b/backend/internal/service/ratelimit_service_401_db_fallback_test.go index e1611425f5..ad2c9c81e7 100644 --- a/backend/internal/service/ratelimit_service_401_db_fallback_test.go +++ b/backend/internal/service/ratelimit_service_401_db_fallback_test.go @@ -28,6 +28,7 @@ func (r *dbFallbackRepoStub) GetByID(ctx context.Context, id int64) (*Account, e func TestCheckErrorPolicy_401_DBFallback_Escalates(t *testing.T) { // Scenario: cache account has empty TempUnschedulableReason (cache miss), // but DB account has a previous 401 record → should escalate to ErrorPolicyNone. + // 注意:Antigravity 跳过 401 升级,此测试使用 Gemini 平台验证升级逻辑。 repo := &dbFallbackRepoStub{ dbAccount: &Account{ ID: 20, @@ -39,7 +40,7 @@ func TestCheckErrorPolicy_401_DBFallback_Escalates(t *testing.T) { account := &Account{ ID: 20, Type: AccountTypeOAuth, - Platform: PlatformAntigravity, + Platform: PlatformGemini, TempUnschedulableReason: "", // cache miss — reason is empty Credentials: map[string]any{ "temp_unschedulable_enabled": true, diff --git a/backend/internal/service/ratelimit_service_401_test.go b/backend/internal/service/ratelimit_service_401_test.go index 7bced46f4f..653e2fba4c 100644 --- a/backend/internal/service/ratelimit_service_401_test.go +++ b/backend/internal/service/ratelimit_service_401_test.go @@ -41,13 +41,37 @@ func (r *tokenCacheInvalidatorRecorder) InvalidateToken(ctx context.Context, acc return r.err } -func TestRateLimitService_HandleUpstreamError_OAuth401SetsTempUnschedulable(t *testing.T) { +// TestRateLimitService_HandleUpstreamError_OpenAIOAuth401SetsTempUnschedulable +// 仅 OpenAI OAuth 账号 401 使用 temp_unschedulable +func TestRateLimitService_HandleUpstreamError_OpenAIOAuth401SetsTempUnschedulable(t *testing.T) { + repo := &rateLimitAccountRepoStub{} + invalidator := &tokenCacheInvalidatorRecorder{} + service := NewRateLimitService(repo, nil, &config.Config{}, nil, nil) + service.SetTokenCacheInvalidator(invalidator) + account := &Account{ + ID: 100, + Platform: PlatformOpenAI, + Type: AccountTypeOAuth, + } + + shouldDisable := service.HandleUpstreamError(context.Background(), account, 401, http.Header{}, []byte("unauthorized")) + + require.True(t, shouldDisable) + require.Equal(t, 0, repo.setErrorCalls) + require.Equal(t, 1, repo.tempCalls) + require.Len(t, invalidator.accounts, 1) +} + +// TestRateLimitService_HandleUpstreamError_NonOpenAIOAuth401SetsError +// 非 OpenAI 的 OAuth 账号 401 走原有 SetError 行为 +func TestRateLimitService_HandleUpstreamError_NonOpenAIOAuth401SetsError(t *testing.T) { tests := []struct { name string platform string }{ {name: "gemini", platform: PlatformGemini}, {name: "antigravity", platform: PlatformAntigravity}, + {name: "anthropic", platform: PlatformAnthropic}, } for _, tt := range tests { @@ -60,29 +84,20 @@ func TestRateLimitService_HandleUpstreamError_OAuth401SetsTempUnschedulable(t *t ID: 100, Platform: tt.platform, Type: AccountTypeOAuth, - Credentials: map[string]any{ - "temp_unschedulable_enabled": true, - "temp_unschedulable_rules": []any{ - map[string]any{ - "error_code": 401, - "keywords": []any{"unauthorized"}, - "duration_minutes": 30, - "description": "custom rule", - }, - }, - }, } shouldDisable := service.HandleUpstreamError(context.Background(), account, 401, http.Header{}, []byte("unauthorized")) require.True(t, shouldDisable) - require.Equal(t, 0, repo.setErrorCalls) - require.Equal(t, 1, repo.tempCalls) - require.Len(t, invalidator.accounts, 1) + require.Equal(t, 1, repo.setErrorCalls) + require.Equal(t, 0, repo.tempCalls) + require.Contains(t, repo.lastErrorMsg, "Authentication failed (401)") }) } } +// TestRateLimitService_HandleUpstreamError_OAuth401InvalidatorError +// OpenAI OAuth 401 缓存失效出错时仍走 temp_unschedulable func TestRateLimitService_HandleUpstreamError_OAuth401InvalidatorError(t *testing.T) { repo := &rateLimitAccountRepoStub{} invalidator := &tokenCacheInvalidatorRecorder{err: errors.New("boom")} @@ -90,7 +105,7 @@ func TestRateLimitService_HandleUpstreamError_OAuth401InvalidatorError(t *testin service.SetTokenCacheInvalidator(invalidator) account := &Account{ ID: 101, - Platform: PlatformGemini, + Platform: PlatformOpenAI, Type: AccountTypeOAuth, } diff --git a/backend/internal/service/setting_service.go b/backend/internal/service/setting_service.go index b77867deb7..c8f4076599 100644 --- a/backend/internal/service/setting_service.go +++ b/backend/internal/service/setting_service.go @@ -186,6 +186,49 @@ func (s *SettingService) SetOnS3UpdateCallback(callback func()) { s.onS3Update = callback } +// SetOnStorageUpdateCallback 设置存储配置变更时的回调函数(用于刷新所有存储客户端缓存)。 +// 替代 SetOnS3UpdateCallback,支持 S3 + GDrive 统一刷新。 +func (s *SettingService) SetOnStorageUpdateCallback(callback func()) { + s.onS3Update = callback +} + +// --- 统一存储 Profile 方法别名 --- + +// ListSoraStorageProfiles 获取 Sora 存储多配置列表(统一方法名)。 +func (s *SettingService) ListSoraStorageProfiles(ctx context.Context) (*SoraS3ProfileList, error) { + return s.ListSoraS3Profiles(ctx) +} + +// CreateSoraStorageProfile 创建 Sora 存储配置(统一方法名)。 +func (s *SettingService) CreateSoraStorageProfile(ctx context.Context, profile *SoraS3Profile, setActive bool) (*SoraS3Profile, error) { + return s.CreateSoraS3Profile(ctx, profile, setActive) +} + +// UpdateSoraStorageProfile 更新 Sora 存储配置(统一方法名)。 +func (s *SettingService) UpdateSoraStorageProfile(ctx context.Context, profileID string, profile *SoraS3Profile) (*SoraS3Profile, error) { + return s.UpdateSoraS3Profile(ctx, profileID, profile) +} + +// DeleteSoraStorageProfile 删除 Sora 存储配置(统一方法名)。 +func (s *SettingService) DeleteSoraStorageProfile(ctx context.Context, profileID string) error { + return s.DeleteSoraS3Profile(ctx, profileID) +} + +// SetActiveSoraStorageProfile 设置激活的 Sora 存储配置(统一方法名)。 +func (s *SettingService) SetActiveSoraStorageProfile(ctx context.Context, profileID string) (*SoraS3Profile, error) { + return s.SetActiveSoraS3Profile(ctx, profileID) +} + +// GetActiveStorageProfile 获取当前激活的存储配置 profile。 +func (s *SettingService) GetActiveStorageProfile(ctx context.Context) (*SoraS3Profile, error) { + profiles, err := s.ListSoraS3Profiles(ctx) + if err != nil { + return nil, err + } + active := pickActiveSoraS3Profile(profiles.Items, profiles.ActiveProfileID) + return active, nil +} + // SetVersion sets the application version for injection into public settings func (s *SettingService) SetVersion(version string) { s.version = version @@ -1341,6 +1384,8 @@ type soraS3ProfilesStore struct { type soraS3ProfileStoreItem struct { ProfileID string `json:"profile_id"` Name string `json:"name"` + Provider string `json:"provider,omitempty"` // "s3" / "gdrive",空值视为 "s3" + AccessMode string `json:"access_mode,omitempty"` // "direct" / "proxy",空值视为 "direct" Enabled bool `json:"enabled"` Endpoint string `json:"endpoint"` Region string `json:"region"` @@ -1352,6 +1397,14 @@ type soraS3ProfileStoreItem struct { CDNURL string `json:"cdn_url"` DefaultStorageQuotaBytes int64 `json:"default_storage_quota_bytes"` UpdatedAt string `json:"updated_at"` + + // --- Google Drive 专属 --- + AuthType string `json:"auth_type,omitempty"` + ClientID string `json:"client_id,omitempty"` + ClientSecret string `json:"client_secret,omitempty"` + RefreshToken string `json:"refresh_token,omitempty"` + ServiceAccountJSON string `json:"service_account_json,omitempty"` + FolderID string `json:"folder_id,omitempty"` } // GetSoraS3Settings 获取 Sora S3 存储配置(兼容旧单配置语义:返回当前激活配置) @@ -1463,6 +1516,8 @@ func (s *SettingService) CreateSoraS3Profile(ctx context.Context, profile *SoraS store.Items = append(store.Items, soraS3ProfileStoreItem{ ProfileID: profileID, Name: name, + Provider: profile.Provider, + AccessMode: profile.AccessMode, Enabled: profile.Enabled, Endpoint: strings.TrimSpace(profile.Endpoint), Region: strings.TrimSpace(profile.Region), @@ -1474,6 +1529,13 @@ func (s *SettingService) CreateSoraS3Profile(ctx context.Context, profile *SoraS CDNURL: strings.TrimSpace(profile.CDNURL), DefaultStorageQuotaBytes: maxInt64(profile.DefaultStorageQuotaBytes, 0), UpdatedAt: now, + // Google Drive 专属 + AuthType: profile.AuthType, + ClientID: strings.TrimSpace(profile.ClientID), + ClientSecret: profile.ClientSecret, + RefreshToken: profile.RefreshToken, + ServiceAccountJSON: profile.ServiceAccountJSON, + FolderID: strings.TrimSpace(profile.FolderID), }) if setActive || store.ActiveProfileID == "" { @@ -1519,6 +1581,8 @@ func (s *SettingService) UpdateSoraS3Profile(ctx context.Context, profileID stri return nil, infraerrors.BadRequest("SORA_S3_PROFILE_NAME_REQUIRED", "name is required") } target.Name = name + target.Provider = profile.Provider + target.AccessMode = profile.AccessMode target.Enabled = profile.Enabled target.Endpoint = strings.TrimSpace(profile.Endpoint) target.Region = strings.TrimSpace(profile.Region) @@ -1531,6 +1595,19 @@ func (s *SettingService) UpdateSoraS3Profile(ctx context.Context, profileID stri if profile.SecretAccessKey != "" { target.SecretAccessKey = profile.SecretAccessKey } + // Google Drive 专属 + target.AuthType = profile.AuthType + target.ClientID = strings.TrimSpace(profile.ClientID) + if profile.ClientSecret != "" { + target.ClientSecret = profile.ClientSecret + } + if profile.RefreshToken != "" { + target.RefreshToken = profile.RefreshToken + } + if profile.ServiceAccountJSON != "" { + target.ServiceAccountJSON = profile.ServiceAccountJSON + } + target.FolderID = strings.TrimSpace(profile.FolderID) target.UpdatedAt = time.Now().UTC().Format(time.RFC3339) store.Items[targetIndex] = target @@ -1833,6 +1910,8 @@ func convertSoraS3ProfilesStore(store *soraS3ProfilesStore) *SoraS3ProfileList { ProfileID: item.ProfileID, Name: item.Name, IsActive: item.ProfileID == store.ActiveProfileID, + Provider: item.Provider, + AccessMode: item.AccessMode, Enabled: item.Enabled, Endpoint: item.Endpoint, Region: item.Region, @@ -1845,6 +1924,16 @@ func convertSoraS3ProfilesStore(store *soraS3ProfilesStore) *SoraS3ProfileList { CDNURL: item.CDNURL, DefaultStorageQuotaBytes: item.DefaultStorageQuotaBytes, UpdatedAt: item.UpdatedAt, + // Google Drive 专属 + AuthType: item.AuthType, + ClientID: item.ClientID, + ClientSecret: item.ClientSecret, + ClientSecretConfigured: item.ClientSecret != "", + RefreshToken: item.RefreshToken, + RefreshTokenConfigured: item.RefreshToken != "", + ServiceAccountJSON: item.ServiceAccountJSON, + ServiceAccountConfigured: item.ServiceAccountJSON != "", + FolderID: item.FolderID, }) } return &SoraS3ProfileList{ diff --git a/backend/internal/service/settings_view.go b/backend/internal/service/settings_view.go index 8734e28aa9..e18d7ed098 100644 --- a/backend/internal/service/settings_view.go +++ b/backend/internal/service/settings_view.go @@ -124,6 +124,8 @@ type SoraS3Profile struct { ProfileID string `json:"profile_id"` Name string `json:"name"` IsActive bool `json:"is_active"` + Provider string `json:"provider"` // "s3" / "gdrive",空值视为 "s3" + AccessMode string `json:"access_mode"` // "direct" / "proxy",空值视为 "direct" Enabled bool `json:"enabled"` Endpoint string `json:"endpoint"` Region string `json:"region"` @@ -136,6 +138,25 @@ type SoraS3Profile struct { CDNURL string `json:"cdn_url"` DefaultStorageQuotaBytes int64 `json:"default_storage_quota_bytes"` UpdatedAt string `json:"updated_at"` + + // --- Google Drive 专属 --- + AuthType string `json:"auth_type,omitempty"` // "oauth2" / "service_account" + ClientID string `json:"client_id,omitempty"` + ClientSecret string `json:"-"` + ClientSecretConfigured bool `json:"client_secret_configured"` + RefreshToken string `json:"-"` + RefreshTokenConfigured bool `json:"refresh_token_configured"` + ServiceAccountJSON string `json:"-"` + ServiceAccountConfigured bool `json:"service_account_configured"` + FolderID string `json:"folder_id,omitempty"` +} + +// GetProvider 返回 Provider,空值视为 "s3"。 +func (p *SoraS3Profile) GetProvider() string { + if p.Provider == "" { + return SoraStorageTypeS3 + } + return p.Provider } // SoraS3ProfileList Sora S3 多配置列表 diff --git a/backend/internal/service/sora_gdrive_oauth.go b/backend/internal/service/sora_gdrive_oauth.go new file mode 100644 index 0000000000..61282053ca --- /dev/null +++ b/backend/internal/service/sora_gdrive_oauth.go @@ -0,0 +1,71 @@ +package service + +import ( + "context" + "crypto/rand" + "encoding/hex" + "fmt" + + "golang.org/x/oauth2" + "golang.org/x/oauth2/google" + "google.golang.org/api/drive/v3" +) + +// SoraGDriveOAuthService 处理 Google Drive OAuth2 授权流程。 +type SoraGDriveOAuthService struct{} + +// NewSoraGDriveOAuthService 创建 GDrive OAuth 服务。 +func NewSoraGDriveOAuthService(_ *SettingService) *SoraGDriveOAuthService { + return &SoraGDriveOAuthService{} +} + +// GenerateAuthURL 生成 Google OAuth 授权 URL。 +func (s *SoraGDriveOAuthService) GenerateAuthURL(clientID, clientSecret, redirectURI string) (authURL, state string, err error) { + if clientID == "" || clientSecret == "" || redirectURI == "" { + return "", "", fmt.Errorf("client_id, client_secret, redirect_uri are required") + } + + config := &oauth2.Config{ + ClientID: clientID, + ClientSecret: clientSecret, + Endpoint: google.Endpoint, + Scopes: []string{drive.DriveFileScope}, + RedirectURL: redirectURI, + } + + // 生成随机 state + stateBytes := make([]byte, 16) + if _, err := rand.Read(stateBytes); err != nil { + return "", "", fmt.Errorf("generate state: %w", err) + } + state = hex.EncodeToString(stateBytes) + + authURL = config.AuthCodeURL(state, oauth2.AccessTypeOffline, oauth2.ApprovalForce) + return authURL, state, nil +} + +// ExchangeCode 用授权码换取 refresh_token。 +func (s *SoraGDriveOAuthService) ExchangeCode(ctx context.Context, clientID, clientSecret, redirectURI, code string) (string, error) { + if code == "" { + return "", fmt.Errorf("authorization code is required") + } + + config := &oauth2.Config{ + ClientID: clientID, + ClientSecret: clientSecret, + Endpoint: google.Endpoint, + Scopes: []string{drive.DriveFileScope}, + RedirectURL: redirectURI, + } + + token, err := config.Exchange(ctx, code) + if err != nil { + return "", fmt.Errorf("exchange code: %w", err) + } + + if token.RefreshToken == "" { + return "", fmt.Errorf("no refresh_token received, please revoke app access and try again") + } + + return token.RefreshToken, nil +} diff --git a/backend/internal/service/sora_gdrive_storage.go b/backend/internal/service/sora_gdrive_storage.go new file mode 100644 index 0000000000..afbc3ea5c9 --- /dev/null +++ b/backend/internal/service/sora_gdrive_storage.go @@ -0,0 +1,433 @@ +package service + +import ( + "context" + "fmt" + "io" + "net/http" + "strings" + "sync" + "time" + + "github.com/google/uuid" + "golang.org/x/oauth2" + "golang.org/x/oauth2/google" + "google.golang.org/api/drive/v3" + "google.golang.org/api/option" + + "github.com/Wei-Shaw/sub2api/internal/pkg/logger" +) + +// SoraGDriveStorage 负责 Sora 媒体文件的 Google Drive 存储操作。 +type SoraGDriveStorage struct { + settingService *SettingService + + mu sync.RWMutex + srv *drive.Service + cfg *SoraS3Profile // 缓存当前 GDrive 配置 + healthCheckedAt time.Time + healthErr error + healthTTL time.Duration +} + +const defaultGDriveHealthTTL = 30 * time.Second + +// NewSoraGDriveStorage 创建 Google Drive 存储服务实例。 +func NewSoraGDriveStorage(settingService *SettingService) *SoraGDriveStorage { + return &SoraGDriveStorage{ + settingService: settingService, + healthTTL: defaultGDriveHealthTTL, + } +} + +// StorageType 返回存储类型标识。 +func (s *SoraGDriveStorage) StorageType() string { + return SoraStorageTypeGDrive +} + +// Enabled 返回 Google Drive 存储是否已启用。 +func (s *SoraGDriveStorage) Enabled(ctx context.Context) bool { + profile := s.getActiveGDriveProfile(ctx) + if profile == nil { + return false + } + return profile.Enabled && s.hasValidCredentials(profile) +} + +// getActiveGDriveProfile 获取当前激活的 GDrive 配置。 +func (s *SoraGDriveStorage) getActiveGDriveProfile(ctx context.Context) *SoraS3Profile { + if s.settingService == nil { + return nil + } + profile, err := s.settingService.GetActiveStorageProfile(ctx) + if err != nil || profile == nil { + return nil + } + if profile.GetProvider() != SoraStorageTypeGDrive { + return nil + } + return profile +} + +// hasValidCredentials 检查 GDrive 配置是否有有效凭证。 +func (s *SoraGDriveStorage) hasValidCredentials(profile *SoraS3Profile) bool { + switch profile.AuthType { + case "oauth2": + return profile.ClientID != "" && profile.ClientSecret != "" && profile.RefreshToken != "" + case "service_account": + return profile.ServiceAccountJSON != "" + default: + return false + } +} + +// getService 获取或初始化 Drive 服务(带缓存)。 +func (s *SoraGDriveStorage) getService(ctx context.Context) (*drive.Service, *SoraS3Profile, error) { + s.mu.RLock() + if s.srv != nil && s.cfg != nil { + srv, cfg := s.srv, s.cfg + s.mu.RUnlock() + return srv, cfg, nil + } + s.mu.RUnlock() + + return s.initService(ctx) +} + +func (s *SoraGDriveStorage) initService(ctx context.Context) (*drive.Service, *SoraS3Profile, error) { + s.mu.Lock() + defer s.mu.Unlock() + + // 双重检查 + if s.srv != nil && s.cfg != nil { + return s.srv, s.cfg, nil + } + + profile := s.getActiveGDriveProfile(ctx) + if profile == nil { + return nil, nil, fmt.Errorf("no active gdrive profile found") + } + if !profile.Enabled { + return nil, nil, fmt.Errorf("gdrive storage is disabled") + } + + srv, err := s.buildDriveService(ctx, profile) + if err != nil { + return nil, nil, fmt.Errorf("build gdrive service: %w", err) + } + + s.srv = srv + s.cfg = profile + logger.LegacyPrintf("service.sora_gdrive", "[SoraGDrive] 客户端已初始化 auth_type=%s folder_id=%s", profile.AuthType, profile.FolderID) + return srv, profile, nil +} + +// buildDriveService 根据认证类型创建 Google Drive 服务。 +func (s *SoraGDriveStorage) buildDriveService(ctx context.Context, profile *SoraS3Profile) (*drive.Service, error) { + switch profile.AuthType { + case "oauth2": + return s.buildOAuth2Service(ctx, profile) + case "service_account": + return s.buildServiceAccountService(ctx, profile) + default: + return nil, fmt.Errorf("unsupported auth_type: %s", profile.AuthType) + } +} + +func (s *SoraGDriveStorage) buildOAuth2Service(ctx context.Context, profile *SoraS3Profile) (*drive.Service, error) { + config := &oauth2.Config{ + ClientID: profile.ClientID, + ClientSecret: profile.ClientSecret, + Endpoint: google.Endpoint, + Scopes: []string{drive.DriveFileScope}, + } + token := &oauth2.Token{ + RefreshToken: profile.RefreshToken, + } + tokenSource := config.TokenSource(ctx, token) + srv, err := drive.NewService(ctx, option.WithTokenSource(tokenSource)) + if err != nil { + return nil, fmt.Errorf("create gdrive oauth2 service: %w", err) + } + return srv, nil +} + +func (s *SoraGDriveStorage) buildServiceAccountService(ctx context.Context, profile *SoraS3Profile) (*drive.Service, error) { + srv, err := drive.NewService(ctx, option.WithCredentialsJSON([]byte(profile.ServiceAccountJSON))) //nolint:staticcheck // SA1019: admin-controlled service account JSON, safe to use + if err != nil { + return nil, fmt.Errorf("create gdrive service account service: %w", err) + } + return srv, nil +} + +// RefreshClient 清除缓存的 Drive 客户端。 +func (s *SoraGDriveStorage) RefreshClient() { + s.mu.Lock() + defer s.mu.Unlock() + s.srv = nil + s.cfg = nil + s.healthCheckedAt = time.Time{} + s.healthErr = nil + logger.LegacyPrintf("service.sora_gdrive", "[SoraGDrive] 客户端缓存已清除") +} + +// GDriveQuotaInfo 包含 Google Drive 配额信息。 +type GDriveQuotaInfo struct { + LimitBytes int64 `json:"limit_bytes"` + UsedBytes int64 `json:"used_bytes"` +} + +// TestConnection 测试 Google Drive 连接。 +func (s *SoraGDriveStorage) TestConnection(ctx context.Context) error { + srv, _, err := s.getService(ctx) + if err != nil { + return err + } + _, err = srv.About.Get().Fields("storageQuota").Context(ctx).Do() + if err != nil { + return fmt.Errorf("gdrive About.Get failed: %w", err) + } + return nil +} + +// GetQuotaInfo 获取 Google Drive 配额信息(总量和已用量)。 +func (s *SoraGDriveStorage) GetQuotaInfo(ctx context.Context) (*GDriveQuotaInfo, error) { + srv, _, err := s.getService(ctx) + if err != nil { + return nil, err + } + about, err := srv.About.Get().Fields("storageQuota").Context(ctx).Do() + if err != nil { + return nil, fmt.Errorf("gdrive About.Get failed: %w", err) + } + if about.StorageQuota == nil { + return nil, fmt.Errorf("storageQuota not available") + } + return &GDriveQuotaInfo{ + LimitBytes: about.StorageQuota.Limit, + UsedBytes: about.StorageQuota.Usage, + }, nil +} + +// TestFullCycle 执行完整的上传→获取链接→删除测试。 +func (s *SoraGDriveStorage) TestFullCycle(ctx context.Context) (map[string]any, error) { + srv, cfg, err := s.getService(ctx) + if err != nil { + return nil, fmt.Errorf("init client: %w", err) + } + + result := map[string]any{} + + // 1. 测试 API 连接 + about, err := srv.About.Get().Fields("storageQuota").Context(ctx).Do() + if err != nil { + return nil, fmt.Errorf("API connection failed: %w", err) + } + if about.StorageQuota != nil { + result["quota_limit_bytes"] = about.StorageQuota.Limit + result["quota_used_bytes"] = about.StorageQuota.Usage + } + + // 2. 上传测试文件 + testContent := "sub2api GDrive test file - " + time.Now().Format(time.RFC3339) + fileMeta := &drive.File{ + Name: "sub2api_test_" + uuid.NewString()[:8] + ".txt", + MimeType: "text/plain", + } + if cfg.FolderID != "" { + fileMeta.Parents = []string{cfg.FolderID} + } + uploaded, err := srv.Files.Create(fileMeta). + Media(strings.NewReader(testContent)). + Fields("id,name,size,webViewLink"). + Context(ctx).Do() + if err != nil { + return nil, fmt.Errorf("upload test file failed: %w", err) + } + result["uploaded_file_id"] = uploaded.Id + result["uploaded_file_name"] = uploaded.Name + result["uploaded_file_size"] = uploaded.Size + result["web_view_link"] = uploaded.WebViewLink + + // 3. 获取访问链接 + accessURL, err := s.GetAccessURL(ctx, uploaded.Id) + if err != nil { + // 即使获取链接失败,仍尝试清理 + _ = srv.Files.Delete(uploaded.Id).Context(ctx).Do() + return nil, fmt.Errorf("get access URL failed: %w", err) + } + result["access_url"] = accessURL + + // 4. 删除测试文件 + if err := srv.Files.Delete(uploaded.Id).Context(ctx).Do(); err != nil { + result["delete_warning"] = fmt.Sprintf("delete failed (manual cleanup needed): %v", err) + } else { + result["deleted"] = true + } + + result["status"] = "ok" + return result, nil +} + +// IsHealthy 返回 Google Drive 健康状态(带短缓存)。 +func (s *SoraGDriveStorage) IsHealthy(ctx context.Context) bool { + if s == nil { + return false + } + now := time.Now() + s.mu.RLock() + lastCheck := s.healthCheckedAt + lastErr := s.healthErr + ttl := s.healthTTL + s.mu.RUnlock() + + if ttl <= 0 { + ttl = defaultGDriveHealthTTL + } + if !lastCheck.IsZero() && now.Sub(lastCheck) < ttl { + return lastErr == nil + } + + err := s.TestConnection(ctx) + s.mu.Lock() + s.healthCheckedAt = time.Now() + s.healthErr = err + s.mu.Unlock() + return err == nil +} + +// UploadFromURL 从上游 URL 下载并上传到 Google Drive。 +// 返回 Google Drive 文件 ID 作为 objectKey、文件大小、存储类型。 +func (s *SoraGDriveStorage) UploadFromURL(ctx context.Context, userID int64, sourceURL string) (string, int64, string, error) { + srv, cfg, err := s.getService(ctx) + if err != nil { + return "", 0, "", err + } + + // 下载源文件 + req, err := http.NewRequestWithContext(ctx, http.MethodGet, sourceURL, nil) + if err != nil { + return "", 0, "", fmt.Errorf("create download request: %w", err) + } + httpClient := &http.Client{Timeout: 5 * time.Minute} + resp, err := httpClient.Do(req) + if err != nil { + return "", 0, "", fmt.Errorf("download from upstream: %w", err) + } + defer func() { + _ = resp.Body.Close() + }() + + if resp.StatusCode != http.StatusOK { + return "", 0, "", &UpstreamDownloadError{StatusCode: resp.StatusCode} + } + + // 推断文件扩展名和 MIME + ext := fileExtFromURL(sourceURL) + if ext == "" { + ext = fileExtFromContentType(resp.Header.Get("Content-Type")) + } + if ext == "" { + ext = ".bin" + } + contentType := resp.Header.Get("Content-Type") + if contentType == "" { + contentType = "application/octet-stream" + } + + // 生成文件名 + datePath := time.Now().Format("2006-01-02") + fileName := fmt.Sprintf("sora_%d_%s_%s%s", userID, datePath, uuid.NewString()[:8], ext) + + // 创建文件元数据 + fileMeta := &drive.File{ + Name: fileName, + MimeType: contentType, + } + if cfg.FolderID != "" { + fileMeta.Parents = []string{cfg.FolderID} + } + + // 使用 CountingReader 统计大小 + cr := &countingReader{Reader: resp.Body} + + // 上传到 Google Drive + created, err := srv.Files.Create(fileMeta). + Media(cr). + Fields("id, size"). + Context(ctx). + Do() + if err != nil { + return "", 0, "", fmt.Errorf("gdrive upload: %w", err) + } + + fileSize := cr.BytesRead + if created.Size > 0 { + fileSize = created.Size + } + + // 根据 access_mode 设置权限 + if cfg.AccessMode == "" || cfg.AccessMode == "direct" { + // 设为任何人可读 + _, permErr := srv.Permissions.Create(created.Id, &drive.Permission{ + Type: "anyone", + Role: "reader", + }).Context(ctx).Do() + if permErr != nil { + logger.LegacyPrintf("service.sora_gdrive", "[SoraGDrive] 设置公开权限失败 fileID=%s err=%v", created.Id, permErr) + } + } + + logger.LegacyPrintf("service.sora_gdrive", "[SoraGDrive] 上传完成 fileID=%s size=%d", created.Id, fileSize) + return created.Id, fileSize, SoraStorageTypeGDrive, nil +} + +// DeleteObjects 删除一组 Google Drive 文件。 +func (s *SoraGDriveStorage) DeleteObjects(ctx context.Context, objectKeys []string) error { + if len(objectKeys) == 0 { + return nil + } + + srv, _, err := s.getService(ctx) + if err != nil { + return err + } + + var lastErr error + for _, fileID := range objectKeys { + if err := srv.Files.Delete(fileID).Context(ctx).Do(); err != nil { + logger.LegacyPrintf("service.sora_gdrive", "[SoraGDrive] 删除失败 fileID=%s err=%v", fileID, err) + lastErr = err + } + } + return lastErr +} + +// GetAccessURL 获取 Google Drive 文件的访问 URL。 +func (s *SoraGDriveStorage) GetAccessURL(ctx context.Context, objectKey string) (string, error) { + _, cfg, err := s.getService(ctx) + if err != nil { + return "", err + } + + // CDN URL 优先 + if cfg.CDNURL != "" { + cdnBase := strings.TrimRight(cfg.CDNURL, "/") + return cdnBase + "/" + objectKey, nil + } + + // 默认使用 Google Drive 直链 + return fmt.Sprintf("https://drive.google.com/uc?export=download&id=%s", objectKey), nil +} + +// countingReader 包装 io.Reader 以统计读取的字节数。 +type countingReader struct { + Reader io.Reader + BytesRead int64 +} + +func (r *countingReader) Read(p []byte) (int, error) { + n, err := r.Reader.Read(p) + r.BytesRead += int64(n) + return n, err +} diff --git a/backend/internal/service/sora_gdrive_storage_test.go b/backend/internal/service/sora_gdrive_storage_test.go new file mode 100644 index 0000000000..e82f70c903 --- /dev/null +++ b/backend/internal/service/sora_gdrive_storage_test.go @@ -0,0 +1,124 @@ +//go:build unit + +package service + +import ( + "context" + "testing" + + "github.com/stretchr/testify/assert" +) + +func TestSoraGDriveStorage_StorageType(t *testing.T) { + s := NewSoraGDriveStorage(nil) + assert.Equal(t, SoraStorageTypeGDrive, s.StorageType()) +} + +func TestSoraGDriveStorage_EnabledWithNilSettingService(t *testing.T) { + s := NewSoraGDriveStorage(nil) + assert.False(t, s.Enabled(context.Background())) +} + +func TestSoraGDriveStorage_IsHealthyWithNilReceiver(t *testing.T) { + var s *SoraGDriveStorage + assert.False(t, s.IsHealthy(context.Background())) +} + +func TestSoraGDriveStorage_GetServiceWithoutProfile(t *testing.T) { + s := NewSoraGDriveStorage(nil) + _, _, err := s.getService(context.Background()) + assert.Error(t, err) + assert.Contains(t, err.Error(), "no active gdrive profile") +} + +func TestSoraGDriveStorage_DeleteObjectsEmpty(t *testing.T) { + s := NewSoraGDriveStorage(nil) + err := s.DeleteObjects(context.Background(), []string{}) + assert.NoError(t, err) +} + +func TestSoraGDriveStorage_RefreshClient(t *testing.T) { + s := NewSoraGDriveStorage(nil) + // 不应 panic + s.RefreshClient() + assert.Nil(t, s.srv) + assert.Nil(t, s.cfg) +} + +func TestSoraGDriveStorage_HasValidCredentials(t *testing.T) { + s := NewSoraGDriveStorage(nil) + + tests := []struct { + name string + profile *SoraS3Profile + want bool + }{ + { + name: "oauth2 with all fields", + profile: &SoraS3Profile{ + AuthType: "oauth2", + ClientID: "id", + ClientSecret: "secret", + RefreshToken: "token", + }, + want: true, + }, + { + name: "oauth2 missing refresh token", + profile: &SoraS3Profile{ + AuthType: "oauth2", + ClientID: "id", + ClientSecret: "secret", + }, + want: false, + }, + { + name: "service_account with json", + profile: &SoraS3Profile{ + AuthType: "service_account", + ServiceAccountJSON: `{"type":"service_account"}`, + }, + want: true, + }, + { + name: "service_account without json", + profile: &SoraS3Profile{ + AuthType: "service_account", + }, + want: false, + }, + { + name: "unknown auth type", + profile: &SoraS3Profile{ + AuthType: "unknown", + }, + want: false, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := s.hasValidCredentials(tt.profile) + assert.Equal(t, tt.want, got) + }) + } +} + +func TestSoraStorageRouter_DefaultsToS3(t *testing.T) { + s3 := NewSoraS3Storage(nil) + router := NewSoraStorageRouter(nil, s3, nil) + // settingService 为 nil,应返回 s3Storage + backend := router.activeBackend(context.Background()) + assert.Equal(t, s3, backend) +} + +func TestSoraStorageRouter_StorageType(t *testing.T) { + router := NewSoraStorageRouter(nil, nil, nil) + assert.Equal(t, SoraStorageTypeS3, router.StorageType()) +} + +func TestSoraStorageRouter_RefreshAllNoPanic(t *testing.T) { + router := NewSoraStorageRouter(nil, nil, nil) + // 不应 panic + router.RefreshAll() +} diff --git a/backend/internal/service/sora_generation.go b/backend/internal/service/sora_generation.go index a704454b82..7323bb38ac 100644 --- a/backend/internal/service/sora_generation.go +++ b/backend/internal/service/sora_generation.go @@ -37,6 +37,7 @@ const ( // Sora 存储类型常量 const ( SoraStorageTypeS3 = "s3" + SoraStorageTypeGDrive = "gdrive" SoraStorageTypeLocal = "local" SoraStorageTypeUpstream = "upstream" SoraStorageTypeNone = "none" @@ -60,4 +61,5 @@ type SoraGenerationRepository interface { Delete(ctx context.Context, id int64) error List(ctx context.Context, params SoraGenerationListParams) ([]*SoraGeneration, int64, error) CountByUserAndStatus(ctx context.Context, userID int64, statuses []string) (int64, error) + CountByStorageType(ctx context.Context, storageType string, statuses []string) (int64, error) } diff --git a/backend/internal/service/sora_generation_service.go b/backend/internal/service/sora_generation_service.go index 22d5b51947..0eba7f91c9 100644 --- a/backend/internal/service/sora_generation_service.go +++ b/backend/internal/service/sora_generation_service.go @@ -35,21 +35,21 @@ type soraGenerationRepoConditionalUpdater interface { // SoraGenerationService 管理 Sora 客户端的生成记录 CRUD。 type SoraGenerationService struct { - genRepo SoraGenerationRepository - s3Storage *SoraS3Storage - quotaService *SoraQuotaService + genRepo SoraGenerationRepository + objectStorage SoraObjectStorage + quotaService *SoraQuotaService } // NewSoraGenerationService 创建生成记录服务。 func NewSoraGenerationService( genRepo SoraGenerationRepository, - s3Storage *SoraS3Storage, + objectStorage SoraObjectStorage, quotaService *SoraQuotaService, ) *SoraGenerationService { return &SoraGenerationService{ - genRepo: genRepo, - s3Storage: s3Storage, - quotaService: quotaService, + genRepo: genRepo, + objectStorage: objectStorage, + quotaService: quotaService, } } @@ -268,15 +268,15 @@ func (s *SoraGenerationService) Delete(ctx context.Context, id, userID int64) er return fmt.Errorf("无权删除此生成记录") } - // 清理 S3 文件 - if gen.StorageType == SoraStorageTypeS3 && len(gen.S3ObjectKeys) > 0 && s.s3Storage != nil { - if err := s.s3Storage.DeleteObjects(ctx, gen.S3ObjectKeys); err != nil { - logger.LegacyPrintf("service.sora_gen", "[SoraGen] S3 清理失败 id=%d err=%v", id, err) + // 清理存储文件(S3 / Google Drive) + if IsObjectStorageType(gen.StorageType) && len(gen.S3ObjectKeys) > 0 && s.objectStorage != nil { + if err := s.objectStorage.DeleteObjects(ctx, gen.S3ObjectKeys); err != nil { + logger.LegacyPrintf("service.sora_gen", "[SoraGen] 存储清理失败 id=%d type=%s err=%v", id, gen.StorageType, err) } } - // 释放配额(S3/本地均释放) - if gen.FileSizeBytes > 0 && (gen.StorageType == SoraStorageTypeS3 || gen.StorageType == SoraStorageTypeLocal) && s.quotaService != nil { + // 释放配额(对象存储/本地均释放) + if gen.FileSizeBytes > 0 && (IsObjectStorageType(gen.StorageType) || gen.StorageType == SoraStorageTypeLocal) && s.quotaService != nil { if err := s.quotaService.ReleaseUsage(ctx, userID, gen.FileSizeBytes); err != nil { logger.LegacyPrintf("service.sora_gen", "[SoraGen] 配额释放失败 id=%d err=%v", id, err) } @@ -290,9 +290,9 @@ func (s *SoraGenerationService) CountActiveByUser(ctx context.Context, userID in return s.genRepo.CountByUserAndStatus(ctx, userID, []string{SoraGenStatusPending, SoraGenStatusGenerating}) } -// ResolveMediaURLs 为 S3 记录动态生成预签名 URL。 +// ResolveMediaURLs 为对象存储记录动态生成访问 URL。 func (s *SoraGenerationService) ResolveMediaURLs(ctx context.Context, gen *SoraGeneration) error { - if gen == nil || gen.StorageType != SoraStorageTypeS3 || s.s3Storage == nil { + if gen == nil || !IsObjectStorageType(gen.StorageType) || s.objectStorage == nil { return nil } if len(gen.S3ObjectKeys) == 0 { @@ -308,7 +308,7 @@ func (s *SoraGenerationService) ResolveMediaURLs(ctx context.Context, gen *SoraG wg.Add(1) go func(i int, objectKey string) { defer wg.Done() - url, err := s.s3Storage.GetAccessURL(ctx, objectKey) + url, err := s.objectStorage.GetAccessURL(ctx, objectKey) if err != nil { errMu.Lock() if firstErr == nil { @@ -330,3 +330,22 @@ func (s *SoraGenerationService) ResolveMediaURLs(ctx context.Context, gen *SoraG return nil } + +// StorageVideoStats 各存储类型的视频统计。 +type StorageVideoStats struct { + Completed int64 `json:"completed"` + InProgress int64 `json:"in_progress"` +} + +// CountByStorageType 按存储类型统计视频数量(completed 和 in_progress)。 +func (s *SoraGenerationService) CountByStorageType(ctx context.Context, storageType string) (completed, inProgress int64, err error) { + completed, err = s.genRepo.CountByStorageType(ctx, storageType, []string{SoraGenStatusCompleted}) + if err != nil { + return 0, 0, fmt.Errorf("count completed: %w", err) + } + inProgress, err = s.genRepo.CountByStorageType(ctx, storageType, []string{SoraGenStatusPending, SoraGenStatusGenerating}) + if err != nil { + return 0, 0, fmt.Errorf("count in_progress: %w", err) + } + return completed, inProgress, nil +} diff --git a/backend/internal/service/sora_generation_service_test.go b/backend/internal/service/sora_generation_service_test.go index 46f322c82c..0bde211f6c 100644 --- a/backend/internal/service/sora_generation_service_test.go +++ b/backend/internal/service/sora_generation_service_test.go @@ -115,6 +115,25 @@ func (r *stubGenRepo) CountByUserAndStatus(_ context.Context, userID int64, stat return count, nil } +func (r *stubGenRepo) CountByStorageType(_ context.Context, storageType string, statuses []string) (int64, error) { + if r.countErr != nil { + return 0, r.countErr + } + var count int64 + statusSet := make(map[string]struct{}) + for _, s := range statuses { + statusSet[s] = struct{}{} + } + for _, gen := range r.gens { + if gen.StorageType == storageType { + if _, ok := statusSet[gen.Status]; ok { + count++ + } + } + } + return count, nil +} + // ==================== Stub: UserRepository (用于 SoraQuotaService) ==================== var _ UserRepository = (*stubUserRepoForQuota)(nil) @@ -519,7 +538,7 @@ func TestDelete_S3Cleanup_NilS3(t *testing.T) { svc := NewSoraGenerationService(repo, nil, nil) err := svc.Delete(context.Background(), 1, 1) - require.NoError(t, err) // s3Storage 为 nil,跳过清理 + require.NoError(t, err) // objectStorage 为 nil,跳过清理 } func TestDelete_QuotaRelease_NilQuota(t *testing.T) { diff --git a/backend/internal/service/sora_object_storage.go b/backend/internal/service/sora_object_storage.go new file mode 100644 index 0000000000..f9e9a4b03e --- /dev/null +++ b/backend/internal/service/sora_object_storage.go @@ -0,0 +1,37 @@ +package service + +import "context" + +// SoraObjectStorage 是 Sora 媒体文件的通用对象存储接口。 +// S3 和 Google Drive 等存储后端均实现此接口。 +type SoraObjectStorage interface { + // Enabled 返回存储是否已启用且配置有效。 + Enabled(ctx context.Context) bool + + // IsHealthy 返回存储健康状态(带短缓存)。 + IsHealthy(ctx context.Context) bool + + // TestConnection 测试存储连接。 + TestConnection(ctx context.Context) error + + // UploadFromURL 从上游 URL 下载并上传到存储。 + // 返回 object key(S3 key 或 GDrive file ID)、文件大小、实际使用的存储类型。 + UploadFromURL(ctx context.Context, userID int64, sourceURL string) (objectKey string, sizeBytes int64, storageType string, err error) + + // DeleteObjects 删除一组存储对象。 + DeleteObjects(ctx context.Context, objectKeys []string) error + + // GetAccessURL 获取存储文件的访问 URL。 + GetAccessURL(ctx context.Context, objectKey string) (string, error) + + // RefreshClient 清除缓存客户端,配置变更时调用。 + RefreshClient() + + // StorageType 返回存储类型标识("s3" / "gdrive")。 + StorageType() string +} + +// IsObjectStorageType 判断是否为对象存储类型(S3 或 Google Drive)。 +func IsObjectStorageType(t string) bool { + return t == SoraStorageTypeS3 || t == SoraStorageTypeGDrive +} diff --git a/backend/internal/service/sora_s3_storage.go b/backend/internal/service/sora_s3_storage.go index 4c57390515..19d8b9c22f 100644 --- a/backend/internal/service/sora_s3_storage.go +++ b/backend/internal/service/sora_s3_storage.go @@ -212,29 +212,29 @@ func (s *SoraS3Storage) GenerateObjectKey(prefix string, userID int64, ext strin } // UploadFromURL 从上游 URL 下载并流式上传到 S3。 -// 返回 S3 object key。 -func (s *SoraS3Storage) UploadFromURL(ctx context.Context, userID int64, sourceURL string) (string, int64, error) { +// 返回 S3 object key、文件大小、存储类型。 +func (s *SoraS3Storage) UploadFromURL(ctx context.Context, userID int64, sourceURL string) (string, int64, string, error) { client, cfg, err := s.getClient(ctx) if err != nil { - return "", 0, err + return "", 0, "", err } // 下载源文件 req, err := http.NewRequestWithContext(ctx, http.MethodGet, sourceURL, nil) if err != nil { - return "", 0, fmt.Errorf("create download request: %w", err) + return "", 0, "", fmt.Errorf("create download request: %w", err) } httpClient := &http.Client{Timeout: 5 * time.Minute} resp, err := httpClient.Do(req) if err != nil { - return "", 0, fmt.Errorf("download from upstream: %w", err) + return "", 0, "", fmt.Errorf("download from upstream: %w", err) } defer func() { _ = resp.Body.Close() }() if resp.StatusCode != http.StatusOK { - return "", 0, &UpstreamDownloadError{StatusCode: resp.StatusCode} + return "", 0, "", &UpstreamDownloadError{StatusCode: resp.StatusCode} } // 推断文件扩展名 @@ -275,14 +275,14 @@ func (s *SoraS3Storage) UploadFromURL(ctx context.Context, userID int64, sourceU _ = writer.CloseWithError(copyErr) uploadErr := <-uploadErrCh if copyErr != nil { - return "", 0, fmt.Errorf("stream upload copy failed: %w", copyErr) + return "", 0, "", fmt.Errorf("stream upload copy failed: %w", copyErr) } if uploadErr != nil { - return "", 0, fmt.Errorf("s3 upload: %w", uploadErr) + return "", 0, "", fmt.Errorf("s3 upload: %w", uploadErr) } logger.LegacyPrintf("service.sora_s3", "[SoraS3] 上传完成 key=%s size=%d", objectKey, written) - return objectKey, written, nil + return objectKey, written, SoraStorageTypeS3, nil } func buildSoraS3Client(ctx context.Context, cfg *SoraS3Settings) (*s3.Client, string, error) { @@ -380,6 +380,11 @@ func (s *SoraS3Storage) GeneratePresignedURL(ctx context.Context, objectKey stri return result.URL, nil } +// StorageType 返回存储类型标识。 +func (s *SoraS3Storage) StorageType() string { + return SoraStorageTypeS3 +} + // GetMediaType 从 object key 推断媒体类型(image/video)。 func GetMediaTypeFromKey(objectKey string) string { ext := strings.ToLower(path.Ext(objectKey)) diff --git a/backend/internal/service/sora_s3_storage_test.go b/backend/internal/service/sora_s3_storage_test.go index 32ff9a6f0a..8ed7137d8e 100644 --- a/backend/internal/service/sora_s3_storage_test.go +++ b/backend/internal/service/sora_s3_storage_test.go @@ -238,7 +238,7 @@ func TestTestConnection_GetClientError(t *testing.T) { func TestUploadFromURL_GetClientError(t *testing.T) { s := NewSoraS3Storage(nil) - _, _, err := s.UploadFromURL(context.Background(), 1, "https://example.com/file.mp4") + _, _, _, err := s.UploadFromURL(context.Background(), 1, "https://example.com/file.mp4") require.Error(t, err) } diff --git a/backend/internal/service/sora_sdk_client.go b/backend/internal/service/sora_sdk_client.go index f9221c5b51..c839be4b03 100644 --- a/backend/internal/service/sora_sdk_client.go +++ b/backend/internal/service/sora_sdk_client.go @@ -316,20 +316,36 @@ func (c *SoraSDKClient) GetCameoStatus(ctx context.Context, account *Account, ca if err != nil { return nil, err } - sdkClient, err := c.getSDKClient(account) - if err != nil { - return nil, err - } - status, err := sdkClient.GetCameoStatus(ctx, token, cameoID) + + // 直接调用 Sora 后端 API 而非 SDK,以获取 SDK 未暴露的字段 + // (status_message、instruction_set_hint、instruction_set)。 + path := "/project_y/cameos/in_progress/" + cameoID + raw, err := c.doSoraBackendJSON(ctx, account, http.MethodGet, path, token, "", nil) if err != nil { return nil, c.wrapSDKError(err, account) } - return &SoraCameoStatus{ - Status: status.Status, - DisplayNameHint: status.DisplayNameHint, - UsernameHint: status.UsernameHint, - ProfileAssetURL: status.ProfileAssetURL, - }, nil + + return parseCameoStatusFromRaw(raw), nil +} + +// parseCameoStatusFromRaw 从原始 JSON 解析 SoraCameoStatus, +// 包含 SDK 未暴露的 status_message / instruction_set_hint / instruction_set 字段。 +func parseCameoStatusFromRaw(raw []byte) *SoraCameoStatus { + result := gjson.ParseBytes(raw) + cameoStatus := &SoraCameoStatus{ + Status: strings.TrimSpace(result.Get("status").String()), + StatusMessage: strings.TrimSpace(result.Get("status_message").String()), + DisplayNameHint: strings.TrimSpace(result.Get("display_name_hint").String()), + UsernameHint: strings.TrimSpace(result.Get("username_hint").String()), + ProfileAssetURL: strings.TrimSpace(result.Get("profile_asset_url").String()), + } + if v := result.Get("instruction_set_hint"); v.Exists() { + cameoStatus.InstructionSetHint = v.Value() + } + if v := result.Get("instruction_set"); v.Exists() { + cameoStatus.InstructionSet = v.Value() + } + return cameoStatus } func (c *SoraSDKClient) DownloadCharacterImage(ctx context.Context, account *Account, imageURL string) ([]byte, error) { @@ -925,26 +941,32 @@ func (c *SoraSDKClient) exchangeSessionToken(ctx context.Context, account *Accou return accessToken, expiresAt, nil } -// applyRecoveredToken 将恢复的 token 写入账号内存和数据库 +// applyRecoveredToken 将恢复的 token 写入账号内存和数据库。 +// 使用 copy-on-write 避免并发 map 写入 panic:创建新 map 后整体替换指针。 func (c *SoraSDKClient) applyRecoveredToken(ctx context.Context, account *Account, accessToken, refreshToken, expiresAt, sessionToken string) { if account == nil { return } - if account.Credentials == nil { - account.Credentials = make(map[string]any) + + // Copy-on-write: 复制旧 map 并写入新值,最后整体替换 + oldCreds := account.Credentials + newCreds := make(map[string]any, len(oldCreds)+4) + for k, v := range oldCreds { + newCreds[k] = v } if strings.TrimSpace(accessToken) != "" { - account.Credentials["access_token"] = accessToken + newCreds["access_token"] = accessToken } if strings.TrimSpace(refreshToken) != "" { - account.Credentials["refresh_token"] = refreshToken + newCreds["refresh_token"] = refreshToken } if strings.TrimSpace(expiresAt) != "" { - account.Credentials["expires_at"] = expiresAt + newCreds["expires_at"] = expiresAt } if strings.TrimSpace(sessionToken) != "" { - account.Credentials["session_token"] = sessionToken + newCreds["session_token"] = sessionToken } + account.Credentials = newCreds if c.accountRepo != nil { if err := c.accountRepo.Update(ctx, account); err != nil && c.debugEnabled() { diff --git a/backend/internal/service/sora_storage_router.go b/backend/internal/service/sora_storage_router.go new file mode 100644 index 0000000000..83e055f5df --- /dev/null +++ b/backend/internal/service/sora_storage_router.go @@ -0,0 +1,129 @@ +package service + +import ( + "context" + "fmt" + + "github.com/Wei-Shaw/sub2api/internal/pkg/logger" +) + +// SoraStorageRouter 根据激活 profile 的 provider 字段路由到对应存储实现。 +// 实现 SoraObjectStorage 接口。 +type SoraStorageRouter struct { + settingService *SettingService + s3Storage *SoraS3Storage + gdriveStorage SoraObjectStorage // 可为 nil(GDrive 未实现时) +} + +// NewSoraStorageRouter 创建存储路由。 +func NewSoraStorageRouter( + settingService *SettingService, + s3Storage *SoraS3Storage, + gdriveStorage SoraObjectStorage, +) *SoraStorageRouter { + return &SoraStorageRouter{ + settingService: settingService, + s3Storage: s3Storage, + gdriveStorage: gdriveStorage, + } +} + +// activeBackend 返回当前激活 profile 对应的存储后端。 +func (r *SoraStorageRouter) activeBackend(ctx context.Context) SoraObjectStorage { + if r.settingService == nil { + return r.s3Storage // 默认 S3 + } + + profile, err := r.settingService.GetActiveStorageProfile(ctx) + if err != nil || profile == nil { + return r.s3Storage // 默认 S3 + } + + switch profile.GetProvider() { + case SoraStorageTypeGDrive: + if r.gdriveStorage != nil { + return r.gdriveStorage + } + logger.LegacyPrintf("service.storage_router", "[StorageRouter] GDrive 后端未初始化,降级到 S3") + return r.s3Storage + default: + return r.s3Storage + } +} + +func (r *SoraStorageRouter) Enabled(ctx context.Context) bool { + backend := r.activeBackend(ctx) + if backend == nil { + return false + } + return backend.Enabled(ctx) +} + +func (r *SoraStorageRouter) IsHealthy(ctx context.Context) bool { + backend := r.activeBackend(ctx) + if backend == nil { + return false + } + return backend.IsHealthy(ctx) +} + +func (r *SoraStorageRouter) TestConnection(ctx context.Context) error { + backend := r.activeBackend(ctx) + if backend == nil { + return fmt.Errorf("no storage backend available") + } + return backend.TestConnection(ctx) +} + +func (r *SoraStorageRouter) UploadFromURL(ctx context.Context, userID int64, sourceURL string) (string, int64, string, error) { + backend := r.activeBackend(ctx) + if backend == nil { + return "", 0, "", fmt.Errorf("no storage backend available") + } + return backend.UploadFromURL(ctx, userID, sourceURL) +} + +func (r *SoraStorageRouter) DeleteObjects(ctx context.Context, objectKeys []string) error { + backend := r.activeBackend(ctx) + if backend == nil { + return fmt.Errorf("no storage backend available") + } + return backend.DeleteObjects(ctx, objectKeys) +} + +func (r *SoraStorageRouter) GetAccessURL(ctx context.Context, objectKey string) (string, error) { + backend := r.activeBackend(ctx) + if backend == nil { + return "", fmt.Errorf("no storage backend available") + } + return backend.GetAccessURL(ctx, objectKey) +} + +func (r *SoraStorageRouter) RefreshClient() { + if r.s3Storage != nil { + r.s3Storage.RefreshClient() + } + if r.gdriveStorage != nil { + r.gdriveStorage.RefreshClient() + } +} + +// RefreshAll 刷新所有后端客户端(用作配置变更回调)。 +func (r *SoraStorageRouter) RefreshAll() { + r.RefreshClient() +} + +func (r *SoraStorageRouter) StorageType() string { + // 不带 context 的方法,返回默认值 + // 真实的 StorageType 在 activeBackend 中动态确定 + return SoraStorageTypeS3 +} + +// StorageTypeWithContext 返回当前激活后端的存储类型。 +func (r *SoraStorageRouter) StorageTypeWithContext(ctx context.Context) string { + backend := r.activeBackend(ctx) + if backend == nil { + return SoraStorageTypeS3 + } + return backend.StorageType() +} diff --git a/backend/internal/service/sora_task.go b/backend/internal/service/sora_task.go new file mode 100644 index 0000000000..6b909b9b2b --- /dev/null +++ b/backend/internal/service/sora_task.go @@ -0,0 +1,63 @@ +package service + +import ( + "context" + "time" +) + +// ── 任务状态常量 ── + +const ( + SoraTaskQueued = "queued" + SoraTaskInProgress = "in_progress" + SoraTaskCompleted = "completed" + SoraTaskFailed = "failed" +) + +// ── 对象类型常量 ── + +const ( + SoraObjectVideo = "video" + SoraObjectCharacter = "character" + SoraObjectImage = "image" +) + +// SoraTask 表示一个 Sora 异步任务记录。 +type SoraTask struct { + ID string + AccountID int64 + APIKeyID *int64 + UpstreamTaskID string + ObjectType string + Model string + Prompt string + Status string + Progress int + VideoURL string // 原始上游 URL + StoredKey string // 存储后的 key(本地路径或 S3 key) + StorageType string // local / s3 / gdrive / 空 + ShareID string + CharacterInfo *SoraCharacter + ErrorMessage string + ErrorType string + RequestBody []byte + Seconds string + Size string + CreatedAt time.Time + CompletedAt *time.Time +} + +// SoraCharacter 角色信息。 +type SoraCharacter struct { + Username string `json:"username"` + DisplayName string `json:"display_name"` +} + +// SoraTaskRepository 持久层接口。 +type SoraTaskRepository interface { + Create(ctx context.Context, task *SoraTask) error + GetByID(ctx context.Context, id string) (*SoraTask, error) + GetByIDAndAPIKey(ctx context.Context, id string, apiKeyID int64) (*SoraTask, error) + Update(ctx context.Context, task *SoraTask) error + ListPending(ctx context.Context) ([]*SoraTask, error) +} diff --git a/backend/internal/service/sora_task_service.go b/backend/internal/service/sora_task_service.go new file mode 100644 index 0000000000..93197c44c8 --- /dev/null +++ b/backend/internal/service/sora_task_service.go @@ -0,0 +1,546 @@ +package service + +import ( + "bytes" + "context" + "crypto/rand" + "encoding/hex" + "encoding/json" + "fmt" + "io" + "net/http" + "strings" + "time" + + "github.com/Wei-Shaw/sub2api/internal/pkg/logger" +) + +// SoraTaskService manages Sora async task creation and status queries. +type SoraTaskService struct { + repo SoraTaskRepository + accountRepo AccountRepository + soraClient SoraClient + httpUpstream HTTPUpstream +} + +func NewSoraTaskService( + repo SoraTaskRepository, + accountRepo AccountRepository, + soraClient SoraClient, + httpUpstream HTTPUpstream, +) *SoraTaskService { + return &SoraTaskService{ + repo: repo, + accountRepo: accountRepo, + soraClient: soraClient, + httpUpstream: httpUpstream, + } +} + +// ── Request structs ── + +type CreateVideoRequest struct { + Model string `json:"model"` + Prompt string `json:"prompt"` + StyleID string `json:"style_id,omitempty"` + Orientation string `json:"orientation,omitempty"` + Image string `json:"image,omitempty"` + Video string `json:"video,omitempty"` + RemixTargetID string `json:"remix_target_id,omitempty"` + Size string `json:"size,omitempty"` +} + +type CreateImageRequest struct { + Model string `json:"model,omitempty"` + Prompt string `json:"prompt"` + Image string `json:"image,omitempty"` + Size string `json:"size,omitempty"` + ResponseFormat string `json:"response_format,omitempty"` + N int `json:"n,omitempty"` +} + +type EditImageRequest struct { + Image string `json:"image"` + Prompt string `json:"prompt"` + Model string `json:"model,omitempty"` + Size string `json:"size,omitempty"` + ResponseFormat string `json:"response_format,omitempty"` +} + +type RemixRequest struct { + Prompt string `json:"prompt"` +} + +// ── Response structs ── + +type SoraTaskResponse struct { + ID string `json:"id"` + Object string `json:"object"` + CreatedAt int64 `json:"created_at"` + Status string `json:"status"` + Model string `json:"model,omitempty"` + Prompt string `json:"prompt,omitempty"` + Progress int `json:"progress"` + VideoURL string `json:"video_url,omitempty"` + ShareID string `json:"share_id,omitempty"` + Seconds string `json:"seconds,omitempty"` + Size string `json:"size,omitempty"` + Character *SoraCharacter `json:"character,omitempty"` + URL string `json:"url,omitempty"` + CompletedAt *int64 `json:"completed_at,omitempty"` + Error *SoraTaskError `json:"error,omitempty"` +} + +type SoraTaskError struct { + Message string `json:"message"` + Type string `json:"type"` +} + +func TaskToResponse(t *SoraTask) *SoraTaskResponse { + resp := &SoraTaskResponse{ + ID: t.ID, + Object: t.ObjectType, + CreatedAt: t.CreatedAt.Unix(), + Status: t.Status, + Model: t.Model, + Prompt: t.Prompt, + Progress: t.Progress, + Seconds: t.Seconds, + Size: t.Size, + } + if t.VideoURL != "" { + resp.VideoURL = t.VideoURL + } + if t.ShareID != "" { + resp.ShareID = t.ShareID + } + if t.CharacterInfo != nil { + resp.Character = t.CharacterInfo + } + if t.CompletedAt != nil { + ts := t.CompletedAt.Unix() + resp.CompletedAt = &ts + } + if t.Status == SoraTaskFailed && t.ErrorMessage != "" { + resp.Error = &SoraTaskError{ + Message: t.ErrorMessage, + Type: t.ErrorType, + } + } + return resp +} + +// ── Task creation ── + +func (s *SoraTaskService) CreateVideoTask( + ctx context.Context, + apiKeyID int64, + account *Account, + req *CreateVideoRequest, + body []byte, +) (*SoraTask, error) { + modelCfg, ok := GetSoraModelConfig(req.Model) + + objectType := SoraObjectVideo + if req.Video != "" && req.Prompt == "" { + objectType = SoraObjectCharacter + } + + taskID := generateTaskID(objectType) + + seconds := "" + size := "" + if ok && modelCfg.Type == "video" { + seconds = fmt.Sprintf("%d", modelCfg.Frames/30) + if modelCfg.Orientation == "landscape" { + size = "1920x1080" + } else { + size = "1080x1920" + } + } + + task := &SoraTask{ + ID: taskID, + AccountID: account.ID, + APIKeyID: &apiKeyID, + ObjectType: objectType, + Model: req.Model, + Prompt: req.Prompt, + Status: SoraTaskQueued, + Progress: 0, + Seconds: seconds, + Size: size, + CreatedAt: time.Now(), + } + + if account.Type == AccountTypeAPIKey { + task.RequestBody = body + upstreamResp, err := s.forwardCreateToUpstream(ctx, account, "/v1/videos", body) + if err != nil { + return nil, fmt.Errorf("forward to upstream: %w", err) + } + s.applyUpstreamResponse(task, upstreamResp) + } else { + if s.soraClient == nil || !s.soraClient.Enabled() { + return nil, fmt.Errorf("sora SDK client not configured") + } + upstreamID, err := s.createViaSdk(ctx, account, req, modelCfg, objectType) + if err != nil { + return nil, fmt.Errorf("sdk create task: %w", err) + } + task.UpstreamTaskID = upstreamID + } + + if err := s.repo.Create(ctx, task); err != nil { + return nil, fmt.Errorf("save task: %w", err) + } + return task, nil +} + +func (s *SoraTaskService) CreateImageGeneration( + ctx context.Context, + apiKeyID int64, + account *Account, + req *CreateImageRequest, + body []byte, +) (*SoraTask, error) { + taskID := generateTaskID(SoraObjectImage) + + task := &SoraTask{ + ID: taskID, + AccountID: account.ID, + APIKeyID: &apiKeyID, + ObjectType: SoraObjectImage, + Model: req.Model, + Prompt: req.Prompt, + Status: SoraTaskQueued, + Progress: 0, + Size: req.Size, + CreatedAt: time.Now(), + } + + if account.Type == AccountTypeAPIKey { + task.RequestBody = body + upstreamResp, err := s.forwardCreateToUpstream(ctx, account, "/v1/images/generations", body) + if err != nil { + return nil, fmt.Errorf("forward to upstream: %w", err) + } + s.applyUpstreamResponse(task, upstreamResp) + } else { + if s.soraClient == nil || !s.soraClient.Enabled() { + return nil, fmt.Errorf("sora SDK client not configured") + } + modelCfg, _ := GetSoraModelConfig(req.Model) + width, height := modelCfg.Width, modelCfg.Height + if width == 0 { + width = 360 + } + if height == 0 { + height = 360 + } + upstreamID, err := s.soraClient.CreateImageTask(ctx, account, SoraImageRequest{ + Prompt: req.Prompt, + Width: width, + Height: height, + }) + if err != nil { + return nil, fmt.Errorf("sdk create image task: %w", err) + } + task.UpstreamTaskID = upstreamID + } + + if err := s.repo.Create(ctx, task); err != nil { + return nil, fmt.Errorf("save task: %w", err) + } + return task, nil +} + +func (s *SoraTaskService) GetTask(ctx context.Context, taskID string, apiKeyID int64) (*SoraTask, error) { + return s.repo.GetByIDAndAPIKey(ctx, taskID, apiKeyID) +} + +func (s *SoraTaskService) GetTaskByID(ctx context.Context, taskID string) (*SoraTask, error) { + return s.repo.GetByID(ctx, taskID) +} + +// ListPendingTasks returns tasks in queued or in_progress status. +func (s *SoraTaskService) ListPendingTasks(ctx context.Context) ([]*SoraTask, error) { + return s.repo.ListPending(ctx) +} + +// UpdateTask persists task state changes. +func (s *SoraTaskService) UpdateTask(ctx context.Context, task *SoraTask) error { + return s.repo.Update(ctx, task) +} + +func (s *SoraTaskService) GetAccountByID(ctx context.Context, accountID int64) (*Account, error) { + return s.accountRepo.GetByID(ctx, accountID) +} + +// ── Internal methods ── + +func (s *SoraTaskService) createViaSdk( + ctx context.Context, + account *Account, + req *CreateVideoRequest, + modelCfg SoraModelConfig, + objectType string, +) (string, error) { + if objectType == SoraObjectCharacter { + return "", fmt.Errorf("character creation via /v1/videos is not yet supported for OAuth accounts") + } + + orientation := modelCfg.Orientation + if req.Orientation != "" { + orientation = req.Orientation + } + + videoReq := SoraVideoRequest{ + Prompt: req.Prompt, + Orientation: orientation, + Frames: modelCfg.Frames, + Model: modelCfg.Model, + Size: modelCfg.Size, + RemixTargetID: req.RemixTargetID, + } + return s.soraClient.CreateVideoTask(ctx, account, videoReq) +} + +func (s *SoraTaskService) forwardCreateToUpstream( + ctx context.Context, + account *Account, + path string, + body []byte, +) (map[string]any, error) { + apiKey := account.GetCredential("api_key") + baseURL := account.GetBaseURL() + if apiKey == "" || baseURL == "" { + return nil, fmt.Errorf("account %d missing api_key or base_url", account.ID) + } + + upstreamURL := strings.TrimRight(baseURL, "/") + path + req, err := http.NewRequestWithContext(ctx, http.MethodPost, upstreamURL, bytes.NewReader(body)) + if err != nil { + return nil, err + } + req.Header.Set("Content-Type", "application/json") + req.Header.Set("Authorization", "Bearer "+apiKey) + + proxyURL := "" + if account.ProxyID != nil && account.Proxy != nil { + proxyURL = account.Proxy.URL() + } + + resp, err := s.httpUpstream.Do(req, proxyURL, account.ID, account.Concurrency) + if err != nil { + return nil, fmt.Errorf("upstream request: %w", err) + } + defer func() { _ = resp.Body.Close() }() + + respBody, err := io.ReadAll(io.LimitReader(resp.Body, 256*1024)) + if err != nil { + return nil, fmt.Errorf("read upstream response: %w", err) + } + if resp.StatusCode >= 400 { + return nil, &SoraUpstreamError{StatusCode: resp.StatusCode, Body: respBody} + } + + var result map[string]any + if err := json.Unmarshal(respBody, &result); err != nil { + return nil, fmt.Errorf("parse upstream response: %w", err) + } + return result, nil +} + +func (s *SoraTaskService) forwardGetToUpstream( + ctx context.Context, + account *Account, + path string, +) (map[string]any, error) { + apiKey := account.GetCredential("api_key") + baseURL := account.GetBaseURL() + if apiKey == "" || baseURL == "" { + return nil, fmt.Errorf("account %d missing api_key or base_url", account.ID) + } + + upstreamURL := strings.TrimRight(baseURL, "/") + path + req, err := http.NewRequestWithContext(ctx, http.MethodGet, upstreamURL, nil) + if err != nil { + return nil, err + } + req.Header.Set("Authorization", "Bearer "+apiKey) + + proxyURL := "" + if account.ProxyID != nil && account.Proxy != nil { + proxyURL = account.Proxy.URL() + } + + resp, err := s.httpUpstream.Do(req, proxyURL, account.ID, account.Concurrency) + if err != nil { + return nil, fmt.Errorf("upstream request: %w", err) + } + defer func() { _ = resp.Body.Close() }() + + respBody, err := io.ReadAll(io.LimitReader(resp.Body, 256*1024)) + if err != nil { + return nil, fmt.Errorf("read upstream response: %w", err) + } + if resp.StatusCode >= 400 { + return nil, fmt.Errorf("upstream error %d: %s", resp.StatusCode, string(respBody)) + } + + var result map[string]any + if err := json.Unmarshal(respBody, &result); err != nil { + return nil, fmt.Errorf("parse upstream response: %w", err) + } + return result, nil +} + +func (s *SoraTaskService) applyUpstreamResponse(task *SoraTask, resp map[string]any) { + if id, ok := resp["id"].(string); ok && id != "" { + task.UpstreamTaskID = id + } + if status, ok := resp["status"].(string); ok && status != "" { + task.Status = status + } + if progress, ok := resp["progress"].(float64); ok { + task.Progress = int(progress) + } + if videoURL, ok := resp["video_url"].(string); ok { + task.VideoURL = videoURL + } + if shareID, ok := resp["share_id"].(string); ok { + task.ShareID = shareID + } + if seconds, ok := resp["seconds"].(string); ok { + task.Seconds = seconds + } + if size, ok := resp["size"].(string); ok { + task.Size = size + } + if obj, ok := resp["object"].(string); ok && obj != "" { + task.ObjectType = obj + } +} + +// PollTask polls a single task's latest status (called by worker). +func (s *SoraTaskService) PollTask(ctx context.Context, task *SoraTask, account *Account) error { + if account.Type == AccountTypeAPIKey { + return s.pollUpstreamTask(ctx, task, account) + } + return s.pollSdkTask(ctx, task, account) +} + +func (s *SoraTaskService) pollUpstreamTask(ctx context.Context, task *SoraTask, account *Account) error { + upstreamID := task.UpstreamTaskID + if upstreamID == "" { + upstreamID = task.ID + } + + path := fmt.Sprintf("/v1/videos/%s", upstreamID) + resp, err := s.forwardGetToUpstream(ctx, account, path) + if err != nil { + logger.LegacyPrintf("service.sora_task", "[PollUpstream] task=%s error=%v", task.ID, err) + return err + } + + s.applyUpstreamResponse(task, resp) + now := time.Now() + if task.Status == SoraTaskCompleted || task.Status == SoraTaskFailed { + task.CompletedAt = &now + } + + if errObj, ok := resp["error"].(map[string]any); ok { + if msg, ok := errObj["message"].(string); ok { + task.ErrorMessage = msg + } + if typ, ok := errObj["type"].(string); ok { + task.ErrorType = typ + } + } + + return s.repo.Update(ctx, task) +} + +func (s *SoraTaskService) pollSdkTask(ctx context.Context, task *SoraTask, account *Account) error { + if s.soraClient == nil { + return fmt.Errorf("sora SDK client not configured") + } + upstreamID := task.UpstreamTaskID + if upstreamID == "" { + return fmt.Errorf("task %s has no upstream_task_id", task.ID) + } + + now := time.Now() + + switch task.ObjectType { + case SoraObjectImage: + status, err := s.soraClient.GetImageTask(ctx, account, upstreamID) + if err != nil { + return err + } + task.Progress = int(status.ProgressPct) + switch status.Status { + case "complete", "completed": + task.Status = SoraTaskCompleted + task.Progress = 100 + task.CompletedAt = &now + if len(status.URLs) > 0 { + task.VideoURL = status.URLs[0] + } + case "failed": + task.Status = SoraTaskFailed + task.CompletedAt = &now + task.ErrorMessage = status.ErrorMsg + task.ErrorType = "server_error" + default: + task.Status = SoraTaskInProgress + } + + case SoraObjectVideo, SoraObjectCharacter: + status, err := s.soraClient.GetVideoTask(ctx, account, upstreamID) + if err != nil { + return err + } + task.Progress = status.ProgressPct + switch status.Status { + case "complete", "completed": + task.Status = SoraTaskCompleted + task.Progress = 100 + task.CompletedAt = &now + if len(status.URLs) > 0 { + task.VideoURL = status.URLs[0] + } + if status.GenerationID != "" { + task.ShareID = status.GenerationID + } + case "failed": + task.Status = SoraTaskFailed + task.CompletedAt = &now + task.ErrorMessage = status.ErrorMsg + task.ErrorType = "server_error" + default: + task.Status = SoraTaskInProgress + } + + default: + return fmt.Errorf("unknown object type: %s", task.ObjectType) + } + + return s.repo.Update(ctx, task) +} + +func generateTaskID(objectType string) string { + b := make([]byte, 8) + if _, err := rand.Read(b); err != nil { + b = []byte(fmt.Sprintf("%x", time.Now().UnixNano())) + } + switch objectType { + case SoraObjectCharacter: + return "char_" + hex.EncodeToString(b) + case SoraObjectImage: + return "img_" + hex.EncodeToString(b) + default: + return "video_" + hex.EncodeToString(b) + } +} diff --git a/backend/internal/service/sora_task_worker.go b/backend/internal/service/sora_task_worker.go new file mode 100644 index 0000000000..059b671031 --- /dev/null +++ b/backend/internal/service/sora_task_worker.go @@ -0,0 +1,183 @@ +package service + +import ( + "context" + "sync" + "time" + + "github.com/Wei-Shaw/sub2api/internal/pkg/logger" +) + +// SoraTaskWorker polls unfinished Sora tasks in the background. +// When a task completes, it downloads media to configured storage if available. +type SoraTaskWorker struct { + taskService *SoraTaskService + accountRepo AccountRepository + objectStorage SoraObjectStorage + mediaStorage *SoraMediaStorage + interval time.Duration + pollTimeout time.Duration + stopCh chan struct{} + stopOnce sync.Once + wg sync.WaitGroup +} + +func NewSoraTaskWorker( + taskService *SoraTaskService, + accountRepo AccountRepository, + objectStorage SoraObjectStorage, + mediaStorage *SoraMediaStorage, + interval time.Duration, +) *SoraTaskWorker { + if interval <= 0 { + interval = 60 * time.Second + } + return &SoraTaskWorker{ + taskService: taskService, + accountRepo: accountRepo, + objectStorage: objectStorage, + mediaStorage: mediaStorage, + interval: interval, + pollTimeout: 30 * time.Second, + stopCh: make(chan struct{}), + } +} + +func (w *SoraTaskWorker) Start() { + if w == nil || w.taskService == nil { + return + } + w.wg.Add(1) + go func() { + defer w.wg.Done() + ticker := time.NewTicker(w.interval) + defer ticker.Stop() + + logger.LegacyPrintf("service.sora_task_worker", "[Start] polling interval=%s", w.interval) + w.pollAll() + + for { + select { + case <-ticker.C: + w.pollAll() + case <-w.stopCh: + return + } + } + }() +} + +func (w *SoraTaskWorker) Stop() { + if w == nil { + return + } + w.stopOnce.Do(func() { + close(w.stopCh) + }) + w.wg.Wait() +} + +func (w *SoraTaskWorker) pollAll() { + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + + tasks, err := w.taskService.ListPendingTasks(ctx) + if err != nil { + logger.LegacyPrintf("service.sora_task_worker", "[PollAll] list pending tasks error: %v", err) + return + } + if len(tasks) == 0 { + return + } + + logger.LegacyPrintf("service.sora_task_worker", "[PollAll] found %d pending tasks", len(tasks)) + + for _, task := range tasks { + select { + case <-w.stopCh: + return + default: + } + w.pollOne(ctx, task) + } +} + +func (w *SoraTaskWorker) pollOne(ctx context.Context, task *SoraTask) { + pollCtx, cancel := context.WithTimeout(ctx, w.pollTimeout) + defer cancel() + + account, err := w.accountRepo.GetByID(pollCtx, task.AccountID) + if err != nil { + logger.LegacyPrintf("service.sora_task_worker", + "[PollOne] task=%s get account=%d error: %v", task.ID, task.AccountID, err) + w.markTaskFailed(pollCtx, task, "account not found") + return + } + + if err := w.taskService.PollTask(pollCtx, task, account); err != nil { + logger.LegacyPrintf("service.sora_task_worker", + "[PollOne] task=%s poll error: %v", task.ID, err) + return + } + + logger.LegacyPrintf("service.sora_task_worker", + "[PollOne] task=%s status=%s progress=%d", task.ID, task.Status, task.Progress) + + if task.Status == SoraTaskCompleted && task.VideoURL != "" && task.StoredKey == "" { + w.tryStoreMedia(pollCtx, task) + } +} + +// tryStoreMedia downloads media to configured storage. +// Priority: S3 > local disk > keep upstream URL. +func (w *SoraTaskWorker) tryStoreMedia(ctx context.Context, task *SoraTask) { + mediaType := "video" + if task.ObjectType == SoraObjectImage { + mediaType = "image" + } + + if w.objectStorage != nil && w.objectStorage.Enabled(ctx) { + key, _, storageType, err := w.objectStorage.UploadFromURL(ctx, 0, task.VideoURL) + if err != nil { + logger.LegacyPrintf("service.sora_task_worker", + "[StoreMedia] task=%s object storage upload error: %v", task.ID, err) + } else { + task.StoredKey = key + task.StorageType = storageType + if updateErr := w.taskService.UpdateTask(ctx, task); updateErr != nil { + logger.LegacyPrintf("service.sora_task_worker", + "[StoreMedia] task=%s update stored_key error: %v", task.ID, updateErr) + } + return + } + } + + if w.mediaStorage != nil && w.mediaStorage.Enabled() { + stored, err := w.mediaStorage.StoreFromURLs(ctx, mediaType, []string{task.VideoURL}) + if err != nil { + logger.LegacyPrintf("service.sora_task_worker", + "[StoreMedia] task=%s local storage error: %v", task.ID, err) + return + } + if len(stored) > 0 && stored[0] != task.VideoURL { + task.StoredKey = stored[0] + task.StorageType = "local" + if updateErr := w.taskService.UpdateTask(ctx, task); updateErr != nil { + logger.LegacyPrintf("service.sora_task_worker", + "[StoreMedia] task=%s update stored_key error: %v", task.ID, updateErr) + } + } + } +} + +func (w *SoraTaskWorker) markTaskFailed(ctx context.Context, task *SoraTask, message string) { + now := time.Now() + task.Status = SoraTaskFailed + task.ErrorMessage = message + task.ErrorType = "server_error" + task.CompletedAt = &now + if updateErr := w.taskService.UpdateTask(ctx, task); updateErr != nil { + logger.LegacyPrintf("service.sora_task_worker", + "[PollOne] task=%s update failed: %v", task.ID, updateErr) + } +} diff --git a/backend/internal/service/token_refresh_service_test.go b/backend/internal/service/token_refresh_service_test.go index bdef0ed747..4b74395033 100644 --- a/backend/internal/service/token_refresh_service_test.go +++ b/backend/internal/service/token_refresh_service_test.go @@ -386,7 +386,7 @@ func TestTokenRefreshService_RefreshWithRetry_ClearsTempUnschedulable(t *testing err := service.refreshWithRetry(context.Background(), account, refresher) require.NoError(t, err) require.Equal(t, 1, repo.updateCalls) - require.Equal(t, 1, repo.clearTempCalls) // DB 清除 + require.Equal(t, 1, repo.clearTempCalls) // DB 清除 require.Equal(t, 1, tempCache.deleteCalls) // Redis 缓存也应清除 } diff --git a/backend/internal/service/user_service_test.go b/backend/internal/service/user_service_test.go index 05fe50560b..7f6c748fa6 100644 --- a/backend/internal/service/user_service_test.go +++ b/backend/internal/service/user_service_test.go @@ -47,8 +47,8 @@ func (m *mockUserRepo) RemoveGroupFromAllowedGroups(context.Context, int64) (int } func (m *mockUserRepo) AddGroupToAllowedGroups(context.Context, int64, int64) error { return nil } func (m *mockUserRepo) UpdateTotpSecret(context.Context, int64, *string) error { return nil } -func (m *mockUserRepo) EnableTotp(context.Context, int64) error { return nil } -func (m *mockUserRepo) DisableTotp(context.Context, int64) error { return nil } +func (m *mockUserRepo) EnableTotp(context.Context, int64) error { return nil } +func (m *mockUserRepo) DisableTotp(context.Context, int64) error { return nil } // --- mock: APIKeyAuthCacheInvalidator --- diff --git a/backend/internal/testutil/stubs.go b/backend/internal/testutil/stubs.go index bc572e1137..229985434a 100644 --- a/backend/internal/testutil/stubs.go +++ b/backend/internal/testutil/stubs.go @@ -100,6 +100,24 @@ func (c StubGatewayCache) RefreshSessionTTL(_ context.Context, _ int64, _ string func (c StubGatewayCache) DeleteSessionAccountID(_ context.Context, _ int64, _ string) error { return nil } +func (c StubGatewayCache) GetClientAffinityAccounts(_ context.Context, _ int64, _ string, _ time.Duration) ([]int64, error) { + return nil, nil +} +func (c StubGatewayCache) UpdateClientAffinity(_ context.Context, _ int64, _ string, _ int64, _ time.Duration) error { + return nil +} +func (c StubGatewayCache) GetAccountAffinityCountBatch(_ context.Context, _ int64, _ []int64, _ time.Duration) (map[int64]int64, error) { + return map[int64]int64{}, nil +} +func (c StubGatewayCache) GetAccountAffinityClientsBatch(_ context.Context, _ map[int64][]int64, _ time.Duration) (map[int64][]string, error) { + return map[int64][]string{}, nil +} +func (c StubGatewayCache) GetAccountAffinityClientsWithScores(_ context.Context, _ int64, _ []int64, _ time.Duration) ([]service.AffinityClient, error) { + return nil, nil +} +func (c StubGatewayCache) ClearAccountAffinity(_ context.Context, _ int64, _ []int64) error { + return nil +} // ============================================================ // StubSessionLimitCache — service.SessionLimitCache 的空实现 diff --git a/backend/migrations/070_add_sora_tasks.sql b/backend/migrations/070_add_sora_tasks.sql new file mode 100644 index 0000000000..3a484976d9 --- /dev/null +++ b/backend/migrations/070_add_sora_tasks.sql @@ -0,0 +1,36 @@ +-- +migrate Up + +-- Sora 异步任务表:记录视频/角色/图片生成任务状态 +CREATE TABLE IF NOT EXISTS sora_tasks ( + id VARCHAR(64) PRIMARY KEY, -- 对外 ID (video_xxx / char_xxx / img_xxx) + account_id BIGINT NOT NULL, -- 使用的账号 ID + api_key_id BIGINT, -- 调用方 API Key ID + upstream_task_id VARCHAR(128) NOT NULL DEFAULT '', -- 上游任务 ID + object_type VARCHAR(16) NOT NULL DEFAULT 'video', -- video / character / image + model VARCHAR(64) NOT NULL, -- 请求模型名 + prompt TEXT NOT NULL DEFAULT '', -- 生成提示词 + status VARCHAR(16) NOT NULL DEFAULT 'queued', -- queued / in_progress / completed / failed + progress INT NOT NULL DEFAULT 0, -- 进度 0-100 + video_url TEXT NOT NULL DEFAULT '', -- 原始上游下载 URL + stored_key TEXT NOT NULL DEFAULT '', -- 存储后的 key(本地相对路径或 S3 object key) + storage_type VARCHAR(16) NOT NULL DEFAULT '', -- 存储类型:local / s3 / gdrive / 空=未存储 + share_id VARCHAR(128) NOT NULL DEFAULT '', -- 可分享 ID + character_info JSONB, -- 角色信息 {username, display_name} + error_message TEXT NOT NULL DEFAULT '', -- 错误信息 + error_type VARCHAR(64) NOT NULL DEFAULT '', -- 错误类型 + request_body JSONB, -- 原始请求体(用于上游重放) + seconds VARCHAR(8) NOT NULL DEFAULT '', -- 视频时长 + size VARCHAR(16) NOT NULL DEFAULT '', -- 分辨率 (如 1920x1080) + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + completed_at TIMESTAMPTZ +); + +-- 仅索引未完成任务,供后台 worker 轮询 +CREATE INDEX IF NOT EXISTS idx_sora_tasks_pending ON sora_tasks(status) WHERE status IN ('queued', 'in_progress'); + +-- 按 API Key 查询 +CREATE INDEX IF NOT EXISTS idx_sora_tasks_api_key ON sora_tasks(api_key_id) WHERE api_key_id IS NOT NULL; + +-- +migrate Down + +DROP TABLE IF EXISTS sora_tasks; diff --git a/deploy/docker-compose.local.yml b/deploy/docker-compose.local.yml index 0ef397df8d..d404ac0be2 100644 --- a/deploy/docker-compose.local.yml +++ b/deploy/docker-compose.local.yml @@ -154,7 +154,7 @@ services: networks: - sub2api-network healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8080/health"] + test: ["CMD", "wget", "-q", "-T", "5", "-O", "/dev/null", "http://localhost:8080/health"] interval: 30s timeout: 10s retries: 3 diff --git a/deploy/docker-compose.standalone.yml b/deploy/docker-compose.standalone.yml index 7676fb9766..df0ccfccc3 100644 --- a/deploy/docker-compose.standalone.yml +++ b/deploy/docker-compose.standalone.yml @@ -94,7 +94,7 @@ services: - GEMINI_CLI_OAUTH_CLIENT_SECRET=${GEMINI_CLI_OAUTH_CLIENT_SECRET:-} - ANTIGRAVITY_OAUTH_CLIENT_SECRET=${ANTIGRAVITY_OAUTH_CLIENT_SECRET:-} healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8080/health"] + test: ["CMD", "wget", "-q", "-T", "5", "-O", "/dev/null", "http://localhost:8080/health"] interval: 30s timeout: 10s retries: 3 diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml index 8715d75df6..d9c2218022 100644 --- a/deploy/docker-compose.yml +++ b/deploy/docker-compose.yml @@ -146,7 +146,7 @@ services: networks: - sub2api-network healthcheck: - test: ["CMD", "curl", "-f", "http://localhost:8080/health"] + test: ["CMD", "wget", "-q", "-T", "5", "-O", "/dev/null", "http://localhost:8080/health"] interval: 30s timeout: 10s retries: 3 diff --git a/frontend/src/api/admin/accounts.ts b/frontend/src/api/admin/accounts.ts index 23d50d3acb..c9584ed5ea 100644 --- a/frontend/src/api/admin/accounts.ts +++ b/frontend/src/api/admin/accounts.ts @@ -618,6 +618,18 @@ export async function batchRefresh(accountIds: number[]): Promise { + const { data } = await apiClient.get<{ client_id: string; last_active: string }[]>( + `/admin/accounts/${id}/affinity-clients` + ) + return data +} + export const accountsAPI = { list, listWithEtag, @@ -654,7 +666,8 @@ export const accountsAPI = { importData, getAntigravityDefaultModelMapping, batchClearError, - batchRefresh + batchRefresh, + getAffinityClients } export default accountsAPI diff --git a/frontend/src/api/admin/settings.ts b/frontend/src/api/admin/settings.ts index 2b156ea1a0..9915979ae8 100644 --- a/frontend/src/api/admin/settings.ts +++ b/frontend/src/api/admin/settings.ts @@ -351,7 +351,7 @@ export async function updateBetaPolicySettings( return data } -// ==================== Sora S3 Settings ==================== +// ==================== Sora Storage Settings ==================== export interface SoraS3Settings { enabled: boolean @@ -370,7 +370,9 @@ export interface SoraS3Profile { profile_id: string name: string is_active: boolean + provider: string // "s3" | "gdrive" enabled: boolean + access_mode: string // "direct" | "proxy" endpoint: string region: string bucket: string @@ -381,6 +383,13 @@ export interface SoraS3Profile { cdn_url: string default_storage_quota_bytes: number updated_at: string + // Google Drive fields + auth_type: string // "oauth2" | "service_account" + client_id: string + client_secret_configured: boolean + refresh_token_configured: boolean + service_account_configured: boolean + folder_id: string } export interface ListSoraS3ProfilesResponse { @@ -406,30 +415,47 @@ export interface CreateSoraS3ProfileRequest { profile_id: string name: string set_active?: boolean + provider?: string enabled: boolean - endpoint: string - region: string - bucket: string - access_key_id: string + access_mode?: string + endpoint?: string + region?: string + bucket?: string + access_key_id?: string secret_access_key?: string - prefix: string - force_path_style: boolean - cdn_url: string - default_storage_quota_bytes: number + prefix?: string + force_path_style?: boolean + cdn_url?: string + default_storage_quota_bytes?: number + // Google Drive fields + auth_type?: string + client_id?: string + client_secret?: string + refresh_token?: string + service_account_json?: string + folder_id?: string } export interface UpdateSoraS3ProfileRequest { name: string enabled: boolean - endpoint: string - region: string - bucket: string - access_key_id: string + access_mode?: string + endpoint?: string + region?: string + bucket?: string + access_key_id?: string secret_access_key?: string - prefix: string - force_path_style: boolean - cdn_url: string - default_storage_quota_bytes: number + prefix?: string + force_path_style?: boolean + cdn_url?: string + default_storage_quota_bytes?: number + // Google Drive fields + auth_type?: string + client_id?: string + client_secret?: string + refresh_token?: string + service_account_json?: string + folder_id?: string } export interface TestSoraS3ConnectionRequest { @@ -446,44 +472,115 @@ export interface TestSoraS3ConnectionRequest { default_storage_quota_bytes?: number } +export interface StartGDriveOAuthRequest { + client_id: string + client_secret: string + redirect_uri: string +} + +export interface StartGDriveOAuthResponse { + auth_url: string + state: string +} + +export interface ExchangeGDriveOAuthCodeRequest { + client_id: string + client_secret: string + redirect_uri: string + code: string + profile_id?: string +} + +export interface ExchangeGDriveOAuthCodeResponse { + refresh_token: string + message: string +} + export async function getSoraS3Settings(): Promise { - const { data } = await apiClient.get('/admin/settings/sora-s3') + const { data } = await apiClient.get('/admin/settings/sora-storage') return data } export async function updateSoraS3Settings(settings: UpdateSoraS3SettingsRequest): Promise { - const { data } = await apiClient.put('/admin/settings/sora-s3', settings) + const { data } = await apiClient.put('/admin/settings/sora-storage', settings) return data } export async function testSoraS3Connection( settings: TestSoraS3ConnectionRequest ): Promise<{ message: string }> { - const { data } = await apiClient.post<{ message: string }>('/admin/settings/sora-s3/test', settings) + const { data } = await apiClient.post<{ message: string }>('/admin/settings/sora-storage/test', settings) return data } export async function listSoraS3Profiles(): Promise { - const { data } = await apiClient.get('/admin/settings/sora-s3/profiles') + const { data } = await apiClient.get('/admin/settings/sora-storage/profiles') return data } export async function createSoraS3Profile(request: CreateSoraS3ProfileRequest): Promise { - const { data } = await apiClient.post('/admin/settings/sora-s3/profiles', request) + const { data } = await apiClient.post('/admin/settings/sora-storage/profiles', request) return data } export async function updateSoraS3Profile(profileID: string, request: UpdateSoraS3ProfileRequest): Promise { - const { data } = await apiClient.put(`/admin/settings/sora-s3/profiles/${profileID}`, request) + const { data } = await apiClient.put(`/admin/settings/sora-storage/profiles/${profileID}`, request) return data } export async function deleteSoraS3Profile(profileID: string): Promise { - await apiClient.delete(`/admin/settings/sora-s3/profiles/${profileID}`) + await apiClient.delete(`/admin/settings/sora-storage/profiles/${profileID}`) } export async function setActiveSoraS3Profile(profileID: string): Promise { - const { data } = await apiClient.post(`/admin/settings/sora-s3/profiles/${profileID}/activate`) + const { data } = await apiClient.post(`/admin/settings/sora-storage/profiles/${profileID}/activate`) + return data +} + +export async function startGDriveOAuth(request: StartGDriveOAuthRequest): Promise { + const { data } = await apiClient.post('/admin/settings/sora-storage/gdrive-oauth/start', request) + return data +} + +export async function exchangeGDriveOAuthCode(request: ExchangeGDriveOAuthCodeRequest): Promise { + const { data } = await apiClient.post('/admin/settings/sora-storage/gdrive-oauth/callback', request) + return data +} + +export interface TestGDriveStorageResponse { + status: string + quota_limit_bytes?: number + quota_used_bytes?: number + uploaded_file_id?: string + uploaded_file_name?: string + uploaded_file_size?: number + access_url?: string + web_view_link?: string + deleted?: boolean + delete_warning?: string +} + +export async function testGDriveStorage(): Promise { + const { data } = await apiClient.post('/admin/settings/sora-storage/gdrive-test') + return data +} + +export interface GDriveQuotaInfo { + limit_bytes: number + used_bytes: number +} + +export interface StorageVideoStats { + [type: string]: { completed: number; in_progress: number } +} + +export async function getGDriveQuota(): Promise { + const { data } = await apiClient.get('/admin/settings/sora-storage/gdrive-quota') + return data +} + +export async function getStorageVideoStats(): Promise { + const { data } = await apiClient.get('/admin/settings/sora-storage/video-stats') return data } @@ -508,7 +605,12 @@ export const settingsAPI = { createSoraS3Profile, updateSoraS3Profile, deleteSoraS3Profile, - setActiveSoraS3Profile + setActiveSoraS3Profile, + startGDriveOAuth, + exchangeGDriveOAuthCode, + testGDriveStorage, + getGDriveQuota, + getStorageVideoStats } export default settingsAPI diff --git a/frontend/src/components/account/AccountCapacityCell.vue b/frontend/src/components/account/AccountCapacityCell.vue index b077264d39..2e00c51ceb 100644 --- a/frontend/src/components/account/AccountCapacityCell.vue +++ b/frontend/src/components/account/AccountCapacityCell.vue @@ -1,76 +1,35 @@