feat: enhance sync workflows and add WES configuration exports
Add the React console, strengthen synchronization and retry handling, and export WES MySQL and MongoDB configuration packages.
This commit is contained in:
曾志威
@@ -31,6 +31,7 @@ config.yaml
|
||||
# Frontend dependencies and build output
|
||||
node_modules/
|
||||
web/dist/
|
||||
frontend/browser-artifacts/
|
||||
|
||||
# Go dependencies are restored from go.mod/go.sum
|
||||
vendor/
|
||||
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
FROM golang:1.23-alpine AS build
|
||||
FROM golang:1.26-alpine AS build
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
COPY cmd ./cmd
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
# syncdb
|
||||
|
||||
单文件 MySQL / MongoDB 数据同步工作台。Go 服务内嵌原生 Web 控制台，无需 Node.js、独立前端服务或 CDN；界面支持中文、英文、日文。
|
||||
单文件 MySQL / MongoDB 数据同步工作台。Go 服务内嵌 React + Ant Design 控制台，运行时无需 Node.js、独立前端服务或 CDN；界面支持中文、英文、日文。
|
||||
|
||||
## 启动
|
||||
|
||||
Windows 安装 Go 1.23 或以上版本后，在项目根目录双击 `start-dev.cmd`，或运行：
|
||||
Windows 安装 Go 1.26 或以上版本后，在项目根目录双击 `start-dev.cmd`，或运行：
|
||||
|
||||
```powershell
|
||||
.\start-dev.cmd
|
||||
@@ -29,7 +29,7 @@ go build -trimpath -ldflags="-s -w" -o dist/syncdb ./cmd/syncdb
|
||||
|
||||
- 首次启动会在数据目录生成 `session-token`。其他设备使用运行机器的局域网 IP 和控制台显示的端口访问，输入这个文件中的令牌。
|
||||
- Windows 自动打开页面时通过 URL fragment 传递令牌，前端立即移除 fragment 并换取 HttpOnly 会话 Cookie；令牌不写入 localStorage。
|
||||
- 所有业务 API、SSE 和导出下载均需认证。API 客户端可发送 `Authorization: Bearer <token>`；健康检查公开。
|
||||
- 本机通过 `127.0.0.1`、`localhost` 或 `[::1]` 访问无需令牌，直接进入工作台；同时检查连接来源与访问地址。局域网访问的业务 API、SSE 和导出下载仍需认证。API 客户端可发送 `Authorization: Bearer <token>`；健康检查公开。
|
||||
- Cookie 有效期 12 小时。退出登录会清除当前浏览器 Cookie，不会撤销已复制的 Cookie；更换访问令牌并重启可使旧 Cookie 失效。
|
||||
- HTTP 不加密浏览器传输的令牌、密码和数据。局域网也需可信，远程部署请配置 HTTPS。数据库 TLS 不保护浏览器到 Web 服务的连接。
|
||||
|
||||
@@ -46,7 +46,7 @@ go build -trimpath -ldflags="-s -w" -o dist/syncdb ./cmd/syncdb
|
||||
| 环境变量 | 默认值 | 作用 |
|
||||
| --- | --- | --- |
|
||||
| `SYNCDB_ADDR` | `:0` | Web 监听地址，0 表示系统分配端口 |
|
||||
| `SYNCDB_DATA_DIR` | `data` | 元数据、密钥、令牌和导出目录，相对可执行文件所在目录 |
|
||||
| `SYNCDB_DATA_DIR` | 用户目录下的 `.syncdb` | 元数据、密钥、令牌和导出目录；显式配置相对路径时相对可执行文件所在目录 |
|
||||
| `SYNCDB_WORKERS` | `2` | 同步/导出共享并发上限，1–16 |
|
||||
| `SYNCDB_EXPORT_RETENTION_HOURS` | `24` | 导出文件保留时长，1–8760 小时 |
|
||||
| `SYNCDB_ENABLE_HIGH_RISK_OPERATIONS` | `false` | 是否开放复制初始化和计划内切换；修改后需重启 |
|
||||
@@ -55,9 +55,9 @@ go build -trimpath -ldflags="-s -w" -o dist/syncdb ./cmd/syncdb
|
||||
|
||||
YAML 支持 `server.addr`、`server.data_dir`、`server.workers`、`export.retention_hours`，未知字段会报错。参考 `config.example.yaml`；示例中的 `server.addr: ":0"` 是显式选择动态端口，不代表直接运行 EXE 的默认值。Docker 或桥接端口映射部署时应显式指定 `SYNCDB_ADDR` 和对应的发布端口。
|
||||
|
||||
数据目录的相对路径（包括 YAML 与环境变量配置）统一相对可执行文件目录解析，不随启动工作目录变化。例如 `dist/syncdb.exe` 默认使用 `dist/data`，脚本启动、双击和从其他目录运行读取同一份数据；绝对路径仍按指定位置使用。启动日志显示实际数据目录。`--config` 文件路径仍按命令行工作目录解析。
|
||||
显式指定数据目录时，相对路径（包括 YAML 与环境变量配置）统一相对可执行文件目录解析，不随启动工作目录变化；绝对路径按指定位置使用。未指定时使用用户目录下的 `.syncdb`。启动日志显示实际数据目录。`--config` 文件路径仍按命令行工作目录解析。
|
||||
|
||||
升级前若数据位于项目根目录的 `data`，请停止旧实例，将其中 `store.json`、`store.key`、`session-token` 及 `exports` 一起复制到 EXE 旁的 `data`，不要覆盖另一套已有数据。移动程序时同时保留旁边的数据目录。使用 `go run` 时程序在临时目录，需显式设置稳定的绝对 `SYNCDB_DATA_DIR`；开发时推荐 `start-dev.cmd`。
|
||||
默认数据目录固定为用户目录下的 `.syncdb`，Windows 示例为 `C:\Users\zoe\.syncdb`，移动程序或使用 `go run` 不会改变默认数据位置。升级前若使用旧数据目录，请停止旧实例，将整个旧目录复制到用户目录下的 `.syncdb`，同时保留 `store.json`、`store.key`、`session-token`、`exports` 和 SQLite 文件，不要覆盖另一套已有数据。仍可使用 `SYNCDB_DATA_DIR` 或 YAML 显式指定其他目录。
|
||||
|
||||
Docker 镜像显式设置 `SYNCDB_DATA_DIR=/data`，可将持久卷挂载到 `/data`。
|
||||
|
||||
@@ -72,6 +72,7 @@ Docker 镜像显式设置 `SYNCDB_DATA_DIR=/data`，可将持久卷挂载到 `/d
|
||||
- 支持关闭 TLS 或开启并验证证书；不跳过证书校验，不自动降级。TLS 关闭仅用于可信开发网络。
|
||||
- 密码从不随创建、编辑、列表、详情或 SSE 返回；编辑时密码留空保留原值。API 更新省略 `password` 保留原值，显式传空字符串可清空。
|
||||
- MySQL 展示列、主键、索引、外键及建表语句；MongoDB 展示集合选项、索引和采样推断字段。MongoDB 推断结构不是强制 Schema。
|
||||
- MySQL 数据库/表名下拉、对象列表、结构检查和写入结构缓存在 information_schema 元数据不可访问时回退到 SHOW 命令。名称下拉最多返回 512 项并报告截断，表名查询明确限定所选数据库且只返回普通表。账号仍需相应 SHOW/读取权限；索引信息不可读取时明确失败，避免误判唯一键。回退检查保留完整建表 SQL，外键关系无法单独列出时页面会提示查看 SQL。
|
||||
- 被任务/导出引用的数据源不能删除，正在使用的数据源不能编辑。
|
||||
|
||||
测试连接、加载库表名称或启动 Binlog 采集返回 `502` 且错误码为 `mysqlAuthFailed` 时，表示 MySQL 拒绝数据源账号登录。请编辑对应数据源，核对用户名、密码及 MySQL 账号允许的客户端 host；如需修正密码，必须重新填写后保存，编辑时留空会保留旧密码。Binlog 的库表筛选不会更换连接账号。`mysqlDatabaseDenied` 表示数据库授权不足，`binlogPermissionDenied` 表示日志读取权限不足，需分别处理。页面登录返回 `invalidToken` 才需要检查 syncdb 访问令牌。
|
||||
@@ -82,21 +83,93 @@ Docker 镜像显式设置 `SYNCDB_DATA_DIR=/data`，可将持久卷挂载到 `/d
|
||||
2. 浏览源/目标对象，必要时先使用结构对比和安全结构计划创建目标表/集合。目标对象必须存在。
|
||||
3. 创建任务，选择对象、冲突策略、过滤条件等。**默认 dry-run**，先预演检查。
|
||||
4. 确认配置后关闭 dry-run，在运行对话框明确确认写入。
|
||||
5. 执行记录通过 SSE 实时更新，连接失败时退回轮询；可取消运行中的任务。
|
||||
5. 执行记录通过 SSE 实时更新，连接失败时退回轮询；可取消运行中的任务。遇到耗时大表时，可在执行记录点击“跳过当前表”，中断该表读写并继续下一张表；已经写入的数据不回滚，跳过表名保存在执行记录中。跳过增量表不会推进水位，下次运行仍会读取该表。
|
||||
|
||||
支持 MySQL→MySQL、MongoDB→MongoDB，以及通过字段映射/转换实现的跨类型同步。支持：
|
||||
|
||||
- 全量、带条件范围读取，以及用户配置水位的字段增量读取。
|
||||
- 同步分为表结构和数据两个阶段。结构并发数 `schema_workers` 默认为 4，可设置为 1–16；所有结构处理结束后才进入数据阶段，数据按任务对象顺序执行。外键表在普通表之后按依赖尝试创建，结构失败的表保留重试入口。表同步状态页显示两个阶段的逐表结果、结构并发进度和失败数量。
|
||||
- 网页实时更新仅在状态变化时推送完整快照，不变时每 15 秒发送轻量心跳；快照列表顺序固定。前端忽略不变快照，不重复渲染列表，心跳不触发页面刷新。隐藏标签页暂停实时连接和全局轮询，重新显示时恢复并读取最新状态，减少空闲浏览器 CPU 占用。
|
||||
- MySQL→MySQL 在结构阶段核对映射、默认值和忽略规则所需的目标字段。缺字段的表标记结构失败并跳过数据读取，其他表继续；修复目标结构后可单独重试。自动克隆源表结构也会先检查是否能接收转换后的字段。字段值的类型转换及数据库约束仍在写入时验证。
|
||||
- 插入、upsert 和 replace 策略还会检查目标非空、无默认值且非自增/生成的必填列是否有对应源字段、映射或任务默认值；缺失时在结构阶段失败，避免先读取大表。update 只更新已有记录，因此允许省略目标额外必填列。对象结构页展示“插入必填”标记。
|
||||
- 源列名含点号时，转换或脱敏可能保留原列并产生嵌套根字段。结构校验按“映射、默认值、转换、忽略、脱敏”的实际顺序检查这些目标字段；忽略嵌套根字段之后执行脱敏，仍可能重新产生该字段。
|
||||
- `insert`/`fail`：插入，冲突失败；`skip`：插入，重复键跳过；`update`：只更新已有记录；`upsert`：更新或插入；`replace`：替换匹配记录内容，不使用删除再插入。
|
||||
- 批次 1–1000 条、任务总上限 1–1,000,000 条、超时 1–3600 秒；不会用一个长事务包裹整个任务。
|
||||
- MySQL 主键匹配的 upsert 可批量写入；存在其他唯一索引时，整数主键批次在短事务内锁定主键，再分别更新已有记录和插入缺失记录，避免通过其他唯一索引更新错行。其他情况仍使用逐行写入。兼容未转发 `CLIENT_FOUND_ROWS` 的代理，无变化的更新会核实匹配记录后计为成功。
|
||||
- MySQL 全量同步可对单列整数键使用 JSON 聚合批量读取，并用键范围分页保留整数、二进制及时间值；只读代理拒绝首批 JSON 查询时自动回退到普通 SELECT。聚合结果超限时缩小尚未交付的页，核对返回行数后才交付，不重放此前批次。普通同步不持有跨表一致性快照，源库持续新增、修改或删除数据时，结束后的源目标差异需结合读取时间核验；upsert 保留目标独有记录。
|
||||
- 批次 1–1000 条、任务总上限 1–1,000,000 条；`max_rows: -1` 表示读取全部数据。超时 1–86400 秒分别应用于每张表，数据阶段读取前的表结构检查也计入该表超时，并支持“跳过当前表”；单表超时记录失败并继续后续表。不会用一个长事务包裹整个任务。
|
||||
- 执行记录的“读取”统计源端已交付的记录，包含转换失败及部分写入的批次；“写入”和“跳过”统计目标处理结果。失败批次同样消耗任务读取上限，后续表仅使用剩余额度，避免失败后超额扫描。重试继续累计原记录的读取和写入数量。
|
||||
- 表同步状态页显示每张表最近一次实际执行的读取、写入和跳过数量，支持按数量排序；未执行的重试表保留之前的数量，重启后仍可查看。旧记录未保存的逐表数量显示为“—”，不推算为零。长表名省略显示，可悬停查看和复制完整名称。
|
||||
- 原生列表排序直接比较十进制文本，避免大整数或高精度小数转为浮点数后被判为相等；此规则保留输入文本的精度，不恢复接口或调用方已舍入的数值。实时刷新保留用户选定的排序。
|
||||
- 最多 3 次有限重试，仅用于允许重试的策略及尚未尝试写入的临时连接失败。部分写入或结果不确定时停止，不盲目重放整批。
|
||||
- 配置至少 60 秒的运行间隔可定时执行；同一目标数据库的同步任务不并发写入。
|
||||
- 任务编辑、删除及取消；运行中不允许编辑/删除。生产环境数据源禁止作为同步目标。
|
||||
- 表同步状态支持单表和批量重试，结果更新原执行记录；批量中单表失败后继续处理其他表，重试不推进增量水位。执行结束回调重复触发时不会重复累计重试数量。
|
||||
- 重试入口按当前任务对象顺序执行所选表，已从任务中移除的历史失败表仍可重试，并按提交顺序追加到末尾；不再按表大小排序。提交前重新校验实际重试对象和目标名称，增量模式不接受多表重试，校验失败不改变执行状态。
|
||||
- 后端不会因旧跳过列表而接受已成功、运行中或重试中的表；等待中的执行/任务及同一任务的活动执行也会阻止重试。重复或并发提交只会启动一次重试，不重复写入该表。
|
||||
- 重试需先完成原生界面的确认；确认组件不可用时显示错误，不直接提交。请求失败后保留所选表并解除按钮加载状态，便于修复问题后再次提交。
|
||||
- 单表和批量重试携带确认时的任务 `revision`，后端在启动前原子校验版本。配置、运行状态或增量水位已改变时拒绝旧请求，不修改原执行结果；不携带版本的旧 API 调用仍兼容。退出登录期间不提交重试，切换页面或会话后忽略迟到响应。
|
||||
- 批量重试期间，表状态、成功/失败数量和状态筛选实时显示当前重试的逐表结果；等待或正在处理的表显示重试中。失败详情查询当前尝试的目标表和 SQL 审计，不使用此前尝试的旧错误。整批结束后仍更新原执行记录，不新增可见记录。
|
||||
- 执行记录及概览中的累计读取、写入、跳过数量实时包含当前重试，已成功的重试表从待补同步数量中移除。重试未结束时不显示历史结束时间；结束后显示合并后的原记录数量，不重复累加内部重试记录。
|
||||
- 重试进入数据阶段后，当前表行和原执行记录行均可跳过该表，操作绑定重试执行编号。确认期间表名或执行状态改变时不提交；结构阶段和等待状态不显示跳过入口。
|
||||
- 顶部与表行的跳过入口共享当前执行的提交状态，确认或请求未结束时不重复提交；取消和请求失败后允许再次操作。页面或登录会话改变后，旧请求的返回结果不覆盖新页面提示。
|
||||
- 任务启动、编辑、删除和重试使用相同的活动检查：任务或关联执行处于运行中、等待中时拒绝修改，即使任务摘要状态尚未更新。界面同步禁用对应操作，只有实际运行中的任务才提供停止操作。
|
||||
- 数据源编辑和删除同样识别等待任务及关联活动执行；界面提示占用或引用原因，已打开的编辑窗口保存前会重新检查。已结束任务或恢复记录的目标数据源仍属于引用，不能直接删除。
|
||||
- 批量选择跨搜索、状态筛选和翻页保留，筛选外的已选数量单独提示；“选择筛选内可重试表”只增加当前筛选内的表。实时更新后，成功或正在重试的表会自动移出选择，不会在再次失败时意外恢复选中。
|
||||
- 普通执行和重试后的原记录按全部表结果汇总：全部成功为“完成”，全部表实际失败为“失败”，成功与失败混合、主动跳过或达到行数上限为“部分完成”；服务重启恢复使用相同规则，并保留未完成表的失败原因及重试入口。
|
||||
- 重试结束后任务状态与原执行记录的全部表汇总一致，不会因所选表成功而掩盖其他失败表；内部重试记录仍保留本次尝试的独立结果。重启时恢复中断任务的执行结果，已编辑的草稿和已结束任务不受历史记录影响。
|
||||
- 执行记录保存当次使用的源表到目标表名称映射。失败详情按实际目标表定位写入 SQL；后续编辑任务或重启不会改变历史执行的目标名称。重试仅更新已尝试表的映射；旧记录缺少映射时使用当前任务配置兼容查询。
|
||||
- 原执行记录同时保存每张表最近一次实际重试的执行编号；内部重试历史被清理后，失败详情仍按该编号查询独立 SQL 审计，避免误显示更早的失败 SQL。未执行的表保留此前编号，旧记录仍使用历史重试信息兼容查询。
|
||||
- 结构阶段自动创建 MySQL 表或 MongoDB 集合会生成 `SCHEMA` 审计，记录实际建表 SQL/创建命令、目标对象、执行结果及安全错误码，并关联任务和执行记录。仅检查失败、结构不兼容或 dry-run 未执行创建时不会虚构写入审计。
|
||||
- 表失败详情只展示失败的 SQL 审计。长语句若被审计存储截断，会明确显示截断提示及“复制 SQL 片段”；完整语句需要从数据库日志或原始结构核对，页面不会把 dry-run 语句当作失败 SQL。
|
||||
- 已打开的失败详情会随新失败结果刷新；表成功或记录被删除后关闭并取消查询，旧请求返回不会重新打开详情。失败信息未变化和切换语言时不重复查询 SQL。
|
||||
- 取消或提前失败时，正在处理的结构/数据状态会收尾为失败；尚未执行的表保留“未执行”状态并立即开放重试，无需重启服务。取消重试时同样更新原记录，保留此前成功表及增量水位。
|
||||
- 表同步状态页可直接停止结构同步或数据同步，包括当前重试。停止确认固定点击时的执行编号，并在提交前重新检查；接口也校验编号，避免确认期间结束的旧执行误取消新执行。切换页面或登录会话后，停止请求的迟到结果不会触发提示或页面刷新。等待中的任务或同一任务的活动执行会禁用重试入口。
|
||||
- 数据阶段每张表结束后立即释放取消上下文和跳过句柄；任务只保留当前表的异常清理，不积累已完成表的延迟清理函数。适配器异常退出时同样清理当前表，保留此前完成的写入与状态。
|
||||
- 结构阶段在源端检查和目标端准备返回后再次检查超时/取消，即使适配器迟到返回成功也不会将过期操作记为成功。源端检查已过期时不再调用目标端；超时表跳过数据阶段，其他表继续。
|
||||
- 删除已结束的执行记录会同时清理其内部重试记录，保留独立 SQL 审计。存在运行中或等待中的关联重试时禁止删除；历史清理保留活动重试所需的原执行记录。
|
||||
- 已删除记录的表状态页显示明确提示，并提供返回执行记录列表的入口；浏览器清理该记录的批量选择、分页、排序及状态筛选缓存。
|
||||
- 服务重启后，中断的重试结果会合并回原记录，包括结构状态、失败原因、逐表时间和已持久化的数量；连续重启不会重复累计。缺失内部记录的旧“重试中”状态会恢复为可单独重试的失败状态，增量水位不变。
|
||||
|
||||
**重要边界**：批次不是跨记录原子事务。取消、失败或超时不会回滚已完成写入；记录保留已尝试、已写入和已跳过数量。dry-run 不写目标，但不能模拟所有服务器约束或并发竞争。MySQL 确定性读取要求主键或完整、非空唯一索引，匹配更新需要主键；MongoDB 匹配键需可靠唯一索引，不使用稀疏、部分或自定义排序规则的唯一索引。
|
||||
**重要边界**：不能将所有批次都视为跨记录原子事务；符合条件的 MySQL 批量 upsert 使用单条语句或短事务，其他写入仍逐行执行。取消、失败或超时不会回滚此前已提交的写入；记录保留已尝试、已写入和已跳过数量。dry-run 不写目标，但不能模拟所有服务器约束或并发竞争。MySQL 确定性读取及匹配更新要求主键或完整、非空唯一索引；MongoDB 匹配键需可靠唯一索引，不使用稀疏、部分或自定义排序规则的唯一索引。
|
||||
|
||||
字段增量不是实时 CDC：仅支持单对象、有初始值且可比较的水位字段，使用包含边界的 `gte` 读取，配合 upsert/update/skip 安全重读边界。仅完整扫描且未达到行数上限的成功写入会推进水位；达到上限时保留原水位，需缩小过滤范围或提高上限。无法保证捕获水位之前迟到的数据或源记录删除。
|
||||
|
||||
时间水位按实际时间比较，保留小数秒精度，并识别 RFC3339 时区偏移；MongoDB 日期和 Go 时间使用同一比较规则。成功执行保存的水位在重启后继续作为包含边界的读取条件；空结果不会重置水位。大整数和十进制数比较不经过浮点转换。
|
||||
|
||||
初始水位必须为有效、非空的标量，不能是数组或非有限数值。MySQL 在结构阶段检查源增量字段是否存在；每批数据在写入前检查水位字段，字段缺失、空值、文档或数组均标记失败并保留重试入口。当前失败批次不写入，此前已完成的写入保留，增量水位不推进。
|
||||
|
||||
任务编辑页在保存前检查增量单表限制、写入策略和水位格式，错误定位到对应字段。任务接口与实时快照提供 `incremental_value_json` 精确文本；编辑页保留并回传原始 JSON 文本，后端精确解析后只保存实际水位，避免浏览器舍入大整数或小数后改变位点。
|
||||
|
||||
任务接口还提供 `revision` 版本标识。编辑页回传打开窗口时的版本；配置、运行状态或水位已更新时，界面或后端拒绝旧表单，避免覆盖新位点或另一窗口的修改。旧版不携带版本的 API 调用仍兼容原行为。
|
||||
|
||||
手动启动任务同样回传运行窗口打开时的 `revision`，前后端拒绝配置、运行状态或水位已变化的旧确认。后端在创建执行记录前完成版本检查，拒绝时不修改任务状态或创建执行；版本校验不替代实际写入确认。定时任务及旧版不携带版本的调用保持原启动行为。
|
||||
|
||||
任务编辑支持同一数据源连接下不同数据库之间的同步；同一连接、同一数据库仍禁止，生产目标仍禁止写入。范围同步须填写筛选条件；自动重试只适用于 `upsert`、`update`、`skip`，其他策略须将重试次数设为 0。界面会在对应字段提示这些配置错误，筛选错误会展开高级配置。
|
||||
|
||||
任务接口通过 `filters_json` 提供可编辑的精确筛选文本，任务编辑和 SQL 导出会回传原始筛选 JSON。后端优先解析该文本并保留大整数、小数的准确值，持久化实际筛选条件；不会保存第二份条件。普通 `filters` 字段仍兼容现有 API。任务详情同样使用精确文本显示筛选条件。
|
||||
|
||||
任务筛选、增量水位及 SQL 导出的 `$date` 在提交前检查 RFC3339 格式、实际日历日期、闰年、时间及偏移范围，不将不存在的日期自动调整为其他日期。合法值保留原始时区和小数秒文本。
|
||||
|
||||
`$numberDecimal` 在前端检查有限 Decimal128 的精度和指数范围，保留可精确表示的尾随零及科学计数法，不通过浮点数转换。前后端使用同一组边界用例验证，超出范围的输入定位到筛选或增量水位字段。
|
||||
|
||||
后端筛选接口同样拒绝 Decimal128 的 `NaN` 和无穷值，包括列表筛选。MySQL 的类型化筛选参数在绑定前规范化，Decimal128 和 JSON 大整数使用精确文本，避免结构体参数被驱动拒绝；MongoDB 仍使用原数值类型。
|
||||
|
||||
默认值同样通过 `defaults_json` 保留精确 JSON 文本，包含嵌套对象和数组；后端保存实际默认值，原始文本不单独持久化。MySQL JSON 列可接收对象、数组默认值及 JSON 转换结果，写入前序列化并保留 JSON 数值精度；结构化值不能写入其他列类型。
|
||||
|
||||
字段的 `json` 转换支持 JSON 文本数组、原生数组及 MongoDB BSON 数组。数组保留嵌套对象、空值和大整数，空数组保持为 `[]`；可用于 MySQL JSON 列或 MongoDB 数组字段。
|
||||
|
||||
MySQL JSON 列中的布尔值以 `true` / `false` JSON 文本传给驱动，单行写入和批量更新均保留布尔类型；普通布尔列仍使用驱动布尔参数。
|
||||
|
||||
`json` 转换得到的字符串在 MySQL JSON 列写入时保留 JSON 引号，例如 `"true"` 不变成布尔值 `true`，空字符串保持为 `""`。写入文本列或 MongoDB 时仍是普通字符串；未经转换的源端 JSON 文本按原文写入，不重复编码。
|
||||
|
||||
文本 `null` 经 `json` 转换后，在 MySQL JSON 列中写为 JSON `null`，与源端空值写成 SQL `NULL` 区分。普通非 JSON 列和 MongoDB 仍写为空值；脱敏保持转换后的空值，不将其散列成字符串。
|
||||
|
||||
重试结果回写原执行记录时，已尝试的表使用最新错误和时间信息；本次跳过或达到行数上限不会残留上一次失败原因，缺失的新时间信息不会沿用旧时间。等待中的表保留原错误、时间和数量，界面不会因排队重试暂时显示为零条。
|
||||
|
||||
数据源列表的同步状态优先显示关联任务的重试中、运行中或等待状态，否则显示最近一次同步任务的结果；没有关联任务时显示“未配置同步”。来源和目标均会更新，状态支持搜索、排序，点击有执行记录的状态可进入表同步状态页。该状态表示一项活动任务或最近任务，不汇总数据源全部历史结果，也不代表导出任务状态。任务列表同样优先显示活动执行记录，避免旧任务摘要仍显示成功。
|
||||
|
||||
列表刷新使用一次扫描得到任务活动、数据源占用和引用关系；执行列表也一次索引活动重试，避免逐行重复扫描历史记录。概览未搜索时只准备最近 8 条执行记录，搜索仍覆盖全部历史记录。
|
||||
|
||||
### 自动主从同步（MySQL 原生复制）
|
||||
|
||||
侧栏「自动主从同步」→「配置与管理」，支持 MySQL 8.0.23 及以上的 8.x 版本。通过 MySQL 的 `syncdb_auto` 命名通道持续复制新增、修改、删除和 DDL；关闭 syncdb 或浏览器不停止复制，配置与复制位置由 MySQL 自身保存。MySQL 重启后是否自动启动取决于其 `skip_replica_start` 配置。
|
||||
@@ -143,6 +216,8 @@ API 沿用 `POST /api/replications/operations/seed/plan` 和 `/seed/apply`，请
|
||||
|
||||
复制采样和运维事件存放在数据目录的独立 SQLite 文件 `replication-monitor.sqlite`，与加密 JSON 元数据及 `binlog-index.sqlite` 分开。每个拓扑的采样和事件固定保留最近 7 天，新记录写入时清理更早数据；该 SQLite 不使用元数据的 AES 加密，备份与访问权限应按运维数据管理。
|
||||
|
||||
正常关闭服务时，两个 SQLite 存储会等待正在打开的连接、取消读取和事务回滚完成，底层连接释放后关闭流程才返回，避免重启或移动数据目录时文件仍被占用。
|
||||
|
||||
「导出状态 CSV」按时间范围创建异步任务，并在「下载中心」提供下载。单个文件最多包含 8,000 条状态采样和 1,900 条运维事件；任一上限被超过时，任务仍完成并发布已截断文件，同时显示 `replicationExportTruncated` 警告。需要完整数据时应缩小时间范围后分段导出。
|
||||
|
||||
### 映射、类型和脱敏
|
||||
@@ -180,9 +255,43 @@ UI 展示计划后需再次确认才能应用；服务端重新计算并核对
|
||||
- 下载需要登录；支持取消、删除、过期清理，失败会删除未完成文件。生产来源需明确确认并默认脱敏。
|
||||
- 不使用长事务，因此导出不是跨表一致快照。达到上限不代表导出了所有记录，请检查清单的 `limit_reached`。
|
||||
|
||||
### WES 配置导出
|
||||
|
||||
在独立的「WES 配置导出」菜单点击「创建 WES 导出」，选择 MySQL 或 MongoDB 数据源及数据库，分别生成配置包。MySQL 默认读取 `ttx_table_model`、`ttx_bill_model`、`ttx_wso_data`、`ttx_navigator`、`config_detail`、`config_header`、`config_value` 七张表，可调整配置数据表清单。异步完成后在该页面或下载中心下载 `.tar.gz`。
|
||||
|
||||
| 数据来源 | 包内路径 |
|
||||
| --- | --- |
|
||||
| `ttx_table_model.structure` | `wes/table/{code}.json` |
|
||||
| `ttx_bill_model.structure` | `wes/bill/{code}.json` |
|
||||
| `ttx_wso_data.structure` | `wes/view/{tid}/{oid}.json` |
|
||||
| `ttx_navigator.structure` | `wes/navigator/{code}.json` |
|
||||
| `config_header`、`config_detail`（按 `recordType` 关联） | `wes/config/{recordType}.json` |
|
||||
| `config_value` | `wes/data/config_value.csv` |
|
||||
| 其他选中的配置表 | `wes/data/{表名}.csv` |
|
||||
| 选中的 `ttx_system` | `wes/sql/ttx_system.sql` |
|
||||
| 当前数据库全部普通表（包含未选择导出数据的业务表） | `schema/{表名}.sql` |
|
||||
|
||||
JSON 文件格式化缩进，导出时直接删除 `structure` JSON 字符串内未转义的 Tab，其余字段和值保留，已有 `\t` 转义不变；CSV 使用数据库列名及 UTF-8 编码，空值输出空单元格，供 WES 配置加载使用。`manifest.json` 记录行数、表及文件清单和 SHA-256。建表 SQL 与 WES 配置分目录，导出过程只读取数据库。配置校验失败时，任务的「错误详情」显示表名、记录标识和字段。
|
||||
|
||||
`config_header` 和 `config_detail` 选择任意一张时自动补齐另一张，不再生成这两张表的 CSV。每个 `recordType` 一份 JSON，格式为 `{ "recordType": "STATUS", "warehouses": [{ "warehouseCode": "*", "header": { ... }, "details": [{ ... }] }] }`。相同类型内按 `warehouseCode` 匹配，避免混合全局和不同仓库的数据；关联不依赖 `id/headerId`，但保留原始字段、ID、空字符串、NULL 和数值精度。没有明细的头保留空数组，没有对应头的明细保留在所属仓库且 `header` 为 `null`；重复头记录会报错，不覆盖数据。两张表的行数分别计入清单，共用 JSON 文件各自列入文件清单。这是供后续导入处理的关联 JSON 格式，现有 WES CSV 加载器不会自动加载 `wes/config/`。
|
||||
|
||||
配置标识中的通配符 `*`、`?` 在 JSON 文件名中分别转义为 `%2A`、`%3F`，`%` 转义为 `%25`，JSON 内的原始标识保持不变。例如 `ClearRedis*` 导出为 `ClearRedis%2A.json`，可在 Windows 解压。
|
||||
|
||||
选择 MongoDB 后无需填写数据库或集合，自动遍历全部可访问数据库，导出所有 `edi_erp_setting_<名称>`、`edi_erp_transform_file_<名称>`（包括 `ess`、`shopee` 等多组配置）、`edijson`、`xslt` 和 `ttx_scheduler` 集合。数据源的默认数据库不会限制遍历范围，原认证库保持有效。总行数上限跨数据库累计；没有匹配集合的数据库不会读取业务文档。清单记录扫描的数据库和每个集合的来源，任一数据库读取失败时不发布不完整包。
|
||||
|
||||
`edi_erp_setting_<名称>` 导出为 `edi/<数据库>/<名称>/edijson.json`；`edi_erp_transform_file_<名称>` 按文档的 `path`、`type`、`text` 导出为 `edi/<数据库>/<名称>/xslt/.../*.xslt`（支持资源路径前导 `/`，已有 `xslt/` 或扩展名时不重复添加）。直接命名为 `edijson`、`xslt` 的集合放入 `edi/<数据库>/`。EDI JSON 去除 `_id`、`_class`，保留嵌套对象和数值精度；XSLT 保留文本原样。同一集合中同路径的 XSLT，主目录保留按 `_id` 升序读取的第一份，其余原文存入 `duplicates/<数据库>/<集合>/<序号>.xslt`，清单的 `duplicate_resources` 记录两份文档标识及文件路径，导出任务显示提示。大小写冲突或同一配置集合存在多份 EDI JSON 仍会报错。
|
||||
|
||||
`ttx_scheduler` 的完整文档导出到 `mongo/<数据库>/ttx_scheduler.json`，格式为 MongoDB Canonical Extended JSON 数组，保留 `_id`、日期和精确数值类型，可用 `mongoimport --jsonArray` 读取。MongoDB 配置包不包含 SQL DDL。将对应 EDI 目录中的文件放入项目 `src/main/resources/` 使用。
|
||||
|
||||
单文件最大 4 MiB，未压缩总内容最大 1 GiB；默认配置记录总上限 100,000，最多可设为 1,000,000。超过行数上限、JSON 损坏、标识不一致、文件名不安全或路径重复时会失败并清理未完成文件，不发布截断包。生产数据源沿用确认和数据库记录脱敏规则。导出不是跨表一致快照。
|
||||
|
||||
参考目录里的 `template/`（Excel、报表模板）和 `yml/` 属于项目资源文件，此数据库导出不包含这些文件。解压后将所需 `wes/` 内容合并到项目配置目录；本功能不会直接覆盖本地项目文件。
|
||||
|
||||
API：`POST /api/exports/wes`。MySQL 请求包含 `source_id`、`database`、`objects`；MongoDB 仅需 `source_id`，自动遍历所有可访问数据库（忽略请求中的 `database`），`objects` 省略或为空时自动发现上述配置集合，也可传入集合名进行筛选。可配置 `batch_size`、`max_rows`、`confirm_production`；完成后复用 `/api/exports/{id}/download` 下载。
|
||||
|
||||
### 数据库本地备份与恢复
|
||||
|
||||
在数据源列表的 MySQL 数据源上选择「快速备份」，为本次操作指定数据库、并行线程和主键/唯一索引分片大小。服务会在主库短暂取得一致性快照并记录快照时的 GTID，随后按可用的主键或非空唯一 B-tree 索引分片并行读取，直接写入本地压缩二进制行帧（`.backup.gz`）。不会生成中间 SQL 文件；`NULL`、空值和二进制内容按原值保存。生产数据的本地制品未加密，创建备份前必须明确确认本地访问权限。
|
||||
在数据源列表的 MySQL 数据源上选择「快速备份」。默认开启「无锁读取」，使用单连接的 InnoDB 一致性快照读取全部分片，不执行 `LOCK INSTANCE FOR BACKUP` 或 `FLUSH TABLES WITH READ LOCK`，不需要相应锁权限；该模式自动使用一个读取线程，不记录 GTID，不能用于原生主从接续。备份期间避免执行 DDL，普通读取仍可能持有 MySQL 元数据锁。取消「无锁读取」后，使用带备份锁及短暂全局读锁的并行 GTID 快照。两种模式均按主键/唯一索引分片，直接写入本地压缩二进制行帧（`.backup.gz`）；`NULL`、空值和二进制内容按原值保存。普通同步任务使用普通 SELECT，不执行上述备份锁。生产数据的本地制品未加密，创建备份前必须明确确认本地访问权限。
|
||||
|
||||
备份任务及每个分片的边界、行数和 SHA-256 校验保存在加密元数据中。完成后可在「下载中心」下载制品，或选择「恢复到新数据库」：目标必须是非生产 MySQL 数据源，目标数据库必须尚不存在，恢复会按分片事务批量写入并校验行数和哈希。恢复制品在作业完成前不能删除或被过期清理；校验失败不会发布成功状态。
|
||||
|
||||
@@ -280,14 +389,21 @@ Binlog CSV 同步兼容接口为 `GET /api/binlog/export.csv?source_id=<id>&from
|
||||
```sh
|
||||
go test ./...
|
||||
go vet ./...
|
||||
cd frontend
|
||||
npm ci
|
||||
npm run build
|
||||
npm test
|
||||
cd ..
|
||||
node --check cmd/syncdb/web/app.js
|
||||
node --check cmd/syncdb/web/auto-replication.js
|
||||
node tests/auto-replication-ui.test.cjs
|
||||
node tests/auth-errors.test.cjs
|
||||
```
|
||||
|
||||
在 `frontend` 中执行 `npm run test:browser` 可使用本机 Edge/Chrome 检查主界面的桌面与窄屏布局、原生样式、搜索、语言切换和登录流程。其他系统可通过 `SYNCDB_BROWSER_PATH` 指定 Chromium 浏览器可执行文件。测试使用临时本地静态服务和模拟 API，不连接配置的数据源；截图输出至 `frontend/browser-artifacts/`。
|
||||
|
||||
开发过程中还使用临时隔离验证套件检查适配器协议/命令、任务生命周期、存储恢复、认证、导出文件和 UI DOM 交互；这些测试不替代真实 MySQL/MongoDB 环境验收。首次正式同步前，必须在测试数据库中验证目标类型、权限、数据约束与脱敏结果。
|
||||
|
||||
原生复制的自动化测试覆盖管理接口、GTID/实例/已有通道保护、TLS、停止流程、各步骤失败及响应丢失后的恢复、重启恢复、并发互斥、密码隐藏与审计持久化；前端控制器测试覆盖保存进度的恢复、剩余步骤确认和过期请求隔离。测试使用模拟 SQL 驱动及最小 DOM，不替代真实 MySQL 主从环境和浏览器的端到端验证。
|
||||
|
||||
`docs/design.md` 保留产品设计目标；前端仍为原生 Web，配置与任务存储仍为加密 JSON。独立 SQLite 已用于 Binlog 事件索引及持续采集检查点，不能据此推断其他设计项已实现。MySQL 原生复制由独立管理入口提供。CI 执行 Go 检查、Linux/Windows 编译和 Docker 发布，静态页面随二进制嵌入。
|
||||
`docs/design.md` 保留产品设计目标；前端使用 React + Ant Design，配置与任务存储仍为加密 JSON。独立 SQLite 已用于 Binlog 事件索引及持续采集检查点，不能据此推断其他设计项已实现。MySQL 原生复制由独立管理入口提供。CI 执行 Go 检查、Linux/Windows 编译和 Docker 发布，静态页面随二进制嵌入。
|
||||
|
||||
+1
-1
@@ -30,7 +30,7 @@ popd
|
||||
exit /b %result%
|
||||
|
||||
:go_missing
|
||||
echo ERROR: Go 1.23 or later is required. Install Go and add it to PATH.
|
||||
echo ERROR: Go 1.26 or later is required. Install Go and add it to PATH.
|
||||
goto failed
|
||||
|
||||
:build_failed
|
||||
|
||||
+18
-3
@@ -27,6 +27,7 @@ type ReadOptions struct {
|
||||
BatchSize int
|
||||
MaxRows int64
|
||||
Filters []Filter
|
||||
JSONBatch bool
|
||||
}
|
||||
|
||||
type WriteOptions struct {
|
||||
@@ -46,6 +47,7 @@ type ColumnInfo struct {
|
||||
Type string `json:"type"`
|
||||
Primary bool `json:"primary"`
|
||||
Nullable bool `json:"nullable"`
|
||||
Required bool `json:"required,omitempty"`
|
||||
}
|
||||
|
||||
type ObjectInfo struct {
|
||||
@@ -66,6 +68,12 @@ type Adapter interface {
|
||||
Close(context.Context) error
|
||||
}
|
||||
|
||||
// EstimatedRowsAdapter optionally provides a cheap row-count estimate for an object.
|
||||
// Implementations may return an error when metadata is unavailable.
|
||||
type EstimatedRowsAdapter interface {
|
||||
EstimatedRows(context.Context, string) (int64, error)
|
||||
}
|
||||
|
||||
const adapterMaxBatch = 1000
|
||||
|
||||
func openAdapter(ctx context.Context, source DataSource, audit func(SQLAudit)) (Adapter, error) {
|
||||
@@ -155,9 +163,13 @@ func adapterScalar(value any) bool {
|
||||
return true
|
||||
}
|
||||
switch typed := value.(type) {
|
||||
case convertedJSONNull:
|
||||
return true
|
||||
case json.Number:
|
||||
return json.Valid([]byte(typed)) && strings.TrimSpace(string(typed)) == string(typed) && len(typed) > 0 && (typed[0] == '-' || typed[0] >= '0' && typed[0] <= '9')
|
||||
case []byte, primitive.ObjectID, primitive.Binary, primitive.Decimal128, primitive.Timestamp, time.Time:
|
||||
case primitive.Decimal128:
|
||||
return !typed.IsNaN() && typed.IsInf() == 0
|
||||
case []byte, primitive.ObjectID, primitive.Binary, primitive.Timestamp, time.Time:
|
||||
return true
|
||||
}
|
||||
reflected := reflect.ValueOf(value)
|
||||
@@ -176,6 +188,9 @@ func adapterNil(value any) bool {
|
||||
if value == nil {
|
||||
return true
|
||||
}
|
||||
if _, ok := value.(convertedJSONNull); ok {
|
||||
return true
|
||||
}
|
||||
switch reflect.TypeOf(value).Kind() {
|
||||
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:
|
||||
return reflect.ValueOf(value).IsNil()
|
||||
@@ -310,9 +325,9 @@ func AdapterErrorCode(err error) string {
|
||||
return "operationTimeout"
|
||||
}
|
||||
switch err.Error() {
|
||||
case "databaseRequired":
|
||||
case "databaseRequired", "syncTargetRequiredColumnsMissing", "incrementalSourceFieldMissing", "incrementalCursorInvalid":
|
||||
return err.Error()
|
||||
case "invalidSource", "invalidEndpoint", "mysqlCredentialsRequired", "mongoDatabaseRequired", "invalidTLSMode", "invalidObject", "objectNotFound", "invalidFilter", "invalidReadOptions", "invalidConflictStrategy", "invalidKeyFields", "invalidRecord", "primaryKeyRequired", "uniqueKeyRequired", "keyValueRequired", "writeConflict", "concurrentModification", "operationCancelled", "operationTimeout", "adapterOperationFailed", "mysqlAuthFailed", "mysqlDatabaseDenied", "mysqlDatabaseMissing", "mysqlTLSFailed", "mysqlConnectionTimeout", "mysqlConnectionFailed", "mysqlOperationFailed", "mongoAuthFailed", "mongoDatabaseDenied", "mongoTLSFailed", "mongoConnectionTimeout", "mongoConnectionFailed", "mongoOperationFailed":
|
||||
case "invalidSource", "invalidEndpoint", "mysqlCredentialsRequired", "mongoDatabaseRequired", "invalidTLSMode", "invalidObject", "objectNotFound", "schemaIncompatible", "syncTargetColumnsMissing", "invalidFilter", "invalidReadOptions", "invalidConflictStrategy", "invalidKeyFields", "invalidRecord", "primaryKeyRequired", "uniqueKeyRequired", "keyValueRequired", "writeConflict", "concurrentModification", "operationCancelled", "operationTimeout", "adapterOperationFailed", "mysqlAuthFailed", "mysqlDatabaseDenied", "mysqlDatabaseMissing", "mysqlTLSFailed", "mysqlConnectionTimeout", "mysqlConnectionFailed", "mysqlOperationFailed", "mongoAuthFailed", "mongoDatabaseDenied", "mongoTLSFailed", "mongoConnectionTimeout", "mongoConnectionFailed", "mongoOperationFailed":
|
||||
return err.Error()
|
||||
}
|
||||
if code := adapterMySQLErrorCode(err); code != "" {
|
||||
|
||||
+184
-22
@@ -1,11 +1,13 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sort"
|
||||
@@ -91,8 +93,8 @@ func (s *server) sourceDatabaseBusyLocked(source DataSource) bool {
|
||||
return true
|
||||
}
|
||||
key := targetKey(source)
|
||||
for _, task := range s.store.Tasks {
|
||||
if task.Status != "RUNNING" {
|
||||
for taskID, task := range s.store.Tasks {
|
||||
if !s.taskBusyLocked(taskID) {
|
||||
continue
|
||||
}
|
||||
reader, readErr := resolveSourceDatabase(s.store.Sources[task.SourceID], task.SourceDatabase)
|
||||
@@ -130,8 +132,8 @@ func (s *server) sourceBusyLocked(sourceID string) bool {
|
||||
return true
|
||||
}
|
||||
}
|
||||
for _, task := range s.store.Tasks {
|
||||
if task.Status == "RUNNING" && (task.SourceID == sourceID || task.TargetID == sourceID) {
|
||||
for taskID, task := range s.store.Tasks {
|
||||
if (task.SourceID == sourceID || task.TargetID == sourceID) && s.taskBusyLocked(taskID) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
@@ -245,6 +247,11 @@ func (s *server) datasource(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
case http.MethodDelete:
|
||||
s.store.Lock()
|
||||
if _, exists := s.store.Sources[item.ID]; !exists {
|
||||
s.store.Unlock()
|
||||
fail(w, 404, "sourceNotFound")
|
||||
return
|
||||
}
|
||||
if s.sourceBusyLocked(item.ID) {
|
||||
s.store.Unlock()
|
||||
fail(w, 409, "sourceBusy")
|
||||
@@ -258,7 +265,7 @@ func (s *server) datasource(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
}
|
||||
for _, job := range s.store.Exports {
|
||||
if job.SourceID == item.ID {
|
||||
if job.SourceID == item.ID || job.TargetID == item.ID {
|
||||
s.store.Unlock()
|
||||
fail(w, 409, "sourceInUse")
|
||||
return
|
||||
@@ -282,8 +289,29 @@ func (s *server) datasource(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
|
||||
func validateTask(task *Task, source, target DataSource) error {
|
||||
task.Revision = ""
|
||||
if task.DefaultsJSON != "" {
|
||||
var defaults map[string]any
|
||||
if !strings.HasPrefix(strings.TrimSpace(task.DefaultsJSON), "{") || decodePersistenceJSON([]byte(task.DefaultsJSON), &defaults) != nil {
|
||||
return errors.New("invalidMapping")
|
||||
}
|
||||
task.Defaults, task.DefaultsJSON = defaults, ""
|
||||
}
|
||||
if task.FiltersJSON != "" {
|
||||
filters, err := parseExactFilters(task.FiltersJSON)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
task.Filters, task.FiltersJSON = filters, ""
|
||||
}
|
||||
if task.IncrementalValueJSON != "" {
|
||||
if err := decodePersistenceJSON([]byte(task.IncrementalValueJSON), &task.IncrementalValue); err != nil {
|
||||
return errors.New("invalidIncremental")
|
||||
}
|
||||
task.IncrementalValueJSON = ""
|
||||
}
|
||||
task.Name = strings.TrimSpace(task.Name)
|
||||
if task.Name == "" || len(task.Objects) == 0 || len(task.Objects) > 128 {
|
||||
if task.Name == "" || len(task.Objects) == 0 {
|
||||
return errors.New("invalidTask")
|
||||
}
|
||||
var err error
|
||||
@@ -321,7 +349,7 @@ func validateTask(task *Task, source, target DataSource) error {
|
||||
if task.Conflict != "upsert" && task.Conflict != "skip" && task.Conflict != "update" {
|
||||
return errors.New("invalidIncremental")
|
||||
}
|
||||
if _, err := filterValue(task.IncrementalValue); err != nil {
|
||||
if value, err := filterValue(task.IncrementalValue); err != nil || !incrementalCursorValid(value) {
|
||||
return errors.New("invalidIncremental")
|
||||
}
|
||||
default:
|
||||
@@ -335,13 +363,19 @@ func validateTask(task *Task, source, target DataSource) error {
|
||||
if task.BatchSize == 0 {
|
||||
task.BatchSize = 200
|
||||
}
|
||||
if task.SchemaWorkers == 0 {
|
||||
task.SchemaWorkers = 4
|
||||
}
|
||||
if task.SchemaWorkers < 1 || task.SchemaWorkers > 16 {
|
||||
return errors.New("invalidTask")
|
||||
}
|
||||
if task.MaxRows == 0 {
|
||||
task.MaxRows = 10000
|
||||
}
|
||||
if task.TimeoutSeconds == 0 {
|
||||
task.TimeoutSeconds = 300
|
||||
}
|
||||
if task.BatchSize < 1 || task.BatchSize > 1000 || task.MaxRows < 1 || task.MaxRows > 1000000 || task.TimeoutSeconds < 1 || task.TimeoutSeconds > 3600 || task.IntervalSeconds < 0 || (task.IntervalSeconds > 0 && task.IntervalSeconds < 60) || task.Retries < 0 || task.Retries > 3 {
|
||||
if task.BatchSize < 1 || task.BatchSize > 1000 || task.MaxRows < -1 || task.MaxRows > 1000000 || task.TimeoutSeconds < 1 || task.TimeoutSeconds > 86400 || task.IntervalSeconds < 0 || (task.IntervalSeconds > 0 && task.IntervalSeconds < 60) || task.Retries < 0 || task.Retries > 3 {
|
||||
return errors.New("invalidTask")
|
||||
}
|
||||
if task.Retries > 0 && task.Conflict != "upsert" && task.Conflict != "skip" && task.Conflict != "update" {
|
||||
@@ -377,7 +411,7 @@ func (s *server) tasks(w http.ResponseWriter, r *http.Request) {
|
||||
s.store.RLock()
|
||||
result := make([]Task, 0, len(s.store.Tasks))
|
||||
for _, item := range s.store.Tasks {
|
||||
result = append(result, item)
|
||||
result = append(result, taskForAPI(item))
|
||||
}
|
||||
s.store.RUnlock()
|
||||
jsonOut(w, 200, result)
|
||||
@@ -407,7 +441,7 @@ func (s *server) tasks(w http.ResponseWriter, r *http.Request) {
|
||||
s.store.Tasks[input.ID] = input
|
||||
s.store.Unlock()
|
||||
if s.save(w) {
|
||||
jsonOut(w, 201, input)
|
||||
jsonOut(w, 201, taskForAPI(input))
|
||||
}
|
||||
default:
|
||||
fail(w, 405, "methodNotAllowed")
|
||||
@@ -431,23 +465,48 @@ func (s *server) task(w http.ResponseWriter, r *http.Request) {
|
||||
switch parts[1] {
|
||||
case "run":
|
||||
var input struct {
|
||||
Confirm bool `json:"confirm"`
|
||||
Confirm bool `json:"confirm"`
|
||||
Revision string `json:"revision"`
|
||||
}
|
||||
if decode(r, &input) != nil {
|
||||
fail(w, 400, "invalidRequest")
|
||||
return
|
||||
}
|
||||
result, err := s.startTask(item.ID, input.Confirm)
|
||||
result, err := s.startReviewedTask(item.ID, input.Confirm, input.Revision)
|
||||
if err != nil {
|
||||
fail(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
jsonOut(w, 202, result)
|
||||
case "skip-table":
|
||||
var input struct {
|
||||
ExecutionID string `json:"execution_id"`
|
||||
Object string `json:"object"`
|
||||
}
|
||||
if decode(r, &input) != nil || input.ExecutionID == "" || input.Object == "" {
|
||||
fail(w, 400, "invalidRequest")
|
||||
return
|
||||
}
|
||||
s.store.RLock()
|
||||
execution, exists := s.store.Executions[input.ExecutionID]
|
||||
s.store.RUnlock()
|
||||
if !exists || execution.TaskID != item.ID || execution.Status != "RUNNING" || !s.skipTable(input.ExecutionID, input.Object) {
|
||||
fail(w, 409, "notRunning")
|
||||
return
|
||||
}
|
||||
jsonOut(w, 202, map[string]bool{"skip_requested": true})
|
||||
case "cancel":
|
||||
var input struct {
|
||||
ExecutionID string `json:"execution_id"`
|
||||
}
|
||||
if err := decode(r, &input); err != nil && !errors.Is(err, io.EOF) {
|
||||
fail(w, 400, "invalidRequest")
|
||||
return
|
||||
}
|
||||
s.store.RLock()
|
||||
executionID := ""
|
||||
for _, execution := range s.store.Executions {
|
||||
if execution.TaskID == item.ID && execution.Status == "RUNNING" {
|
||||
if execution.TaskID == item.ID && execution.Status == "RUNNING" && (input.ExecutionID == "" || execution.ID == input.ExecutionID) {
|
||||
executionID = execution.ID
|
||||
break
|
||||
}
|
||||
@@ -465,7 +524,7 @@ func (s *server) task(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
switch r.Method {
|
||||
case http.MethodGet:
|
||||
jsonOut(w, 200, item)
|
||||
jsonOut(w, 200, taskForAPI(item))
|
||||
case http.MethodPut:
|
||||
input := Task{DryRun: true}
|
||||
if decode(r, &input) != nil {
|
||||
@@ -479,11 +538,16 @@ func (s *server) task(w http.ResponseWriter, r *http.Request) {
|
||||
fail(w, 404, "taskNotFound")
|
||||
return
|
||||
}
|
||||
if current.Status == "RUNNING" {
|
||||
if s.taskBusyLocked(item.ID) {
|
||||
s.store.Unlock()
|
||||
fail(w, 409, "taskBusy")
|
||||
return
|
||||
}
|
||||
if input.Revision != "" && input.Revision != taskRevision(current) {
|
||||
s.store.Unlock()
|
||||
fail(w, 409, "taskReviewChanged")
|
||||
return
|
||||
}
|
||||
source, sourceOK := s.store.Sources[input.SourceID]
|
||||
target, targetOK := s.store.Sources[input.TargetID]
|
||||
if !sourceOK || !targetOK {
|
||||
@@ -503,11 +567,16 @@ func (s *server) task(w http.ResponseWriter, r *http.Request) {
|
||||
s.store.Tasks[item.ID] = input
|
||||
s.store.Unlock()
|
||||
if s.save(w) {
|
||||
jsonOut(w, 200, input)
|
||||
jsonOut(w, 200, taskForAPI(input))
|
||||
}
|
||||
case http.MethodDelete:
|
||||
s.store.Lock()
|
||||
if s.store.Tasks[item.ID].Status == "RUNNING" {
|
||||
if _, exists := s.store.Tasks[item.ID]; !exists {
|
||||
s.store.Unlock()
|
||||
fail(w, 404, "taskNotFound")
|
||||
return
|
||||
}
|
||||
if s.taskBusyLocked(item.ID) {
|
||||
s.store.Unlock()
|
||||
fail(w, 409, "taskBusy")
|
||||
return
|
||||
@@ -539,7 +608,7 @@ func (s *server) executions(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
|
||||
func (s *server) execution(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodGet {
|
||||
if r.Method != http.MethodGet && r.Method != http.MethodPost && r.Method != http.MethodDelete {
|
||||
fail(w, 405, "methodNotAllowed")
|
||||
return
|
||||
}
|
||||
@@ -551,6 +620,69 @@ func (s *server) execution(w http.ResponseWriter, r *http.Request) {
|
||||
fail(w, 404, "notFound")
|
||||
return
|
||||
}
|
||||
if r.Method == http.MethodDelete {
|
||||
if len(parts) != 1 {
|
||||
fail(w, 405, "methodNotAllowed")
|
||||
return
|
||||
}
|
||||
s.store.Lock()
|
||||
current, exists := s.store.Executions[item.ID]
|
||||
if !exists {
|
||||
s.store.Unlock()
|
||||
fail(w, 404, "notFound")
|
||||
return
|
||||
}
|
||||
family := executionFamily(s.store.Executions, current.ID)
|
||||
busy := false
|
||||
for _, executionID := range family {
|
||||
if unfinishedStatus(s.store.Executions[executionID].Status) {
|
||||
busy = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if busy {
|
||||
s.store.Unlock()
|
||||
fail(w, 409, "taskBusy")
|
||||
return
|
||||
}
|
||||
for _, executionID := range family {
|
||||
delete(s.store.Executions, executionID)
|
||||
}
|
||||
s.store.Unlock()
|
||||
if s.save(w) {
|
||||
jsonOut(w, 200, map[string]bool{"deleted": true})
|
||||
}
|
||||
return
|
||||
}
|
||||
if len(parts) == 2 && (parts[1] == "retry" || parts[1] == "retry-object") {
|
||||
if r.Method != http.MethodPost {
|
||||
fail(w, 405, "methodNotAllowed")
|
||||
return
|
||||
}
|
||||
var input struct {
|
||||
Object string `json:"object"`
|
||||
Objects []string `json:"objects"`
|
||||
Revision string `json:"revision"`
|
||||
}
|
||||
if decode(r, &input) != nil || (input.Object == "" && len(input.Objects) == 0) || (input.Object != "" && len(input.Objects) != 0) {
|
||||
fail(w, 400, "invalidRequest")
|
||||
return
|
||||
}
|
||||
if input.Object != "" {
|
||||
input.Objects = []string{input.Object}
|
||||
}
|
||||
result, err := s.retryReviewedObjects(item.ID, input.Objects, input.Revision)
|
||||
if err != nil {
|
||||
fail(w, 409, err.Error())
|
||||
return
|
||||
}
|
||||
jsonOut(w, 202, result)
|
||||
return
|
||||
}
|
||||
if r.Method != http.MethodGet {
|
||||
fail(w, 405, "methodNotAllowed")
|
||||
return
|
||||
}
|
||||
if len(parts) == 2 && parts[1] == "events" {
|
||||
s.streamResourceEvents(w, r, "execution", item.ID)
|
||||
return
|
||||
@@ -671,18 +803,34 @@ func (s *server) streamEvents(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("X-Accel-Buffering", "no")
|
||||
ticker := time.NewTicker(time.Second)
|
||||
defer ticker.Stop()
|
||||
var previous []byte
|
||||
lastHeartbeat := time.Now()
|
||||
for {
|
||||
s.store.RLock()
|
||||
snapshot := map[string]any{"sources": values(s.store.Sources), "tasks": values(s.store.Tasks), "executions": values(s.store.Executions), "exports": values(s.store.Exports), "audits": values(s.store.Audits)}
|
||||
tasks := snapshotValues(s.store.Tasks)
|
||||
for index := range tasks {
|
||||
tasks[index] = taskForAPI(tasks[index])
|
||||
}
|
||||
snapshot := map[string]any{"sources": snapshotValues(s.store.Sources), "tasks": tasks, "executions": snapshotValues(s.store.Executions), "exports": snapshotValues(s.store.Exports), "audits": snapshotValues(s.store.Audits)}
|
||||
data, err := json.Marshal(snapshot)
|
||||
s.store.RUnlock()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
if _, err = w.Write(append(append([]byte("event: snapshot\ndata: "), data...), []byte("\n\n")...)); err != nil {
|
||||
return
|
||||
if !bytes.Equal(previous, data) {
|
||||
if _, err = w.Write(append(append([]byte("event: snapshot\ndata: "), data...), []byte("\n\n")...)); err != nil {
|
||||
return
|
||||
}
|
||||
previous = data
|
||||
lastHeartbeat = time.Now()
|
||||
flusher.Flush()
|
||||
} else if time.Since(lastHeartbeat) >= 15*time.Second {
|
||||
if _, err = io.WriteString(w, "event: heartbeat\ndata: {}\n\n"); err != nil {
|
||||
return
|
||||
}
|
||||
lastHeartbeat = time.Now()
|
||||
flusher.Flush()
|
||||
}
|
||||
flusher.Flush()
|
||||
select {
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
@@ -693,6 +841,20 @@ func (s *server) streamEvents(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
}
|
||||
|
||||
// Map iteration order must not make an unchanged snapshot look different.
|
||||
func snapshotValues[Value any](items map[string]Value) []Value {
|
||||
keys := make([]string, 0, len(items))
|
||||
for key := range items {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
result := make([]Value, 0, len(keys))
|
||||
for _, key := range keys {
|
||||
result = append(result, items[key])
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func values[Value any](items map[string]Value) []Value {
|
||||
result := make([]Value, 0, len(items))
|
||||
for _, item := range items {
|
||||
|
||||
@@ -2,6 +2,7 @@ package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -13,6 +14,13 @@ import (
|
||||
const binlogCSVMaximumEventCells = 100000
|
||||
const binlogCSVMaximumPayloadDepth = 64
|
||||
|
||||
// Go JSON implementations may encode invalid UTF-8 as either a literal
|
||||
// replacement rune or its escape sequence. Match the active encoder once.
|
||||
var binlogCSVInvalidUTF8Bytes = func() int {
|
||||
encoded, _ := json.Marshal(string([]byte{0xff}))
|
||||
return len(encoded) - 2
|
||||
}()
|
||||
|
||||
// CSV visits matching events independently, so unrelated compressed events do not
|
||||
// consume the row budget or inherit the preview's truncation flags.
|
||||
func visitBinlogCSVEvent(ctx context.Context, raw *replication.BinlogEvent, query binlogQuery, file string, position, end uint64, depth int, emit func(binlogEvent) string) string {
|
||||
@@ -245,9 +253,12 @@ func consumeBinlogCSVJSONString(value string, remaining *int) bool {
|
||||
case '<', '>', '&', '\u2028', '\u2029':
|
||||
size = 6
|
||||
default:
|
||||
if character < 0x20 || (character == utf8.RuneError && width == 1) {
|
||||
if character < 0x20 {
|
||||
size = 6
|
||||
}
|
||||
if character == utf8.RuneError && width == 1 {
|
||||
size = binlogCSVInvalidUTF8Bytes
|
||||
}
|
||||
}
|
||||
if !consumeBinlogCSVBytes(size, remaining) {
|
||||
return false
|
||||
|
||||
@@ -140,7 +140,7 @@ func TestBinlogCSVJSONStringBudgetMatchesEncoding(t *testing.T) {
|
||||
}
|
||||
remaining := len(encoded)
|
||||
if !consumeBinlogCSVJSONString(value, &remaining) || remaining != 0 {
|
||||
t.Fatal("JSON size estimate differs from encoder")
|
||||
t.Fatalf("JSON size estimate differs from encoder for %q: encoded=%q remaining=%d", value, encoded, remaining)
|
||||
}
|
||||
remaining = len(encoded) - 1
|
||||
if consumeBinlogCSVJSONString(value, &remaining) {
|
||||
|
||||
@@ -13,8 +13,6 @@ import (
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
_ "modernc.org/sqlite"
|
||||
)
|
||||
|
||||
type binlogIndexState struct {
|
||||
@@ -43,7 +41,7 @@ type binlogIndexState struct {
|
||||
}
|
||||
|
||||
type binlogIndexStore struct {
|
||||
db *sql.DB
|
||||
db *localSQLiteDB
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
@@ -59,10 +57,7 @@ func openBinlogIndexStore(path string) (*binlogIndexStore, error) {
|
||||
return nil, err
|
||||
}
|
||||
f.Close()
|
||||
db, err := sql.Open("sqlite", filepath.ToSlash(path)+"?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)&_pragma=synchronous(FULL)")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
db := openLocalSQLite(filepath.ToSlash(path) + "?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)&_pragma=synchronous(FULL)")
|
||||
db.SetMaxOpenConns(4)
|
||||
statements := []string{
|
||||
`CREATE TABLE IF NOT EXISTS index_sources(source_id TEXT PRIMARY KEY, fingerprint TEXT NOT NULL, state_json BLOB NOT NULL, checkpoint_json BLOB NOT NULL)`,
|
||||
|
||||
@@ -42,7 +42,11 @@ func resolveDataDir(configured string) (string, error) {
|
||||
func readConfig(arguments []string) (runtimeConfig, error) {
|
||||
var config runtimeConfig
|
||||
config.Server.Addr = ":0"
|
||||
config.Server.DataDir = "data"
|
||||
homeDir, err := os.UserHomeDir()
|
||||
if err != nil {
|
||||
return config, err
|
||||
}
|
||||
config.Server.DataDir = filepath.Join(homeDir, ".syncdb")
|
||||
config.Server.Workers = 2
|
||||
config.Export.RetentionHours = 24
|
||||
flags := flag.NewFlagSet("syncdb", flag.ContinueOnError)
|
||||
|
||||
@@ -80,6 +80,7 @@ type databaseBackupInput struct {
|
||||
ShardRows int `json:"shard_rows"`
|
||||
BatchRows int `json:"batch_rows"`
|
||||
ConfirmProduction bool `json:"confirm_production"`
|
||||
NoLock *bool `json:"no_lock,omitempty"`
|
||||
}
|
||||
|
||||
type databaseRestoreInput struct {
|
||||
@@ -278,6 +279,10 @@ func (s *server) createDatabaseBackup(w http.ResponseWriter, r *http.Request) {
|
||||
backupRespond(w, http.StatusBadRequest, "invalidBackup")
|
||||
return
|
||||
}
|
||||
config.NoLock = input.NoLock == nil || *input.NoLock
|
||||
if config.NoLock {
|
||||
config.Workers = 1
|
||||
}
|
||||
s.store.Lock()
|
||||
source, exists := s.store.Sources[input.SourceID]
|
||||
if !exists {
|
||||
@@ -503,12 +508,20 @@ func (s *server) runDatabaseBackup(ctx context.Context, source DataSource, job E
|
||||
}
|
||||
|
||||
func (s *server) writeDatabaseBackup(ctx context.Context, source DataSource, job Export, config replicationSeedConfig) (int64, string, int64, error) {
|
||||
if config.NoLock {
|
||||
config.Workers = 1
|
||||
}
|
||||
adapter, err := s.openSeedDatabase(ctx, source, config.Workers+2)
|
||||
if err != nil {
|
||||
return 0, "", 0, err
|
||||
}
|
||||
defer adapter.Close(context.Background())
|
||||
snapshot, err := acquireSeedSnapshot(ctx, adapter.database, source.Database, config.Workers)
|
||||
var snapshot *seedSnapshot
|
||||
if config.NoLock {
|
||||
snapshot, err = acquireUnlockedSnapshot(ctx, adapter.database, source.Database)
|
||||
} else {
|
||||
snapshot, err = acquireSeedSnapshot(ctx, adapter.database, source.Database, config.Workers)
|
||||
}
|
||||
if err != nil {
|
||||
return 0, "", 0, err
|
||||
}
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"net/http"
|
||||
"net/url"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
@@ -145,6 +146,9 @@ func boundedDatabaseNames(input []string, sourceType string) ([]string, bool, er
|
||||
|
||||
func readMySQLDatabases(ctx context.Context, database *sql.DB) ([]string, bool, error) {
|
||||
rows, err := database.QueryContext(ctx, "SELECT SCHEMA_NAME FROM information_schema.SCHEMATA ORDER BY SCHEMA_NAME LIMIT 513")
|
||||
if err != nil && mysqlMetadataFallback(err) {
|
||||
rows, err = database.QueryContext(ctx, "SHOW DATABASES")
|
||||
}
|
||||
if err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
@@ -157,21 +161,39 @@ func readMySQLTables(ctx context.Context, database *sql.DB, name string) ([]stri
|
||||
}
|
||||
rows, err := database.QueryContext(ctx, "SELECT TABLE_NAME FROM information_schema.TABLES WHERE TABLE_SCHEMA = ? AND TABLE_TYPE = 'BASE TABLE' ORDER BY TABLE_NAME LIMIT 513", name)
|
||||
if err != nil {
|
||||
if mysqlMetadataFallback(err) {
|
||||
rows, err = database.QueryContext(ctx, "SHOW FULL TABLES FROM "+QuoteIdentifier(name)+" WHERE Table_type = 'BASE TABLE'")
|
||||
if err == nil {
|
||||
return readMySQLNameRows(rows, true)
|
||||
}
|
||||
}
|
||||
return nil, false, err
|
||||
}
|
||||
return readMySQLNames(rows)
|
||||
}
|
||||
|
||||
func readMySQLNames(rows *sql.Rows) ([]string, bool, error) {
|
||||
return readMySQLNameRows(rows, false)
|
||||
}
|
||||
|
||||
func readMySQLNameRows(rows *sql.Rows, tableTypes bool) ([]string, bool, error) {
|
||||
defer rows.Close()
|
||||
names := make([]string, 0)
|
||||
seen := make(map[string]bool)
|
||||
truncated := false
|
||||
for rows.Next() {
|
||||
var name string
|
||||
if err := rows.Scan(&name); err != nil {
|
||||
var kind string
|
||||
pointers := []any{&name}
|
||||
if tableTypes {
|
||||
pointers = append(pointers, &kind)
|
||||
}
|
||||
if err := rows.Scan(pointers...); err != nil {
|
||||
return nil, false, err
|
||||
}
|
||||
if tableTypes && !strings.EqualFold(kind, "BASE TABLE") {
|
||||
continue
|
||||
}
|
||||
if !adapterValidName(name) {
|
||||
return nil, false, errors.New("invalidObject")
|
||||
}
|
||||
|
||||
@@ -9,8 +9,12 @@ import (
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/go-sql-driver/mysql"
|
||||
)
|
||||
|
||||
type scopeNamesConnector struct {
|
||||
@@ -134,3 +138,128 @@ func TestSourceDatabaseNamesRejectsMissingSelection(t *testing.T) {
|
||||
t.Fatal(w.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestSourceDatabaseNamesSHOWFallbackUsesSelectedScope(t *testing.T) {
|
||||
for _, tables := range []bool{false, true} {
|
||||
t.Run(fmt.Sprint(tables), func(t *testing.T) {
|
||||
selected := "chosen_%`db"
|
||||
queries := []string{}
|
||||
connector := scopeNamesConnector{query: func(query string, args []driver.NamedValue) (driver.Rows, error) {
|
||||
queries = append(queries, query)
|
||||
if strings.Contains(query, "information_schema") {
|
||||
if tables && (len(args) != 1 || args[0].Value != selected) {
|
||||
t.Fatalf("wrong initial database scope: %v", args)
|
||||
}
|
||||
return nil, &mysql.MySQLError{Number: 1142, Message: "metadata denied"}
|
||||
}
|
||||
if tables {
|
||||
if query != "SHOW FULL TABLES FROM `chosen_%``db` WHERE Table_type = 'BASE TABLE'" || len(args) != 0 {
|
||||
t.Fatalf("unsafe or unscoped SHOW query: %s %v", query, args)
|
||||
}
|
||||
return &showSchemaRows{names: []string{"Tables_in_chosen", "Table_type"}, rows: [][]driver.Value{{"view", "VIEW"}, {"z", "BASE TABLE"}, {"a", "BASE TABLE"}, {"a", "BASE TABLE"}}}, nil
|
||||
}
|
||||
if query != "SHOW DATABASES" || len(args) != 0 {
|
||||
t.Fatalf("unexpected database fallback query: %s %v", query, args)
|
||||
}
|
||||
return &scopeNamesRows{names: []string{"z", "a", "a"}}, nil
|
||||
}}
|
||||
s := &server{adapterFactory: func(_ context.Context, connected DataSource, _ func(SQLAudit)) (Adapter, error) {
|
||||
if connected.Database != "" {
|
||||
t.Fatal("name lookup connected to a previously selected business database")
|
||||
}
|
||||
return &mysqlAdapter{database: sql.OpenDB(connector)}, nil
|
||||
}}
|
||||
path := "/api/datasources/server/databases"
|
||||
key := "databases"
|
||||
if tables {
|
||||
path, key = "/api/datasources/server/tables?database="+url.QueryEscape(selected), "tables"
|
||||
}
|
||||
response := httptest.NewRecorder()
|
||||
s.sourceDatabaseNames(response, httptest.NewRequest(http.MethodGet, path, nil), DataSource{Name: "server", Type: "mysql", Endpoint: "db.example", Username: "reader", Database: "unavailable"}, tables)
|
||||
if response.Code != http.StatusOK || !strings.Contains(response.Body.String(), `"`+key+`":["a","z"]`) || !strings.Contains(response.Body.String(), `"truncated":false`) || len(queries) != 2 {
|
||||
t.Fatalf("SHOW name lookup failed: %d %s queries=%v", response.Code, response.Body.String(), queries)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type closedNameRows struct {
|
||||
showSchemaRows
|
||||
closed bool
|
||||
}
|
||||
|
||||
func (rows *closedNameRows) Close() error { rows.closed = true; return nil }
|
||||
|
||||
func TestMySQLSHOWNameLimitsCloseRows(t *testing.T) {
|
||||
for _, tables := range []bool{false, true} {
|
||||
rows := &closedNameRows{}
|
||||
rows.names = []string{"Name"}
|
||||
if tables {
|
||||
rows.names = append(rows.names, "Table_type")
|
||||
rows.rows = append(rows.rows, []driver.Value{"ignored_view", "VIEW"})
|
||||
}
|
||||
for i := 0; i < databaseNameLimit+5; i++ {
|
||||
values := []driver.Value{fmt.Sprintf("name%03d", i)}
|
||||
if tables {
|
||||
values = append(values, "BASE TABLE")
|
||||
}
|
||||
rows.rows = append(rows.rows, values)
|
||||
}
|
||||
database := sql.OpenDB(scopeNamesConnector{query: func(query string, _ []driver.NamedValue) (driver.Rows, error) {
|
||||
if strings.Contains(query, "information_schema") {
|
||||
return nil, &mysql.MySQLError{Number: 1142}
|
||||
}
|
||||
return rows, nil
|
||||
}})
|
||||
var names []string
|
||||
var truncated bool
|
||||
var err error
|
||||
if tables {
|
||||
names, truncated, err = readMySQLTables(context.Background(), database, "selected")
|
||||
} else {
|
||||
names, truncated, err = readMySQLDatabases(context.Background(), database)
|
||||
}
|
||||
database.Close()
|
||||
if err != nil || !truncated || len(names) != databaseNameLimit || names[0] != "name000" || names[len(names)-1] != "name511" || !rows.closed {
|
||||
t.Fatalf("unbounded or unclosed SHOW result: tables=%t count=%d truncated=%t closed=%t err=%v", tables, len(names), truncated, rows.closed, err)
|
||||
}
|
||||
if len(rows.rows) == 0 {
|
||||
t.Fatal("truncated SHOW result unnecessarily consumed all rows")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLNameFallbackPreservesTerminalErrors(t *testing.T) {
|
||||
for _, expected := range []error{context.Canceled, context.DeadlineExceeded, &mysql.MySQLError{Number: 1045}} {
|
||||
for _, tables := range []bool{false, true} {
|
||||
queries := []string{}
|
||||
database := sql.OpenDB(scopeNamesConnector{query: func(query string, _ []driver.NamedValue) (driver.Rows, error) {
|
||||
queries = append(queries, query)
|
||||
return nil, expected
|
||||
}})
|
||||
var err error
|
||||
if tables {
|
||||
_, _, err = readMySQLTables(context.Background(), database, "selected")
|
||||
} else {
|
||||
_, _, err = readMySQLDatabases(context.Background(), database)
|
||||
}
|
||||
database.Close()
|
||||
if err == nil || AdapterErrorCode(err) != AdapterErrorCode(expected) || len(queries) != 1 {
|
||||
t.Fatalf("terminal error retried or changed: %v %v", err, queries)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLNameFallbackFailureIsNotAnEmptySuccess(t *testing.T) {
|
||||
queries := []string{}
|
||||
database := sql.OpenDB(scopeNamesConnector{query: func(query string, _ []driver.NamedValue) (driver.Rows, error) {
|
||||
queries = append(queries, query)
|
||||
return nil, &mysql.MySQLError{Number: 1142}
|
||||
}})
|
||||
defer database.Close()
|
||||
names, truncated, err := readMySQLDatabases(context.Background(), database)
|
||||
if err == nil || names != nil || truncated || !reflect.DeepEqual(queries, []string{"SELECT SCHEMA_NAME FROM information_schema.SCHEMATA ORDER BY SCHEMA_NAME LIMIT 513", "SHOW DATABASES"}) {
|
||||
t.Fatalf("denied SHOW became an empty successful picker: names=%v truncated=%t err=%v queries=%v", names, truncated, err, queries)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson/primitive"
|
||||
)
|
||||
|
||||
func TestFrontendDecimal128BoundaryCasesMatchBackendParser(t *testing.T) {
|
||||
data, err := os.ReadFile("../../frontend/testdata/decimal128.json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var cases []struct {
|
||||
Value string `json:"value"`
|
||||
Valid bool `json:"valid"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &cases); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, test := range cases {
|
||||
value, err := primitive.ParseDecimal128(test.Value)
|
||||
valid := err == nil && !value.IsNaN() && value.IsInf() == 0
|
||||
if valid != test.Valid {
|
||||
t.Errorf("%q: finite backend validity=%v expected=%v", test.Value, valid, test.Valid)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"database/sql/driver"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/bson/primitive"
|
||||
)
|
||||
|
||||
func TestDecimalFiltersRejectNonFiniteTypedAndWireValues(t *testing.T) {
|
||||
for _, text := range []string{"NaN", "Inf", "Infinity", "-Infinity"} {
|
||||
value, err := primitive.ParseDecimal128(text)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := filterValue(map[string]any{"$numberDecimal": text}); err == nil {
|
||||
t.Errorf("non-finite filter decoded: %s", text)
|
||||
}
|
||||
for _, filter := range []Filter{{Field: "amount", Operator: "eq", Value: value}, {Field: "amount", Operator: "in", Value: []any{1, value}}} {
|
||||
if err := ValidateFilters([]Filter{filter}); err == nil {
|
||||
t.Errorf("non-finite filter validated: %s", text)
|
||||
}
|
||||
if _, _, err := mysqlFilterSQL([]Filter{filter}); err == nil {
|
||||
t.Errorf("MySQL accepted non-finite filter: %s", text)
|
||||
}
|
||||
if _, err := mongoFilterDocument([]Filter{filter}); err == nil {
|
||||
t.Errorf("MongoDB accepted non-finite filter: %s", text)
|
||||
}
|
||||
}
|
||||
data, _ := json.Marshal(map[string]any{"field": "amount", "operator": "eq", "value": map[string]string{"$numberDecimal": text}})
|
||||
var filter Filter
|
||||
if err := json.Unmarshal(data, &filter); err == nil {
|
||||
t.Errorf("non-finite wire filter accepted: %s", text)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTypedDecimalFiltersReachSQLDriverExactlyAndRemainNumericInMongoDB(t *testing.T) {
|
||||
const exact = "12345678901234567890.0001"
|
||||
value, err := primitive.ParseDecimal128(exact)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
filters := []Filter{{Field: "amount", Operator: "gte", Value: value}, {Field: "amount", Operator: "in", Value: []any{value, json.Number("9007199254740993")}}}
|
||||
where, args, err := mysqlFilterSQL(filters)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
calls := 0
|
||||
db := sql.OpenDB(scopeNamesConnector{query: func(query string, values []driver.NamedValue) (driver.Rows, error) {
|
||||
calls++
|
||||
if query != "SELECT `id` FROM `items` WHERE `amount` >= ? AND `amount` IN (?,?)" || len(values) != 3 || values[0].Value != exact || values[1].Value != exact || values[2].Value != "9007199254740993" {
|
||||
t.Fatalf("filter precision changed at driver boundary: %s %+v", query, values)
|
||||
}
|
||||
return &scopeNamesRows{names: []string{"found"}}, nil
|
||||
}})
|
||||
defer db.Close()
|
||||
rows, err := db.QueryContext(context.Background(), "SELECT `id` FROM `items`"+where, args...)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = rows.Close()
|
||||
if calls != 1 {
|
||||
t.Fatal("SQL filter never reached the driver")
|
||||
}
|
||||
document, err := mongoFilterDocument(filters)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
encoded, err := bson.MarshalWithRegistry(mongoRegistry, document)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
clauses := bson.Raw(encoded).Lookup("$and").Array()
|
||||
if clauses.Index(0).Value().Document().Lookup("amount").Document().Lookup("$gte").Decimal128().String() != exact {
|
||||
t.Fatal("MongoDB decimal filter changed type or precision")
|
||||
}
|
||||
if _, ok := filters[0].Value.(primitive.Decimal128); !ok {
|
||||
t.Fatal("SQL normalization mutated shared filter values")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"math"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
)
|
||||
|
||||
const exactDefaultsText = `{"amount":12345678901234567890.0001,"large":9007199254740993,"nested":{"values":[9007199254740995,0.00000000000000000012345]}}`
|
||||
|
||||
func TestDefaultValuesBrowserRoundTripAndWriteValuesStayExact(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
var defaults map[string]any
|
||||
if err := decodePersistenceJSON([]byte(exactDefaultsText), &defaults); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task := Task{ID: "task", Name: "task", SourceID: "source", TargetID: "target", SourceDatabase: "app", TargetDatabase: "app", Objects: []string{"table"}, Defaults: defaults}
|
||||
s.store.Tasks[task.ID] = task
|
||||
input := taskForAPI(task)
|
||||
if input.DefaultsJSON != exactDefaultsText || taskRevision(input) != taskRevision(task) {
|
||||
t.Fatal("invalid exact defaults transport")
|
||||
}
|
||||
input.Defaults = map[string]any{"large": float64(9007199254740992), "stale": true}
|
||||
data, err := json.Marshal(input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
s.task(rec, httptest.NewRequest(http.MethodPut, "/api/tasks/task", bytes.NewReader(data)))
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("edit rejected: %s", rec.Body.String())
|
||||
}
|
||||
stored := s.store.Tasks[task.ID]
|
||||
if stored.DefaultsJSON != "" {
|
||||
t.Fatal("transport text persisted")
|
||||
}
|
||||
if _, exists := stored.Defaults["stale"]; exists {
|
||||
t.Fatal("exact defaults merged with stale convenience values")
|
||||
}
|
||||
loaded, _, err := loadStore(s.dataDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if taskForAPI(loaded.Tasks[task.ID]).DefaultsJSON != exactDefaultsText {
|
||||
t.Fatal("default precision changed on restart")
|
||||
}
|
||||
result, err := transformRecord(Record{"id": int64(1)}, nil, loaded.Tasks[task.ID].Defaults, nil, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for field, kind := range map[string]string{"large": "BIGINT", "amount": "DECIMAL(24,4)", "nested": "JSON"} {
|
||||
value, err := mysqlWriteValue(result[field], kind)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
switch field {
|
||||
case "large":
|
||||
if value != "9007199254740993" {
|
||||
t.Fatalf("integer rounded: %v", value)
|
||||
}
|
||||
case "amount":
|
||||
if value != "12345678901234567890.0001" {
|
||||
t.Fatalf("decimal rounded: %v", value)
|
||||
}
|
||||
case "nested":
|
||||
if value != `{"values":[9007199254740995,0.00000000000000000012345]}` {
|
||||
t.Fatalf("nested default rounded: %v", value)
|
||||
}
|
||||
}
|
||||
}
|
||||
encoded, err := bson.MarshalWithRegistry(mongoRegistry, result)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
document := bson.Raw(encoded)
|
||||
if document.Lookup("large").Int64() != 9007199254740993 || document.Lookup("amount").Decimal128().String() != "12345678901234567890.0001" {
|
||||
t.Fatal("MongoDB defaults lost precision")
|
||||
}
|
||||
converted, err := transformRecord(Record{}, nil, stored.Defaults, map[string]string{"large": "int", "amount": "decimal"}, nil, nil)
|
||||
if err != nil || converted["large"] != int64(9007199254740993) {
|
||||
t.Fatalf("conversion changed default: %+v %v", converted, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTaskExactDefaultsRejectMalformedAndReplaceExistingMap(t *testing.T) {
|
||||
source := DataSource{Type: "mysql", Endpoint: "source:3306", Database: "app"}
|
||||
target := DataSource{Type: "mysql", Endpoint: "target:3306", Database: "app"}
|
||||
for _, raw := range []string{"null", "[]", `{} trailing`, `{"bad..path":1}`} {
|
||||
task := Task{Name: "task", Objects: []string{"table"}, DefaultsJSON: raw}
|
||||
if err := validateTask(&task, source, target); err == nil || err.Error() != "invalidMapping" {
|
||||
t.Fatalf("invalid defaults accepted: %q %v", raw, err)
|
||||
}
|
||||
}
|
||||
task := Task{Name: "task", Objects: []string{"table"}, Defaults: map[string]any{"old": 1}, DefaultsJSON: "{}"}
|
||||
if err := validateTask(&task, source, target); err != nil || len(task.Defaults) != 0 {
|
||||
t.Fatalf("clearing defaults failed: %+v %v", task, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLJSONWriteSupportsStructuredConversionsAndRejectsInvalidValues(t *testing.T) {
|
||||
for _, input := range []any{map[string]any{"id": json.Number("9007199254740993")}, Record{"id": json.Number("9007199254740993")}, bson.M{"id": json.Number("9007199254740993")}, bson.D{{Key: "id", Value: json.Number("9007199254740993")}}, []any{json.Number("9007199254740993")}, bson.A{json.Number("9007199254740993")}} {
|
||||
value, err := mysqlWriteValue(input, "json")
|
||||
if err != nil || (value != `{"id":9007199254740993}` && value != `[9007199254740993]`) {
|
||||
t.Fatalf("structured JSON rejected or rounded: %T %v %v", input, value, err)
|
||||
}
|
||||
if _, err := mysqlWriteValue(input, "TEXT"); err == nil {
|
||||
t.Fatal("structured value accepted for a non-JSON column")
|
||||
}
|
||||
}
|
||||
converted, err := convertValue(`{"id":9007199254740993,"values":[0.0000000000000000001]}`, "json")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
value, err := mysqlWriteValue(converted, "JSON")
|
||||
if err != nil || value != `{"id":9007199254740993,"values":[0.0000000000000000001]}` {
|
||||
t.Fatalf("JSON conversion did not reach writable SQL value: %v %v", value, err)
|
||||
}
|
||||
object := ObjectInfo{Name: "table", Columns: []ColumnInfo{{Name: "id", Type: "BIGINT", Primary: true}, {Name: "doc", Type: "JSON"}}}
|
||||
statement, arguments, handled, err := mysqlBatchUpsertSQL(object, []Record{{"id": int64(1), "doc": converted}}, []string{"id"}, []string{"id"})
|
||||
if err != nil || !handled || statement == "" || len(arguments) != 2 || arguments[0] != value || arguments[1] != int64(1) {
|
||||
t.Fatalf("batch upsert rejected converted JSON: %s %+v %v", statement, arguments, err)
|
||||
}
|
||||
cycle := map[string]any{}
|
||||
cycle["self"] = cycle
|
||||
for _, input := range []any{cycle, map[string]any{"n": math.NaN()}, []any{math.Inf(1)}, map[string]any{"n": json.Number("bad")}} {
|
||||
if _, err := mysqlWriteValue(input, "JSON"); err == nil || err.Error() != "invalidRecord" {
|
||||
t.Fatalf("invalid JSON accepted: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
+469
-81
@@ -4,9 +4,28 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"log"
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
|
||||
type tableRun struct {
|
||||
object string
|
||||
cancel context.CancelFunc
|
||||
skipped bool
|
||||
}
|
||||
|
||||
func (s *server) skipTable(executionID, object string) bool {
|
||||
s.jobsMu.Lock()
|
||||
defer s.jobsMu.Unlock()
|
||||
run := s.tableRuns[executionID]
|
||||
if run == nil || run.object != object || run.skipped {
|
||||
return false
|
||||
}
|
||||
run.skipped = true
|
||||
run.cancel()
|
||||
return true
|
||||
}
|
||||
|
||||
func (s *server) connect(ctx context.Context, source DataSource, audit func(SQLAudit)) (Adapter, error) {
|
||||
if s.adapterFactory != nil {
|
||||
return s.adapterFactory(ctx, source, audit)
|
||||
@@ -90,16 +109,24 @@ func (s *server) cancelJob(jobID string) bool {
|
||||
}
|
||||
|
||||
func (s *server) startTask(taskID string, confirmed bool) (Execution, error) {
|
||||
return s.startReviewedTask(taskID, confirmed, "")
|
||||
}
|
||||
|
||||
func (s *server) startReviewedTask(taskID string, confirmed bool, revision string) (Execution, error) {
|
||||
s.store.Lock()
|
||||
task, exists := s.store.Tasks[taskID]
|
||||
if !exists {
|
||||
s.store.Unlock()
|
||||
return Execution{}, errors.New("taskNotFound")
|
||||
}
|
||||
if task.Status == "RUNNING" {
|
||||
if s.taskBusyLocked(taskID) {
|
||||
s.store.Unlock()
|
||||
return Execution{}, errors.New("taskBusy")
|
||||
}
|
||||
if revision != "" && revision != taskRevision(task) {
|
||||
s.store.Unlock()
|
||||
return Execution{}, errors.New("runReviewChanged")
|
||||
}
|
||||
if !task.DryRun && !confirmed {
|
||||
s.store.Unlock()
|
||||
return Execution{}, errors.New("confirmationRequired")
|
||||
@@ -121,8 +148,8 @@ func (s *server) startTask(taskID string, confirmed bool) (Execution, error) {
|
||||
s.store.Unlock()
|
||||
return Execution{}, errors.New("targetBusy")
|
||||
}
|
||||
for _, running := range s.store.Tasks {
|
||||
if running.Status != "RUNNING" {
|
||||
for runningID, running := range s.store.Tasks {
|
||||
if !s.taskBusyLocked(runningID) {
|
||||
continue
|
||||
}
|
||||
other, err := resolveSourceDatabase(s.store.Sources[running.TargetID], running.TargetDatabase)
|
||||
@@ -142,7 +169,148 @@ func (s *server) startTask(taskID string, confirmed bool) (Execution, error) {
|
||||
s.finishExecution(execution.ID, "FAILED", "persistenceFailed", nil)
|
||||
return Execution{}, errors.New("persistenceFailed")
|
||||
}
|
||||
if err := s.startJob(execution.ID, time.Duration(task.TimeoutSeconds)*time.Second, func(ctx context.Context) { s.runTask(ctx, task, source, target, execution.ID) }); err != nil {
|
||||
// Budget setup, schema preparation and each data table independently.
|
||||
timeout := time.Duration(task.TimeoutSeconds) * time.Second * time.Duration(2*len(task.Objects)+1)
|
||||
if err := s.startJob(execution.ID, timeout, func(ctx context.Context) { s.runTask(ctx, task, source, target, execution.ID) }); err != nil {
|
||||
s.finishExecution(execution.ID, "FAILED", err.Error(), nil)
|
||||
return Execution{}, err
|
||||
}
|
||||
return execution, nil
|
||||
}
|
||||
|
||||
// retrySkippedObject starts a new execution containing only an object that was
|
||||
// skipped by a previous execution. Retry executions never advance the task's
|
||||
// incremental watermark.
|
||||
func (s *server) retrySkippedObject(executionID, object string) (Execution, error) {
|
||||
return s.retrySkippedObjects(executionID, []string{object})
|
||||
}
|
||||
|
||||
func (s *server) retrySkippedObjects(executionID string, objects []string) (Execution, error) {
|
||||
return s.retryReviewedObjects(executionID, objects, "")
|
||||
}
|
||||
|
||||
func (s *server) retryReviewedObjects(executionID string, objects []string, revision string) (Execution, error) {
|
||||
s.store.Lock()
|
||||
previous, ok := s.store.Executions[executionID]
|
||||
if !ok || unfinishedStatus(previous.Status) {
|
||||
s.store.Unlock()
|
||||
return Execution{}, errors.New("executionNotFound")
|
||||
}
|
||||
if len(objects) == 0 {
|
||||
s.store.Unlock()
|
||||
return Execution{}, errors.New("objectNotSkipped")
|
||||
}
|
||||
selected := make(map[string]bool)
|
||||
for _, object := range objects {
|
||||
found := false
|
||||
switch previous.ObjectStatuses[object] {
|
||||
case "COMPLETED", "RUNNING", "RETRYING":
|
||||
s.store.Unlock()
|
||||
return Execution{}, errors.New("objectNotSkipped")
|
||||
case "FAILED", "SKIPPED", "LIMIT_REACHED", "PENDING":
|
||||
found = true
|
||||
}
|
||||
for _, name := range previous.SkippedObjects {
|
||||
if name == object {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found || selected[object] {
|
||||
s.store.Unlock()
|
||||
return Execution{}, errors.New("objectNotSkipped")
|
||||
}
|
||||
selected[object] = true
|
||||
}
|
||||
task, ok := s.store.Tasks[previous.TaskID]
|
||||
if !ok {
|
||||
s.store.Unlock()
|
||||
return Execution{}, errors.New("taskNotFound")
|
||||
}
|
||||
if revision != "" && revision != taskRevision(task) {
|
||||
s.store.Unlock()
|
||||
return Execution{}, errors.New("retryReviewChanged")
|
||||
}
|
||||
if s.taskBusyLocked(task.ID) {
|
||||
s.store.Unlock()
|
||||
return Execution{}, errors.New("taskBusy")
|
||||
}
|
||||
source, sok := s.store.Sources[task.SourceID]
|
||||
target, tok := s.store.Sources[task.TargetID]
|
||||
if !sok || !tok {
|
||||
s.store.Unlock()
|
||||
return Execution{}, errors.New("sourceNotFound")
|
||||
}
|
||||
if err := validateTask(&task, source, target); err != nil {
|
||||
s.store.Unlock()
|
||||
return Execution{}, err
|
||||
}
|
||||
// Follow the configured data order; tables removed from the edited task
|
||||
// remain independently retryable and follow it in their requested order.
|
||||
ordered := make([]string, 0, len(objects))
|
||||
for _, object := range task.Objects {
|
||||
if selected[object] {
|
||||
ordered = append(ordered, object)
|
||||
delete(selected, object)
|
||||
}
|
||||
}
|
||||
for _, object := range objects {
|
||||
if selected[object] {
|
||||
ordered = append(ordered, object)
|
||||
}
|
||||
}
|
||||
objects = ordered
|
||||
retryTask := task
|
||||
retryTask.Objects = append([]string(nil), objects...)
|
||||
if err := validateTask(&retryTask, source, target); err != nil {
|
||||
s.store.Unlock()
|
||||
return Execution{}, err
|
||||
}
|
||||
source, _ = resolveSourceDatabase(source, task.SourceDatabase)
|
||||
target, _ = resolveSourceDatabase(target, task.TargetDatabase)
|
||||
if s.maintenanceBusyLocked(target) || s.maintenanceBusyLocked(source) {
|
||||
s.store.Unlock()
|
||||
return Execution{}, errors.New("targetBusy")
|
||||
}
|
||||
for runningID, running := range s.store.Tasks {
|
||||
if !s.taskBusyLocked(runningID) {
|
||||
continue
|
||||
}
|
||||
other, err := resolveSourceDatabase(s.store.Sources[running.TargetID], running.TargetDatabase)
|
||||
if err == nil && targetKey(other) == targetKey(target) {
|
||||
s.store.Unlock()
|
||||
return Execution{}, errors.New("targetBusy")
|
||||
}
|
||||
}
|
||||
now := time.Now().UTC()
|
||||
execution := Execution{ID: id(s.store, "exec"), TaskID: task.ID, Status: "RUNNING", StartedAt: now, DryRun: task.DryRun, RetryOf: executionID}
|
||||
execution.RetryOriginalStatuses = make(map[string]string, len(objects))
|
||||
execution.ObjectStatuses = make(map[string]string, len(objects))
|
||||
if previous.ObjectStatuses == nil {
|
||||
previous.ObjectStatuses = make(map[string]string)
|
||||
}
|
||||
for _, object := range objects {
|
||||
original := previous.ObjectStatuses[object]
|
||||
if original == "" {
|
||||
original = "SKIPPED"
|
||||
}
|
||||
execution.RetryOriginalStatuses[object] = original
|
||||
execution.ObjectStatuses[object] = "PENDING"
|
||||
previous.ObjectStatuses[object] = "RETRYING"
|
||||
}
|
||||
s.store.Executions[executionID] = previous
|
||||
task.Status = "RUNNING"
|
||||
task.LastRunAt = &now
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions[execution.ID] = execution
|
||||
s.store.Unlock()
|
||||
if err := s.persist(); err != nil {
|
||||
s.finishExecution(execution.ID, "FAILED", "persistenceFailed", nil)
|
||||
return Execution{}, errors.New("persistenceFailed")
|
||||
}
|
||||
// Each retry table receives its own timeout, plus one interval for setup.
|
||||
retryTimeout := time.Duration(task.TimeoutSeconds) * time.Second * time.Duration(2*len(objects)+1)
|
||||
if err := s.startJob(execution.ID, retryTimeout, func(ctx context.Context) { s.runTask(ctx, retryTask, source, target, execution.ID) }); err != nil {
|
||||
s.finishExecution(execution.ID, "FAILED", err.Error(), nil)
|
||||
return Execution{}, err
|
||||
}
|
||||
@@ -152,14 +320,59 @@ func (s *server) startTask(taskID string, confirmed bool) (Execution, error) {
|
||||
func (s *server) finishExecution(executionID, status, code string, cursor any) {
|
||||
now := time.Now().UTC()
|
||||
s.store.Lock()
|
||||
execution := s.store.Executions[executionID]
|
||||
execution, exists := s.store.Executions[executionID]
|
||||
if !exists || (execution.FinishedAt != nil && !unfinishedStatus(execution.Status)) {
|
||||
s.store.Unlock()
|
||||
return
|
||||
}
|
||||
execution.Status = status
|
||||
execution.Error = code
|
||||
execution.FinishedAt = &now
|
||||
for object, state := range execution.SchemaStatuses {
|
||||
if state == "RUNNING" {
|
||||
execution.SchemaStatuses[object] = "FAILED"
|
||||
}
|
||||
}
|
||||
retryable := make(map[string]bool, len(execution.SkippedObjects))
|
||||
for _, object := range execution.SkippedObjects {
|
||||
retryable[object] = true
|
||||
}
|
||||
var unfinished []string
|
||||
for object, state := range execution.ObjectStatuses {
|
||||
// A cancelled schema worker has not started its data reader, but its
|
||||
// table failed rather than merely remaining in the waiting queue.
|
||||
if state == "PENDING" && execution.SchemaStatuses[object] == "FAILED" && code != "" {
|
||||
state = "RUNNING"
|
||||
}
|
||||
if (state == "RUNNING" || state == "PENDING") && !retryable[object] {
|
||||
unfinished = append(unfinished, object)
|
||||
}
|
||||
if state == "RUNNING" {
|
||||
execution.ObjectStatuses[object] = "FAILED"
|
||||
if execution.ObjectErrors == nil {
|
||||
execution.ObjectErrors = make(map[string]string)
|
||||
}
|
||||
execution.ObjectErrors[object] = code
|
||||
if execution.ObjectFinishedAt == nil {
|
||||
execution.ObjectFinishedAt = make(map[string]time.Time)
|
||||
}
|
||||
execution.ObjectFinishedAt[object] = now
|
||||
}
|
||||
}
|
||||
sort.Strings(unfinished)
|
||||
execution.SkippedObjects = append(execution.SkippedObjects, unfinished...)
|
||||
s.store.Executions[executionID] = execution
|
||||
taskStatus := status
|
||||
if execution.RetryOf != "" {
|
||||
if previous, exists := s.store.Executions[execution.RetryOf]; exists {
|
||||
previous, _ = mergeRetryResult(previous, execution, false)
|
||||
s.store.Executions[execution.RetryOf] = previous
|
||||
taskStatus = previous.Status
|
||||
}
|
||||
}
|
||||
if task, exists := s.store.Tasks[execution.TaskID]; exists {
|
||||
task.Status = status
|
||||
if status == "COMPLETED" && !task.DryRun && task.Mode == "incremental" && cursor != nil && execution.Rows < task.MaxRows {
|
||||
task.Status = taskStatus
|
||||
if status == "COMPLETED" && execution.RetryOf == "" && len(execution.SkippedObjects) == 0 && !task.DryRun && task.Mode == "incremental" && cursor != nil && (task.MaxRows == -1 || execution.Rows < task.MaxRows) {
|
||||
task.IncrementalValue = filterWireValue(cursor)
|
||||
}
|
||||
s.store.Tasks[task.ID] = task
|
||||
@@ -179,6 +392,30 @@ func executionError(err error) string {
|
||||
}
|
||||
|
||||
func (s *server) runTask(ctx context.Context, task Task, source, target DataSource, executionID string) {
|
||||
s.store.Lock()
|
||||
initial := s.store.Executions[executionID]
|
||||
initial.ObjectStatuses = make(map[string]string, len(task.Objects))
|
||||
initial.Stage = "SCHEMA"
|
||||
initial.SchemaStatuses = make(map[string]string, len(task.Objects))
|
||||
initial.ObjectErrors = make(map[string]string)
|
||||
initial.ObjectStartedAt = make(map[string]time.Time)
|
||||
initial.ObjectFinishedAt = make(map[string]time.Time)
|
||||
initial.ObjectRows = make(map[string]int64, len(task.Objects))
|
||||
initial.ObjectWritten = make(map[string]int64, len(task.Objects))
|
||||
initial.ObjectSkipped = make(map[string]int64, len(task.Objects))
|
||||
initial.TargetObjects = make(map[string]string, len(task.Objects))
|
||||
for _, object := range task.Objects {
|
||||
targetName := task.TargetObjects[object]
|
||||
if targetName == "" {
|
||||
targetName = object
|
||||
}
|
||||
initial.TargetObjects[object] = targetName
|
||||
initial.ObjectStatuses[object] = "PENDING"
|
||||
initial.SchemaStatuses[object] = "PENDING"
|
||||
initial.ObjectRows[object], initial.ObjectWritten[object], initial.ObjectSkipped[object] = 0, 0, 0
|
||||
}
|
||||
s.store.Executions[executionID] = initial
|
||||
s.store.Unlock()
|
||||
var runError error
|
||||
var cursor any
|
||||
defer func() {
|
||||
@@ -196,6 +433,12 @@ func (s *server) runTask(ctx context.Context, task Task, source, target DataSour
|
||||
status = "CANCELLED"
|
||||
}
|
||||
}
|
||||
if status == "COMPLETED" {
|
||||
s.store.RLock()
|
||||
result := s.store.Executions[executionID]
|
||||
status, code = executionTableSummary(result)
|
||||
s.store.RUnlock()
|
||||
}
|
||||
s.finishExecution(executionID, status, code, cursor)
|
||||
}()
|
||||
reader, err := s.connectRetry(ctx, source, nil, task.Retries)
|
||||
@@ -204,29 +447,53 @@ func (s *server) runTask(ctx context.Context, task Task, source, target DataSour
|
||||
return
|
||||
}
|
||||
defer reader.Close(context.Background())
|
||||
if mysql, ok := reader.(*mysqlAdapter); ok {
|
||||
// The adapter closes with this job. Keep healthy gateway sessions
|
||||
// through long scans instead of rotating them halfway through a table.
|
||||
mysql.database.SetConnMaxLifetime(0)
|
||||
mysql.database.SetMaxIdleConns(4)
|
||||
}
|
||||
writer, err := s.connectRetry(ctx, target, func(entry SQLAudit) { entry.TaskID = task.ID; entry.ExecutionID = executionID; s.recordAudit(entry) }, task.Retries)
|
||||
if err != nil {
|
||||
runError = err
|
||||
return
|
||||
}
|
||||
defer writer.Close(context.Background())
|
||||
for _, object := range task.Objects {
|
||||
if _, err = reader.Inspect(ctx, object); err != nil {
|
||||
runError = err
|
||||
return
|
||||
}
|
||||
targetName := task.TargetObjects[object]
|
||||
if targetName == "" {
|
||||
targetName = object
|
||||
}
|
||||
if _, err = writer.Inspect(ctx, targetName); err != nil {
|
||||
runError = err
|
||||
return
|
||||
}
|
||||
if mysql, ok := writer.(*mysqlAdapter); ok {
|
||||
mysql.database.SetConnMaxLifetime(0)
|
||||
mysql.database.SetMaxIdleConns(4)
|
||||
}
|
||||
objects := append([]string(nil), task.Objects...)
|
||||
if err := s.runSchemaPhase(ctx, task, reader, writer, source, target, objects, executionID); err != nil {
|
||||
runError = err
|
||||
return
|
||||
}
|
||||
s.store.Lock()
|
||||
phase := s.store.Executions[executionID]
|
||||
phase.Stage = "DATA"
|
||||
s.store.Executions[executionID] = phase
|
||||
s.store.Unlock()
|
||||
|
||||
remaining := task.MaxRows
|
||||
for _, object := range task.Objects {
|
||||
if remaining <= 0 {
|
||||
if task.MaxRows == -1 {
|
||||
remaining = 0 // Adapter read limit 0 means scan to EOF.
|
||||
}
|
||||
// Keep one fallback for the current table rather than retaining a deferred
|
||||
// closure and context for every completed table until the task ends.
|
||||
var activeTable *tableRun
|
||||
defer func() {
|
||||
if activeTable == nil {
|
||||
return
|
||||
}
|
||||
activeTable.cancel()
|
||||
s.jobsMu.Lock()
|
||||
if s.tableRuns[executionID] == activeTable {
|
||||
delete(s.tableRuns, executionID)
|
||||
}
|
||||
s.jobsMu.Unlock()
|
||||
}()
|
||||
for _, object := range objects {
|
||||
if task.MaxRows != -1 && remaining <= 0 {
|
||||
break
|
||||
}
|
||||
targetName := task.TargetObjects[object]
|
||||
@@ -236,84 +503,205 @@ func (s *server) runTask(ctx context.Context, task Task, source, target DataSour
|
||||
s.store.Lock()
|
||||
current := s.store.Executions[executionID]
|
||||
current.Object = object
|
||||
if current.ObjectStatuses[object] == "FAILED" {
|
||||
s.store.Unlock()
|
||||
continue
|
||||
}
|
||||
current.ObjectStatuses[object] = "RUNNING"
|
||||
current.ObjectStartedAt[object] = time.Now().UTC()
|
||||
s.store.Executions[executionID] = current
|
||||
s.store.Unlock()
|
||||
// Register cancellation before inspecting data metadata, so a blocked
|
||||
// inspection shares the table's timeout and can be skipped immediately.
|
||||
tableCtx, tableCancel := context.WithCancel(ctx)
|
||||
if task.TimeoutSeconds > 0 {
|
||||
tableCancel()
|
||||
tableCtx, tableCancel = context.WithTimeout(ctx, time.Duration(task.TimeoutSeconds)*time.Second)
|
||||
}
|
||||
run := &tableRun{object: object, cancel: tableCancel}
|
||||
activeTable = run
|
||||
s.jobsMu.Lock()
|
||||
if s.tableRuns == nil {
|
||||
s.tableRuns = make(map[string]*tableRun)
|
||||
}
|
||||
s.tableRuns[executionID] = run
|
||||
s.jobsMu.Unlock()
|
||||
filters := append([]Filter(nil), task.Filters...)
|
||||
sourceInfo, inspectErr := reader.Inspect(ctx, object)
|
||||
if inspectErr != nil {
|
||||
runError = inspectErr
|
||||
return
|
||||
}
|
||||
if task.Mode == "incremental" {
|
||||
sourceInfo, inspectErr := inspectSyncObject(tableCtx, reader, object)
|
||||
err = inspectErr
|
||||
if err == nil && task.Mode == "incremental" {
|
||||
value, valueErr := filterValue(task.IncrementalValue)
|
||||
if valueErr != nil {
|
||||
runError = valueErr
|
||||
return
|
||||
err = valueErr
|
||||
if err == nil {
|
||||
filters = append(filters, Filter{Field: task.IncrementalField, Operator: "gte", Value: value})
|
||||
}
|
||||
filters = append(filters, Filter{Field: task.IncrementalField, Operator: "gte", Value: value})
|
||||
}
|
||||
err = reader.Read(ctx, object, ReadOptions{BatchSize: task.BatchSize, MaxRows: remaining, Filters: filters}, func(batch []Record) error {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
converted := make([]Record, 0, len(batch))
|
||||
for _, record := range batch {
|
||||
if task.Mode == "incremental" {
|
||||
value, ok := recordPath(record, task.IncrementalField)
|
||||
if ok && (cursor == nil || compareCursor(value, cursor) > 0) {
|
||||
cursor = value
|
||||
}
|
||||
}
|
||||
if isProduction(source) {
|
||||
masked := automaticMask(record)
|
||||
for _, column := range sourceInfo.Columns {
|
||||
if column.Primary {
|
||||
before, _ := recordPath(record, column.Name)
|
||||
after, _ := recordPath(masked, column.Name)
|
||||
if scalarText(before) != scalarText(after) {
|
||||
return errors.New("maskingKeyBlocked")
|
||||
}
|
||||
}
|
||||
}
|
||||
record = masked
|
||||
}
|
||||
result, err := transformRecord(record, task.Mapping, task.Defaults, task.Conversions, task.Omit, task.Masking)
|
||||
if err != nil {
|
||||
if err == nil {
|
||||
err = tableCtx.Err()
|
||||
}
|
||||
if err == nil {
|
||||
err = reader.Read(tableCtx, object, ReadOptions{BatchSize: task.BatchSize, MaxRows: remaining, Filters: filters, JSONBatch: task.Mode == "full" && source.Type == "mysql" && target.Type == "mysql"}, func(batch []Record) error {
|
||||
if err := tableCtx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
converted = append(converted, result)
|
||||
}
|
||||
var result WriteResult
|
||||
var writeErr error
|
||||
for attempt := 0; attempt <= task.Retries; attempt++ {
|
||||
result, writeErr = writer.Write(ctx, targetName, converted, WriteOptions{Conflict: task.Conflict, KeyFields: task.KeyFields, DryRun: task.DryRun})
|
||||
if writeErr == nil || result.Rows > 0 || result.Written > 0 || !transientError(writeErr) || attempt == task.Retries {
|
||||
break
|
||||
// Count records received from the source, including batches that
|
||||
// later fail conversion or are only partially written.
|
||||
s.store.Lock()
|
||||
read := s.store.Executions[executionID]
|
||||
read.Rows += int64(len(batch))
|
||||
read.ObjectRows[object] += int64(len(batch))
|
||||
s.store.Executions[executionID] = read
|
||||
s.store.Unlock()
|
||||
if task.MaxRows != -1 {
|
||||
remaining -= int64(len(batch))
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
case <-time.After(time.Duration(attempt+1) * 250 * time.Millisecond):
|
||||
converted := make([]Record, 0, len(batch))
|
||||
for _, record := range batch {
|
||||
if task.Mode == "incremental" {
|
||||
value, ok := recordPath(record, task.IncrementalField)
|
||||
if !ok || !incrementalCursorValid(value) {
|
||||
return errors.New("incrementalCursorInvalid")
|
||||
}
|
||||
if cursor == nil || compareCursor(value, cursor) > 0 {
|
||||
cursor = value
|
||||
}
|
||||
}
|
||||
if isProduction(source) {
|
||||
masked := automaticMask(record)
|
||||
for _, column := range sourceInfo.Columns {
|
||||
if column.Primary {
|
||||
before, _ := recordPath(record, column.Name)
|
||||
after, _ := recordPath(masked, column.Name)
|
||||
if scalarText(before) != scalarText(after) {
|
||||
return errors.New("maskingKeyBlocked")
|
||||
}
|
||||
}
|
||||
}
|
||||
record = masked
|
||||
}
|
||||
result, err := transformRecord(record, task.Mapping, task.Defaults, task.Conversions, task.Omit, task.Masking)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
converted = append(converted, result)
|
||||
}
|
||||
}
|
||||
var result WriteResult
|
||||
var writeErr error
|
||||
for attempt := 0; attempt <= task.Retries; attempt++ {
|
||||
result, writeErr = writer.Write(tableCtx, targetName, converted, WriteOptions{Conflict: task.Conflict, KeyFields: task.KeyFields, DryRun: task.DryRun})
|
||||
if writeErr == nil || result.Rows > 0 || result.Written > 0 || !transientError(writeErr) || attempt == task.Retries {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-tableCtx.Done():
|
||||
return tableCtx.Err()
|
||||
case <-time.After(time.Duration(attempt+1) * 250 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
s.store.Lock()
|
||||
current := s.store.Executions[executionID]
|
||||
current.Written += result.Written
|
||||
current.Skipped += result.Skipped
|
||||
current.ObjectWritten[object] += result.Written
|
||||
current.ObjectSkipped[object] += result.Skipped
|
||||
s.store.Executions[executionID] = current
|
||||
s.store.Unlock()
|
||||
if err := s.persist(); err != nil {
|
||||
return errors.New("persistenceFailed")
|
||||
}
|
||||
return writeErr
|
||||
})
|
||||
}
|
||||
tableError := tableCtx.Err()
|
||||
if err == nil && tableError != nil {
|
||||
err = tableError
|
||||
}
|
||||
tableCancel()
|
||||
s.jobsMu.Lock()
|
||||
skipped := run.skipped
|
||||
delete(s.tableRuns, executionID)
|
||||
s.jobsMu.Unlock()
|
||||
activeTable = nil
|
||||
if skipped && ctx.Err() == nil {
|
||||
s.store.Lock()
|
||||
current := s.store.Executions[executionID]
|
||||
current.Rows += result.Rows
|
||||
current.Written += result.Written
|
||||
current.Skipped += result.Skipped
|
||||
current.SkippedObjects = append(current.SkippedObjects, object)
|
||||
current.ObjectStatuses[object] = "SKIPPED"
|
||||
current.ObjectFinishedAt[object] = time.Now().UTC()
|
||||
s.store.Executions[executionID] = current
|
||||
s.store.Unlock()
|
||||
remaining -= int64(len(batch))
|
||||
if err := s.persist(); err != nil {
|
||||
return errors.New("persistenceFailed")
|
||||
runError = errors.New("persistenceFailed")
|
||||
return
|
||||
}
|
||||
return writeErr
|
||||
})
|
||||
continue
|
||||
}
|
||||
if err != nil {
|
||||
runError = err
|
||||
return
|
||||
if taskFatalError(err) && !(errors.Is(tableError, context.DeadlineExceeded) && ctx.Err() == nil) {
|
||||
runError = err
|
||||
return
|
||||
}
|
||||
if markErr := s.markSkippedObject(executionID, object, err); markErr != nil {
|
||||
runError = markErr
|
||||
return
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
s.store.Lock()
|
||||
completed := s.store.Executions[executionID]
|
||||
completed.ObjectStatuses[object] = "COMPLETED"
|
||||
completed.ObjectFinishedAt[object] = time.Now().UTC()
|
||||
if task.MaxRows != -1 && remaining <= 0 {
|
||||
completed.ObjectStatuses[object] = "LIMIT_REACHED"
|
||||
completed.SkippedObjects = append(completed.SkippedObjects, object)
|
||||
}
|
||||
s.store.Executions[executionID] = completed
|
||||
s.store.Unlock()
|
||||
}
|
||||
}
|
||||
|
||||
// taskFatalError identifies failures that make continuing unsafe. Errors tied
|
||||
// to one source/target object are recorded as skipped so later objects run.
|
||||
func taskFatalError(err error) bool {
|
||||
if err == nil {
|
||||
return false
|
||||
}
|
||||
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||
return true
|
||||
}
|
||||
return AdapterErrorCode(err) == "persistenceFailed"
|
||||
}
|
||||
|
||||
func (s *server) markSkippedObject(executionID, object string, failures ...error) error {
|
||||
s.store.Lock()
|
||||
current := s.store.Executions[executionID]
|
||||
if current.ObjectErrors == nil {
|
||||
current.ObjectErrors = make(map[string]string)
|
||||
}
|
||||
if len(failures) > 0 && failures[0] != nil {
|
||||
current.ObjectErrors[object] = executionError(failures[0])
|
||||
}
|
||||
if current.ObjectFinishedAt != nil {
|
||||
current.ObjectFinishedAt[object] = time.Now().UTC()
|
||||
}
|
||||
if current.ObjectStatuses != nil {
|
||||
current.ObjectStatuses[object] = "FAILED"
|
||||
s.store.Executions[executionID] = current
|
||||
}
|
||||
for _, existing := range current.SkippedObjects {
|
||||
if existing == object {
|
||||
s.store.Unlock()
|
||||
return nil
|
||||
}
|
||||
}
|
||||
current.SkippedObjects = append(current.SkippedObjects, object)
|
||||
s.store.Executions[executionID] = current
|
||||
s.store.Unlock()
|
||||
if err := s.persist(); err != nil {
|
||||
return errors.New("persistenceFailed")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func transientError(err error) bool {
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type countedRead struct {
|
||||
object string
|
||||
limit int64
|
||||
}
|
||||
type executionCountsAdapter struct {
|
||||
tableSkipAdapter
|
||||
objects map[string][]Record
|
||||
reads *[]countedRead
|
||||
partial bool
|
||||
}
|
||||
|
||||
func (a *executionCountsAdapter) Read(_ context.Context, object string, options ReadOptions, consume func([]Record) error) error {
|
||||
*a.reads = append(*a.reads, countedRead{object, options.MaxRows})
|
||||
records := a.objects[object]
|
||||
if options.MaxRows > 0 && int64(len(records)) > options.MaxRows {
|
||||
records = records[:options.MaxRows]
|
||||
}
|
||||
return consume(records)
|
||||
}
|
||||
|
||||
func (a *executionCountsAdapter) Write(ctx context.Context, object string, records []Record, options WriteOptions) (WriteResult, error) {
|
||||
if a.partial && object == "bad" {
|
||||
return WriteResult{Rows: 1, Written: 1}, errors.New("mysqlOperationFailed")
|
||||
}
|
||||
return a.tableSkipAdapter.Write(ctx, object, records, options)
|
||||
}
|
||||
|
||||
func TestExecutionReadsIncludePartiallyWrittenBatch(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
var reads []countedRead
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
return &executionCountsAdapter{objects: map[string][]Record{"bad": {{"id": 1}, {"id": 2}}, "good": {{"id": 3}, {"id": 4}}}, reads: &reads, partial: true}, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"bad", "good"}, MaxRows: -1, BatchSize: 10}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
result := s.store.Executions["exec"]
|
||||
if result.Rows != 4 || result.Written != 3 || result.Status != "PARTIAL" || result.ObjectStatuses["bad"] != "FAILED" || result.ObjectStatuses["good"] != "COMPLETED" {
|
||||
t.Fatalf("read count confused with write attempts: %+v", result)
|
||||
}
|
||||
if result.ObjectRows["bad"] != 2 || result.ObjectWritten["bad"] != 1 || result.ObjectRows["good"] != 2 || result.ObjectWritten["good"] != 2 {
|
||||
t.Fatalf("table counts not recorded: %+v / %+v", result.ObjectRows, result.ObjectWritten)
|
||||
}
|
||||
restored, _, err := loadStore(s.dataDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if restored.Executions["exec"].Rows != 4 || restored.Executions["exec"].Written != 3 {
|
||||
t.Fatal("restart lost actual read/write counts")
|
||||
}
|
||||
if restored.Executions["exec"].ObjectRows["bad"] != 2 || restored.Executions["exec"].ObjectWritten["good"] != 2 {
|
||||
t.Fatal("restart lost table counts")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFailedConversionConsumesTaskReadBudget(t *testing.T) {
|
||||
for _, limit := range []int64{2, 3} {
|
||||
s := databaseOperationServer(t)
|
||||
var reads []countedRead
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
return &executionCountsAdapter{objects: map[string][]Record{"bad": {{"id": "not an integer"}, {"id": 2}}, "good": {{"id": 3}, {"id": 4}}}, reads: &reads}, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"bad", "good"}, MaxRows: limit, BatchSize: 10, Conversions: map[string]string{"id": "int"}}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
result := s.store.Executions["exec"]
|
||||
if result.Rows != limit || result.Written != limit-2 || result.ObjectStatuses["bad"] != "FAILED" {
|
||||
t.Fatalf("failed batch escaped read budget %d: %+v", limit, result)
|
||||
}
|
||||
if limit == 2 && (len(reads) != 1 || result.ObjectStatuses["good"] != "PENDING") {
|
||||
t.Fatal("read next table after exhausted budget")
|
||||
}
|
||||
if limit == 3 && (len(reads) != 2 || reads[1].limit != 1 || result.ObjectStatuses["good"] != "LIMIT_REACHED") {
|
||||
t.Fatalf("next table received wrong remaining budget: %+v", reads)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,136 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestExecutionDeletion(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name, status string
|
||||
retry bool
|
||||
want int
|
||||
}{
|
||||
{"finished", "COMPLETED", false, 200},
|
||||
{"failed", "FAILED", false, 200},
|
||||
{"running", "RUNNING", false, 409},
|
||||
{"pending", "PENDING", false, 409},
|
||||
{"retry running", "COMPLETED", true, 409},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Executions["original"] = Execution{ID: "original", Status: tc.status}
|
||||
if tc.retry {
|
||||
s.store.Executions["retry"] = Execution{ID: "retry", RetryOf: "original", Status: "RUNNING"}
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
s.execution(rec, httptest.NewRequest(http.MethodDelete, "/api/executions/original", nil))
|
||||
if rec.Code != tc.want {
|
||||
t.Fatalf("status=%d body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
_, exists := s.store.Executions["original"]
|
||||
if exists != (tc.want != 200) {
|
||||
t.Fatal("unexpected deletion result")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutionDeletionRemovesRetryFamilyAndPersists(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
for _, execution := range []Execution{
|
||||
{ID: "original", Status: "PARTIAL"},
|
||||
{ID: "retry-1", RetryOf: "original", Status: "FAILED"},
|
||||
{ID: "retry-2", RetryOf: "original", Status: "COMPLETED"},
|
||||
{ID: "nested", RetryOf: "retry-1", Status: "COMPLETED"},
|
||||
{ID: "unrelated", Status: "COMPLETED"},
|
||||
{ID: "unrelated-retry", RetryOf: "unrelated", Status: "FAILED"},
|
||||
} {
|
||||
s.store.Executions[execution.ID] = execution
|
||||
}
|
||||
s.store.Audits["audit"] = SQLAudit{ID: "audit", ExecutionID: "original"}
|
||||
rec := httptest.NewRecorder()
|
||||
s.execution(rec, httptest.NewRequest(http.MethodDelete, "/api/executions/original", nil))
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status=%d body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
reloaded, _, err := loadStore(s.dataDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(reloaded.Executions) != 2 || reloaded.Executions["unrelated"].ID == "" || reloaded.Executions["unrelated-retry"].ID == "" {
|
||||
t.Fatalf("retry family was not deleted atomically: %v", reloaded.Executions)
|
||||
}
|
||||
if reloaded.Audits["audit"].ExecutionID != "original" {
|
||||
t.Fatal("execution deletion must preserve SQL audit history")
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutionDeletionBlocksActiveNestedRetry(t *testing.T) {
|
||||
for _, status := range []string{"RUNNING", "PENDING"} {
|
||||
t.Run(status, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Executions["original"] = Execution{ID: "original", Status: "FAILED"}
|
||||
s.store.Executions["retry"] = Execution{ID: "retry", RetryOf: "original", Status: "FAILED"}
|
||||
s.store.Executions["nested"] = Execution{ID: "nested", RetryOf: "retry", Status: status}
|
||||
rec := httptest.NewRecorder()
|
||||
s.execution(rec, httptest.NewRequest(http.MethodDelete, "/api/executions/original", nil))
|
||||
if rec.Code != http.StatusConflict || len(s.store.Executions) != 3 {
|
||||
t.Fatalf("active family was deleted: status=%d records=%v", rec.Code, s.store.Executions)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutionHistoryPreservesActiveRetryAncestry(t *testing.T) {
|
||||
records := make(map[string]Execution)
|
||||
for sequence := 1; sequence <= historyLimit+5; sequence++ {
|
||||
recordID := fmt.Sprintf("exec-%d", sequence)
|
||||
records[recordID] = Execution{ID: recordID, Status: "COMPLETED"}
|
||||
}
|
||||
records["exec-2"] = Execution{ID: "exec-2", RetryOf: "exec-1", Status: "FAILED"}
|
||||
records["exec-3"] = Execution{ID: "exec-3", RetryOf: "exec-2", Status: "RUNNING"}
|
||||
trimExecutionHistory(records)
|
||||
for _, recordID := range []string{"exec-1", "exec-2", "exec-3"} {
|
||||
if records[recordID].ID == "" {
|
||||
t.Fatalf("active retry lost ancestor %s", recordID)
|
||||
}
|
||||
}
|
||||
if len(records) != historyLimit+3 {
|
||||
t.Fatalf("unexpected retained history: %d", len(records))
|
||||
}
|
||||
// Once the retry finishes the usual history bound applies again.
|
||||
record := records["exec-3"]
|
||||
record.Status = "COMPLETED"
|
||||
records[record.ID] = record
|
||||
trimExecutionHistory(records)
|
||||
if len(records) > historyLimit {
|
||||
t.Fatalf("terminal history remains unbounded: %d", len(records))
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutionHistoryRemovesTerminalOrphansAndKeepsActiveOnes(t *testing.T) {
|
||||
records := map[string]Execution{
|
||||
"orphan": {ID: "orphan", RetryOf: "missing", Status: "FAILED"},
|
||||
"nested": {ID: "nested", RetryOf: "orphan", Status: "COMPLETED"},
|
||||
"active": {ID: "active", RetryOf: "missing", Status: "RUNNING"},
|
||||
"root": {ID: "root", Status: "COMPLETED"},
|
||||
}
|
||||
trimExecutionHistory(records)
|
||||
if len(records) != 2 || records["active"].ID == "" || records["root"].ID == "" {
|
||||
t.Fatalf("unexpected orphan cleanup: %v", records)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutionFamilyToleratesCyclicHistoricalLinks(t *testing.T) {
|
||||
records := map[string]Execution{
|
||||
"a": {ID: "a", RetryOf: "b", Status: "COMPLETED"},
|
||||
"b": {ID: "b", RetryOf: "a", Status: "COMPLETED"},
|
||||
}
|
||||
if family := executionFamily(records, "a"); len(family) != 2 {
|
||||
t.Fatalf("unexpected cyclic family: %v", family)
|
||||
}
|
||||
trimExecutionHistory(records)
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
package main
|
||||
|
||||
// executionFamily returns the selected execution and all of its retries. The
|
||||
// caller holds the store lock so checking and deleting a family is atomic.
|
||||
func executionFamily(records map[string]Execution, executionID string) []string {
|
||||
children := make(map[string][]string)
|
||||
for recordID, record := range records {
|
||||
if record.RetryOf != "" {
|
||||
children[record.RetryOf] = append(children[record.RetryOf], recordID)
|
||||
}
|
||||
}
|
||||
family := []string{executionID}
|
||||
seen := map[string]bool{executionID: true}
|
||||
for index := 0; index < len(family); index++ {
|
||||
for _, child := range children[family[index]] {
|
||||
if !seen[child] {
|
||||
seen[child] = true
|
||||
family = append(family, child)
|
||||
}
|
||||
}
|
||||
}
|
||||
return family
|
||||
}
|
||||
|
||||
func trimExecutionHistory(records map[string]Execution) {
|
||||
// Keep the entire ancestry of active retries; completion still needs the
|
||||
// original record even when it is older than the normal history window.
|
||||
protected := make(map[string]bool)
|
||||
queue := make([]string, 0)
|
||||
children := make(map[string][]string)
|
||||
for recordID, record := range records {
|
||||
if unfinishedStatus(record.Status) {
|
||||
protected[recordID] = true
|
||||
queue = append(queue, recordID)
|
||||
}
|
||||
if record.RetryOf != "" {
|
||||
children[record.RetryOf] = append(children[record.RetryOf], recordID)
|
||||
}
|
||||
}
|
||||
for index := 0; index < len(queue); index++ {
|
||||
parent := records[queue[index]].RetryOf
|
||||
if _, exists := records[parent]; parent != "" && exists && !protected[parent] {
|
||||
protected[parent] = true
|
||||
queue = append(queue, parent)
|
||||
}
|
||||
}
|
||||
trimHistory(records, func(record Execution) bool { return protected[record.ID] })
|
||||
// Terminal retries without an original are hidden by the UI and cannot be
|
||||
// retried meaningfully. Remove them and their terminal descendants.
|
||||
queue = queue[:0]
|
||||
for recordID, record := range records {
|
||||
if _, exists := records[record.RetryOf]; record.RetryOf != "" && !exists && !protected[recordID] {
|
||||
queue = append(queue, recordID)
|
||||
}
|
||||
}
|
||||
for index := 0; index < len(queue); index++ {
|
||||
recordID := queue[index]
|
||||
if _, exists := records[recordID]; !exists || protected[recordID] {
|
||||
continue
|
||||
}
|
||||
delete(records, recordID)
|
||||
queue = append(queue, children[recordID]...)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
package main
|
||||
|
||||
import "sort"
|
||||
|
||||
// A deliberate skip or a row limit is incomplete work, not a failed table.
|
||||
// Use the same summary for a normal run and the original record after retry.
|
||||
func executionTableSummary(execution Execution) (string, string) {
|
||||
incomplete := len(execution.SkippedObjects) > 0
|
||||
allFailed := len(execution.ObjectStatuses) > 0
|
||||
for _, state := range execution.ObjectStatuses {
|
||||
if state != "COMPLETED" {
|
||||
incomplete = true
|
||||
}
|
||||
if state != "FAILED" {
|
||||
allFailed = false
|
||||
}
|
||||
}
|
||||
if !incomplete {
|
||||
return "COMPLETED", ""
|
||||
}
|
||||
status := "PARTIAL"
|
||||
if allFailed {
|
||||
status = "FAILED"
|
||||
}
|
||||
objects := make([]string, 0, len(execution.ObjectErrors))
|
||||
for object, code := range execution.ObjectErrors {
|
||||
if code != "" && execution.ObjectStatuses[object] != "COMPLETED" {
|
||||
objects = append(objects, object)
|
||||
}
|
||||
}
|
||||
sort.Strings(objects)
|
||||
if len(objects) > 0 {
|
||||
return status, execution.ObjectErrors[objects[0]]
|
||||
}
|
||||
return status, ""
|
||||
}
|
||||
@@ -0,0 +1,26 @@
|
||||
package main
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestExecutionTableSummary(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
execution Execution
|
||||
status, code string
|
||||
}{
|
||||
{"complete ignores stale errors", Execution{ObjectStatuses: map[string]string{"a": "COMPLETED"}, ObjectErrors: map[string]string{"a": "oldError"}}, "COMPLETED", ""},
|
||||
{"all failed uses deterministic error", Execution{ObjectStatuses: map[string]string{"a": "FAILED", "b": "FAILED"}, SkippedObjects: []string{"b", "a"}, ObjectErrors: map[string]string{"b": "secondError", "a": "firstError"}}, "FAILED", "firstError"},
|
||||
{"mixed results", Execution{ObjectStatuses: map[string]string{"a": "COMPLETED", "b": "FAILED"}, ObjectErrors: map[string]string{"a": "oldError", "b": "currentError"}}, "PARTIAL", "currentError"},
|
||||
{"deliberate skip", Execution{ObjectStatuses: map[string]string{"a": "SKIPPED"}, SkippedObjects: []string{"a"}}, "PARTIAL", ""},
|
||||
{"row limit", Execution{ObjectStatuses: map[string]string{"a": "LIMIT_REACHED", "b": "PENDING"}}, "PARTIAL", ""},
|
||||
{"failed and skipped", Execution{ObjectStatuses: map[string]string{"a": "FAILED", "b": "SKIPPED"}, ObjectErrors: map[string]string{"a": "currentError"}}, "PARTIAL", "currentError"},
|
||||
{"legacy skipped object", Execution{SkippedObjects: []string{"a"}}, "PARTIAL", ""},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
status, code := executionTableSummary(test.execution)
|
||||
if status != test.status || code != test.code {
|
||||
t.Fatalf("got %s/%s, want %s/%s", status, code, test.status, test.code)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
+68
-7
@@ -55,6 +55,7 @@ func (s *server) createExport(w http.ResponseWriter, r *http.Request) {
|
||||
Mode string `json:"mode"`
|
||||
Objects []string `json:"objects"`
|
||||
Filters []Filter `json:"filters"`
|
||||
FiltersJSON string `json:"filters_json"`
|
||||
Masking map[string]string `json:"masking"`
|
||||
MaxRows int64 `json:"max_rows"`
|
||||
BatchSize int `json:"batch_size"`
|
||||
@@ -71,6 +72,22 @@ func (s *server) createExport(w http.ResponseWriter, r *http.Request) {
|
||||
exportRespond(w, http.StatusBadRequest, "invalidExport")
|
||||
return
|
||||
}
|
||||
if input.FiltersJSON != "" {
|
||||
filters, err := parseExactFilters(input.FiltersJSON)
|
||||
if err != nil {
|
||||
exportRespond(w, http.StatusBadRequest, "invalidExportFilter")
|
||||
return
|
||||
}
|
||||
input.Filters = filters
|
||||
}
|
||||
wes := r.URL.Path == "/api/exports/wes"
|
||||
if wes {
|
||||
if input.Mode != "" && input.Mode != "wes" || len(input.Filters) != 0 || len(input.Masking) != 0 {
|
||||
exportRespond(w, http.StatusBadRequest, "invalidExport")
|
||||
return
|
||||
}
|
||||
input.Mode = "wes"
|
||||
}
|
||||
if input.Mode == "" {
|
||||
input.Mode = "both"
|
||||
}
|
||||
@@ -83,7 +100,7 @@ func (s *server) createExport(w http.ResponseWriter, r *http.Request) {
|
||||
if input.BatchSize == 0 {
|
||||
input.BatchSize = 500
|
||||
}
|
||||
if input.SourceID == "" || input.Format != "tar.gz" || (input.Mode != "schema" && input.Mode != "data" && input.Mode != "both") || input.MaxRows < 1 || input.MaxRows > exportMaximumRows || input.BatchSize < 1 || input.BatchSize > adapterMaxBatch || len(input.Objects) == 0 || len(input.Objects) > exportMaximumTables || len(input.Filters) > 100 || len(input.Masking) > 4096 {
|
||||
if input.SourceID == "" || input.Format != "tar.gz" || (!wes && input.Mode != "schema" && input.Mode != "data" && input.Mode != "both") || input.MaxRows < 1 || input.MaxRows > exportMaximumRows || input.BatchSize < 1 || input.BatchSize > adapterMaxBatch || len(input.Objects) > exportMaximumTables || len(input.Filters) > 100 || len(input.Masking) > 4096 {
|
||||
exportRespond(w, http.StatusBadRequest, "invalidExport")
|
||||
return
|
||||
}
|
||||
@@ -120,13 +137,43 @@ func (s *server) createExport(w http.ResponseWriter, r *http.Request) {
|
||||
exportRespond(w, http.StatusNotFound, "sourceNotFound")
|
||||
return
|
||||
}
|
||||
if !strings.EqualFold(source.Type, "mysql") {
|
||||
if !strings.EqualFold(source.Type, "mysql") && !(wes && strings.EqualFold(source.Type, "mongodb")) {
|
||||
s.store.Unlock()
|
||||
exportRespond(w, http.StatusBadRequest, "exportMySQLOnly")
|
||||
return
|
||||
}
|
||||
mongoWES := wes && strings.EqualFold(source.Type, "mongodb")
|
||||
if wes && !mongoWES {
|
||||
input.Objects = wesDictionaryObjects(input.Objects)
|
||||
if len(input.Objects) > exportMaximumTables {
|
||||
s.store.Unlock()
|
||||
exportRespond(w, http.StatusBadRequest, "invalidExportObjects")
|
||||
return
|
||||
}
|
||||
}
|
||||
if !mongoWES && len(input.Objects) == 0 {
|
||||
s.store.Unlock()
|
||||
exportRespond(w, http.StatusBadRequest, "invalidExport")
|
||||
return
|
||||
}
|
||||
if mongoWES {
|
||||
for _, name := range input.Objects {
|
||||
if kind, _ := wesMongoCollection(name); kind == "" {
|
||||
s.store.Unlock()
|
||||
exportRespond(w, http.StatusBadRequest, "wesMongoCollectionsOnly")
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
var err error
|
||||
source, err = resolveSourceDatabase(source, input.Database)
|
||||
if mongoWES {
|
||||
// Authentication may still use the source's configured database. Freeze
|
||||
// it before clearing the business scope for discovery of all databases.
|
||||
source.AuthSource = mongoAuthDatabase(source)
|
||||
source.Database = ""
|
||||
} else {
|
||||
source, err = resolveSourceDatabase(source, input.Database)
|
||||
}
|
||||
if err != nil {
|
||||
s.store.Unlock()
|
||||
exportRespond(w, http.StatusBadRequest, AdapterErrorCode(err))
|
||||
@@ -153,7 +200,13 @@ func (s *server) createExport(w http.ResponseWriter, r *http.Request) {
|
||||
exportRespond(w, http.StatusInternalServerError, "exportPersistenceFailed")
|
||||
return
|
||||
}
|
||||
if err := s.startJob(job.ID, 30*time.Minute, func(ctx context.Context) { s.runSQLExport(ctx, source, job) }); err != nil {
|
||||
if err := s.startJob(job.ID, 30*time.Minute, func(ctx context.Context) {
|
||||
if wes {
|
||||
s.runWESExport(ctx, source, job)
|
||||
} else {
|
||||
s.runSQLExport(ctx, source, job)
|
||||
}
|
||||
}); err != nil {
|
||||
s.store.Lock()
|
||||
delete(s.store.Exports, job.ID)
|
||||
s.store.Unlock()
|
||||
@@ -486,9 +539,10 @@ func (s *server) cleanupExports(maxAge time.Duration) {
|
||||
}
|
||||
|
||||
type exportManifestTable struct {
|
||||
Name string `json:"name"`
|
||||
Rows int64 `json:"rows"`
|
||||
Files []string `json:"files"`
|
||||
Name string `json:"name"`
|
||||
Database string `json:"database,omitempty"`
|
||||
Rows int64 `json:"rows"`
|
||||
Files []string `json:"files"`
|
||||
}
|
||||
|
||||
type exportManifest struct {
|
||||
@@ -503,6 +557,8 @@ type exportManifest struct {
|
||||
ProductionMasking bool `json:"production_masking"`
|
||||
ConsistentSnapshot bool `json:"consistent_snapshot"`
|
||||
Tables []exportManifestTable `json:"tables"`
|
||||
Databases []string `json:"databases,omitempty"`
|
||||
Duplicates []wesMongoDuplicate `json:"duplicate_resources,omitempty"`
|
||||
Checksums map[string]string `json:"sha256"`
|
||||
}
|
||||
|
||||
@@ -565,6 +621,7 @@ func (s *server) finishExport(ctx context.Context, job Export, rows int64, err e
|
||||
}
|
||||
now := time.Now().UTC()
|
||||
current.Rows, current.FinishedAt = rows, &now
|
||||
current.ErrorDetail = ""
|
||||
current.Warnings = append([]string(nil), job.Warnings...)
|
||||
switch {
|
||||
case current.Status == "CANCELLED" || errors.Is(ctx.Err(), context.Canceled):
|
||||
@@ -579,6 +636,10 @@ func (s *server) finishExport(ctx context.Context, job Export, rows int64, err e
|
||||
} else {
|
||||
current.Error = AdapterErrorCode(err)
|
||||
}
|
||||
var configError *wesConfigError
|
||||
if errors.As(err, &configError) {
|
||||
current.ErrorDetail = configError.detail
|
||||
}
|
||||
default:
|
||||
format := exportArtifactFormat(job)
|
||||
partial, pathErr := s.exportArtifactPath(job.ID, format, true)
|
||||
|
||||
@@ -27,7 +27,11 @@ func filterValue(value any) (any, error) {
|
||||
case "$date":
|
||||
return time.Parse(time.RFC3339Nano, text)
|
||||
case "$numberDecimal":
|
||||
return primitive.ParseDecimal128(text)
|
||||
decimal, err := primitive.ParseDecimal128(text)
|
||||
if err != nil || !adapterScalar(decimal) {
|
||||
return nil, errors.New("invalidFilter")
|
||||
}
|
||||
return decimal, nil
|
||||
case "$numberLong":
|
||||
return strconv.ParseInt(text, 10, 64)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
const exactFilterText = `[{"field":"id","operator":"gt","value":9007199254740993},{"field":"amount","operator":"lte","value":12345678901234567890.0001}]`
|
||||
|
||||
func checkExactFilters(t *testing.T, filters []Filter) {
|
||||
t.Helper()
|
||||
if len(filters) != 2 || filters[0].Value != json.Number("9007199254740993") || filters[1].Value != json.Number("12345678901234567890.0001") {
|
||||
t.Fatalf("filter values changed: %+v", filters)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTaskFiltersRemainExactThroughBrowserEditAndRestart(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
filters, err := parseExactFilters(exactFilterText)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task := Task{ID: "task", Name: "task", SourceID: "source", TargetID: "target", SourceDatabase: "app", TargetDatabase: "app", Mode: "range", Objects: []string{"table"}, Filters: filters}
|
||||
s.store.Tasks[task.ID] = task
|
||||
input := taskForAPI(task)
|
||||
if input.FiltersJSON != exactFilterText || taskRevision(input) != taskRevision(task) {
|
||||
t.Fatalf("invalid exact transport: %+v", input)
|
||||
}
|
||||
// Simulate a browser decoding ordinary JSON numeric properties as float64.
|
||||
input.Filters = []Filter{{Field: "id", Operator: "gt", Value: float64(9007199254740992)}}
|
||||
data, err := json.Marshal(input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
s.task(rec, httptest.NewRequest(http.MethodPut, "/api/tasks/task", bytes.NewReader(data)))
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("edit rejected: %s", rec.Body.String())
|
||||
}
|
||||
checkExactFilters(t, s.store.Tasks[task.ID].Filters)
|
||||
if s.store.Tasks[task.ID].FiltersJSON != "" {
|
||||
t.Fatal("transport text persisted")
|
||||
}
|
||||
loaded, _, err := loadStore(s.dataDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
checkExactFilters(t, loaded.Tasks[task.ID].Filters)
|
||||
if taskForAPI(loaded.Tasks[task.ID]).FiltersJSON != exactFilterText {
|
||||
t.Fatal("restart changed editable filter text")
|
||||
}
|
||||
for _, path := range []string{"/api/tasks", "/api/tasks/task"} {
|
||||
rec = httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodGet, path, nil)
|
||||
if path == "/api/tasks" {
|
||||
s.tasks(rec, req)
|
||||
} else {
|
||||
s.task(rec, req)
|
||||
}
|
||||
if rec.Code != 200 || !strings.Contains(rec.Body.String(), `9007199254740993`) || !strings.Contains(rec.Body.String(), `"filters_json"`) {
|
||||
t.Fatalf("missing exact transport: %s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestExactFilterTransportRejectsMalformedOrUnsafeValues(t *testing.T) {
|
||||
for _, raw := range []string{"null", "{}", "[] trailing", `[{"field":"id","operator":"in","value":[]}]`, `[{"field":"id","operator":"gt","value":null}]`, `[{"field":"id","operator":"eq","value":1,"extra":true}]`, `[{"field":"amount","operator":"eq","value":{"$numberDecimal":"NaN"}}]`, `[{"field":"amount","operator":"in","value":[{"$numberDecimal":"Infinity"}]}]`} {
|
||||
t.Run(raw, func(t *testing.T) {
|
||||
if _, err := parseExactFilters(raw); err == nil {
|
||||
t.Fatal("invalid filters accepted")
|
||||
}
|
||||
task := Task{Name: "task", Objects: []string{"table"}, FiltersJSON: raw}
|
||||
if err := validateTask(&task, DataSource{}, DataSource{}); err == nil || err.Error() != "invalidFilter" {
|
||||
t.Fatalf("task accepted raw filters: %v", err)
|
||||
}
|
||||
s := databaseOperationServer(t)
|
||||
data, _ := json.Marshal(map[string]any{"source_id": "source", "database": "app", "objects": []string{"table"}, "filters_json": raw})
|
||||
rec := httptest.NewRecorder()
|
||||
s.createExport(rec, httptest.NewRequest(http.MethodPost, "/api/exports", bytes.NewReader(data)))
|
||||
if rec.Code != 400 || !strings.Contains(rec.Body.String(), "invalidExportFilter") || len(s.store.Exports) != 0 {
|
||||
t.Fatalf("export accepted raw filters: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestSQLExportSavesExactBrowserFilters(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.runtimeCtx = context.Background()
|
||||
s.jobs = map[string]context.CancelFunc{}
|
||||
s.slots = make(chan struct{}, 1)
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
return nil, errors.New("mysqlConnectionFailed")
|
||||
}
|
||||
data, _ := json.Marshal(map[string]any{"source_id": "source", "database": "app", "objects": []string{"table"}, "filters": []Filter{{Field: "id", Operator: "gt", Value: float64(9007199254740992)}}, "filters_json": exactFilterText})
|
||||
rec := httptest.NewRecorder()
|
||||
s.createExport(rec, httptest.NewRequest(http.MethodPost, "/api/exports", bytes.NewReader(data)))
|
||||
s.workers.Wait()
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("export rejected: %s", rec.Body.String())
|
||||
}
|
||||
for _, job := range s.store.Exports {
|
||||
checkExactFilters(t, job.Filters)
|
||||
}
|
||||
loaded, _, err := loadStore(s.dataDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, job := range loaded.Exports {
|
||||
checkExactFilters(t, job.Filters)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type fullScanAdapter struct {
|
||||
tableSkipAdapter
|
||||
limits *[]int64
|
||||
}
|
||||
|
||||
func (a *fullScanAdapter) Read(_ context.Context, _ string, options ReadOptions, consume func([]Record) error) error {
|
||||
*a.limits = append(*a.limits, options.MaxRows)
|
||||
return consume([]Record{{"id": 1}, {"id": 2}})
|
||||
}
|
||||
|
||||
func TestUnlimitedTaskReadsEveryTableToEOF(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
var limits []int64
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
return &fullScanAdapter{limits: &limits}, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"a", "b"}, MaxRows: -1, BatchSize: 10}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
result := s.store.Executions["exec"]
|
||||
if len(limits) != 2 || limits[0] != 0 || limits[1] != 0 || result.Status != "COMPLETED" || result.Written != 4 {
|
||||
t.Fatalf("limits=%v execution=%+v", limits, result)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalTaskContinuesAfterTableTimeout(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) { return &retryTimeoutAdapter{}, nil }
|
||||
task := Task{ID: "task", Objects: []string{"slow", "good"}, MaxRows: 100, BatchSize: 10, TimeoutSeconds: 1}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
result := s.store.Executions["exec"]
|
||||
if result.ObjectErrors["slow"] != "operationTimeout" || result.ObjectStatuses["good"] != "COMPLETED" || result.Status != "PARTIAL" {
|
||||
t.Fatalf("execution=%+v", result)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,238 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Explicitly enabled read-only comparison; logs column names and counts only.
|
||||
func TestFullSyncDifferenceLive(t *testing.T) {
|
||||
dir := os.Getenv("SYNCDB_DIFFERENCE_DATA_DIR")
|
||||
if dir == "" {
|
||||
t.Skip("requires explicit live diagnostic configuration")
|
||||
}
|
||||
store, _, err := loadStore(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task := fullSyncLiveTask(t, store, "SYNCDB_DIFFERENCE_TASK_ID")
|
||||
open := func(id, database string) *mysqlAdapter {
|
||||
resolved, err := resolveSourceDatabase(store.Sources[id], database)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
adapter, err := openMySQLAdapter(resolved, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { adapter.Close(context.Background()) })
|
||||
return adapter
|
||||
}
|
||||
source, target := open(task.SourceID, task.SourceDatabase), open(task.TargetID, task.TargetDatabase)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
|
||||
defer cancel()
|
||||
for _, name := range strings.Split(os.Getenv("SYNCDB_DIFFERENCE_TABLES"), ",") {
|
||||
read := func(adapter *mysqlAdapter) map[string]Record {
|
||||
rows := map[string]Record{}
|
||||
err := adapter.Read(ctx, name, ReadOptions{BatchSize: 1000, JSONBatch: true}, func(batch []Record) error {
|
||||
for _, row := range batch {
|
||||
rows[scalarText(row["id"])] = row
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return rows
|
||||
}
|
||||
s, d := read(source), read(target)
|
||||
columns := map[string]int{}
|
||||
missing, extra, different := 0, 0, 0
|
||||
for id, row := range s {
|
||||
other, exists := d[id]
|
||||
if !exists {
|
||||
missing++
|
||||
continue
|
||||
}
|
||||
changed := false
|
||||
for key, value := range row {
|
||||
a, _ := json.Marshal(value)
|
||||
b, _ := json.Marshal(other[key])
|
||||
if !reflect.DeepEqual(a, b) {
|
||||
columns[key]++
|
||||
changed = true
|
||||
}
|
||||
}
|
||||
if changed {
|
||||
different++
|
||||
}
|
||||
}
|
||||
for id := range d {
|
||||
if _, exists := s[id]; !exists {
|
||||
extra++
|
||||
}
|
||||
}
|
||||
t.Logf("table=%s source=%d target=%d missing=%d extra=%d differing_rows=%d columns=%v", name, len(s), len(d), missing, extra, different, columns)
|
||||
}
|
||||
}
|
||||
|
||||
// Catch up explicitly selected tables and verify the exact records read from
|
||||
// the source. Target-only records are retained; the source may keep changing.
|
||||
func TestFullSyncCatchUpLive(t *testing.T) {
|
||||
dir := os.Getenv("SYNCDB_CATCHUP_DATA_DIR")
|
||||
if dir == "" {
|
||||
t.Skip("requires explicit live write configuration")
|
||||
}
|
||||
store, _, err := loadStore(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task := fullSyncLiveTask(t, store, "SYNCDB_CATCHUP_TASK_ID")
|
||||
if task.DryRun || task.Mode != "full" || task.Conflict != "upsert" || store.Sources[task.TargetID].Environment == "production" {
|
||||
t.Fatal("real full upsert into a nonproduction target required")
|
||||
}
|
||||
open := func(id, database string) *mysqlAdapter {
|
||||
resolved, err := resolveSourceDatabase(store.Sources[id], database)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
adapter, err := openMySQLAdapter(resolved, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
adapter.database.SetConnMaxLifetime(0)
|
||||
t.Cleanup(func() { adapter.Close(context.Background()) })
|
||||
return adapter
|
||||
}
|
||||
source, target := open(task.SourceID, task.SourceDatabase), open(task.TargetID, task.TargetDatabase)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 24*time.Hour)
|
||||
defer cancel()
|
||||
type result struct {
|
||||
Table string `json:"table"`
|
||||
SourceRows int64 `json:"source_rows"`
|
||||
TargetRows int64 `json:"target_rows"`
|
||||
Written int64 `json:"written"`
|
||||
Missing int64 `json:"missing"`
|
||||
Different int64 `json:"different"`
|
||||
RetainedExtra int64 `json:"retained_target_only"`
|
||||
Verified bool `json:"verified_source_snapshot"`
|
||||
FinishedAt time.Time `json:"finished_at"`
|
||||
}
|
||||
var results []result
|
||||
for _, name := range strings.Split(os.Getenv("SYNCDB_CATCHUP_TABLES"), ",") {
|
||||
if name == "" || name == "shipment_header" {
|
||||
t.Fatal("deferred or empty table cannot be written")
|
||||
}
|
||||
allowed := false
|
||||
for _, object := range task.Objects {
|
||||
allowed = allowed || name == object
|
||||
}
|
||||
if !allowed {
|
||||
t.Fatal("table is outside synchronization scope")
|
||||
}
|
||||
info, err := source.writeSchema(ctx, name)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
keys, _, err := mysqlWriteKeys(info, nil)
|
||||
if err != nil || len(keys) != 1 {
|
||||
t.Fatal("catch-up requires a single stable identity")
|
||||
}
|
||||
key := keys[0]
|
||||
read := func(adapter *mysqlAdapter, consume func([]Record) error) error {
|
||||
options := ReadOptions{BatchSize: 1000, JSONBatch: true}
|
||||
kind := ""
|
||||
for _, column := range info.Columns {
|
||||
if column.Name == key {
|
||||
kind = strings.ToUpper(column.Type)
|
||||
}
|
||||
}
|
||||
if (strings.HasPrefix(kind, "CHAR(") || strings.HasPrefix(kind, "VARCHAR(")) && adapter.supportsJSONTables(ctx) && !adapter.jsonReadDisabled.Load() {
|
||||
return adapter.readJSONBatches(ctx, info, key, QuoteIdentifier(key), options, 1000, consume)
|
||||
}
|
||||
return adapter.Read(ctx, name, options, consume)
|
||||
}
|
||||
fingerprint := func(row Record) [32]byte {
|
||||
payload, err := json.Marshal(row)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return sha256.Sum256(payload)
|
||||
}
|
||||
before := map[string][32]byte{}
|
||||
err = read(target, func(batch []Record) error {
|
||||
for _, row := range batch {
|
||||
before[scalarText(row[key])] = fingerprint(row)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(name, "target read", err)
|
||||
}
|
||||
captured := map[string][32]byte{}
|
||||
r := result{Table: name}
|
||||
err = read(source, func(batch []Record) error {
|
||||
var pending []Record
|
||||
for _, row := range batch {
|
||||
id, hash := scalarText(row[key]), fingerprint(row)
|
||||
captured[id] = hash
|
||||
if old, exists := before[id]; !exists || old != hash {
|
||||
pending = append(pending, row)
|
||||
}
|
||||
r.SourceRows++
|
||||
}
|
||||
if len(pending) > 0 {
|
||||
write, err := target.Write(ctx, name, pending, WriteOptions{Conflict: "upsert"})
|
||||
r.Written += write.Written
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(name, "catch-up write", err)
|
||||
}
|
||||
err = read(target, func(batch []Record) error {
|
||||
for _, row := range batch {
|
||||
r.TargetRows++
|
||||
id := scalarText(row[key])
|
||||
if expected, exists := captured[id]; exists {
|
||||
if fingerprint(row) != expected {
|
||||
r.Different++
|
||||
}
|
||||
delete(captured, id)
|
||||
} else {
|
||||
r.RetainedExtra++
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(name, "target verification", err)
|
||||
}
|
||||
r.Missing = int64(len(captured))
|
||||
r.Verified = r.Missing == 0 && r.Different == 0
|
||||
r.FinishedAt = time.Now().UTC()
|
||||
results = append(results, r)
|
||||
if path := os.Getenv("SYNCDB_CATCHUP_REPORT"); path != "" {
|
||||
payload, err := json.MarshalIndent(results, "", " ")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(path, payload, 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
t.Logf("table=%s source=%d target=%d written=%d missing=%d different=%d retained_extra=%d verified=%v", name, r.SourceRows, r.TargetRows, r.Written, r.Missing, r.Different, r.RetainedExtra, r.Verified)
|
||||
if !r.Verified {
|
||||
t.Errorf("table %s differs from captured source records", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,409 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func fullSyncLiveTask(t *testing.T, store *Store, variable string) Task {
|
||||
t.Helper()
|
||||
taskID := os.Getenv(variable)
|
||||
if taskID == "" {
|
||||
if len(store.Tasks) != 1 {
|
||||
t.Fatalf("set %s to select one synchronization task", variable)
|
||||
}
|
||||
for _, task := range store.Tasks {
|
||||
return task
|
||||
}
|
||||
}
|
||||
task, exists := store.Tasks[taskID]
|
||||
if !exists {
|
||||
t.Fatal("synchronization task not found")
|
||||
}
|
||||
return task
|
||||
}
|
||||
|
||||
func TestFullSyncConflictInspectLive(t *testing.T) {
|
||||
dir := os.Getenv("SYNCDB_CONFLICT_DATA_DIR")
|
||||
if dir == "" {
|
||||
t.Skip("explicit read-only conflict inspection")
|
||||
}
|
||||
store, _, err := loadStore(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task := fullSyncLiveTask(t, store, "SYNCDB_CONFLICT_TASK_ID")
|
||||
open := func(id, db string) *mysqlAdapter {
|
||||
source, err := resolveSourceDatabase(store.Sources[id], db)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
adapter, err := openMySQLAdapter(source, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { adapter.Close(context.Background()) })
|
||||
return adapter
|
||||
}
|
||||
source := open(task.SourceID, task.SourceDatabase)
|
||||
target := open(task.TargetID, task.TargetDatabase)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
|
||||
defer cancel()
|
||||
name := "shipment_header"
|
||||
sourceInfo, err := source.writeSchema(ctx, name)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
targetInfo, err := target.writeSchema(ctx, name)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Logf("source indexes=%+v target indexes=%+v", sourceInfo.Indexes, targetInfo.Indexes)
|
||||
var records []Record
|
||||
if err := source.Read(ctx, name, ReadOptions{BatchSize: 500, JSONBatch: true}, func(batch []Record) error { records = append(records, batch...); return nil }); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ids := make(map[string]bool)
|
||||
for _, record := range records {
|
||||
ids[scalarText(record["id"])] = true
|
||||
}
|
||||
for _, index := range targetInfo.Indexes {
|
||||
if index["unique"] != true || index["primary"] == true {
|
||||
continue
|
||||
}
|
||||
columns := index["columns"].([]string)
|
||||
keyOf := func(record Record) string {
|
||||
parts := make([]string, len(columns))
|
||||
for i, column := range columns {
|
||||
parts[i] = strings.ToLower(scalarText(record[column]))
|
||||
}
|
||||
data, _ := json.Marshal(parts)
|
||||
return string(data)
|
||||
}
|
||||
sourceKeys := make(map[string]string)
|
||||
for _, record := range records {
|
||||
sourceKeys[keyOf(record)] = scalarText(record["id"])
|
||||
}
|
||||
conflicts, extras := 0, 0
|
||||
for start := 0; start < len(records); start += 500 {
|
||||
end := min(start+500, len(records))
|
||||
params := []any{}
|
||||
keys := []string{}
|
||||
values := []string{}
|
||||
fields := []string{"'id'", "`id`"}
|
||||
for _, column := range columns {
|
||||
keys = append(keys, QuoteIdentifier(column))
|
||||
fields = append(fields, "?", QuoteIdentifier(column))
|
||||
params = append(params, column)
|
||||
}
|
||||
for _, record := range records[start:end] {
|
||||
row := make([]string, len(columns))
|
||||
for i, column := range columns {
|
||||
row[i] = "?"
|
||||
params = append(params, record[column])
|
||||
}
|
||||
values = append(values, "("+strings.Join(row, ",")+")")
|
||||
}
|
||||
query := "SELECT JSON_ARRAYAGG(JSON_OBJECT(" + strings.Join(fields, ",") + ")) FROM " + QuoteIdentifier(name) + " WHERE (" + strings.Join(keys, ",") + ") IN (" + strings.Join(values, ",") + ")"
|
||||
var payload []byte
|
||||
if err := target.database.QueryRowContext(ctx, query, params...).Scan(&payload); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if payload == nil {
|
||||
continue
|
||||
}
|
||||
decoder := json.NewDecoder(strings.NewReader(string(payload)))
|
||||
decoder.UseNumber()
|
||||
var matches []Record
|
||||
if err := decoder.Decode(&matches); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, record := range matches {
|
||||
id := scalarText(record["id"])
|
||||
if sourceKeys[keyOf(record)] != id {
|
||||
conflicts++
|
||||
if !ids[id] {
|
||||
extras++
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
t.Logf("table=%s source_rows=%d index=%v conflicting identities=%d conflicting target-only identities=%d", name, len(records), index["name"], conflicts, extras)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFullSyncInspectLive(t *testing.T) {
|
||||
dir := os.Getenv("SYNCDB_INSPECT_DATA_DIR")
|
||||
if dir == "" {
|
||||
t.Skip("explicit read-only inspection only")
|
||||
}
|
||||
store, _, err := loadStore(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task := fullSyncLiveTask(t, store, "SYNCDB_INSPECT_TASK_ID")
|
||||
target, err := resolveSourceDatabase(store.Sources[task.TargetID], task.TargetDatabase)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
adapter, err := openMySQLAdapter(target, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer adapter.Close(context.Background())
|
||||
source, err := resolveSourceDatabase(store.Sources[task.SourceID], task.SourceDatabase)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reader, err := openMySQLAdapter(source, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer reader.Close(context.Background())
|
||||
for _, name := range []string{"preference_user_right", "shipment_history", "shopee_wave", "wave", "inventory_comparison", "inventory_comparison_wes"} {
|
||||
info, err := adapter.Inspect(context.Background(), name)
|
||||
if err != nil {
|
||||
t.Fatal(name, err)
|
||||
}
|
||||
if name == "preference_user_right" {
|
||||
t.Logf("schema columns=%+v indexes=%+v create=%s", info.Columns, info.Indexes, info.CreateSQL)
|
||||
}
|
||||
primary, keys, err := mysqlWriteKeys(info, nil)
|
||||
if err != nil {
|
||||
t.Fatal(name, err)
|
||||
}
|
||||
fast := true
|
||||
for _, index := range info.Indexes {
|
||||
if index["unique"] == true && !adapterSameFields(index["columns"].([]string), primary) {
|
||||
fast = false
|
||||
}
|
||||
}
|
||||
t.Logf("table=%s primary=%v keys=%v batch_upsert=%v", name, primary, keys, fast)
|
||||
shape := info
|
||||
shape.Indexes = nil
|
||||
for _, index := range info.Indexes {
|
||||
if index["primary"] == true {
|
||||
shape.Indexes = append(shape.Indexes, index)
|
||||
}
|
||||
}
|
||||
err = reader.Read(context.Background(), name, ReadOptions{BatchSize: 1000, MaxRows: 1000}, func(batch []Record) error {
|
||||
statement, args, handled, err := mysqlBatchUpsertSQL(shape, batch, primary, keys)
|
||||
t.Logf("table=%s sample_rows=%d fields=%d sql_bytes=%d arguments=%d batch_shape=%v error=%v", name, len(batch), len(info.Columns), len(statement), len(args), handled, err)
|
||||
return err
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Explicitly enabled read-only verification for a local full synchronization.
|
||||
func TestFullSyncVerifyLive(t *testing.T) {
|
||||
dir := os.Getenv("SYNCDB_VERIFY_DATA_DIR")
|
||||
if dir == "" {
|
||||
t.Skip("requires explicit live verification configuration")
|
||||
}
|
||||
store, _, err := loadStore(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task := fullSyncLiveTask(t, store, "SYNCDB_VERIFY_TASK_ID")
|
||||
open := func(source DataSource, database string) *mysqlAdapter {
|
||||
resolved, err := resolveSourceDatabase(source, database)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
adapter, err := openMySQLAdapter(resolved, nil)
|
||||
if err != nil {
|
||||
t.Fatal("verification adapter configuration failed")
|
||||
}
|
||||
adapter.database.SetConnMaxLifetime(0)
|
||||
t.Cleanup(func() { adapter.Close(context.Background()) })
|
||||
return adapter
|
||||
}
|
||||
source := open(store.Sources[task.SourceID], task.SourceDatabase)
|
||||
target := open(store.Sources[task.TargetID], task.TargetDatabase)
|
||||
type tableResult struct {
|
||||
Table string `json:"table"`
|
||||
SourceRows int64 `json:"source_rows"`
|
||||
TargetRows int64 `json:"target_rows"`
|
||||
SourceChecksum string `json:"source_sha256"`
|
||||
TargetChecksum string `json:"target_sha256"`
|
||||
Match bool `json:"match"`
|
||||
Deferred bool `json:"deferred,omitempty"`
|
||||
}
|
||||
results := make([]tableResult, 0, len(task.Objects))
|
||||
previous := make(map[string]tableResult)
|
||||
if os.Getenv("SYNCDB_VERIFY_RESUME") == "true" {
|
||||
payload, err := os.ReadFile(os.Getenv("SYNCDB_VERIFY_REPORT"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var saved []tableResult
|
||||
if err := json.Unmarshal(payload, &saved); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, result := range saved {
|
||||
previous[result.Table] = result
|
||||
}
|
||||
}
|
||||
saveReport := func() {
|
||||
if path := os.Getenv("SYNCDB_VERIFY_REPORT"); path != "" {
|
||||
data, err := json.MarshalIndent(results, "", " ")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.WriteFile(path, data, 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
var total int64
|
||||
failures := 0
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 24*time.Hour)
|
||||
defer cancel()
|
||||
readHash := func(adapter *mysqlAdapter, name string) (int64, string, error) {
|
||||
hash := sha256.New()
|
||||
var count int64
|
||||
consume := func(batch []Record) error {
|
||||
for _, record := range batch {
|
||||
payload, err := json.Marshal(record)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
hash.Write(payload)
|
||||
hash.Write([]byte{'\n'})
|
||||
count++
|
||||
}
|
||||
return nil
|
||||
}
|
||||
options := ReadOptions{BatchSize: 1000, JSONBatch: true}
|
||||
object, err := adapter.writeSchema(ctx, name)
|
||||
if err != nil {
|
||||
return 0, "", err
|
||||
}
|
||||
order, err := mysqlReadOrder(object)
|
||||
if err != nil {
|
||||
return 0, "", err
|
||||
}
|
||||
stringKey := ""
|
||||
for _, column := range object.Columns {
|
||||
kind := strings.ToUpper(column.Type)
|
||||
if order == QuoteIdentifier(column.Name) && (strings.HasPrefix(kind, "CHAR(") || strings.HasPrefix(kind, "VARCHAR(")) {
|
||||
stringKey = column.Name
|
||||
}
|
||||
}
|
||||
if stringKey != "" && adapter.supportsJSONTables(ctx) && !adapter.jsonReadDisabled.Load() {
|
||||
err = adapter.readJSONBatches(ctx, object, stringKey, order, options, 1000, consume)
|
||||
} else {
|
||||
err = adapter.Read(ctx, name, options, consume)
|
||||
}
|
||||
return count, fmt.Sprintf("%x", hash.Sum(nil)), err
|
||||
}
|
||||
deferred := make(map[string]bool)
|
||||
for _, name := range strings.Split(os.Getenv("SYNCDB_VERIFY_DEFERRED_TABLES"), ",") {
|
||||
deferred[name] = true
|
||||
}
|
||||
waitForTable := func(name string) {
|
||||
executionID := os.Getenv("SYNCDB_VERIFY_EXECUTION_ID")
|
||||
if executionID == "" {
|
||||
return
|
||||
}
|
||||
client := &http.Client{Timeout: 10 * time.Second}
|
||||
for {
|
||||
fetch := func(id string) Execution {
|
||||
request, err := http.NewRequestWithContext(ctx, http.MethodGet, "http://127.0.0.1:42923/api/executions/"+id, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
response, err := client.Do(request)
|
||||
if err != nil {
|
||||
t.Fatal("execution status unavailable", err)
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.StatusCode != http.StatusOK {
|
||||
t.Fatal("execution status request failed")
|
||||
}
|
||||
var execution Execution
|
||||
if err := json.NewDecoder(response.Body).Decode(&execution); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return execution
|
||||
}
|
||||
execution := fetch(executionID)
|
||||
status, exists := execution.ObjectStatuses[name]
|
||||
if !exists && execution.RetryOf != "" {
|
||||
status = fetch(execution.RetryOf).ObjectStatuses[name]
|
||||
}
|
||||
if status == "COMPLETED" {
|
||||
return
|
||||
}
|
||||
if execution.Status != "RUNNING" && execution.Status != "PENDING" {
|
||||
t.Fatalf("table %s did not complete: %s", name, status)
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
t.Fatal(ctx.Err())
|
||||
case <-time.After(5 * time.Second):
|
||||
}
|
||||
}
|
||||
}
|
||||
for _, name := range task.Objects {
|
||||
if saved, exists := previous[name]; exists {
|
||||
results = append(results, saved)
|
||||
if !saved.Deferred {
|
||||
total += saved.SourceRows
|
||||
if !saved.Match {
|
||||
failures++
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
mapped := task.TargetObjects[name]
|
||||
if mapped == "" {
|
||||
mapped = name
|
||||
}
|
||||
result := tableResult{Table: name}
|
||||
if deferred[name] {
|
||||
result.Deferred = true
|
||||
results = append(results, result)
|
||||
saveReport()
|
||||
continue
|
||||
}
|
||||
waitForTable(name)
|
||||
result.SourceRows, result.SourceChecksum, err = readHash(source, name)
|
||||
if err != nil {
|
||||
t.Fatal(name, "source content", err)
|
||||
}
|
||||
result.TargetRows, result.TargetChecksum, err = readHash(target, mapped)
|
||||
if err != nil {
|
||||
t.Fatal(name, "target content", err)
|
||||
}
|
||||
result.Match = result.SourceRows == result.TargetRows && result.SourceChecksum == result.TargetChecksum
|
||||
if !result.Match {
|
||||
failures++
|
||||
t.Logf("mismatch %s source_rows=%d target_rows=%d source_checksum=%v target_checksum=%v", name, result.SourceRows, result.TargetRows, result.SourceChecksum, result.TargetChecksum)
|
||||
}
|
||||
total += result.SourceRows
|
||||
results = append(results, result)
|
||||
saveReport()
|
||||
t.Logf("verified %s source_rows=%d target_rows=%d match=%v", name, result.SourceRows, result.TargetRows, result.Match)
|
||||
}
|
||||
verified := 0
|
||||
for _, result := range results {
|
||||
if !result.Deferred {
|
||||
verified++
|
||||
}
|
||||
}
|
||||
t.Logf("verified tables=%d deferred=%d source rows=%d mismatches=%d", verified, len(results)-verified, total, failures)
|
||||
if failures > 0 {
|
||||
t.Fatalf("%d tables differ", failures)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type idleStreamWriter struct {
|
||||
header http.Header
|
||||
frames chan string
|
||||
}
|
||||
|
||||
func (w *idleStreamWriter) Header() http.Header { return w.header }
|
||||
func (w *idleStreamWriter) WriteHeader(int) {}
|
||||
func (w *idleStreamWriter) Flush() {}
|
||||
func (w *idleStreamWriter) Write(data []byte) (int, error) {
|
||||
w.frames <- string(data)
|
||||
return len(data), nil
|
||||
}
|
||||
|
||||
func TestIdleStreamSendsOnlyChangedSnapshotsAndKeepsConnectionAlive(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.streams = make(chan struct{}, 1)
|
||||
s.runtimeCtx = context.Background()
|
||||
s.store.Executions["run"] = Execution{ID: "run", Status: "RUNNING", Written: 10}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
w := &idleStreamWriter{header: make(http.Header), frames: make(chan string, 32)}
|
||||
var worker sync.WaitGroup
|
||||
worker.Add(1)
|
||||
go func() {
|
||||
defer worker.Done()
|
||||
s.streamEvents(w, httptest.NewRequest(http.MethodGet, "/api/events", nil).WithContext(ctx))
|
||||
}()
|
||||
t.Cleanup(func() { cancel(); worker.Wait() })
|
||||
next := func() string {
|
||||
select {
|
||||
case frame := <-w.frames:
|
||||
return frame
|
||||
case <-time.After(18 * time.Second):
|
||||
t.Fatal("stream did not send snapshot or heartbeat")
|
||||
return ""
|
||||
}
|
||||
}
|
||||
first := next()
|
||||
if !strings.HasPrefix(first, "event: snapshot\n") || !strings.Contains(first, `"written":10`) {
|
||||
t.Fatalf("missing initial state: %s", first)
|
||||
}
|
||||
if frame := next(); frame != "event: heartbeat\ndata: {}\n\n" {
|
||||
t.Fatalf("idle stream resent full snapshot: %s", frame)
|
||||
}
|
||||
s.store.Lock()
|
||||
execution := s.store.Executions["run"]
|
||||
execution.Written = 11
|
||||
s.store.Executions["run"] = execution
|
||||
s.store.Unlock()
|
||||
if frame := next(); !strings.HasPrefix(frame, "event: snapshot\n") || !strings.Contains(frame, `"written":11`) {
|
||||
t.Fatalf("changed state was not delivered: %s", frame)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSnapshotMapOrderIsStable(t *testing.T) {
|
||||
items := map[string]Task{"b": {ID: "b"}, "a": {ID: "a"}, "c": {ID: "c"}}
|
||||
for i := 0; i < 100; i++ {
|
||||
data, err := json.Marshal(snapshotValues(items))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var tasks []Task
|
||||
if err = json.Unmarshal(data, &tasks); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if tasks[0].ID != "a" || tasks[1].ID != "b" || tasks[2].ID != "c" {
|
||||
t.Fatal("unstable snapshot order")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson/primitive"
|
||||
)
|
||||
|
||||
func TestIncrementalCursorComparison(t *testing.T) {
|
||||
base := time.Date(2026, 10, 5, 1, 2, 3, 0, time.UTC)
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
first, second any
|
||||
want int
|
||||
}{
|
||||
{"fractional second", base.Add(time.Millisecond), base, 1},
|
||||
{"different fractional widths", base.Add(100 * time.Millisecond), base.Add(110 * time.Millisecond), -1},
|
||||
{"Mongo millisecond", primitive.NewDateTimeFromTime(base.Add(time.Millisecond)), primitive.NewDateTimeFromTime(base), 1},
|
||||
{"mixed time representations", primitive.NewDateTimeFromTime(base), base, 0},
|
||||
{"timestamp strings", "2026-10-05T01:02:03.001Z", "2026-10-05T01:02:03Z", 1},
|
||||
{"timezone offsets", "2026-10-05T09:02:03+08:00", "2026-10-05T01:02:03Z", 0},
|
||||
{"large integer", json.Number("9007199254740993"), json.Number("9007199254740992"), 1},
|
||||
{"decimal", "100.01", "100.001", 1},
|
||||
{"ordinary string", "b", "a", 1},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
got := compareCursor(test.first, test.second)
|
||||
if got != test.want || compareCursor(test.second, test.first) != -test.want {
|
||||
t.Fatalf("cursor order: got %d, want %d", got, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type incrementalCursorAdapter struct {
|
||||
tableSkipAdapter
|
||||
values []any
|
||||
filters *[]Filter
|
||||
}
|
||||
|
||||
func (a *incrementalCursorAdapter) Read(_ context.Context, _ string, options ReadOptions, consume func([]Record) error) error {
|
||||
*a.filters = append(*a.filters, options.Filters...)
|
||||
for _, value := range a.values {
|
||||
if err := consume([]Record{{"id": value}}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestIncrementalCursorPersistsAndResumesAfterRestart(t *testing.T) {
|
||||
base := time.Date(2026, 10, 5, 1, 2, 3, 0, time.UTC)
|
||||
later := base.Add(123 * time.Millisecond)
|
||||
decimalFirst, _ := primitive.ParseDecimal128("12345678901234567890.0001")
|
||||
decimalLast, _ := primitive.ParseDecimal128("12345678901234567890.0002")
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
first, last any
|
||||
}{
|
||||
{"time", base, later},
|
||||
{"MongoDate", primitive.NewDateTimeFromTime(base), primitive.NewDateTimeFromTime(later)},
|
||||
{"largeInteger", json.Number("9007199254740992"), json.Number("9007199254740993")},
|
||||
{"decimal", decimalFirst, decimalLast},
|
||||
{"ObjectID", primitive.ObjectID{11: 1}, primitive.ObjectID{11: 2}},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
first, last := test.first, test.last
|
||||
var filters []Filter
|
||||
values := []any{last, first} // Readers need not order by the watermark.
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
return &incrementalCursorAdapter{values: values, filters: &filters}, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"table"}, Mode: "incremental", IncrementalField: "id", IncrementalValue: filterWireValue(first), MaxRows: -1, BatchSize: 10}
|
||||
// Decode the same typed initial value that an API request provides.
|
||||
encoded, err := json.Marshal(task.IncrementalValue)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := decodePersistenceJSON(encoded, &task.IncrementalValue); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec-1"] = Execution{ID: "exec-1", TaskID: task.ID, Status: "RUNNING"}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "exec-1")
|
||||
if s.store.Executions["exec-1"].Status != "COMPLETED" {
|
||||
t.Fatalf("initial scan failed: %+v", s.store.Executions["exec-1"])
|
||||
}
|
||||
restored, key, err := loadStore(s.dataDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s.store, s.persistenceKey = restored, key
|
||||
task = restored.Tasks[task.ID]
|
||||
cursor, err := filterValue(task.IncrementalValue)
|
||||
if err != nil || compareCursor(cursor, last) != 0 {
|
||||
t.Fatalf("saved watermark lost precision: %v / %v", cursor, err)
|
||||
}
|
||||
filters = nil
|
||||
values = nil // A successful empty scan must keep the saved watermark.
|
||||
s.store.Executions["exec-2"] = Execution{ID: "exec-2", TaskID: task.ID, Status: "RUNNING"}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "exec-2")
|
||||
if len(filters) != 1 || filters[0].Operator != "gte" || compareCursor(filters[0].Value, last) != 0 {
|
||||
t.Fatalf("restart did not resume at saved boundary: %+v", filters)
|
||||
}
|
||||
cursor, err = filterValue(s.store.Tasks[task.ID].IncrementalValue)
|
||||
if err != nil || compareCursor(cursor, last) != 0 {
|
||||
t.Fatal("empty scan reset watermark")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"math"
|
||||
"testing"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson/primitive"
|
||||
)
|
||||
|
||||
func TestIncrementalInitialValueMustBeFiniteScalar(t *testing.T) {
|
||||
for _, value := range []any{[]any{1, 2}, map[string]any{"$date": "bad"}, map[string]any{"$numberDecimal": "NaN"}, math.Inf(1), nil} {
|
||||
source := DataSource{Type: "mysql", Endpoint: "source:3306", Database: "app"}
|
||||
target := DataSource{Type: "mysql", Endpoint: "target:3306", Database: "app"}
|
||||
task := Task{Name: "incremental", Objects: []string{"table"}, Mode: "incremental", IncrementalField: "id", IncrementalValue: value}
|
||||
if err := validateTask(&task, source, target); err == nil || err.Error() != "invalidIncremental" {
|
||||
t.Fatalf("invalid initial watermark accepted: %v, error=%v", value, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type incrementalGuardAdapter struct {
|
||||
tableSkipAdapter
|
||||
batches [][]Record
|
||||
reads, writes *int
|
||||
}
|
||||
|
||||
func (a *incrementalGuardAdapter) Read(_ context.Context, _ string, _ ReadOptions, consume func([]Record) error) error {
|
||||
*a.reads++
|
||||
for _, batch := range a.batches {
|
||||
if err := consume(batch); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (a *incrementalGuardAdapter) Write(ctx context.Context, name string, records []Record, options WriteOptions) (WriteResult, error) {
|
||||
*a.writes++
|
||||
return a.tableSkipAdapter.Write(ctx, name, records, options)
|
||||
}
|
||||
|
||||
func TestInvalidIncrementalFieldAndRecordsFailWithoutAdvancingWatermark(t *testing.T) {
|
||||
nan, _ := primitive.ParseDecimal128("NaN")
|
||||
for _, test := range []struct {
|
||||
name, field, code, schema string
|
||||
batches [][]Record
|
||||
reads, writes int
|
||||
}{
|
||||
{"missing source field", "missing", "incrementalSourceFieldMissing", "FAILED", nil, 0, 0},
|
||||
{"missing record field", "id", "incrementalCursorInvalid", "COMPLETED", [][]Record{{{"id": 11}, {"other": 12}}}, 1, 0},
|
||||
{"null", "id", "incrementalCursorInvalid", "COMPLETED", [][]Record{{{"id": nil}}}, 1, 0},
|
||||
{"array", "id", "incrementalCursorInvalid", "COMPLETED", [][]Record{{{"id": []any{12}}}}, 1, 0},
|
||||
{"document", "id", "incrementalCursorInvalid", "COMPLETED", [][]Record{{{"id": map[string]any{"nested": 12}}}}, 1, 0},
|
||||
{"nonfinite decimal", "id", "incrementalCursorInvalid", "COMPLETED", [][]Record{{{"id": nan}}}, 1, 0},
|
||||
{"previous writes retained", "id", "incrementalCursorInvalid", "COMPLETED", [][]Record{{{"id": 11}}, {{"id": nil}}}, 1, 1},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
reads, writes := 0, 0
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
return &incrementalGuardAdapter{batches: test.batches, reads: &reads, writes: &writes}, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"table"}, Mode: "incremental", IncrementalField: test.field, IncrementalValue: 10, MaxRows: -1, BatchSize: 10}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
result := s.store.Executions["exec"]
|
||||
if result.Status != "FAILED" || result.ObjectErrors["table"] != test.code || result.SchemaStatuses["table"] != test.schema || result.Written != int64(test.writes) || reads != test.reads || writes != test.writes {
|
||||
t.Fatalf("unexpected guard outcome: %+v reads=%d writes=%d", result, reads, writes)
|
||||
}
|
||||
if s.store.Tasks[task.ID].IncrementalValue != 10 || len(result.SkippedObjects) != 1 || result.SkippedObjects[0] != "table" {
|
||||
t.Fatal("invalid watermark advanced or lost retry entry")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
)
|
||||
|
||||
func TestJSONArrayConversionThroughTransformAndDatabaseWrites(t *testing.T) {
|
||||
const document = `[9007199254740993,{"items":[true,null,"text"]}]`
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
input any
|
||||
}{
|
||||
{"text", document},
|
||||
{"bytes", []byte(document)},
|
||||
{"raw JSON", json.RawMessage(document)},
|
||||
{"native array", []any{json.Number("9007199254740993"), map[string]any{"items": []any{true, nil, "text"}}}},
|
||||
{"BSON array", bson.A{int64(9007199254740993), bson.M{"items": bson.A{true, nil, "text"}}}},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
input := Record{"payload": test.input}
|
||||
converted, err := transformRecord(input, map[string]string{"payload": "doc"}, nil, map[string]string{"doc": "json"}, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
value, err := mysqlWriteValue(converted["doc"], "JSON")
|
||||
if err != nil || value != document {
|
||||
t.Fatalf("MySQL array changed: %v %v", value, err)
|
||||
}
|
||||
object := ObjectInfo{Name: "table", Columns: []ColumnInfo{{Name: "id", Type: "BIGINT", Primary: true}, {Name: "doc", Type: "JSON"}}}
|
||||
converted["id"] = int64(1)
|
||||
_, arguments, handled, err := mysqlBatchUpsertSQL(object, []Record{converted}, []string{"id"}, []string{"id"})
|
||||
if err != nil || !handled || len(arguments) != 2 || arguments[0] != document {
|
||||
t.Fatalf("batch JSON array changed: %+v %v", arguments, err)
|
||||
}
|
||||
encoded, err := bson.MarshalWithRegistry(mongoRegistry, converted)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
array := bson.Raw(encoded).Lookup("doc").Array()
|
||||
if array.Index(0).Value().Int64() != 9007199254740993 {
|
||||
t.Fatal("MongoDB array integer lost precision")
|
||||
}
|
||||
items := array.Index(1).Value().Document().Lookup("items").Array()
|
||||
if !items.Index(0).Value().Boolean() || items.Index(1).Value().Type != bson.TypeNull || items.Index(2).Value().StringValue() != "text" {
|
||||
t.Fatal("MongoDB nested array changed")
|
||||
}
|
||||
if _, exists := input["doc"]; exists {
|
||||
t.Fatal("transform mutated source record")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestJSONConversionEmptyNativeArrays(t *testing.T) {
|
||||
for _, input := range []any{[]any{}, bson.A{}} {
|
||||
converted, err := transformRecord(Record{"doc": input}, nil, nil, map[string]string{"doc": "json"}, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
value, err := mysqlWriteValue(converted["doc"], "JSON")
|
||||
if err != nil || value != "[]" {
|
||||
t.Fatalf("empty array changed: %v %v", value, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"database/sql/driver"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestMySQLJSONBooleanWritePreservesTypeInSingleAndBatchWrites(t *testing.T) {
|
||||
for _, boolean := range []bool{true, false} {
|
||||
t.Run(fmt.Sprint(boolean), func(t *testing.T) {
|
||||
want := fmt.Sprint(boolean)
|
||||
for _, input := range []any{boolean, want} {
|
||||
record, err := transformRecord(Record{"id": int64(1), "doc": input}, nil, nil, map[string]string{"doc": "json"}, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
object := ObjectInfo{Name: "items", Type: "table", Columns: []ColumnInfo{{Name: "id", Type: "BIGINT", Primary: true}, {Name: "doc", Type: "JSON"}}}
|
||||
calls := 0
|
||||
db := sql.OpenDB(schemaAuditConnector{execute: func(_ context.Context, statement string, args []driver.NamedValue) (driver.Result, error) {
|
||||
calls++
|
||||
if statement != "INSERT INTO `items` (`doc`, `id`) VALUES (?,?)" || len(args) != 2 || args[0].Value != want || args[1].Value != int64(1) {
|
||||
t.Fatalf("JSON boolean was not passed as JSON text: %s %+v", statement, args)
|
||||
}
|
||||
return driver.RowsAffected(1), nil
|
||||
}})
|
||||
writer := &mysqlAdapter{database: db}
|
||||
written, skipped, err := writer.mysqlWriteRecord(context.Background(), object, record, WriteOptions{Conflict: "insert"}, "insert", nil, nil)
|
||||
_ = db.Close()
|
||||
if err != nil || calls != 1 || written != 1 || skipped != 0 {
|
||||
t.Fatalf("single boolean write failed: %d %d %v", written, skipped, err)
|
||||
}
|
||||
_, arguments, handled, err := mysqlBatchUpsertSQL(object, []Record{record}, []string{"id"}, []string{"id"})
|
||||
if err != nil || !handled || len(arguments) != 2 || arguments[0] != want {
|
||||
t.Fatalf("batch boolean write changed type: %+v %v", arguments, err)
|
||||
}
|
||||
_, parameters, handled, err := mysqlJSONUpdateSQL(object, "id", []string{"doc", "id"}, arguments, []int{0})
|
||||
if err != nil || !handled || len(parameters) != 1 {
|
||||
t.Fatalf("JSON_TABLE boolean update failed: %+v %v", parameters, err)
|
||||
}
|
||||
var rows [][]any
|
||||
if err := json.Unmarshal([]byte(parameters[0].(string)), &rows); err != nil || len(rows) != 1 || rows[0][0] != want {
|
||||
t.Fatalf("JSON_TABLE boolean became a SQL number: %+v %v", rows, err)
|
||||
}
|
||||
}
|
||||
for _, column := range []string{"BOOLEAN", "TINYINT(1)"} {
|
||||
value, err := mysqlWriteValue(boolean, column)
|
||||
if err != nil || value != boolean {
|
||||
t.Fatalf("non-JSON boolean changed: %v %v", value, err)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"database/sql/driver"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
)
|
||||
|
||||
func TestConvertedJSONNullDiffersFromSQLNullInSingleAndBatchWrites(t *testing.T) {
|
||||
jsonNull, err := transformRecord(Record{"doc": "null", "id": int64(1)}, nil, nil, map[string]string{"doc": "json"}, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !adapterNil(jsonNull["doc"]) || incrementalCursorValid(jsonNull["doc"]) {
|
||||
t.Fatal("converted JSON null must remain invalid as an incremental cursor")
|
||||
}
|
||||
sqlNull, err := transformRecord(Record{"doc": nil, "id": int64(2)}, nil, nil, map[string]string{"doc": "json"}, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
object := ObjectInfo{Name: "items", Type: "table", Columns: []ColumnInfo{{Name: "id", Type: "BIGINT", Primary: true}, {Name: "doc", Type: "JSON"}}}
|
||||
calls := 0
|
||||
db := sql.OpenDB(schemaAuditConnector{execute: func(_ context.Context, _ string, args []driver.NamedValue) (driver.Result, error) {
|
||||
var want any = "null"
|
||||
if calls == 1 {
|
||||
want = nil
|
||||
}
|
||||
calls++
|
||||
if len(args) != 2 || args[0].Value != want {
|
||||
t.Fatalf("single write confused JSON and SQL null: %+v", args)
|
||||
}
|
||||
return driver.RowsAffected(1), nil
|
||||
}})
|
||||
defer db.Close()
|
||||
writer := &mysqlAdapter{database: db}
|
||||
for _, record := range []Record{jsonNull, sqlNull} {
|
||||
if _, _, err := writer.mysqlWriteRecord(context.Background(), object, record, WriteOptions{Conflict: "insert"}, "insert", nil, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if calls != 2 {
|
||||
t.Fatal("missing single writes")
|
||||
}
|
||||
_, args, handled, err := mysqlBatchUpsertSQL(object, []Record{jsonNull, sqlNull}, []string{"id"}, []string{"id"})
|
||||
if err != nil || !handled || len(args) != 4 || args[0] != "null" || args[2] != nil {
|
||||
t.Fatalf("batch write confused JSON and SQL null: %+v %v", args, err)
|
||||
}
|
||||
_, parameters, handled, err := mysqlJSONUpdateSQL(object, "id", []string{"doc", "id"}, args, []int{0, 1})
|
||||
if err != nil || !handled || len(parameters) != 1 {
|
||||
t.Fatalf("JSON_TABLE null update failed: %+v %v", parameters, err)
|
||||
}
|
||||
var rows [][]any
|
||||
if err := json.Unmarshal([]byte(parameters[0].(string)), &rows); err != nil || len(rows) != 2 || rows[0][0] != "null" || rows[1][0] != nil {
|
||||
t.Fatalf("JSON_TABLE confused null values: %+v %v", rows, err)
|
||||
}
|
||||
for _, column := range []string{"TEXT", "INT"} {
|
||||
value, err := mysqlWriteValue(jsonNull["doc"], column)
|
||||
if err != nil || value != nil {
|
||||
t.Fatalf("non-JSON column must retain SQL null: %v %v", value, err)
|
||||
}
|
||||
}
|
||||
mongo, err := bson.MarshalWithRegistry(mongoRegistry, jsonNull)
|
||||
if err != nil || bson.Raw(mongo).Lookup("doc").Type != bson.TypeNull || mongoValueType(jsonNull["doc"]) != "null" {
|
||||
t.Fatalf("MongoDB null changed: %v", err)
|
||||
}
|
||||
encoded, err := json.Marshal(jsonNull)
|
||||
if err != nil || string(encoded) != `{"doc":null,"id":1}` {
|
||||
t.Fatalf("JSON serialization changed null: %s %v", encoded, err)
|
||||
}
|
||||
masked, err := transformRecord(Record{"doc": "null"}, nil, nil, map[string]string{"doc": "json"}, nil, map[string]string{"doc": "hash"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
value, err := mysqlWriteValue(masked["doc"], "JSON")
|
||||
if err != nil || value != "null" {
|
||||
t.Fatalf("masking must preserve the converted null: %v %v", value, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"database/sql/driver"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
)
|
||||
|
||||
func TestConvertedJSONStringPreservesTypeInDatabaseWrites(t *testing.T) {
|
||||
for _, original := range []string{"", "text", "true", "null", "9007199254740993", "quote\"slash\\\n中文", `{"id":1}`} {
|
||||
t.Run(original, func(t *testing.T) {
|
||||
encoded, err := json.Marshal(original)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
record, err := transformRecord(Record{"doc": string(encoded), "id": int64(1)}, nil, nil, map[string]string{"doc": "json"}, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := string(encoded)
|
||||
value, err := mysqlWriteValue(record["doc"], "JSON")
|
||||
if err != nil || value != want {
|
||||
t.Fatalf("JSON string lost its quotes or changed type: %v %v", value, err)
|
||||
}
|
||||
value, err = mysqlWriteValue(record["doc"], "TEXT")
|
||||
if err != nil || value != original {
|
||||
t.Fatalf("ordinary text gained JSON quotes: %v %v", value, err)
|
||||
}
|
||||
object := ObjectInfo{Name: "items", Type: "table", Columns: []ColumnInfo{{Name: "id", Type: "BIGINT", Primary: true}, {Name: "doc", Type: "JSON"}}}
|
||||
calls := 0
|
||||
db := sql.OpenDB(schemaAuditConnector{execute: func(_ context.Context, _ string, args []driver.NamedValue) (driver.Result, error) {
|
||||
calls++
|
||||
if len(args) != 2 || args[0].Value != want {
|
||||
t.Fatalf("single write lost JSON string type: %+v", args)
|
||||
}
|
||||
return driver.RowsAffected(1), nil
|
||||
}})
|
||||
writer := &mysqlAdapter{database: db}
|
||||
written, _, err := writer.mysqlWriteRecord(context.Background(), object, record, WriteOptions{Conflict: "insert"}, "insert", nil, nil)
|
||||
_ = db.Close()
|
||||
if err != nil || calls != 1 || written != 1 {
|
||||
t.Fatalf("single write failed: %d %v", written, err)
|
||||
}
|
||||
_, args, handled, err := mysqlBatchUpsertSQL(object, []Record{record}, []string{"id"}, []string{"id"})
|
||||
if err != nil || !handled || len(args) != 2 || args[0] != want {
|
||||
t.Fatalf("batch write lost JSON string type: %+v %v", args, err)
|
||||
}
|
||||
_, parameters, handled, err := mysqlJSONUpdateSQL(object, "id", []string{"doc", "id"}, args, []int{0})
|
||||
if err != nil || !handled || len(parameters) != 1 {
|
||||
t.Fatalf("JSON_TABLE string update failed: %+v %v", parameters, err)
|
||||
}
|
||||
var rows [][]any
|
||||
if err := json.Unmarshal([]byte(parameters[0].(string)), &rows); err != nil || len(rows) != 1 || rows[0][0] != want {
|
||||
t.Fatalf("JSON_TABLE lost JSON string type: %+v %v", rows, err)
|
||||
}
|
||||
mongo, err := bson.MarshalWithRegistry(mongoRegistry, record)
|
||||
if err != nil || bson.Raw(mongo).Lookup("doc").StringValue() != original || mongoValueType(record["doc"]) != "string" {
|
||||
t.Fatalf("MongoDB string changed: %v", err)
|
||||
}
|
||||
text, err := convertValue(record["doc"], "string")
|
||||
if err != nil || text != original {
|
||||
t.Fatalf("subsequent string conversion changed: %v %v", text, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnconvertedMySQLJSONTextRemainsRaw(t *testing.T) {
|
||||
for _, raw := range []string{`"text"`, `{"id":1}`, `[1,true,null]`, "true", "null"} {
|
||||
value, err := mysqlWriteValue(raw, "JSON")
|
||||
if err != nil || value != raw {
|
||||
t.Fatalf("raw JSON text was encoded twice: %v %v", value, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,55 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestLocalAuthentication(t *testing.T) {
|
||||
s := &server{sessionToken: "test-session-token"}
|
||||
handler := s.auth(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}))
|
||||
for _, tc := range []struct {
|
||||
host, peer string
|
||||
allowed bool
|
||||
}{
|
||||
{"127.0.0.1:18081", "127.0.0.1:12345", true},
|
||||
{"localhost:18081", "127.0.0.1:12345", true},
|
||||
{"[::1]:18081", "[::1]:12345", true},
|
||||
{"127.0.0.1:18081", "192.168.1.2:12345", false},
|
||||
{"example.com:18081", "127.0.0.1:12345", false},
|
||||
{"192.168.1.10:18081", "192.168.1.2:12345", false},
|
||||
} {
|
||||
t.Run(tc.host+"/"+tc.peer, func(t *testing.T) {
|
||||
r := httptest.NewRequest(http.MethodGet, "http://"+tc.host+"/api/tasks", nil)
|
||||
r.RemoteAddr = tc.peer
|
||||
w := httptest.NewRecorder()
|
||||
handler.ServeHTTP(w, r)
|
||||
want := http.StatusUnauthorized
|
||||
if tc.allowed {
|
||||
want = http.StatusNoContent
|
||||
}
|
||||
if w.Code != want {
|
||||
t.Fatalf("status %d, want %d", w.Code, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
r := httptest.NewRequest(http.MethodGet, "http://127.0.0.1/api/session", nil)
|
||||
r.RemoteAddr = "127.0.0.1:12345"
|
||||
w := httptest.NewRecorder()
|
||||
handler.ServeHTTP(w, r)
|
||||
if w.Code != 200 || !strings.Contains(w.Body.String(), `"authenticated":true`) {
|
||||
t.Fatalf("local session: %d %s", w.Code, w.Body.String())
|
||||
}
|
||||
r = httptest.NewRequest(http.MethodPost, "http://127.0.0.1/api/tasks", nil)
|
||||
r.RemoteAddr = "127.0.0.1:12345"
|
||||
r.Header.Set("Origin", "http://example.com")
|
||||
w = httptest.NewRecorder()
|
||||
handler.ServeHTTP(w, r)
|
||||
if w.Code != http.StatusForbidden {
|
||||
t.Fatalf("cross-origin local request accepted: %d", w.Code)
|
||||
}
|
||||
}
|
||||
+61
-40
@@ -56,49 +56,68 @@ type DataSource struct {
|
||||
}
|
||||
|
||||
type Task struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
SourceID string `json:"source_id"`
|
||||
TargetID string `json:"target_id"`
|
||||
SourceDatabase string `json:"source_database,omitempty"`
|
||||
TargetDatabase string `json:"target_database,omitempty"`
|
||||
Mode string `json:"mode"`
|
||||
Conflict string `json:"conflict"`
|
||||
Objects []string `json:"objects"`
|
||||
TargetObjects map[string]string `json:"target_objects,omitempty"`
|
||||
Mapping map[string]string `json:"mapping,omitempty"`
|
||||
Defaults map[string]any `json:"defaults,omitempty"`
|
||||
Conversions map[string]string `json:"conversions,omitempty"`
|
||||
Omit []string `json:"omit,omitempty"`
|
||||
Masking map[string]string `json:"masking,omitempty"`
|
||||
Filters []Filter `json:"filters,omitempty"`
|
||||
KeyFields []string `json:"key_fields,omitempty"`
|
||||
BatchSize int `json:"batch_size"`
|
||||
MaxRows int64 `json:"max_rows"`
|
||||
TimeoutSeconds int `json:"timeout_seconds"`
|
||||
Retries int `json:"retries"`
|
||||
DryRun bool `json:"dry_run"`
|
||||
IntervalSeconds int `json:"interval_seconds"`
|
||||
Enabled bool `json:"enabled"`
|
||||
IncrementalField string `json:"incremental_field,omitempty"`
|
||||
IncrementalValue any `json:"incremental_value,omitempty"`
|
||||
Status string `json:"status"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
LastRunAt *time.Time `json:"last_run_at,omitempty"`
|
||||
Revision string `json:"revision,omitempty"`
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
SourceID string `json:"source_id"`
|
||||
TargetID string `json:"target_id"`
|
||||
SourceDatabase string `json:"source_database,omitempty"`
|
||||
TargetDatabase string `json:"target_database,omitempty"`
|
||||
Mode string `json:"mode"`
|
||||
Conflict string `json:"conflict"`
|
||||
Objects []string `json:"objects"`
|
||||
TargetObjects map[string]string `json:"target_objects,omitempty"`
|
||||
Mapping map[string]string `json:"mapping,omitempty"`
|
||||
Defaults map[string]any `json:"defaults,omitempty"`
|
||||
DefaultsJSON string `json:"defaults_json,omitempty"`
|
||||
Conversions map[string]string `json:"conversions,omitempty"`
|
||||
Omit []string `json:"omit,omitempty"`
|
||||
Masking map[string]string `json:"masking,omitempty"`
|
||||
Filters []Filter `json:"filters,omitempty"`
|
||||
FiltersJSON string `json:"filters_json,omitempty"`
|
||||
KeyFields []string `json:"key_fields,omitempty"`
|
||||
BatchSize int `json:"batch_size"`
|
||||
SchemaWorkers int `json:"schema_workers"`
|
||||
MaxRows int64 `json:"max_rows"`
|
||||
TimeoutSeconds int `json:"timeout_seconds"`
|
||||
Retries int `json:"retries"`
|
||||
DryRun bool `json:"dry_run"`
|
||||
IntervalSeconds int `json:"interval_seconds"`
|
||||
Enabled bool `json:"enabled"`
|
||||
IncrementalField string `json:"incremental_field,omitempty"`
|
||||
IncrementalValue any `json:"incremental_value,omitempty"`
|
||||
IncrementalValueJSON string `json:"incremental_value_json,omitempty"`
|
||||
Status string `json:"status"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
LastRunAt *time.Time `json:"last_run_at,omitempty"`
|
||||
}
|
||||
|
||||
type Execution struct {
|
||||
ID string `json:"id"`
|
||||
TaskID string `json:"task_id"`
|
||||
Status string `json:"status"`
|
||||
Rows int64 `json:"rows"`
|
||||
Written int64 `json:"written"`
|
||||
Skipped int64 `json:"skipped"`
|
||||
Error string `json:"error,omitempty"`
|
||||
DryRun bool `json:"dry_run"`
|
||||
Object string `json:"object,omitempty"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
FinishedAt *time.Time `json:"finished_at,omitempty"`
|
||||
Stage string `json:"stage,omitempty"`
|
||||
SchemaStatuses map[string]string `json:"schema_statuses,omitempty"`
|
||||
ObjectErrors map[string]string `json:"object_errors,omitempty"`
|
||||
ObjectStartedAt map[string]time.Time `json:"object_started_at,omitempty"`
|
||||
ObjectFinishedAt map[string]time.Time `json:"object_finished_at,omitempty"`
|
||||
RetryOriginalStatuses map[string]string `json:"retry_original_statuses,omitempty"`
|
||||
ObjectStatuses map[string]string `json:"object_statuses,omitempty"`
|
||||
ObjectAttemptIDs map[string]string `json:"object_attempt_ids,omitempty"`
|
||||
ObjectRows map[string]int64 `json:"object_rows,omitempty"`
|
||||
ObjectWritten map[string]int64 `json:"object_written,omitempty"`
|
||||
ObjectSkipped map[string]int64 `json:"object_skipped,omitempty"`
|
||||
TargetObjects map[string]string `json:"target_objects,omitempty"`
|
||||
ID string `json:"id"`
|
||||
TaskID string `json:"task_id"`
|
||||
Status string `json:"status"`
|
||||
Rows int64 `json:"rows"`
|
||||
Written int64 `json:"written"`
|
||||
Skipped int64 `json:"skipped"`
|
||||
Error string `json:"error,omitempty"`
|
||||
DryRun bool `json:"dry_run"`
|
||||
Object string `json:"object,omitempty"`
|
||||
SkippedObjects []string `json:"skipped_objects,omitempty"`
|
||||
RetryOf string `json:"retry_of,omitempty"`
|
||||
StartedAt time.Time `json:"started_at"`
|
||||
FinishedAt *time.Time `json:"finished_at,omitempty"`
|
||||
}
|
||||
|
||||
type Export struct {
|
||||
@@ -115,6 +134,7 @@ type Export struct {
|
||||
Progress *binlogExportProgress `json:"progress,omitempty"`
|
||||
File string `json:"file,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
ErrorDetail string `json:"error_detail,omitempty"`
|
||||
Mode string `json:"mode"`
|
||||
SnapshotGTID string `json:"snapshot_gtid,omitempty"`
|
||||
PrepareReplication bool `json:"prepare_replication,omitempty"`
|
||||
@@ -162,6 +182,7 @@ type server struct {
|
||||
cancel context.CancelFunc
|
||||
jobsMu sync.Mutex
|
||||
jobs map[string]context.CancelFunc
|
||||
tableRuns map[string]*tableRun
|
||||
slots chan struct{}
|
||||
workers sync.WaitGroup
|
||||
streams chan struct{}
|
||||
|
||||
+22
-2
@@ -28,6 +28,26 @@ type mongoAdapter struct {
|
||||
|
||||
var _ Adapter = (*mongoAdapter)(nil)
|
||||
|
||||
func (adapter *mongoAdapter) EstimatedRows(ctx context.Context, name string) (int64, error) {
|
||||
if !mongoValidObject(name) {
|
||||
return 0, errors.New("invalidObject")
|
||||
}
|
||||
value, err := adapter.database.RunCommand(ctx, bson.D{{Key: "collStats", Value: name}}).DecodeBytes()
|
||||
if err != nil {
|
||||
return 0, mongoOperationError(err)
|
||||
}
|
||||
var result struct {
|
||||
Count int64 `bson:"count"`
|
||||
}
|
||||
if err := bson.Unmarshal(value, &result); err != nil {
|
||||
return 0, mongoOperationError(err)
|
||||
}
|
||||
if result.Count < 0 {
|
||||
return 0, errors.New("rowEstimateUnavailable")
|
||||
}
|
||||
return result.Count, nil
|
||||
}
|
||||
|
||||
var mongoRegistry = bson.NewRegistryBuilder().RegisterTypeEncoder(reflect.TypeOf(json.Number("")), bsoncodec.ValueEncoderFunc(func(_ bsoncodec.EncodeContext, writer bsonrw.ValueWriter, value reflect.Value) error {
|
||||
number := value.Interface().(json.Number)
|
||||
if !adapterScalar(number) {
|
||||
@@ -286,7 +306,7 @@ func (adapter *mongoAdapter) Inspect(ctx context.Context, name string) (ObjectIn
|
||||
|
||||
func mongoValueType(value any) string {
|
||||
switch value.(type) {
|
||||
case nil:
|
||||
case nil, convertedJSONNull:
|
||||
return "null"
|
||||
case primitive.ObjectID:
|
||||
return "objectId"
|
||||
@@ -304,7 +324,7 @@ func mongoValueType(value any) string {
|
||||
return "object"
|
||||
case bson.A, []any:
|
||||
return "array"
|
||||
case string:
|
||||
case string, convertedJSONString:
|
||||
return "string"
|
||||
case bool:
|
||||
return "bool"
|
||||
|
||||
+179
-14
@@ -12,10 +12,13 @@ import (
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
"unicode"
|
||||
|
||||
"github.com/go-sql-driver/mysql"
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/bson/primitive"
|
||||
)
|
||||
|
||||
@@ -106,9 +109,13 @@ func pingMySQL(ctx context.Context, source DataSource) error {
|
||||
}
|
||||
|
||||
type mysqlAdapter struct {
|
||||
database *sql.DB
|
||||
source DataSource
|
||||
audit func(SQLAudit)
|
||||
database *sql.DB
|
||||
source DataSource
|
||||
audit func(SQLAudit)
|
||||
writeSchemas sync.Map
|
||||
jsonTablesOnce sync.Once
|
||||
jsonTablesSupported bool
|
||||
jsonReadDisabled atomic.Bool
|
||||
}
|
||||
|
||||
var _ Adapter = (*mysqlAdapter)(nil)
|
||||
@@ -121,8 +128,10 @@ func openMySQLAdapter(source DataSource, audit func(SQLAudit)) (*mysqlAdapter, e
|
||||
config.Passwd = source.Password
|
||||
config.DBName = source.Database
|
||||
config.Timeout = 5 * time.Second
|
||||
config.ReadTimeout = 5 * time.Second
|
||||
config.WriteTimeout = 5 * time.Second
|
||||
// Large result packets and audited SQL gateways can take longer than a
|
||||
// connection handshake. Operation contexts still bound each table/API call.
|
||||
config.ReadTimeout = 60 * time.Second
|
||||
config.WriteTimeout = 60 * time.Second
|
||||
config.ClientFoundRows = true
|
||||
if source.TLSMode == "required" {
|
||||
config.TLSConfig = "true"
|
||||
@@ -235,6 +244,9 @@ func mysqlOperationError(err error) error {
|
||||
func (adapter *mysqlAdapter) ListObjects(ctx context.Context) ([]ObjectInfo, error) {
|
||||
rows, err := adapter.database.QueryContext(ctx, "SELECT TABLE_NAME, TABLE_TYPE FROM information_schema.TABLES WHERE TABLE_SCHEMA = ? ORDER BY TABLE_NAME", adapter.source.Database)
|
||||
if err != nil {
|
||||
if mysqlMetadataFallback(err) {
|
||||
return adapter.listShowObjects(ctx)
|
||||
}
|
||||
return nil, mysqlOperationError(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
@@ -254,7 +266,61 @@ func (adapter *mysqlAdapter) ListObjects(ctx context.Context) ([]ObjectInfo, err
|
||||
return objects, mysqlOperationError(rows.Err())
|
||||
}
|
||||
|
||||
func (adapter *mysqlAdapter) EstimatedRows(ctx context.Context, name string) (int64, error) {
|
||||
if !adapterValidName(name) {
|
||||
return 0, errors.New("invalidObject")
|
||||
}
|
||||
var rows sql.NullInt64
|
||||
err := adapter.database.QueryRowContext(ctx, "SELECT TABLE_ROWS FROM information_schema.TABLES WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ?", adapter.source.Database, name).Scan(&rows)
|
||||
if err == nil && rows.Valid && rows.Int64 >= 0 {
|
||||
return rows.Int64, nil
|
||||
}
|
||||
// SHOW TABLE STATUS can work when information_schema access is restricted.
|
||||
status, statusErr := adapter.database.QueryContext(ctx, "SHOW TABLE STATUS LIKE ?", name)
|
||||
if statusErr == nil {
|
||||
defer status.Close()
|
||||
if status.Next() {
|
||||
columns, columnErr := status.Columns()
|
||||
values := make([]any, len(columns))
|
||||
pointers := make([]any, len(columns))
|
||||
for i := range values {
|
||||
pointers[i] = &values[i]
|
||||
}
|
||||
if columnErr == nil && status.Scan(pointers...) == nil {
|
||||
for i, column := range columns {
|
||||
if strings.EqualFold(column, "Rows") {
|
||||
switch value := values[i].(type) {
|
||||
case int64:
|
||||
return value, nil
|
||||
case []byte:
|
||||
if parsed, parseErr := strconv.ParseInt(string(value), 10, 64); parseErr == nil && parsed >= 0 {
|
||||
return parsed, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
return 0, mysqlOperationError(err)
|
||||
}
|
||||
return 0, errors.New("rowEstimateUnavailable")
|
||||
}
|
||||
|
||||
func (adapter *mysqlAdapter) Inspect(ctx context.Context, name string) (ObjectInfo, error) {
|
||||
return adapter.inspect(ctx, name, true)
|
||||
}
|
||||
|
||||
func (adapter *mysqlAdapter) inspect(ctx context.Context, name string, full bool) (ObjectInfo, error) {
|
||||
object, err := adapter.inspectInformationSchema(ctx, name, full)
|
||||
if err != nil && mysqlMetadataFallback(err) {
|
||||
return adapter.inspectShow(ctx, name, full)
|
||||
}
|
||||
return object, err
|
||||
}
|
||||
|
||||
func (adapter *mysqlAdapter) inspectInformationSchema(ctx context.Context, name string, full bool) (ObjectInfo, error) {
|
||||
object := ObjectInfo{Name: name, Columns: []ColumnInfo{}, Indexes: []map[string]any{}, Options: map[string]any{}}
|
||||
if !adapterValidName(name) {
|
||||
return object, errors.New("invalidObject")
|
||||
@@ -271,7 +337,7 @@ func (adapter *mysqlAdapter) Inspect(ctx context.Context, name string) (ObjectIn
|
||||
}
|
||||
object.Options["engine"] = engine.String
|
||||
object.Options["collation"] = collation.String
|
||||
columns, err := adapter.database.QueryContext(ctx, "SELECT COLUMN_NAME, COLUMN_TYPE, COLUMN_KEY, IS_NULLABLE, EXTRA FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ? ORDER BY ORDINAL_POSITION", adapter.source.Database, name)
|
||||
columns, err := adapter.database.QueryContext(ctx, "SELECT COLUMN_NAME, COLUMN_TYPE, COLUMN_KEY, IS_NULLABLE, EXTRA, COLUMN_DEFAULT FROM information_schema.COLUMNS WHERE TABLE_SCHEMA = ? AND TABLE_NAME = ? ORDER BY ORDINAL_POSITION", adapter.source.Database, name)
|
||||
if err != nil {
|
||||
return object, mysqlOperationError(err)
|
||||
}
|
||||
@@ -279,12 +345,14 @@ func (adapter *mysqlAdapter) Inspect(ctx context.Context, name string) (ObjectIn
|
||||
for columns.Next() {
|
||||
var column ColumnInfo
|
||||
var key, nullable, extra string
|
||||
if err := columns.Scan(&column.Name, &column.Type, &key, &nullable, &extra); err != nil {
|
||||
var defaultValue sql.NullString
|
||||
if err := columns.Scan(&column.Name, &column.Type, &key, &nullable, &extra, &defaultValue); err != nil {
|
||||
columns.Close()
|
||||
return object, mysqlOperationError(err)
|
||||
}
|
||||
column.Primary = key == "PRI"
|
||||
column.Nullable = nullable == "YES"
|
||||
column.Required = mysqlColumnRequired(column.Nullable, defaultValue.Valid, extra)
|
||||
if strings.Contains(extra, "VIRTUAL GENERATED") || strings.Contains(extra, "STORED GENERATED") {
|
||||
generated = append(generated, column.Name)
|
||||
}
|
||||
@@ -327,6 +395,9 @@ func (adapter *mysqlAdapter) Inspect(ctx context.Context, name string) (ObjectIn
|
||||
if err != nil {
|
||||
return object, mysqlOperationError(err)
|
||||
}
|
||||
if !full {
|
||||
return object, nil
|
||||
}
|
||||
foreignKeys, err := adapter.database.QueryContext(ctx, "SELECT k.CONSTRAINT_NAME, k.COLUMN_NAME, k.REFERENCED_TABLE_SCHEMA, k.REFERENCED_TABLE_NAME, k.REFERENCED_COLUMN_NAME, r.UPDATE_RULE, r.DELETE_RULE FROM information_schema.KEY_COLUMN_USAGE k JOIN information_schema.REFERENTIAL_CONSTRAINTS r ON r.CONSTRAINT_SCHEMA = k.CONSTRAINT_SCHEMA AND r.CONSTRAINT_NAME = k.CONSTRAINT_NAME AND r.TABLE_NAME = k.TABLE_NAME WHERE k.TABLE_SCHEMA = ? AND k.TABLE_NAME = ? AND k.REFERENCED_TABLE_NAME IS NOT NULL ORDER BY k.CONSTRAINT_NAME, k.ORDINAL_POSITION", adapter.source.Database, name)
|
||||
if err != nil {
|
||||
return object, mysqlOperationError(err)
|
||||
@@ -411,6 +482,15 @@ func mysqlFilterSQL(filters []Filter) (string, []any, error) {
|
||||
predicates = append(predicates, field+" "+operators[operator]+" ?")
|
||||
arguments = append(arguments, filter.Value)
|
||||
}
|
||||
// Normalize typed filter values before database/sql binds them. In
|
||||
// particular Decimal128 is a struct, so pass its exact decimal text.
|
||||
for index, value := range arguments {
|
||||
converted, err := mysqlWriteValue(value, "")
|
||||
if err != nil {
|
||||
return "", nil, errors.New("invalidFilter")
|
||||
}
|
||||
arguments[index] = converted
|
||||
}
|
||||
if len(predicates) == 0 {
|
||||
return "", arguments, nil
|
||||
}
|
||||
@@ -429,7 +509,7 @@ func (adapter *mysqlAdapter) Read(ctx context.Context, name string, options Read
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
object, err := adapter.Inspect(ctx, name)
|
||||
object, err := adapter.inspect(ctx, name, false)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -437,6 +517,11 @@ func (adapter *mysqlAdapter) Read(ctx context.Context, name string, options Read
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if options.JSONBatch && !adapter.jsonReadDisabled.Load() && adapter.supportsJSONTables(ctx) {
|
||||
if key, ok := mysqlJSONReadKey(object); ok {
|
||||
return adapter.readJSONBatches(ctx, object, key, order, options, batchSize, consume)
|
||||
}
|
||||
}
|
||||
statement := "SELECT * FROM " + QuoteIdentifier(name) + where + " ORDER BY " + order
|
||||
if options.MaxRows > 0 {
|
||||
statement += " LIMIT ?"
|
||||
@@ -539,7 +624,24 @@ func mysqlWriteKeys(object ObjectInfo, requested []string) ([]string, []string,
|
||||
}
|
||||
}
|
||||
if len(primary) == 0 {
|
||||
return nil, nil, errors.New("primaryKeyRequired")
|
||||
nonNullable := make(map[string]bool, len(object.Columns))
|
||||
for _, column := range object.Columns {
|
||||
nonNullable[column.Name] = !column.Nullable
|
||||
}
|
||||
for _, index := range object.Indexes {
|
||||
columns, valid := index["columns"].([]string)
|
||||
valid = valid && len(columns) > 0 && index["unique"] == true && index["full_columns"] == true
|
||||
for _, column := range columns {
|
||||
valid = valid && nonNullable[column]
|
||||
}
|
||||
if valid {
|
||||
primary = append(primary, columns...)
|
||||
break
|
||||
}
|
||||
}
|
||||
if len(primary) == 0 {
|
||||
return nil, nil, errors.New("primaryKeyRequired")
|
||||
}
|
||||
}
|
||||
keys := requested
|
||||
if len(keys) == 0 {
|
||||
@@ -608,7 +710,7 @@ func (adapter *mysqlAdapter) Write(ctx context.Context, name string, records []R
|
||||
if len(records) == 0 {
|
||||
return result, nil
|
||||
}
|
||||
object, err := adapter.Inspect(ctx, name)
|
||||
object, err := adapter.writeSchema(ctx, name)
|
||||
if err != nil {
|
||||
return result, err
|
||||
}
|
||||
@@ -622,6 +724,11 @@ func (adapter *mysqlAdapter) Write(ctx context.Context, name string, records []R
|
||||
return result, err
|
||||
}
|
||||
}
|
||||
if strategy == "upsert" && !options.DryRun {
|
||||
if result, handled, err := adapter.mysqlBatchUpsert(ctx, object, records, primary, keys); handled {
|
||||
return result, err
|
||||
}
|
||||
}
|
||||
for _, record := range records {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return result, adapterSafeError(err)
|
||||
@@ -683,6 +790,14 @@ func (adapter *mysqlAdapter) mysqlWriteRecord(ctx context.Context, object Object
|
||||
}
|
||||
var ids []any
|
||||
found := false
|
||||
primaryIndex := "PRIMARY"
|
||||
for _, index := range object.Indexes {
|
||||
columns, _ := index["columns"].([]string)
|
||||
if index["unique"] == true && index["full_columns"] == true && adapterSameFields(columns, primary) {
|
||||
primaryIndex, _ = index["name"].(string)
|
||||
break
|
||||
}
|
||||
}
|
||||
if len(primary) > 0 {
|
||||
indexName := ""
|
||||
for _, index := range object.Indexes {
|
||||
@@ -700,7 +815,7 @@ func (adapter *mysqlAdapter) mysqlWriteRecord(ctx context.Context, object Object
|
||||
return 0, 0, err
|
||||
}
|
||||
if found {
|
||||
if err := adapter.mysqlCheckPrimary(ctx, object.Name, primary, keys, ids, record); err != nil {
|
||||
if err := adapter.mysqlCheckPrimary(ctx, object.Name, primary, keys, ids, record, primaryIndex); err != nil {
|
||||
return 0, 0, err
|
||||
}
|
||||
}
|
||||
@@ -747,7 +862,7 @@ func (adapter *mysqlAdapter) mysqlWriteRecord(ctx context.Context, object Object
|
||||
arguments = append(arguments, supplied)
|
||||
}
|
||||
}
|
||||
statement = "UPDATE " + QuoteIdentifier(object.Name) + " FORCE INDEX (`PRIMARY`) SET " + strings.Join(assignments, ", ") + " WHERE " + strings.Join(predicates, " AND ") + " LIMIT 1"
|
||||
statement = "UPDATE " + QuoteIdentifier(object.Name) + " FORCE INDEX (" + QuoteIdentifier(primaryIndex) + ") SET " + strings.Join(assignments, ", ") + " WHERE " + strings.Join(predicates, " AND ") + " LIMIT 1"
|
||||
} else {
|
||||
operation = "INSERT"
|
||||
quoted := make([]string, len(writable))
|
||||
@@ -771,13 +886,28 @@ func (adapter *mysqlAdapter) mysqlWriteRecord(ctx context.Context, object Object
|
||||
if err != nil {
|
||||
return 0, 0, mysqlOperationError(err)
|
||||
}
|
||||
if affected == 0 && found {
|
||||
// Proxies may not forward CLIENT_FOUND_ROWS. Confirm the exact
|
||||
// match still exists before accepting an unchanged UPDATE.
|
||||
where := strings.Index(statement, " WHERE ")
|
||||
predicateSQL := statement[where:]
|
||||
predicateCount := strings.Count(predicateSQL, "?")
|
||||
var matched int
|
||||
checkErr := adapter.database.QueryRowContext(ctx, "SELECT 1 FROM "+QuoteIdentifier(object.Name)+predicateSQL, arguments[len(arguments)-predicateCount:]...).Scan(&matched)
|
||||
if checkErr == nil {
|
||||
return 1, 0, nil
|
||||
}
|
||||
if !errors.Is(checkErr, sql.ErrNoRows) {
|
||||
return 0, 0, mysqlOperationError(checkErr)
|
||||
}
|
||||
}
|
||||
if affected != 1 {
|
||||
return 0, 0, errors.New("concurrentModification")
|
||||
}
|
||||
return 1, 0, nil
|
||||
}
|
||||
|
||||
func (adapter *mysqlAdapter) mysqlCheckPrimary(ctx context.Context, name string, primary, keys []string, ids []any, record Record) error {
|
||||
func (adapter *mysqlAdapter) mysqlCheckPrimary(ctx context.Context, name string, primary, keys []string, ids []any, record Record, primaryIndex string) error {
|
||||
keySet := make(map[string]bool, len(keys))
|
||||
for _, key := range keys {
|
||||
keySet[key] = true
|
||||
@@ -801,7 +931,7 @@ func (adapter *mysqlAdapter) mysqlCheckPrimary(ctx context.Context, name string,
|
||||
return nil
|
||||
}
|
||||
var matched int
|
||||
err := adapter.database.QueryRowContext(ctx, "SELECT 1 FROM "+QuoteIdentifier(name)+" FORCE INDEX (`PRIMARY`) WHERE "+strings.Join(predicates, " AND ")+" LIMIT 1", arguments...).Scan(&matched)
|
||||
err := adapter.database.QueryRowContext(ctx, "SELECT 1 FROM "+QuoteIdentifier(name)+" FORCE INDEX ("+QuoteIdentifier(primaryIndex)+") WHERE "+strings.Join(predicates, " AND ")+" LIMIT 1", arguments...).Scan(&matched)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return errors.New("writeConflict")
|
||||
}
|
||||
@@ -809,10 +939,45 @@ func (adapter *mysqlAdapter) mysqlCheckPrimary(ctx context.Context, name string,
|
||||
}
|
||||
|
||||
func mysqlWriteValue(value any, columnType string) (any, error) {
|
||||
if _, ok := value.(convertedJSONNull); ok && strings.EqualFold(strings.TrimSpace(columnType), "JSON") {
|
||||
return "null", nil
|
||||
}
|
||||
if adapterNil(value) {
|
||||
return nil, nil
|
||||
}
|
||||
if strings.EqualFold(strings.TrimSpace(columnType), "JSON") {
|
||||
// MySQL's driver binds bool as SQL 1/0. JSON columns must receive
|
||||
// JSON boolean text to preserve the original scalar's type.
|
||||
if boolean, ok := value.(bool); ok {
|
||||
value = strconv.FormatBool(boolean)
|
||||
}
|
||||
var document any
|
||||
if object, ok := recordMap(value); ok {
|
||||
document = object
|
||||
} else {
|
||||
switch value.(type) {
|
||||
case []any, bson.A:
|
||||
document = value
|
||||
}
|
||||
}
|
||||
if document != nil {
|
||||
encoded, err := json.Marshal(document)
|
||||
if err != nil {
|
||||
return nil, errors.New("invalidRecord")
|
||||
}
|
||||
value = json.RawMessage(encoded)
|
||||
}
|
||||
}
|
||||
switch typed := value.(type) {
|
||||
case convertedJSONString:
|
||||
value = string(typed)
|
||||
if strings.EqualFold(strings.TrimSpace(columnType), "JSON") {
|
||||
encoded, err := json.Marshal(string(typed))
|
||||
if err != nil {
|
||||
return nil, errors.New("invalidRecord")
|
||||
}
|
||||
value = string(encoded)
|
||||
}
|
||||
case json.Number:
|
||||
if !adapterScalar(typed) {
|
||||
return nil, errors.New("invalidRecord")
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Batch upsert is safe only when every unique constraint is the match key.
|
||||
// Otherwise MySQL may update a row matched by an unrelated unique index.
|
||||
func mysqlBatchUpsertSQL(object ObjectInfo, records []Record, primary, keys []string) (string, []any, bool, error) {
|
||||
if len(records) == 0 || len(primary) == 0 || !adapterSameFields(primary, keys) {
|
||||
return "", nil, false, nil
|
||||
}
|
||||
for _, index := range object.Indexes {
|
||||
if index["unique"] != true {
|
||||
continue
|
||||
}
|
||||
columns, ok := index["columns"].([]string)
|
||||
if !ok || index["full_columns"] != true || !adapterSameFields(columns, primary) {
|
||||
return "", nil, false, nil
|
||||
}
|
||||
}
|
||||
fields, err := adapterRecordFields(records[0])
|
||||
if err != nil {
|
||||
return "", nil, true, err
|
||||
}
|
||||
known := make(map[string]ColumnInfo, len(object.Columns))
|
||||
for _, column := range object.Columns {
|
||||
known[column.Name] = column
|
||||
}
|
||||
generated := make(map[string]bool)
|
||||
generatedColumns, _ := object.Options["generated_columns"].([]string)
|
||||
for _, field := range generatedColumns {
|
||||
generated[field] = true
|
||||
}
|
||||
writable := make([]string, 0, len(fields))
|
||||
for _, field := range fields {
|
||||
if _, ok := known[field]; !ok {
|
||||
return "", nil, true, errors.New("invalidRecord")
|
||||
}
|
||||
if !generated[field] {
|
||||
writable = append(writable, field)
|
||||
}
|
||||
}
|
||||
if len(writable) == 0 || len(writable)*len(records) > 60000 {
|
||||
return "", nil, false, nil
|
||||
}
|
||||
primarySet := make(map[string]bool)
|
||||
for _, field := range primary {
|
||||
primarySet[field] = true
|
||||
}
|
||||
quoted := make([]string, len(writable))
|
||||
updates := make([]string, 0, len(writable))
|
||||
for i, field := range writable {
|
||||
quoted[i] = QuoteIdentifier(field)
|
||||
if !primarySet[field] {
|
||||
updates = append(updates, quoted[i]+" = VALUES("+quoted[i]+")")
|
||||
}
|
||||
}
|
||||
if len(updates) == 0 {
|
||||
return "", nil, false, nil
|
||||
}
|
||||
arguments := make([]any, 0, len(writable)*len(records))
|
||||
payload := 0
|
||||
for _, record := range records {
|
||||
rowFields, err := adapterRecordFields(record)
|
||||
if err != nil {
|
||||
return "", nil, true, err
|
||||
}
|
||||
if !adapterSameFields(fields, rowFields) {
|
||||
return "", nil, false, nil
|
||||
}
|
||||
for _, key := range primary {
|
||||
value, ok := record[key]
|
||||
if !ok || adapterNil(value) || !adapterScalar(value) {
|
||||
return "", nil, true, errors.New("keyValueRequired")
|
||||
}
|
||||
}
|
||||
for _, field := range writable {
|
||||
value, err := mysqlWriteValue(record[field], known[field].Type)
|
||||
if err != nil {
|
||||
return "", nil, true, err
|
||||
}
|
||||
switch v := value.(type) {
|
||||
case string:
|
||||
payload += len(v)
|
||||
case []byte:
|
||||
payload += len(v)
|
||||
default:
|
||||
payload += 32
|
||||
}
|
||||
arguments = append(arguments, value)
|
||||
}
|
||||
}
|
||||
if payload > 1<<20 {
|
||||
return "", nil, false, nil
|
||||
}
|
||||
tuple := "(" + strings.TrimSuffix(strings.Repeat("?,", len(writable)), ",") + ")"
|
||||
statement := "INSERT INTO " + QuoteIdentifier(object.Name) + " (" + strings.Join(quoted, ", ") + ") VALUES " + strings.TrimSuffix(strings.Repeat(tuple+",", len(records)), ",") + " ON DUPLICATE KEY UPDATE " + strings.Join(updates, ", ")
|
||||
return statement, arguments, true, nil
|
||||
}
|
||||
|
||||
func (adapter *mysqlAdapter) mysqlBatchUpsert(ctx context.Context, object ObjectInfo, records []Record, primary, keys []string) (result WriteResult, handled bool, failure error) {
|
||||
result, handled, failure = adapter.mysqlTryBatchUpsert(ctx, object, records, primary, keys)
|
||||
if handled || len(records) < 3 {
|
||||
return result, handled, failure
|
||||
}
|
||||
// Split oversized/wide batches instead of dropping the entire batch back
|
||||
// to millions of single-row network round trips. Preserve partial results.
|
||||
mid := len(records) / 2
|
||||
for _, part := range [][]Record{records[:mid], records[mid:]} {
|
||||
partial, handled, err := adapter.mysqlBatchUpsert(ctx, object, part, primary, keys)
|
||||
if !handled {
|
||||
for _, record := range part {
|
||||
if ctx.Err() != nil {
|
||||
err = ctx.Err()
|
||||
break
|
||||
}
|
||||
partial.Rows++
|
||||
written, skipped, writeErr := adapter.mysqlWriteRecord(ctx, object, record, WriteOptions{Conflict: "upsert", KeyFields: keys}, "upsert", primary, keys)
|
||||
partial.Written += written
|
||||
partial.Skipped += skipped
|
||||
if writeErr != nil {
|
||||
err = writeErr
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
result.Rows += partial.Rows
|
||||
result.Written += partial.Written
|
||||
result.Skipped += partial.Skipped
|
||||
if err != nil {
|
||||
return result, true, err
|
||||
}
|
||||
}
|
||||
return result, true, nil
|
||||
}
|
||||
|
||||
func (adapter *mysqlAdapter) mysqlTryBatchUpsert(ctx context.Context, object ObjectInfo, records []Record, primary, keys []string) (result WriteResult, handled bool, failure error) {
|
||||
if len(records) < 2 {
|
||||
return result, false, nil
|
||||
}
|
||||
statement, arguments, handled, err := mysqlBatchUpsertSQL(object, records, primary, keys)
|
||||
if !handled {
|
||||
return adapter.mysqlGuardedBatchUpsert(ctx, object, records, primary, keys)
|
||||
}
|
||||
if !handled || err != nil {
|
||||
return result, handled, err
|
||||
}
|
||||
started := time.Now()
|
||||
defer func() {
|
||||
adapterAudit(adapter.audit, adapter.source.ID, object.Name, "INSERT", statement, started, result.Written, false, failure)
|
||||
}()
|
||||
// Mark the entire batch attempted before sending. A lost response must not
|
||||
// cause blind automatic replay of a potentially committed statement.
|
||||
result.Rows = int64(len(records))
|
||||
if _, err := adapter.database.ExecContext(ctx, statement, arguments...); err != nil {
|
||||
return result, true, mysqlOperationError(err)
|
||||
}
|
||||
result.Written = int64(len(records))
|
||||
return result, true, nil
|
||||
}
|
||||
@@ -0,0 +1,217 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
func mysqlIntegerKey(value any) (string, bool) {
|
||||
raw := scalarText(value)
|
||||
if strings.HasPrefix(raw, "-") {
|
||||
v, err := strconv.ParseInt(raw, 10, 64)
|
||||
return strconv.FormatInt(v, 10), err == nil
|
||||
}
|
||||
v, err := strconv.ParseUint(raw, 10, 64)
|
||||
return strconv.FormatUint(v, 10), err == nil
|
||||
}
|
||||
|
||||
// Tables with alternate unique indexes cannot use ON DUPLICATE KEY UPDATE:
|
||||
// the conflicting index may identify a different row. Lock primary keys in a
|
||||
// short transaction, UPDATE existing keys, and plain INSERT absent keys.
|
||||
func (adapter *mysqlAdapter) mysqlGuardedBatchUpsert(ctx context.Context, object ObjectInfo, records []Record, primary, keys []string) (result WriteResult, handled bool, failure error) {
|
||||
if len(primary) != 1 || !adapterSameFields(primary, keys) {
|
||||
return result, false, nil
|
||||
}
|
||||
physicalPrimary := false
|
||||
for _, index := range object.Indexes {
|
||||
physicalPrimary = physicalPrimary || index["primary"] == true
|
||||
}
|
||||
if !physicalPrimary {
|
||||
return result, false, nil
|
||||
}
|
||||
key := primary[0]
|
||||
numeric := false
|
||||
for _, column := range object.Columns {
|
||||
if column.Name == key {
|
||||
kind := strings.ToLower(column.Type)
|
||||
for _, prefix := range []string{"tinyint", "smallint", "mediumint", "int", "bigint"} {
|
||||
if strings.HasPrefix(kind, prefix) {
|
||||
numeric = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if !numeric {
|
||||
return result, false, nil
|
||||
}
|
||||
identities := make([]string, len(records))
|
||||
lookup := make([]any, len(records))
|
||||
seen := make(map[string]bool)
|
||||
for i, record := range records {
|
||||
identity, ok := mysqlIntegerKey(record[key])
|
||||
if !ok || seen[identity] {
|
||||
return result, false, nil
|
||||
}
|
||||
identities[i] = identity
|
||||
lookup[i] = record[key]
|
||||
seen[identity] = true
|
||||
}
|
||||
// Reuse field, value, generated-column and payload validation. Only the
|
||||
// SQL builder sees the reduced indexes; actual writes use guarded SQL.
|
||||
shape := object
|
||||
shape.Indexes = nil
|
||||
for _, index := range object.Indexes {
|
||||
if index["primary"] == true {
|
||||
shape.Indexes = append(shape.Indexes, index)
|
||||
}
|
||||
}
|
||||
statement, arguments, handled, err := mysqlBatchUpsertSQL(shape, records, primary, keys)
|
||||
if !handled || err != nil {
|
||||
return result, handled, err
|
||||
}
|
||||
fields, _ := adapterRecordFields(records[0])
|
||||
generated := make(map[string]bool)
|
||||
generatedColumns, _ := object.Options["generated_columns"].([]string)
|
||||
for _, field := range generatedColumns {
|
||||
generated[field] = true
|
||||
}
|
||||
writable := make([]string, 0, len(fields))
|
||||
for _, field := range fields {
|
||||
if !generated[field] {
|
||||
writable = append(writable, field)
|
||||
}
|
||||
}
|
||||
tx, err := adapter.database.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
return result, true, mysqlOperationError(err)
|
||||
}
|
||||
defer tx.Rollback()
|
||||
started := time.Now()
|
||||
defer func() {
|
||||
adapterAudit(adapter.audit, adapter.source.ID, object.Name, "INSERT", statement, started, result.Written, false, failure)
|
||||
}()
|
||||
matchSQL := " FROM " + QuoteIdentifier(object.Name) + " FORCE INDEX (`PRIMARY`) WHERE " + QuoteIdentifier(key) + " IN (" + strings.TrimSuffix(strings.Repeat("?,", len(records)), ",") + ")"
|
||||
existing := make(map[string]bool)
|
||||
jsonTables := adapter.supportsJSONTables(ctx)
|
||||
if jsonTables {
|
||||
var payload sql.NullString
|
||||
if err := tx.QueryRowContext(ctx, "SELECT JSON_ARRAYAGG("+QuoteIdentifier(key)+")"+matchSQL+" FOR UPDATE", lookup...).Scan(&payload); err != nil {
|
||||
return result, true, mysqlOperationError(err)
|
||||
}
|
||||
if payload.Valid {
|
||||
var identities []json.Number
|
||||
if err := json.Unmarshal([]byte(payload.String), &identities); err != nil {
|
||||
return result, true, errors.New("invalidRecord")
|
||||
}
|
||||
for _, value := range identities {
|
||||
identity, ok := mysqlIntegerKey(value)
|
||||
if !ok {
|
||||
return result, true, errors.New("keyValueRequired")
|
||||
}
|
||||
existing[identity] = true
|
||||
}
|
||||
}
|
||||
} else {
|
||||
rows, err := tx.QueryContext(ctx, "SELECT "+QuoteIdentifier(key)+matchSQL+" ORDER BY "+QuoteIdentifier(key)+" FOR UPDATE", lookup...)
|
||||
if err != nil {
|
||||
return result, true, mysqlOperationError(err)
|
||||
}
|
||||
for rows.Next() {
|
||||
var value any
|
||||
if err := rows.Scan(&value); err != nil {
|
||||
rows.Close()
|
||||
return result, true, mysqlOperationError(err)
|
||||
}
|
||||
identity, ok := mysqlIntegerKey(value)
|
||||
if !ok {
|
||||
rows.Close()
|
||||
return result, true, errors.New("keyValueRequired")
|
||||
}
|
||||
existing[identity] = true
|
||||
}
|
||||
rowErr := rows.Err()
|
||||
rows.Close()
|
||||
if rowErr != nil {
|
||||
return result, true, mysqlOperationError(rowErr)
|
||||
}
|
||||
}
|
||||
updates := make([]int, 0, len(records))
|
||||
inserts := make([]int, 0, len(records))
|
||||
for i, identity := range identities {
|
||||
if existing[identity] {
|
||||
updates = append(updates, i)
|
||||
} else {
|
||||
inserts = append(inserts, i)
|
||||
}
|
||||
}
|
||||
result.Rows = int64(len(records))
|
||||
if len(updates) > 0 {
|
||||
selects := make([]string, 0, len(updates))
|
||||
updateArgs := make([]any, 0, len(updates)*len(writable))
|
||||
for n, index := range updates {
|
||||
placeholders := make([]string, len(writable))
|
||||
for j, field := range writable {
|
||||
placeholders[j] = "?"
|
||||
if n == 0 {
|
||||
placeholders[j] += " AS " + QuoteIdentifier(field)
|
||||
}
|
||||
}
|
||||
selects = append(selects, "SELECT "+strings.Join(placeholders, ", "))
|
||||
updateArgs = append(updateArgs, arguments[index*len(writable):(index+1)*len(writable)]...)
|
||||
}
|
||||
assignments := make([]string, 0, len(writable))
|
||||
for _, field := range writable {
|
||||
if field != key {
|
||||
quoted := QuoteIdentifier(field)
|
||||
assignments = append(assignments, "target."+quoted+" = incoming."+quoted)
|
||||
}
|
||||
}
|
||||
updateSQL := "UPDATE " + QuoteIdentifier(object.Name) + " AS target JOIN (" + strings.Join(selects, " UNION ALL ") + ") AS incoming ON target." + QuoteIdentifier(key) + " = incoming." + QuoteIdentifier(key) + " SET " + strings.Join(assignments, ", ") + " WHERE target." + QuoteIdentifier(key) + " = incoming." + QuoteIdentifier(key)
|
||||
if jsonTables {
|
||||
jsonSQL, jsonArgs, handled, err := mysqlJSONUpdateSQL(object, key, writable, arguments, updates)
|
||||
if err != nil {
|
||||
return result, true, err
|
||||
}
|
||||
if handled {
|
||||
updateSQL, updateArgs = jsonSQL, jsonArgs
|
||||
}
|
||||
}
|
||||
updated, err := tx.ExecContext(ctx, updateSQL, updateArgs...)
|
||||
if err != nil {
|
||||
return result, true, mysqlOperationError(err)
|
||||
}
|
||||
matched, err := updated.RowsAffected()
|
||||
if err != nil {
|
||||
return result, true, mysqlOperationError(err)
|
||||
}
|
||||
// Primary rows remain locked. Some gateways omit CLIENT_FOUND_ROWS,
|
||||
// reporting changed rows rather than matched rows for no-op updates.
|
||||
if matched < 0 || matched > int64(len(updates)) {
|
||||
return result, true, errors.New("concurrentModification")
|
||||
}
|
||||
}
|
||||
if len(inserts) > 0 {
|
||||
missing := make([]Record, 0, len(inserts))
|
||||
for _, index := range inserts {
|
||||
missing = append(missing, records[index])
|
||||
}
|
||||
insertSQL, insertArgs, _, err := mysqlBatchUpsertSQL(shape, missing, primary, keys)
|
||||
if err != nil {
|
||||
return result, true, err
|
||||
}
|
||||
insertSQL = strings.Split(insertSQL, " ON DUPLICATE KEY UPDATE ")[0]
|
||||
if _, err := tx.ExecContext(ctx, insertSQL, insertArgs...); err != nil {
|
||||
return result, true, mysqlOperationError(err)
|
||||
}
|
||||
}
|
||||
if err := tx.Commit(); err != nil {
|
||||
return result, true, mysqlOperationError(err)
|
||||
}
|
||||
result.Written = int64(len(records))
|
||||
return result, true, nil
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
func (adapter *mysqlAdapter) supportsJSONTables(ctx context.Context) bool {
|
||||
adapter.jsonTablesOnce.Do(func() {
|
||||
var version string
|
||||
if adapter.database.QueryRowContext(ctx, "SELECT VERSION()").Scan(&version) != nil || strings.Contains(version, "MariaDB") {
|
||||
return
|
||||
}
|
||||
major, err := strconv.Atoi(strings.Split(version, ".")[0])
|
||||
adapter.jsonTablesSupported = err == nil && major >= 8
|
||||
})
|
||||
return adapter.jsonTablesSupported
|
||||
}
|
||||
|
||||
// JSON_TABLE avoids constructing hundreds of UNION branches and tens of
|
||||
// thousands of prepared parameters for wide rows. Put incoming rows first
|
||||
// so each target match uses a primary-index lookup instead of a table scan.
|
||||
func mysqlJSONUpdateSQL(object ObjectInfo, key string, writable []string, arguments []any, updates []int) (string, []any, bool, error) {
|
||||
types := make(map[string]string, len(object.Columns))
|
||||
for _, column := range object.Columns {
|
||||
types[column.Name] = column.Type
|
||||
}
|
||||
binary := make(map[string]bool)
|
||||
columns := make([]string, len(writable))
|
||||
assignments := make([]string, 0, len(writable))
|
||||
for i, field := range writable {
|
||||
kind := strings.ToUpper(types[field])
|
||||
dataType := "LONGTEXT"
|
||||
if field == key {
|
||||
dataType = types[field]
|
||||
}
|
||||
for _, prefix := range []string{"BINARY", "VARBINARY", "BLOB", "TINYBLOB", "MEDIUMBLOB", "LONGBLOB", "BIT"} {
|
||||
if strings.HasPrefix(kind, prefix) {
|
||||
binary[field] = true
|
||||
}
|
||||
}
|
||||
quoted := QuoteIdentifier(field)
|
||||
columns[i] = quoted + " " + dataType + " PATH '$[" + strconv.Itoa(i) + "]' ERROR ON ERROR"
|
||||
if field != key {
|
||||
value := "incoming." + quoted
|
||||
if binary[field] {
|
||||
value = "UNHEX(" + value + ")"
|
||||
}
|
||||
assignments = append(assignments, "target."+quoted+" = "+value)
|
||||
}
|
||||
}
|
||||
incoming := make([][]any, 0, len(updates))
|
||||
for _, index := range updates {
|
||||
row := append([]any(nil), arguments[index*len(writable):(index+1)*len(writable)]...)
|
||||
for i, field := range writable {
|
||||
if row[i] == nil {
|
||||
continue
|
||||
}
|
||||
if binary[field] {
|
||||
switch value := row[i].(type) {
|
||||
case []byte:
|
||||
row[i] = hex.EncodeToString(value)
|
||||
case string:
|
||||
row[i] = hex.EncodeToString([]byte(value))
|
||||
default:
|
||||
return "", nil, false, nil
|
||||
}
|
||||
} else {
|
||||
switch value := row[i].(type) {
|
||||
case bool:
|
||||
if value {
|
||||
row[i] = 1
|
||||
} else {
|
||||
row[i] = 0
|
||||
}
|
||||
case time.Time:
|
||||
row[i] = value.UTC().Format("2006-01-02 15:04:05.999999")
|
||||
}
|
||||
}
|
||||
}
|
||||
incoming = append(incoming, row)
|
||||
}
|
||||
payload, err := json.Marshal(incoming)
|
||||
if err != nil {
|
||||
return "", nil, true, err
|
||||
}
|
||||
statement := "UPDATE JSON_TABLE(?, '$[*]' COLUMNS (" + strings.Join(columns, ", ") + ")) AS incoming STRAIGHT_JOIN " + QuoteIdentifier(object.Name) + " AS target FORCE INDEX (`PRIMARY`) ON target." + QuoteIdentifier(key) + " = incoming." + QuoteIdentifier(key) + " SET " + strings.Join(assignments, ", ") + " WHERE target." + QuoteIdentifier(key) + " = incoming." + QuoteIdentifier(key)
|
||||
return statement, []any{string(payload)}, true, nil
|
||||
}
|
||||
@@ -0,0 +1,219 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestMySQLBatchUpsertRejectsUnrelatedUniqueIndex(t *testing.T) {
|
||||
object := ObjectInfo{Name: "items", Columns: []ColumnInfo{{Name: "id", Type: "int", Primary: true}, {Name: "code", Type: "varchar(20)"}}, Indexes: []map[string]any{
|
||||
{"name": "PRIMARY", "primary": true, "unique": true, "full_columns": true, "columns": []string{"id"}},
|
||||
}}
|
||||
records := []Record{{"id": 1, "code": "a"}, {"id": 2, "code": "b"}}
|
||||
statement, args, handled, err := mysqlBatchUpsertSQL(object, records, []string{"id"}, []string{"id"})
|
||||
if !handled || err != nil || len(args) != 4 || !strings.Contains(statement, "ON DUPLICATE KEY UPDATE `code` = VALUES(`code`)") {
|
||||
t.Fatalf("sql=%s args=%v handled=%v err=%v", statement, args, handled, err)
|
||||
}
|
||||
object.Indexes = append(object.Indexes, map[string]any{"unique": true, "full_columns": true, "columns": []string{"code"}})
|
||||
if _, _, handled, _ := mysqlBatchUpsertSQL(object, records, []string{"id"}, []string{"id"}); handled {
|
||||
t.Fatal("unrelated unique index accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLWriteKeysAcceptsOnlyNonNullableCompleteUniqueKeys(t *testing.T) {
|
||||
object := ObjectInfo{Columns: []ColumnInfo{{Name: "a", Nullable: false}, {Name: "b", Nullable: false}}, Indexes: []map[string]any{{"name": "match", "unique": true, "full_columns": true, "columns": []string{"a", "b"}}}}
|
||||
primary, keys, err := mysqlWriteKeys(object, nil)
|
||||
if err != nil || !adapterSameFields(primary, []string{"a", "b"}) || !adapterSameFields(keys, primary) {
|
||||
t.Fatalf("unique fallback: primary=%v keys=%v err=%v", primary, keys, err)
|
||||
}
|
||||
object.Columns[1].Nullable = true
|
||||
if _, _, err := mysqlWriteKeys(object, nil); err == nil {
|
||||
t.Fatal("nullable match index accepted")
|
||||
}
|
||||
object.Columns[1].Nullable = false
|
||||
object.Indexes[0]["full_columns"] = false
|
||||
if _, _, err := mysqlWriteKeys(object, nil); err == nil {
|
||||
t.Fatal("prefix match index accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLBatchUpsertLive(t *testing.T) {
|
||||
dir := os.Getenv("SYNCDB_BATCH_TEST_DATA_DIR")
|
||||
if dir == "" {
|
||||
t.Skip("explicit development database test only")
|
||||
}
|
||||
store, _, err := loadStore(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
task := store.Tasks["task-7"]
|
||||
target, err := resolveSourceDatabase(store.Sources[task.TargetID], task.TargetDatabase)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if isProduction(target) {
|
||||
t.Fatal("development target required")
|
||||
}
|
||||
adapter, err := openMySQLAdapter(target, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer adapter.Close(context.Background())
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
|
||||
defer cancel()
|
||||
name := fmt.Sprintf("syncdb_batch_check_%d", time.Now().UnixNano())
|
||||
_, err = adapter.database.ExecContext(ctx, "CREATE TABLE "+QuoteIdentifier(name)+" (`id` bigint PRIMARY KEY, `value` varchar(30), `retained` int DEFAULT 9, `derived` bigint GENERATED ALWAYS AS (id+1) STORED)")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer adapter.database.ExecContext(context.Background(), "DROP TABLE "+QuoteIdentifier(name))
|
||||
options := WriteOptions{Conflict: "upsert"}
|
||||
for _, records := range [][]Record{{{"id": 1, "value": "old"}, {"id": 2, "value": "two"}}, {{"id": 1, "value": "new"}, {"id": 3, "value": "three"}}} {
|
||||
result, err := adapter.Write(ctx, name, records, options)
|
||||
if err != nil || result.Rows != 2 || result.Written != 2 {
|
||||
t.Fatalf("result=%+v error=%v", result, err)
|
||||
}
|
||||
}
|
||||
var value string
|
||||
var retained, derived, count int
|
||||
if err := adapter.database.QueryRowContext(ctx, "SELECT value,retained,derived FROM "+QuoteIdentifier(name)+" WHERE id=1").Scan(&value, &retained, &derived); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := adapter.database.QueryRowContext(ctx, "SELECT COUNT(*) FROM "+QuoteIdentifier(name)).Scan(&count); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if value != "new" || retained != 9 || derived != 2 || count != 3 {
|
||||
t.Fatalf("value=%s retained=%d derived=%d count=%d", value, retained, derived, count)
|
||||
}
|
||||
tx, err := adapter.database.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var lockedIDs string
|
||||
if err := tx.QueryRowContext(ctx, "SELECT JSON_ARRAYAGG(id) FROM "+QuoteIdentifier(name)+" WHERE id IN (1,2) FOR UPDATE").Scan(&lockedIDs); err != nil {
|
||||
tx.Rollback()
|
||||
t.Fatal(err)
|
||||
}
|
||||
lockCtx, lockCancel := context.WithTimeout(ctx, 300*time.Millisecond)
|
||||
var lockedID int
|
||||
lockErr := adapter.database.QueryRowContext(lockCtx, "SELECT id FROM "+QuoteIdentifier(name)+" WHERE id=1 FOR UPDATE").Scan(&lockedID)
|
||||
lockCancel()
|
||||
tx.Rollback()
|
||||
if !errors.Is(lockErr, context.DeadlineExceeded) {
|
||||
t.Fatalf("aggregate read did not lock primary rows: %v", lockErr)
|
||||
}
|
||||
// An alternate unique conflict must use the guarded single-row path.
|
||||
if _, err := adapter.database.ExecContext(ctx, "ALTER TABLE "+QuoteIdentifier(name)+" ADD UNIQUE KEY unique_value (value)"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
result, err := adapter.Write(ctx, name, []Record{{"id": 1, "value": "guarded"}, {"id": 4, "value": "four"}}, options)
|
||||
if err != nil || result.Written != 2 {
|
||||
t.Fatalf("guarded batch failed: %+v %v", result, err)
|
||||
}
|
||||
_, err = adapter.Write(ctx, name, []Record{{"id": 1, "value": "rolled-back"}, {"id": 5, "value": "two"}}, options)
|
||||
if err == nil || AdapterErrorCode(err) != "writeConflict" {
|
||||
t.Fatalf("expected conflict and rollback: %v", err)
|
||||
}
|
||||
if err := adapter.database.QueryRowContext(ctx, "SELECT value FROM "+QuoteIdentifier(name)+" WHERE id=1").Scan(&value); err != nil || value != "guarded" {
|
||||
t.Fatalf("batch did not roll back: value=%s error=%v", value, err)
|
||||
}
|
||||
_, err = adapter.Write(ctx, name, []Record{{"id": 6, "value": "guarded"}, {"id": 7, "value": "seven"}}, options)
|
||||
if err == nil || AdapterErrorCode(err) != "writeConflict" {
|
||||
t.Fatalf("expected guarded unique conflict: %v", err)
|
||||
}
|
||||
result, err = adapter.Write(ctx, name, []Record{{"id": 1, "value": "guarded"}}, options)
|
||||
if err != nil || result.Written != 1 {
|
||||
t.Fatalf("unchanged single update rejected: %+v %v", result, err)
|
||||
}
|
||||
bulk := make([]Record, 1000)
|
||||
for i := range bulk {
|
||||
bulk[i] = Record{"id": i + 100, "value": fmt.Sprintf("bulk-%d", i)}
|
||||
}
|
||||
for pass := 0; pass < 2; pass++ {
|
||||
if pass == 1 {
|
||||
for i, record := range bulk {
|
||||
record["value"] = fmt.Sprintf("updated-bulk-%d", i)
|
||||
}
|
||||
}
|
||||
started := time.Now()
|
||||
result, err = adapter.Write(ctx, name, bulk, options)
|
||||
t.Logf("1000-row pass=%d duration=%s json_tables=%v", pass, time.Since(started), adapter.jsonTablesSupported)
|
||||
if err != nil || result.Written != 1000 {
|
||||
t.Fatalf("1000-row guarded batch pass=%d result=%+v error=%v", pass, result, err)
|
||||
}
|
||||
}
|
||||
var updatedCount int
|
||||
if err := adapter.database.QueryRowContext(ctx, "SELECT COUNT(*) FROM "+QuoteIdentifier(name)+" WHERE value LIKE 'updated-bulk-%'").Scan(&updatedCount); err != nil || updatedCount != 1000 {
|
||||
t.Fatalf("bulk update missed rows: updated=%d error=%v", updatedCount, err)
|
||||
}
|
||||
if _, err := adapter.database.ExecContext(ctx, "ALTER TABLE "+QuoteIdentifier(name)+" ADD COLUMN payload LONGBLOB"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, record := range bulk {
|
||||
record["payload"] = []byte(strings.Repeat("x", 1600))
|
||||
}
|
||||
started := time.Now()
|
||||
result, err = adapter.Write(ctx, name, bulk, options)
|
||||
t.Logf("wide-row split duration=%s", time.Since(started))
|
||||
if err != nil || result.Written != 1000 {
|
||||
t.Fatalf("oversized batch did not split: result=%+v error=%v", result, err)
|
||||
}
|
||||
if _, err := adapter.database.ExecContext(ctx, "ALTER TABLE "+QuoteIdentifier(name)+" ADD COLUMN precise DECIMAL(30,9) DEFAULT 1.123456789, ADD COLUMN instant TIMESTAMP(6) DEFAULT CURRENT_TIMESTAMP(6), ADD COLUMN doc JSON, ADD COLUMN flag BIT(9) DEFAULT b'1', ADD COLUMN fraction DOUBLE DEFAULT 1.2"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := adapter.database.ExecContext(ctx, "INSERT INTO "+QuoteIdentifier(name)+" (id,value) VALUES (9007199254740993,'high-1'),(9007199254740994,'high-2'),(9007199254740995,'high-3')"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var original, aggregated []Record
|
||||
if err := adapter.Read(ctx, name, ReadOptions{BatchSize: 137}, func(batch []Record) error { original = append(original, batch...); return nil }); err != nil {
|
||||
t.Fatal("original read", err)
|
||||
}
|
||||
started = time.Now()
|
||||
if err := adapter.Read(ctx, name, ReadOptions{BatchSize: 137, JSONBatch: true}, func(batch []Record) error { aggregated = append(aggregated, batch...); return nil }); err != nil {
|
||||
t.Fatal("JSON read", err)
|
||||
}
|
||||
t.Logf("JSON read rows=%d duration=%s", len(aggregated), time.Since(started))
|
||||
if !reflect.DeepEqual(original, aggregated) {
|
||||
t.Fatal("JSON transport changed row values, types or order")
|
||||
}
|
||||
// Read-only verification also batches character identities. Compare all
|
||||
// values across several pages, including keys with a nonbinary collation.
|
||||
if _, err := adapter.database.ExecContext(ctx, "ALTER TABLE "+QuoteIdentifier(name)+" ADD UNIQUE KEY value_key (value)"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
info, err := adapter.writeSchema(ctx, name)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var stringRows []Record
|
||||
err = adapter.readJSONBatches(ctx, info, "value", QuoteIdentifier("value"), ReadOptions{BatchSize: 137}, 137, func(batch []Record) error { stringRows = append(stringRows, batch...); return nil })
|
||||
if err != nil {
|
||||
t.Fatal("character JSON cursor", err)
|
||||
}
|
||||
byID := func(rows []Record) map[string]Record {
|
||||
result := map[string]Record{}
|
||||
for _, row := range rows {
|
||||
result[scalarText(row["id"])] = row
|
||||
}
|
||||
return result
|
||||
}
|
||||
if len(stringRows) != len(original) || !reflect.DeepEqual(byID(stringRows), byID(original)) {
|
||||
t.Fatal("character keyset paging lost, duplicated or changed records")
|
||||
}
|
||||
if _, err := adapter.database.ExecContext(ctx, "ALTER TABLE "+QuoteIdentifier(name)+" DROP INDEX value_key"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := adapter.database.ExecContext(ctx, "ALTER TABLE "+QuoteIdentifier(name)+" DROP PRIMARY KEY, ADD UNIQUE KEY match_id (id)"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
result, err = adapter.Write(ctx, name, []Record{{"id": 1, "value": "guarded"}, {"id": 8, "value": "eight"}}, options)
|
||||
if err != nil || result.Written != 2 {
|
||||
t.Fatalf("unique-only identity rejected: %+v %v", result, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,210 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"github.com/go-sql-driver/mysql"
|
||||
"math/big"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func mysqlJSONReadKey(object ObjectInfo) (string, bool) {
|
||||
order, err := mysqlReadOrder(object)
|
||||
if err != nil {
|
||||
return "", false
|
||||
}
|
||||
for _, column := range object.Columns {
|
||||
if order != QuoteIdentifier(column.Name) {
|
||||
continue
|
||||
}
|
||||
kind := strings.ToUpper(column.Type)
|
||||
for _, prefix := range []string{"TINYINT", "SMALLINT", "MEDIUMINT", "INT", "BIGINT"} {
|
||||
if strings.HasPrefix(kind, prefix) {
|
||||
return column.Name, true
|
||||
}
|
||||
}
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// Return a bounded batch as one JSON result row. This preserves all values
|
||||
// while avoiding per-record result-packet/audit overhead through SQL gateways.
|
||||
// Numeric keyset pagination never uses OFFSET or advances past unread rows.
|
||||
func (adapter *mysqlAdapter) readJSONBatches(ctx context.Context, object ObjectInfo, key, order string, options ReadOptions, batchSize int, consume func([]Record) error) error {
|
||||
expressions := make([]string, 0, len(object.Columns)*2)
|
||||
names := make([]any, 0, len(object.Columns))
|
||||
kinds := make(map[string]string, len(object.Columns))
|
||||
binary := make(map[string]bool)
|
||||
for _, column := range object.Columns {
|
||||
kind := strings.ToUpper(column.Type)
|
||||
if at := strings.IndexAny(kind, "( "); at >= 0 {
|
||||
kind = kind[:at]
|
||||
}
|
||||
kinds[column.Name] = kind
|
||||
value := "sync_rows." + QuoteIdentifier(column.Name)
|
||||
switch kind {
|
||||
case "BINARY", "VARBINARY", "BLOB", "TINYBLOB", "MEDIUMBLOB", "LONGBLOB":
|
||||
value = "HEX(" + value + ")"
|
||||
binary[column.Name] = true
|
||||
case "BIT":
|
||||
bits := 1
|
||||
if start := strings.Index(column.Type, "("); start >= 0 {
|
||||
if end := strings.Index(column.Type, ")"); end > start {
|
||||
parsed, err := strconv.Atoi(column.Type[start+1 : end])
|
||||
if err == nil {
|
||||
bits = parsed
|
||||
}
|
||||
}
|
||||
}
|
||||
value = "LPAD(HEX(" + value + "), " + strconv.Itoa(2*((bits+7)/8)) + ", '0')"
|
||||
binary[column.Name] = true
|
||||
case "DECIMAL", "NEWDECIMAL", "DATE", "DATETIME", "TIMESTAMP", "TIME", "JSON":
|
||||
value = "CAST(" + value + " AS CHAR)"
|
||||
}
|
||||
expressions = append(expressions, "?", value)
|
||||
names = append(names, column.Name)
|
||||
}
|
||||
var cursor any
|
||||
var total int64
|
||||
for {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
filters := append([]Filter(nil), options.Filters...)
|
||||
if cursor != nil {
|
||||
filters = append(filters, Filter{Field: key, Operator: "gt", Value: cursor})
|
||||
}
|
||||
where, args, err := mysqlFilterSQL(filters)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
limit := int64(batchSize)
|
||||
if options.MaxRows > 0 && options.MaxRows-total < limit {
|
||||
limit = options.MaxRows - total
|
||||
}
|
||||
if limit <= 0 {
|
||||
return nil
|
||||
}
|
||||
parameters := append([]any(nil), names...)
|
||||
parameters = append(parameters, args...)
|
||||
parameters = append(parameters, limit)
|
||||
statement := "SELECT JSON_ARRAYAGG(JSON_OBJECT(" + strings.Join(expressions, ", ") + ")), MAX(sync_rows." + QuoteIdentifier(key) + "), COUNT(*) FROM (SELECT * FROM " + QuoteIdentifier(object.Name) + where + " ORDER BY " + order + " LIMIT ?) AS sync_rows"
|
||||
var payload []byte
|
||||
var last sql.NullString
|
||||
var expectedRows int64
|
||||
if err := adapter.database.QueryRowContext(ctx, statement, parameters...).Scan(&payload, &last, &expectedRows); err != nil {
|
||||
var rejected *mysql.MySQLError
|
||||
if total == 0 && errors.As(err, &rejected) && rejected.Number == 1290 {
|
||||
// Respect a gateway's SQL policy. No batch was delivered, so
|
||||
// ordinary SELECT streaming is safe without replaying writes.
|
||||
adapter.jsonReadDisabled.Store(true)
|
||||
options.JSONBatch = false
|
||||
return adapter.Read(ctx, object.Name, options, consume)
|
||||
}
|
||||
return mysqlOperationError(err)
|
||||
}
|
||||
if !last.Valid {
|
||||
if expectedRows != 0 {
|
||||
return errors.New("invalidRecord")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
// A server may return NULL (with a warning) when the aggregate exceeds
|
||||
// its packet limit. Retry only this undelivered page at a smaller size;
|
||||
// keep the previous cursor and never replay already consumed records.
|
||||
if len(payload) == 0 && batchSize > 1 {
|
||||
batchSize = max(1, batchSize/2)
|
||||
continue
|
||||
}
|
||||
decoder := json.NewDecoder(strings.NewReader(string(payload)))
|
||||
decoder.UseNumber()
|
||||
var records []Record
|
||||
if err := decoder.Decode(&records); err != nil {
|
||||
if batchSize > 1 {
|
||||
batchSize = max(1, batchSize/2)
|
||||
continue
|
||||
}
|
||||
return errors.New("invalidRecord")
|
||||
}
|
||||
if int64(len(records)) != expectedRows || len(records) == 0 {
|
||||
if batchSize > 1 {
|
||||
batchSize = max(1, batchSize/2)
|
||||
continue
|
||||
}
|
||||
return errors.New("invalidRecord")
|
||||
}
|
||||
for _, record := range records {
|
||||
for _, column := range object.Columns {
|
||||
value := record[column.Name]
|
||||
if value == nil {
|
||||
continue
|
||||
}
|
||||
if binary[column.Name] {
|
||||
data, err := hex.DecodeString(scalarText(value))
|
||||
if err != nil {
|
||||
return errors.New("invalidRecord")
|
||||
}
|
||||
record[column.Name] = data
|
||||
continue
|
||||
}
|
||||
switch number := value.(type) {
|
||||
case json.Number:
|
||||
switch kinds[column.Name] {
|
||||
case "TINYINT", "SMALLINT", "MEDIUMINT", "INT", "INTEGER", "BIGINT", "YEAR":
|
||||
signed, err := strconv.ParseInt(string(number), 10, 64)
|
||||
if err == nil {
|
||||
record[column.Name] = signed
|
||||
} else {
|
||||
unsigned, err := strconv.ParseUint(string(number), 10, 64)
|
||||
if err != nil {
|
||||
return errors.New("invalidRecord")
|
||||
}
|
||||
record[column.Name] = unsigned
|
||||
}
|
||||
case "FLOAT", "DOUBLE", "REAL":
|
||||
number, err := strconv.ParseFloat(string(number), 64)
|
||||
if err != nil {
|
||||
return errors.New("invalidRecord")
|
||||
}
|
||||
record[column.Name] = number
|
||||
default:
|
||||
record[column.Name] = string(number)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Slice(records, func(i, j int) bool {
|
||||
if kinds[key] == "CHAR" || kinds[key] == "VARCHAR" {
|
||||
return scalarText(records[i][key]) < scalarText(records[j][key])
|
||||
}
|
||||
left, _ := new(big.Int).SetString(scalarText(records[i][key]), 10)
|
||||
right, _ := new(big.Int).SetString(scalarText(records[j][key]), 10)
|
||||
return left.Cmp(right) < 0
|
||||
})
|
||||
if err := consume(records); err != nil {
|
||||
return err
|
||||
}
|
||||
total += int64(len(records))
|
||||
if kinds[key] == "CHAR" || kinds[key] == "VARCHAR" {
|
||||
// MAX and the subsequent range predicate use the server's key
|
||||
// collation. A string cursor must never be parsed as a number.
|
||||
cursor = last.String
|
||||
} else if signed, err := strconv.ParseInt(last.String, 10, 64); err == nil {
|
||||
cursor = signed
|
||||
} else {
|
||||
unsigned, err := strconv.ParseUint(last.String, 10, 64)
|
||||
if err != nil {
|
||||
return errors.New("invalidRecord")
|
||||
}
|
||||
cursor = unsigned
|
||||
}
|
||||
if int64(len(records)) < limit || options.MaxRows > 0 && total >= options.MaxRows {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func mysqlMetadataFallback(err error) bool {
|
||||
switch AdapterErrorCode(err) {
|
||||
case "mysqlDatabaseDenied", "mysqlOperationFailed", "objectNotFound":
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func mysqlColumnRequired(nullable, hasDefault bool, extra string) bool {
|
||||
extra = strings.ToUpper(extra)
|
||||
return !nullable && !hasDefault && !strings.Contains(extra, "AUTO_INCREMENT") && !strings.Contains(extra, "VIRTUAL GENERATED") && !strings.Contains(extra, "STORED GENERATED")
|
||||
}
|
||||
|
||||
// SHOW result columns vary by server version. Read by name so optional fields
|
||||
// never shift primary, generated-column or unique-index safety checks.
|
||||
func (adapter *mysqlAdapter) showMetadata(ctx context.Context, statement string, args ...any) ([]Record, error) {
|
||||
rows, err := adapter.database.QueryContext(ctx, statement, args...)
|
||||
if err != nil {
|
||||
return nil, mysqlOperationError(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
names, err := rows.Columns()
|
||||
if err != nil {
|
||||
return nil, mysqlOperationError(err)
|
||||
}
|
||||
result := []Record{}
|
||||
for rows.Next() {
|
||||
values := make([]any, len(names))
|
||||
pointers := make([]any, len(names))
|
||||
for i := range values {
|
||||
pointers[i] = &values[i]
|
||||
}
|
||||
if err := rows.Scan(pointers...); err != nil {
|
||||
return nil, mysqlOperationError(err)
|
||||
}
|
||||
record := Record{}
|
||||
for i, name := range names {
|
||||
record[strings.ToLower(name)] = values[i]
|
||||
}
|
||||
result = append(result, record)
|
||||
}
|
||||
return result, mysqlOperationError(rows.Err())
|
||||
}
|
||||
|
||||
func mysqlMetadataText(row Record, field string) string {
|
||||
if row[field] == nil {
|
||||
return ""
|
||||
}
|
||||
return scalarText(row[field])
|
||||
}
|
||||
|
||||
func (adapter *mysqlAdapter) listShowObjects(ctx context.Context) ([]ObjectInfo, error) {
|
||||
rows, err := adapter.database.QueryContext(ctx, "SHOW FULL TABLES")
|
||||
if err != nil {
|
||||
return nil, mysqlOperationError(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
objects := []ObjectInfo{}
|
||||
for rows.Next() {
|
||||
var name, kind string
|
||||
if err := rows.Scan(&name, &kind); err != nil {
|
||||
return nil, mysqlOperationError(err)
|
||||
}
|
||||
object := ObjectInfo{Name: name, Type: "table", Columns: []ColumnInfo{}, Indexes: []map[string]any{}}
|
||||
if strings.EqualFold(kind, "VIEW") {
|
||||
object.Type = "view"
|
||||
}
|
||||
objects = append(objects, object)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, mysqlOperationError(err)
|
||||
}
|
||||
sort.Slice(objects, func(i, j int) bool { return objects[i].Name < objects[j].Name })
|
||||
return objects, nil
|
||||
}
|
||||
|
||||
func (adapter *mysqlAdapter) inspectShow(ctx context.Context, name string, full bool) (ObjectInfo, error) {
|
||||
object := ObjectInfo{Name: name, Type: "table", Columns: []ColumnInfo{}, Indexes: []map[string]any{}, Options: map[string]any{}}
|
||||
if !adapterValidName(name) {
|
||||
return object, errors.New("invalidObject")
|
||||
}
|
||||
create, err := adapter.showMetadata(ctx, "SHOW CREATE TABLE "+QuoteIdentifier(name))
|
||||
if err != nil {
|
||||
return object, err
|
||||
}
|
||||
if len(create) != 1 {
|
||||
return object, errors.New("objectNotFound")
|
||||
}
|
||||
object.CreateSQL = mysqlMetadataText(create[0], "create table")
|
||||
if value := mysqlMetadataText(create[0], "create view"); value != "" {
|
||||
object.Type, object.CreateSQL = "view", value
|
||||
}
|
||||
if object.CreateSQL == "" {
|
||||
return object, errors.New("invalidObject")
|
||||
}
|
||||
columns, err := adapter.showMetadata(ctx, "SHOW FULL COLUMNS FROM "+QuoteIdentifier(name))
|
||||
if err != nil {
|
||||
return object, err
|
||||
}
|
||||
generated := []string{}
|
||||
for _, row := range columns {
|
||||
column := ColumnInfo{Name: mysqlMetadataText(row, "field"), Type: mysqlMetadataText(row, "type"), Primary: mysqlMetadataText(row, "key") == "PRI", Nullable: mysqlMetadataText(row, "null") == "YES"}
|
||||
if column.Name == "" || column.Type == "" {
|
||||
return object, errors.New("invalidObject")
|
||||
}
|
||||
extra := strings.ToUpper(mysqlMetadataText(row, "extra"))
|
||||
column.Required = mysqlColumnRequired(column.Nullable, row["default"] != nil, extra)
|
||||
object.Columns = append(object.Columns, column)
|
||||
if strings.Contains(extra, "VIRTUAL GENERATED") || strings.Contains(extra, "STORED GENERATED") {
|
||||
generated = append(generated, column.Name)
|
||||
}
|
||||
}
|
||||
if len(object.Columns) == 0 {
|
||||
return object, errors.New("objectNotFound")
|
||||
}
|
||||
object.Options["generated_columns"] = generated
|
||||
if object.Type == "table" {
|
||||
indexes, err := adapter.showMetadata(ctx, "SHOW INDEX FROM "+QuoteIdentifier(name))
|
||||
if err != nil {
|
||||
return object, err
|
||||
}
|
||||
sort.SliceStable(indexes, func(i, j int) bool {
|
||||
a, b := mysqlMetadataText(indexes[i], "key_name"), mysqlMetadataText(indexes[j], "key_name")
|
||||
if a != b {
|
||||
return a < b
|
||||
}
|
||||
x, _ := strconv.Atoi(mysqlMetadataText(indexes[i], "seq_in_index"))
|
||||
y, _ := strconv.Atoi(mysqlMetadataText(indexes[j], "seq_in_index"))
|
||||
return x < y
|
||||
})
|
||||
positions := map[string]int{}
|
||||
for _, row := range indexes {
|
||||
key, kind := mysqlMetadataText(row, "key_name"), mysqlMetadataText(row, "index_type")
|
||||
unique, err := strconv.Atoi(mysqlMetadataText(row, "non_unique"))
|
||||
sequence, seqErr := strconv.Atoi(mysqlMetadataText(row, "seq_in_index"))
|
||||
if key == "" || kind == "" || err != nil || (unique != 0 && unique != 1) || seqErr != nil || sequence < 1 {
|
||||
return object, errors.New("invalidObject")
|
||||
}
|
||||
position, exists := positions[key]
|
||||
if !exists {
|
||||
position = len(object.Indexes)
|
||||
positions[key] = position
|
||||
object.Indexes = append(object.Indexes, map[string]any{"name": key, "unique": unique == 0, "primary": key == "PRIMARY", "type": kind, "columns": []string{}, "full_columns": true})
|
||||
}
|
||||
index := object.Indexes[position]
|
||||
if index["unique"] != (unique == 0) || index["type"] != kind {
|
||||
return object, errors.New("invalidObject")
|
||||
}
|
||||
column := mysqlMetadataText(row, "column_name")
|
||||
index["columns"] = append(index["columns"].([]string), column)
|
||||
_, prefixKnown := row["sub_part"]
|
||||
if column == "" || !prefixKnown || row["sub_part"] != nil || strings.EqualFold(mysqlMetadataText(row, "visible"), "NO") || sequence != len(index["columns"].([]string)) {
|
||||
index["full_columns"] = false
|
||||
}
|
||||
}
|
||||
}
|
||||
if full {
|
||||
status, err := adapter.showMetadata(ctx, "SHOW TABLE STATUS WHERE Name = ?", name)
|
||||
if err != nil {
|
||||
return object, err
|
||||
}
|
||||
for _, row := range status {
|
||||
if mysqlMetadataText(row, "name") == name {
|
||||
object.Options["engine"] = mysqlMetadataText(row, "engine")
|
||||
object.Options["collation"] = mysqlMetadataText(row, "collation")
|
||||
}
|
||||
}
|
||||
// SHOW CREATE retains complete foreign-key definitions, but SHOW has
|
||||
// no separate foreign-key rows. Do not report an empty list as proof
|
||||
// that the table has no foreign keys.
|
||||
object.Options["metadata_warnings"] = []string{"mysqlForeignKeyDetailsInSQL"}
|
||||
}
|
||||
return object, nil
|
||||
}
|
||||
@@ -0,0 +1,203 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"database/sql/driver"
|
||||
"errors"
|
||||
"io"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/go-sql-driver/mysql"
|
||||
)
|
||||
|
||||
type showSchemaRows struct {
|
||||
names []string
|
||||
rows [][]driver.Value
|
||||
}
|
||||
|
||||
func (rows *showSchemaRows) Columns() []string { return rows.names }
|
||||
func (rows *showSchemaRows) Close() error { return nil }
|
||||
func (rows *showSchemaRows) Next(values []driver.Value) error {
|
||||
if len(rows.rows) == 0 {
|
||||
return io.EOF
|
||||
}
|
||||
copy(values, rows.rows[0])
|
||||
rows.rows = rows.rows[1:]
|
||||
return nil
|
||||
}
|
||||
|
||||
func showSchemaFixtureAdapter(t *testing.T, fail string) (*mysqlAdapter, map[string]int) {
|
||||
t.Helper()
|
||||
calls := map[string]int{}
|
||||
connector := scopeNamesConnector{query: func(statement string, args []driver.NamedValue) (driver.Rows, error) {
|
||||
calls[statement]++
|
||||
if strings.Contains(statement, "information_schema") || statement == fail {
|
||||
return nil, &mysql.MySQLError{Number: 1142, Message: "metadata denied"}
|
||||
}
|
||||
switch statement {
|
||||
case "SHOW CREATE TABLE `odd_%``name`":
|
||||
return &showSchemaRows{names: []string{"Table", "Create Table"}, rows: [][]driver.Value{{"odd_%`name", "CREATE TABLE `odd_%``name` (`id` int PRIMARY KEY, `part` int, `generated` int GENERATED ALWAYS AS (`id`+1) STORED) ENGINE=InnoDB"}}}, nil
|
||||
case "SHOW FULL COLUMNS FROM `odd_%``name`":
|
||||
return &showSchemaRows{names: []string{"Extra", "Type", "Field", "Key", "Null", "Default", "Collation"}, rows: [][]driver.Value{
|
||||
{"auto_increment", "int", "id", "PRI", "NO", nil, nil},
|
||||
{"DEFAULT_GENERATED", "int", "part", "", "NO", "1", nil},
|
||||
{"STORED GENERATED", "int", "generated", "", "YES", nil, nil},
|
||||
}}, nil
|
||||
case "SHOW INDEX FROM `odd_%``name`":
|
||||
return &showSchemaRows{names: []string{"Key_name", "Non_unique", "Seq_in_index", "Column_name", "Sub_part", "Index_type", "Visible", "Expression"}, rows: [][]driver.Value{
|
||||
{"composite", int64(0), int64(2), "part", nil, "BTREE", "YES", nil},
|
||||
{"PRIMARY", int64(0), int64(1), "id", nil, "BTREE", "YES", nil},
|
||||
{"composite", int64(0), int64(1), "id", nil, "BTREE", "YES", nil},
|
||||
{"prefix", int64(0), int64(1), "part", int64(4), "BTREE", "YES", nil},
|
||||
{"functional", int64(0), int64(1), nil, nil, "BTREE", "YES", "id+1"},
|
||||
{"invisible", int64(0), int64(1), "part", nil, "BTREE", "NO", nil},
|
||||
}}, nil
|
||||
case "SHOW TABLE STATUS WHERE Name = ?":
|
||||
if len(args) != 1 || args[0].Value != "odd_%`name" {
|
||||
t.Fatalf("status lookup did not use the exact parameterized name: %v", args)
|
||||
}
|
||||
return &showSchemaRows{names: []string{"Name", "Engine", "Collation"}, rows: [][]driver.Value{{"odd_%`name", "InnoDB", "utf8mb4_0900_ai_ci"}}}, nil
|
||||
case "SHOW FULL TABLES":
|
||||
return &showSchemaRows{names: []string{"Tables_in_app", "Table_type"}, rows: [][]driver.Value{{"z", "BASE TABLE"}, {"a", "VIEW"}}}, nil
|
||||
default:
|
||||
t.Fatalf("unexpected query: %s", statement)
|
||||
return nil, errors.New("unexpected query")
|
||||
}
|
||||
}}
|
||||
database := sql.OpenDB(connector)
|
||||
t.Cleanup(func() { database.Close() })
|
||||
return &mysqlAdapter{database: database, source: DataSource{Database: "app"}}, calls
|
||||
}
|
||||
|
||||
func TestMySQLShowMetadataRetainsWriteSafetyAndCache(t *testing.T) {
|
||||
adapter, calls := showSchemaFixtureAdapter(t, "")
|
||||
info, err := adapter.writeSchema(context.Background(), "odd_%`name")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if info.Type != "table" || len(info.Columns) != 3 || !info.Columns[0].Primary || info.Columns[0].Nullable || !info.Columns[2].Nullable {
|
||||
t.Fatalf("column flags changed: %+v", info.Columns)
|
||||
}
|
||||
if !reflect.DeepEqual(info.Options["generated_columns"], []string{"generated"}) {
|
||||
t.Fatalf("default expression was mistaken for a generated column: %v", info.Options)
|
||||
}
|
||||
indexes := map[string]map[string]any{}
|
||||
for _, index := range info.Indexes {
|
||||
indexes[index["name"].(string)] = index
|
||||
}
|
||||
if !reflect.DeepEqual(indexes["composite"]["columns"], []string{"id", "part"}) || indexes["composite"]["full_columns"] != true {
|
||||
t.Fatalf("composite key order changed: %v", indexes["composite"])
|
||||
}
|
||||
for _, name := range []string{"prefix", "functional", "invisible"} {
|
||||
if indexes[name]["full_columns"] != false {
|
||||
t.Fatalf("unsafe index accepted: %v", indexes[name])
|
||||
}
|
||||
}
|
||||
if _, _, err := mysqlWriteKeys(info, []string{"id", "part"}); err != nil {
|
||||
t.Fatalf("valid complete unique key rejected: %v", err)
|
||||
}
|
||||
if _, _, err := mysqlWriteKeys(info, []string{"part"}); err == nil {
|
||||
t.Fatal("prefix/invisible indexes enabled unsafe matching")
|
||||
}
|
||||
if _, err := adapter.writeSchema(context.Background(), "odd_%`name"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if calls["SHOW FULL COLUMNS FROM `odd_%``name`"] != 1 || calls["SHOW INDEX FROM `odd_%``name`"] != 1 {
|
||||
t.Fatalf("unchanged schema did not reuse parsed metadata: %v", calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLShowInspectionAndObjectList(t *testing.T) {
|
||||
adapter, _ := showSchemaFixtureAdapter(t, "")
|
||||
info, err := adapter.Inspect(context.Background(), "odd_%`name")
|
||||
if err != nil || info.Options["engine"] != "InnoDB" || info.Options["collation"] != "utf8mb4_0900_ai_ci" || info.CreateSQL == "" {
|
||||
t.Fatalf("full SHOW inspection: %+v %v", info, err)
|
||||
}
|
||||
if _, exists := info.Options["foreign_keys"]; exists || !reflect.DeepEqual(info.Options["metadata_warnings"], []string{"mysqlForeignKeyDetailsInSQL"}) {
|
||||
t.Fatal("unavailable relationship rows were presented as an empty foreign-key list")
|
||||
}
|
||||
objects, err := adapter.ListObjects(context.Background())
|
||||
if err != nil || len(objects) != 2 || objects[0].Name != "a" || objects[0].Type != "view" || objects[1].Type != "table" {
|
||||
t.Fatalf("SHOW object list: %+v %v", objects, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLShowInspectionFailsWhenIndexesAreDenied(t *testing.T) {
|
||||
adapter, _ := showSchemaFixtureAdapter(t, "SHOW INDEX FROM `odd_%``name`")
|
||||
if _, err := adapter.writeSchema(context.Background(), "odd_%`name"); err == nil {
|
||||
t.Fatal("incomplete index metadata was accepted for writing")
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLMetadataDoesNotFallbackAfterCancellationOrAuthenticationFailure(t *testing.T) {
|
||||
for _, expected := range []error{context.Canceled, context.DeadlineExceeded, &mysql.MySQLError{Number: 1045}} {
|
||||
calls := 0
|
||||
database := sql.OpenDB(scopeNamesConnector{query: func(statement string, _ []driver.NamedValue) (driver.Rows, error) {
|
||||
calls++
|
||||
if !strings.Contains(statement, "information_schema") {
|
||||
t.Fatal("terminal error triggered another metadata request")
|
||||
}
|
||||
return nil, expected
|
||||
}})
|
||||
adapter := &mysqlAdapter{database: database}
|
||||
_, err := adapter.Inspect(context.Background(), "table")
|
||||
database.Close()
|
||||
if err == nil || calls != 1 || AdapterErrorCode(err) != AdapterErrorCode(expected) {
|
||||
t.Fatalf("terminal error changed: %v calls=%d", err, calls)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMySQLRequiredColumnMetadataFromInformationSchemaAndShow(t *testing.T) {
|
||||
for _, fallback := range []bool{false, true} {
|
||||
t.Run(map[bool]string{false: "information_schema", true: "SHOW"}[fallback], func(t *testing.T) {
|
||||
values := [][]driver.Value{
|
||||
{"required", "int", "", "NO", "", nil},
|
||||
{"zero_default", "int", "", "NO", "", int64(0)},
|
||||
{"empty_default", "varchar(8)", "", "NO", "", ""},
|
||||
{"auto_id", "int", "PRI", "NO", "auto_increment", nil},
|
||||
{"virtual", "int", "", "NO", "VIRTUAL GENERATED", nil},
|
||||
{"stored", "int", "", "NO", "STORED GENERATED", nil},
|
||||
{"optional", "int", "", "YES", "", nil},
|
||||
{"expression_default", "timestamp", "", "NO", "DEFAULT_GENERATED", "CURRENT_TIMESTAMP"},
|
||||
}
|
||||
database := sql.OpenDB(scopeNamesConnector{query: func(statement string, _ []driver.NamedValue) (driver.Rows, error) {
|
||||
if fallback && strings.Contains(statement, "information_schema") {
|
||||
return nil, &mysql.MySQLError{Number: 1142}
|
||||
}
|
||||
switch {
|
||||
case strings.Contains(statement, "information_schema.TABLES"):
|
||||
return &showSchemaRows{names: []string{"TABLE_TYPE", "ENGINE", "TABLE_COLLATION"}, rows: [][]driver.Value{{"BASE TABLE", "InnoDB", "utf8mb4_bin"}}}, nil
|
||||
case strings.Contains(statement, "information_schema.COLUMNS"):
|
||||
if !strings.Contains(statement, "COLUMN_DEFAULT") {
|
||||
t.Fatal("default metadata was not requested")
|
||||
}
|
||||
return &showSchemaRows{names: []string{"COLUMN_NAME", "COLUMN_TYPE", "COLUMN_KEY", "IS_NULLABLE", "EXTRA", "COLUMN_DEFAULT"}, rows: values}, nil
|
||||
case strings.HasPrefix(statement, "SHOW FULL COLUMNS"):
|
||||
return &showSchemaRows{names: []string{"Field", "Type", "Key", "Null", "Extra", "Default"}, rows: values}, nil
|
||||
case strings.HasPrefix(statement, "SHOW CREATE TABLE"):
|
||||
return &showSchemaRows{names: []string{"Table", "Create Table"}, rows: [][]driver.Value{{"table", "CREATE TABLE `table` (`required` int NOT NULL)"}}}, nil
|
||||
case strings.HasPrefix(statement, "SHOW INDEX"), strings.Contains(statement, "information_schema.STATISTICS"):
|
||||
return &showSchemaRows{names: []string{"Key_name"}}, nil
|
||||
default:
|
||||
t.Fatalf("unexpected query: %s", statement)
|
||||
return nil, errors.New("unexpected query")
|
||||
}
|
||||
}})
|
||||
defer database.Close()
|
||||
adapter := &mysqlAdapter{database: database, source: DataSource{Database: "app"}}
|
||||
info, err := adapter.inspect(context.Background(), "table", false)
|
||||
if err != nil || len(info.Columns) != len(values) {
|
||||
t.Fatalf("metadata failed: %+v %v", info, err)
|
||||
}
|
||||
for _, column := range info.Columns {
|
||||
if column.Required != (column.Name == "required") {
|
||||
t.Fatalf("incorrect required flag: %+v", column)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
)
|
||||
|
||||
type mysqlWriteSchema struct {
|
||||
createSQL string
|
||||
object ObjectInfo
|
||||
}
|
||||
|
||||
// SHOW CREATE is a cheap schema fingerprint. Reuse parsed metadata only while
|
||||
// it is identical, so added unique indexes never enable an unsafe upsert.
|
||||
func (adapter *mysqlAdapter) writeSchema(ctx context.Context, name string) (ObjectInfo, error) {
|
||||
rows, err := adapter.database.QueryContext(ctx, "SHOW CREATE TABLE "+QuoteIdentifier(name))
|
||||
if err != nil {
|
||||
return ObjectInfo{}, mysqlOperationError(err)
|
||||
}
|
||||
columns, err := rows.Columns()
|
||||
if err != nil {
|
||||
rows.Close()
|
||||
return ObjectInfo{}, mysqlOperationError(err)
|
||||
}
|
||||
if !rows.Next() {
|
||||
rowErr := rows.Err()
|
||||
rows.Close()
|
||||
if rowErr != nil {
|
||||
return ObjectInfo{}, mysqlOperationError(rowErr)
|
||||
}
|
||||
return ObjectInfo{}, errors.New("objectNotFound")
|
||||
}
|
||||
values := make([]any, len(columns))
|
||||
pointers := make([]any, len(values))
|
||||
for i := range values {
|
||||
pointers[i] = &values[i]
|
||||
}
|
||||
if err := rows.Scan(pointers...); err != nil {
|
||||
rows.Close()
|
||||
return ObjectInfo{}, mysqlOperationError(err)
|
||||
}
|
||||
rowErr := rows.Err()
|
||||
rows.Close()
|
||||
if rowErr != nil {
|
||||
return ObjectInfo{}, mysqlOperationError(rowErr)
|
||||
}
|
||||
if len(values) < 2 {
|
||||
return ObjectInfo{}, errors.New("invalidObject")
|
||||
}
|
||||
createSQL := scalarText(values[1])
|
||||
if cached, ok := adapter.writeSchemas.Load(name); ok {
|
||||
schema := cached.(mysqlWriteSchema)
|
||||
if schema.createSQL == createSQL {
|
||||
return schema.object, nil
|
||||
}
|
||||
}
|
||||
object, err := adapter.inspect(ctx, name, false)
|
||||
if err != nil {
|
||||
return object, err
|
||||
}
|
||||
object.CreateSQL = createSQL
|
||||
adapter.writeSchemas.Store(name, mysqlWriteSchema{createSQL: createSQL, object: object})
|
||||
return object, nil
|
||||
}
|
||||
+122
-4
@@ -216,11 +216,13 @@ func restoreStore(snapshot diskStore, key []byte) (*Store, []byte, error) {
|
||||
store.ReplicationJobs[jobID] = job
|
||||
}
|
||||
}
|
||||
recoveringTasks := make(map[string]bool)
|
||||
for taskID, task := range store.Tasks {
|
||||
if taskID == "" || task.ID != taskID {
|
||||
return nil, nil, errors.New("invalid task identity in store")
|
||||
}
|
||||
if task.Status == "RUNNING" {
|
||||
if unfinishedStatus(task.Status) {
|
||||
recoveringTasks[taskID] = true
|
||||
task.Status = "DRAFT"
|
||||
}
|
||||
if task.SourceDatabase == "" {
|
||||
@@ -235,11 +237,114 @@ func restoreStore(snapshot diskStore, key []byte) (*Store, []byte, error) {
|
||||
if executionID == "" || execution.ID != executionID {
|
||||
return nil, nil, errors.New("invalid execution identity in store")
|
||||
}
|
||||
if unfinishedStatus(execution.Status) {
|
||||
execution.Status, execution.Error, execution.FinishedAt = "FAILED", "interruptedOnRestart", &now
|
||||
if unfinishedStatus(execution.Status) || execution.Error == "interruptedOnRestart" {
|
||||
if unfinishedStatus(execution.Status) {
|
||||
execution.Status, execution.Error, execution.FinishedAt = "FAILED", "interruptedOnRestart", &now
|
||||
}
|
||||
if execution.ObjectErrors == nil {
|
||||
execution.ObjectErrors = make(map[string]string)
|
||||
}
|
||||
for object, status := range execution.ObjectStatuses {
|
||||
if status == "RUNNING" {
|
||||
execution.ObjectStatuses[object] = "FAILED"
|
||||
execution.ObjectErrors[object] = "interruptedOnRestart"
|
||||
if execution.ObjectFinishedAt == nil {
|
||||
execution.ObjectFinishedAt = make(map[string]time.Time)
|
||||
}
|
||||
execution.ObjectFinishedAt[object] = now
|
||||
}
|
||||
if status == "RUNNING" || status == "PENDING" {
|
||||
found := false
|
||||
for _, skipped := range execution.SkippedObjects {
|
||||
if skipped == object {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
execution.SkippedObjects = append(execution.SkippedObjects, object)
|
||||
}
|
||||
}
|
||||
}
|
||||
for object, status := range execution.SchemaStatuses {
|
||||
if status == "RUNNING" {
|
||||
execution.SchemaStatuses[object] = "FAILED"
|
||||
execution.ObjectErrors[object] = "interruptedOnRestart"
|
||||
}
|
||||
}
|
||||
store.Executions[executionID] = execution
|
||||
}
|
||||
}
|
||||
// Newer retries resolve RETRYING markers first. Historical attempts must
|
||||
// not overwrite the current attempt or add their counts again.
|
||||
retries := make([]string, 0)
|
||||
for executionID, execution := range store.Executions {
|
||||
if execution.RetryOf != "" {
|
||||
retries = append(retries, executionID)
|
||||
}
|
||||
}
|
||||
sort.Slice(retries, func(i, j int) bool {
|
||||
left, right := store.Executions[retries[i]], store.Executions[retries[j]]
|
||||
if !left.StartedAt.Equal(right.StartedAt) {
|
||||
return left.StartedAt.After(right.StartedAt)
|
||||
}
|
||||
if persistenceSequence(left.ID) != persistenceSequence(right.ID) {
|
||||
return persistenceSequence(left.ID) > persistenceSequence(right.ID)
|
||||
}
|
||||
return left.ID > right.ID
|
||||
})
|
||||
for _, executionID := range retries {
|
||||
retry := store.Executions[executionID]
|
||||
previous, exists := store.Executions[retry.RetryOf]
|
||||
if !exists || unfinishedStatus(retry.Status) {
|
||||
continue
|
||||
}
|
||||
if merged, applied := mergeRetryResult(previous, retry, true); applied {
|
||||
store.Executions[retry.RetryOf] = merged
|
||||
}
|
||||
}
|
||||
// Older snapshots can contain RETRYING markers whose internal retry was
|
||||
// already pruned. There is no live worker after restart; expose a failed,
|
||||
// independently retryable table instead of leaving the UI permanently busy.
|
||||
for executionID, execution := range store.Executions {
|
||||
unresolved := make([]string, 0)
|
||||
for object, status := range execution.ObjectStatuses {
|
||||
if status == "RETRYING" {
|
||||
unresolved = append(unresolved, object)
|
||||
}
|
||||
}
|
||||
sort.Strings(unresolved)
|
||||
for _, object := range unresolved {
|
||||
retry := Execution{ObjectStatuses: map[string]string{object: "FAILED"}, ObjectErrors: map[string]string{object: "interruptedOnRestart"}, ObjectFinishedAt: map[string]time.Time{object: now}, FinishedAt: &now}
|
||||
if execution.SchemaStatuses[object] == "RUNNING" {
|
||||
retry.SchemaStatuses = map[string]string{object: "FAILED"}
|
||||
}
|
||||
execution, _ = mergeRetryResult(execution, retry, true)
|
||||
}
|
||||
if len(unresolved) > 0 {
|
||||
store.Executions[executionID] = execution
|
||||
}
|
||||
}
|
||||
// Only recover tasks that were active in this snapshot. Edited drafts and
|
||||
// settled tasks must not be overwritten by older execution history.
|
||||
latest := make(map[string]Execution)
|
||||
for _, execution := range store.Executions {
|
||||
if !recoveringTasks[execution.TaskID] {
|
||||
continue
|
||||
}
|
||||
previous, found := latest[execution.TaskID]
|
||||
if !found || execution.StartedAt.After(previous.StartedAt) || (execution.StartedAt.Equal(previous.StartedAt) && (persistenceSequence(execution.ID) > persistenceSequence(previous.ID) || (persistenceSequence(execution.ID) == persistenceSequence(previous.ID) && execution.ID > previous.ID))) {
|
||||
latest[execution.TaskID] = execution
|
||||
}
|
||||
}
|
||||
for taskID, execution := range latest {
|
||||
if original, found := store.Executions[execution.RetryOf]; found && original.TaskID == taskID {
|
||||
execution = original
|
||||
}
|
||||
task := store.Tasks[taskID]
|
||||
task.Status = execution.Status
|
||||
store.Tasks[taskID] = task
|
||||
}
|
||||
for exportID, export := range store.Exports {
|
||||
if exportID == "" || export.ID != exportID {
|
||||
return nil, nil, errors.New("invalid export identity in store")
|
||||
@@ -322,6 +427,19 @@ func (s *server) persist() error {
|
||||
snapshot.Tasks[taskID] = task
|
||||
}
|
||||
for executionID, execution := range snapshot.Executions {
|
||||
execution.SkippedObjects = append([]string(nil), execution.SkippedObjects...)
|
||||
execution.SchemaStatuses = maps.Clone(execution.SchemaStatuses)
|
||||
execution.ObjectErrors = maps.Clone(execution.ObjectErrors)
|
||||
execution.ObjectStartedAt = maps.Clone(execution.ObjectStartedAt)
|
||||
execution.ObjectFinishedAt = maps.Clone(execution.ObjectFinishedAt)
|
||||
execution.RetryOriginalStatuses = maps.Clone(execution.RetryOriginalStatuses)
|
||||
execution.ObjectStatuses = maps.Clone(execution.ObjectStatuses)
|
||||
execution.ObjectAttemptIDs = maps.Clone(execution.ObjectAttemptIDs)
|
||||
execution.ObjectRows = maps.Clone(execution.ObjectRows)
|
||||
execution.ObjectWritten = maps.Clone(execution.ObjectWritten)
|
||||
execution.ObjectSkipped = maps.Clone(execution.ObjectSkipped)
|
||||
execution.TargetObjects = maps.Clone(execution.TargetObjects)
|
||||
snapshot.Executions[executionID] = execution
|
||||
if execution.FinishedAt != nil {
|
||||
finishedAt := *execution.FinishedAt
|
||||
execution.FinishedAt = &finishedAt
|
||||
@@ -486,7 +604,7 @@ func trimHistory[Value any](records map[string]Value, retain func(Value) bool) {
|
||||
}
|
||||
|
||||
func trimStoreHistories(store *Store) {
|
||||
trimHistory(store.Executions, func(execution Execution) bool { return unfinishedStatus(execution.Status) })
|
||||
trimExecutionHistory(store.Executions)
|
||||
trimHistory(store.Exports, func(export Export) bool {
|
||||
if unfinishedStatus(export.Status) {
|
||||
return true
|
||||
|
||||
@@ -39,7 +39,7 @@ type replicationEvent struct {
|
||||
}
|
||||
|
||||
type replicationMonitorStore struct {
|
||||
db *sql.DB
|
||||
db *localSQLiteDB
|
||||
mu sync.Mutex
|
||||
}
|
||||
|
||||
@@ -57,10 +57,7 @@ func openReplicationMonitorStore(path string) (*replicationMonitorStore, error)
|
||||
if err := file.Close(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
database, err := sql.Open("sqlite", filepath.ToSlash(path)+"?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)&_pragma=synchronous(FULL)")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
database := openLocalSQLite(filepath.ToSlash(path) + "?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)&_pragma=synchronous(FULL)")
|
||||
database.SetMaxOpenConns(4)
|
||||
statements := []string{
|
||||
`CREATE TABLE IF NOT EXISTS replication_samples(id INTEGER PRIMARY KEY AUTOINCREMENT, source_id TEXT NOT NULL, target_id TEXT NOT NULL, observed_at_ns INTEGER NOT NULL, available INTEGER NOT NULL, configured INTEGER NOT NULL, io_running TEXT NOT NULL, sql_running TEXT NOT NULL, lag_seconds INTEGER, io_error INTEGER NOT NULL, sql_error INTEGER NOT NULL, healthy INTEGER NOT NULL, code TEXT NOT NULL)`,
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
)
|
||||
|
||||
type replicationSeedConfig struct {
|
||||
NoLock bool `json:"no_lock,omitempty"`
|
||||
Workers int `json:"workers"`
|
||||
ShardRows int `json:"shard_rows"`
|
||||
BatchRows int `json:"batch_rows"`
|
||||
@@ -271,10 +272,46 @@ type seedSnapshot struct {
|
||||
}
|
||||
|
||||
func (snapshot *seedSnapshot) close() {
|
||||
closed := make(map[*sql.Conn]bool)
|
||||
for _, conn := range snapshot.Readers {
|
||||
discardSeedConn(conn)
|
||||
if !closed[conn] {
|
||||
discardSeedConn(conn)
|
||||
closed[conn] = true
|
||||
}
|
||||
}
|
||||
discardSeedConn(snapshot.Lock)
|
||||
if !closed[snapshot.Lock] {
|
||||
discardSeedConn(snapshot.Lock)
|
||||
}
|
||||
}
|
||||
|
||||
// One connection gives every shard the same read view without a global lock.
|
||||
// No GTID is attached: this snapshot cannot seed native replication safely.
|
||||
func acquireUnlockedSnapshot(ctx context.Context, db *sql.DB, database string) (_ *seedSnapshot, resultErr error) {
|
||||
conn, err := db.Conn(ctx)
|
||||
if err != nil {
|
||||
return nil, errors.New("backupSnapshotFailed")
|
||||
}
|
||||
snapshot := &seedSnapshot{Lock: conn, Readers: []*sql.Conn{conn, conn}}
|
||||
defer func() {
|
||||
if resultErr != nil {
|
||||
snapshot.close()
|
||||
}
|
||||
}()
|
||||
for _, statement := range []string{
|
||||
"SET SESSION wait_timeout=21660",
|
||||
"SET SESSION time_zone='+00:00'",
|
||||
"SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ",
|
||||
"START TRANSACTION WITH CONSISTENT SNAPSHOT, READ ONLY",
|
||||
} {
|
||||
if _, err := conn.ExecContext(ctx, statement); err != nil {
|
||||
return nil, errors.New("backupSnapshotFailed")
|
||||
}
|
||||
}
|
||||
snapshot.Tables, err = readSeedTables(ctx, conn, database)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return snapshot, nil
|
||||
}
|
||||
|
||||
func acquireSeedSnapshot(ctx context.Context, db *sql.DB, database string, workers int) (_ *seedSnapshot, resultErr error) {
|
||||
|
||||
@@ -77,6 +77,7 @@ type seedFixture struct {
|
||||
executed []string
|
||||
nextID int
|
||||
locked bool
|
||||
allowUnlocked bool
|
||||
snapshotStarts int
|
||||
failSnapshotAt int
|
||||
localCalls, inserts, commits, rollbacks int
|
||||
@@ -162,7 +163,7 @@ func (c *seedConn) ExecContext(ctx context.Context, query string, args []driver.
|
||||
case query == "UNLOCK TABLES":
|
||||
f.locked = false
|
||||
case strings.HasPrefix(query, "START TRANSACTION WITH CONSISTENT SNAPSHOT"):
|
||||
if !f.locked {
|
||||
if !f.locked && !f.allowUnlocked {
|
||||
return nil, errors.New("read view outside global lock")
|
||||
}
|
||||
f.snapshotStarts++
|
||||
|
||||
@@ -0,0 +1,190 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"reflect"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestConcurrentRetrySubmissionStartsOnlyOneAttempt(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
s.runtimeCtx = ctx
|
||||
s.jobs = make(map[string]context.CancelFunc)
|
||||
s.slots = make(chan struct{}, 4)
|
||||
t.Cleanup(func() { cancel(); s.workers.Wait() })
|
||||
reader := &schemaParallelReader{gate: make(chan struct{}), entered: make(chan struct{}, 4)}
|
||||
s.adapterFactory = func(_ context.Context, source DataSource, _ func(SQLAudit)) (Adapter, error) {
|
||||
if source.ID == "source" {
|
||||
return reader, nil
|
||||
}
|
||||
return &tableSkipAdapter{}, nil
|
||||
}
|
||||
task := Task{ID: "task", Name: "task", SourceID: "source", TargetID: "target", SourceDatabase: "app", TargetDatabase: "app", Objects: []string{"a"}, Status: "PARTIAL", MaxRows: -1, BatchSize: 10}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["original"] = Execution{ID: "original", TaskID: task.ID, Status: "PARTIAL", ObjectStatuses: map[string]string{"a": "FAILED"}, SkippedObjects: []string{"a"}}
|
||||
var accepted atomic.Int32
|
||||
start := make(chan struct{})
|
||||
failures := make(chan error, 8)
|
||||
var submitters sync.WaitGroup
|
||||
for i := 0; i < 8; i++ {
|
||||
submitters.Add(1)
|
||||
go func() {
|
||||
defer submitters.Done()
|
||||
<-start
|
||||
_, err := s.retrySkippedObjects("original", []string{"a"})
|
||||
if err == nil {
|
||||
accepted.Add(1)
|
||||
} else if err.Error() != "objectNotSkipped" && err.Error() != "taskBusy" {
|
||||
failures <- err
|
||||
}
|
||||
}()
|
||||
}
|
||||
close(start)
|
||||
submitters.Wait()
|
||||
close(failures)
|
||||
for err := range failures {
|
||||
t.Fatalf("unexpected retry rejection: %v", err)
|
||||
}
|
||||
s.store.RLock()
|
||||
count := len(s.store.Executions)
|
||||
state := s.store.Executions["original"].ObjectStatuses["a"]
|
||||
s.store.RUnlock()
|
||||
if accepted.Load() != 1 || count != 2 || state != "RETRYING" {
|
||||
t.Fatalf("duplicate attempts: accepted=%d records=%d table=%s", accepted.Load(), count, state)
|
||||
}
|
||||
close(reader.gate)
|
||||
done := make(chan struct{})
|
||||
go func() { s.workers.Wait(); close(done) }()
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("retry did not complete")
|
||||
}
|
||||
if reader.reads.Load() != 1 || s.store.Executions["original"].Status != "COMPLETED" {
|
||||
t.Fatalf("table was repeated or original did not settle: reads=%d result=%+v", reader.reads.Load(), s.store.Executions["original"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryCannotUseStaleSkippedListToRepeatCompletedOrActiveTables(t *testing.T) {
|
||||
for _, status := range []string{"COMPLETED", "RUNNING", "RETRYING"} {
|
||||
t.Run(status, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
task := Task{ID: "task", Name: "task", SourceID: "source", TargetID: "target", SourceDatabase: "app", TargetDatabase: "app", Objects: []string{"a"}, Status: "PARTIAL"}
|
||||
s.store.Tasks[task.ID] = task
|
||||
original := Execution{ID: "original", TaskID: task.ID, Status: "PARTIAL", ObjectStatuses: map[string]string{"a": status}, SkippedObjects: []string{"a"}}
|
||||
s.store.Executions[original.ID] = original
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
t.Fatal("ineligible retry opened database connection")
|
||||
return nil, errors.New("unexpected connection")
|
||||
}
|
||||
_, err := s.retrySkippedObjects(original.ID, []string{"a"})
|
||||
if err == nil || err.Error() != "objectNotSkipped" || len(s.store.Executions) != 1 || !reflect.DeepEqual(s.store.Executions[original.ID], original) || !reflect.DeepEqual(s.store.Tasks[task.ID], task) {
|
||||
t.Fatalf("ineligible retry changed state: %v %+v", err, s.store)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryRejectsPendingOriginalAndActiveExecutionWithStaleTaskStatus(t *testing.T) {
|
||||
for _, state := range []string{"original-pending", "task-pending", "child-running", "child-pending", "other-running"} {
|
||||
t.Run(state, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
task := Task{ID: "task", Name: "task", SourceID: "source", TargetID: "target", SourceDatabase: "app", TargetDatabase: "app", Objects: []string{"a"}, Status: "PARTIAL"}
|
||||
original := Execution{ID: "original", TaskID: task.ID, Status: "PARTIAL", ObjectStatuses: map[string]string{"a": "FAILED"}, SkippedObjects: []string{"a"}}
|
||||
code := "taskBusy"
|
||||
switch state {
|
||||
case "original-pending":
|
||||
original.Status = "PENDING"
|
||||
code = "executionNotFound"
|
||||
case "task-pending":
|
||||
task.Status = "PENDING"
|
||||
default:
|
||||
active := Execution{ID: "active", TaskID: task.ID, Status: "RUNNING", RetryOf: original.ID}
|
||||
if state == "child-pending" {
|
||||
active.Status = "PENDING"
|
||||
}
|
||||
if state == "other-running" {
|
||||
active.RetryOf = ""
|
||||
}
|
||||
s.store.Executions[active.ID] = active
|
||||
}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions[original.ID] = original
|
||||
count := len(s.store.Executions)
|
||||
_, err := s.retrySkippedObjects(original.ID, []string{"a"})
|
||||
if err == nil || err.Error() != code || len(s.store.Executions) != count || !reflect.DeepEqual(s.store.Tasks[task.ID], task) || !reflect.DeepEqual(s.store.Executions[original.ID], original) {
|
||||
t.Fatalf("busy retry changed state: %v %+v", err, s.store)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryBatchRejectsInvalidSelection(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Executions["original"] = Execution{ID: "original", Status: "COMPLETED", SkippedObjects: []string{"a", "b"}}
|
||||
for _, objects := range [][]string{nil, {"a", "a"}, {"a", "other"}} {
|
||||
if _, err := s.retrySkippedObjects("original", objects); err == nil {
|
||||
t.Fatalf("accepted invalid selection: %v", objects)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryBatchKeepsFailedTables(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Executions["original"] = Execution{ID: "original", Status: "COMPLETED", SkippedObjects: []string{"a", "b", "c"}, ObjectStatuses: map[string]string{"a": "FAILED", "b": "FAILED", "c": "SKIPPED"}}
|
||||
s.store.Executions["retry"] = Execution{ID: "retry", RetryOf: "original", ObjectStatuses: map[string]string{"a": "COMPLETED", "b": "FAILED"}}
|
||||
s.finishExecution("retry", "COMPLETED", "", nil)
|
||||
previous := s.store.Executions["original"]
|
||||
if len(previous.SkippedObjects) != 2 || previous.SkippedObjects[0] != "b" || previous.SkippedObjects[1] != "c" || previous.ObjectStatuses["a"] != "COMPLETED" {
|
||||
t.Fatalf("incorrect retry results: %+v", previous)
|
||||
}
|
||||
}
|
||||
func TestRetryRestoresUnstartedTableStatus(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Executions["original"] = Execution{ID: "original", SkippedObjects: []string{"a", "b"}, ObjectStatuses: map[string]string{"a": "RETRYING", "b": "RETRYING"}}
|
||||
s.store.Executions["retry"] = Execution{ID: "retry", RetryOf: "original", RetryOriginalStatuses: map[string]string{"a": "FAILED", "b": "SKIPPED"}, ObjectStatuses: map[string]string{"a": "RUNNING", "b": "PENDING"}}
|
||||
s.finishExecution("retry", "FAILED", "", nil)
|
||||
states := s.store.Executions["original"].ObjectStatuses
|
||||
if states["a"] != "FAILED" || states["b"] != "SKIPPED" {
|
||||
t.Fatalf("stale retry status: %v", states)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryUpdatesOriginalExecution(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Executions["original"] = Execution{ID: "original", Status: "FAILED", Error: "mysqlOperationFailed", Rows: 2, Written: 2, SkippedObjects: []string{"a"}, ObjectStatuses: map[string]string{"a": "RETRYING"}, ObjectErrors: map[string]string{"a": "mysqlOperationFailed"}}
|
||||
s.store.Executions["retry"] = Execution{ID: "retry", RetryOf: "original", Rows: 3, Written: 3, ObjectStatuses: map[string]string{"a": "COMPLETED"}, RetryOriginalStatuses: map[string]string{"a": "FAILED"}}
|
||||
s.finishExecution("retry", "COMPLETED", "", nil)
|
||||
original := s.store.Executions["original"]
|
||||
if original.Status != "COMPLETED" || original.Error != "" || original.Written != 5 || original.Rows != 5 || original.FinishedAt == nil || len(original.SkippedObjects) != 0 {
|
||||
t.Fatalf("original not updated: %+v", original)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryCompletionIsIdempotent(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Executions["original"] = Execution{ID: "original", TaskID: "task", Status: "FAILED", Rows: 2, Written: 2, SkippedObjects: []string{"a"}, ObjectStatuses: map[string]string{"a": "RETRYING"}}
|
||||
s.store.Executions["retry"] = Execution{ID: "retry", TaskID: "task", RetryOf: "original", Status: "RUNNING", Rows: 3, Written: 3, ObjectStatuses: map[string]string{"a": "COMPLETED"}, RetryOriginalStatuses: map[string]string{"a": "FAILED"}}
|
||||
s.store.Tasks["task"] = Task{ID: "task", Status: "RUNNING"}
|
||||
s.finishExecution("retry", "COMPLETED", "", nil)
|
||||
finished := *s.store.Executions["retry"].FinishedAt
|
||||
s.finishExecution("retry", "FAILED", "adapterOperationFailed", nil)
|
||||
original, retry := s.store.Executions["original"], s.store.Executions["retry"]
|
||||
if original.Rows != 5 || original.Written != 5 || original.Status != "COMPLETED" || retry.Status != "COMPLETED" || !retry.FinishedAt.Equal(finished) || s.store.Tasks["task"].Status != "COMPLETED" {
|
||||
t.Fatalf("repeated completion changed the result: original=%+v retry=%+v", original, retry)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLateExecutionCompletionDoesNotRestoreDeletedRecords(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Tasks["task"] = Task{ID: "task", Status: "COMPLETED"}
|
||||
s.finishExecution("removed", "FAILED", "adapterOperationFailed", nil)
|
||||
if len(s.store.Executions) != 0 || s.store.Tasks["task"].Status != "COMPLETED" {
|
||||
t.Fatal("late completion resurrected a deleted record or changed task status")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type retryContinueAdapter struct {
|
||||
tableSkipAdapter
|
||||
failure string
|
||||
visited *[]string
|
||||
}
|
||||
|
||||
func (a *retryContinueAdapter) Inspect(ctx context.Context, name string) (ObjectInfo, error) {
|
||||
if a.failure == "inspect" && name == "bad" {
|
||||
return ObjectInfo{}, errors.New("targetObjectMissing")
|
||||
}
|
||||
return a.tableSkipAdapter.Inspect(ctx, name)
|
||||
}
|
||||
func (a *retryContinueAdapter) Read(_ context.Context, name string, _ ReadOptions, consume func([]Record) error) error {
|
||||
*a.visited = append(*a.visited, name)
|
||||
if a.failure == "read" && name == "bad" {
|
||||
return errors.New("mysqlOperationFailed")
|
||||
}
|
||||
return consume([]Record{{"id": 1}})
|
||||
}
|
||||
func (a *retryContinueAdapter) Write(ctx context.Context, name string, records []Record, options WriteOptions) (WriteResult, error) {
|
||||
if a.failure == "write" && name == "bad" {
|
||||
return WriteResult{}, errors.New("mysqlOperationFailed")
|
||||
}
|
||||
return a.tableSkipAdapter.Write(ctx, name, records, options)
|
||||
}
|
||||
func TestBatchRetryContinuesAfterTableFailure(t *testing.T) {
|
||||
for _, failure := range []string{"inspect", "read", "write"} {
|
||||
t.Run(failure, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
var visited []string
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
return &retryContinueAdapter{failure: failure, visited: &visited}, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"bad", "good"}, MaxRows: 100, BatchSize: 10}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["original"] = Execution{ID: "original", SkippedObjects: []string{"bad", "good"}, ObjectStatuses: map[string]string{"bad": "RETRYING", "good": "RETRYING"}}
|
||||
s.store.Executions["retry"] = Execution{ID: "retry", TaskID: task.ID, RetryOf: "original", Status: "RUNNING", RetryOriginalStatuses: map[string]string{"bad": "FAILED", "good": "FAILED"}}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "retry")
|
||||
retry := s.store.Executions["retry"]
|
||||
if retry.ObjectStatuses["bad"] != "FAILED" || retry.ObjectStatuses["good"] != "COMPLETED" || retry.Written != 1 {
|
||||
t.Fatalf("retry stopped prematurely: %+v", retry)
|
||||
}
|
||||
original := s.store.Executions["original"]
|
||||
if original.Status != "PARTIAL" || original.ObjectStatuses["good"] != "COMPLETED" || original.ObjectStatuses["bad"] != "FAILED" || len(original.SkippedObjects) != 1 || original.SkippedObjects[0] != "bad" {
|
||||
t.Fatalf("incorrect retry outcome: %+v", original)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestOrdinaryRunAllTableFailuresPreserveWatermark(t *testing.T) {
|
||||
for _, failure := range []string{"read", "write"} {
|
||||
t.Run(failure, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
var visited []string
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
return &retryContinueAdapter{failure: failure, visited: &visited}, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"bad"}, MaxRows: -1, BatchSize: 10, Mode: "incremental", IncrementalField: "id", IncrementalValue: 42}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
result := s.store.Executions["exec"]
|
||||
if result.Status != "FAILED" || result.Error != "mysqlOperationFailed" || result.ObjectStatuses["bad"] != "FAILED" || len(result.SkippedObjects) != 1 || result.SkippedObjects[0] != "bad" {
|
||||
t.Fatalf("all-table data failure was not failed and retryable: %+v", result)
|
||||
}
|
||||
if s.store.Tasks[task.ID].Status != "FAILED" || s.store.Tasks[task.ID].IncrementalValue != 42 {
|
||||
t.Fatalf("failed task advanced its watermark or hid its failure: %+v", s.store.Tasks[task.ID])
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type retryTimeoutAdapter struct{ tableSkipAdapter }
|
||||
|
||||
func (a *retryTimeoutAdapter) Read(ctx context.Context, name string, _ ReadOptions, consume func([]Record) error) error {
|
||||
if name == "slow" {
|
||||
<-ctx.Done()
|
||||
return ctx.Err()
|
||||
}
|
||||
return consume([]Record{{"id": 1}})
|
||||
}
|
||||
func TestBatchRetryContinuesAfterTableTimeout(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) { return &retryTimeoutAdapter{}, nil }
|
||||
task := Task{ID: "task", Objects: []string{"slow", "good"}, MaxRows: 100, BatchSize: 10, TimeoutSeconds: 1}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["original"] = Execution{ID: "original", SkippedObjects: []string{"slow", "good"}, ObjectStatuses: map[string]string{"slow": "RETRYING", "good": "RETRYING"}}
|
||||
s.store.Executions["retry"] = Execution{ID: "retry", TaskID: task.ID, RetryOf: "original", Status: "RUNNING", RetryOriginalStatuses: map[string]string{"slow": "FAILED", "good": "FAILED"}}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "retry")
|
||||
retry := s.store.Executions["retry"]
|
||||
if retry.ObjectErrors["slow"] != "operationTimeout" || retry.ObjectStatuses["good"] != "COMPLETED" || retry.Written != 1 {
|
||||
t.Fatalf("timeout stopped batch: %+v", retry)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestRetryLatestResultClearsOldErrorsAndMissingTimestamps(t *testing.T) {
|
||||
for _, recovering := range []bool{false, true} {
|
||||
for _, status := range []string{"SKIPPED", "LIMIT_REACHED", "COMPLETED", "FAILED"} {
|
||||
t.Run(status+map[bool]string{false: "/live", true: "/restart"}[recovering], func(t *testing.T) {
|
||||
old := time.Now().UTC().Add(-time.Hour)
|
||||
finished := old.Add(time.Minute)
|
||||
previous := Execution{ID: "original", Status: "PARTIAL", ObjectStatuses: map[string]string{"a": "RETRYING", "waiting": "RETRYING", "done": "COMPLETED"}, ObjectErrors: map[string]string{"a": "oldFailure", "waiting": "waitingFailure"}, ObjectStartedAt: map[string]time.Time{"a": old, "waiting": old}, ObjectFinishedAt: map[string]time.Time{"a": finished, "waiting": finished}, SkippedObjects: []string{"a", "waiting"}}
|
||||
retry := Execution{ID: "latest", ObjectStatuses: map[string]string{"a": status, "waiting": "PENDING"}, RetryOriginalStatuses: map[string]string{"a": "FAILED", "waiting": "FAILED"}}
|
||||
merged, applied := mergeRetryResult(previous, retry, recovering)
|
||||
if !applied || merged.ObjectErrors["a"] != "" || !merged.ObjectStartedAt["a"].IsZero() || !merged.ObjectFinishedAt["a"].IsZero() {
|
||||
t.Fatalf("latest attempt retained previous metadata: %+v", merged)
|
||||
}
|
||||
if merged.ObjectErrors["waiting"] != "waitingFailure" || !merged.ObjectStartedAt["waiting"].Equal(old) || !merged.ObjectFinishedAt["waiting"].Equal(finished) || merged.ObjectStatuses["waiting"] != "FAILED" || merged.ObjectStatuses["done"] != "COMPLETED" || merged.Error != "waitingFailure" {
|
||||
t.Fatalf("unattempted metadata changed: %+v", merged)
|
||||
}
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Executions[merged.ID] = merged
|
||||
if err := s.persist(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
loaded, _, err := loadStore(s.dataDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if loaded.Executions[merged.ID].ObjectErrors["a"] != "" || loaded.Executions[merged.ID].ObjectErrors["waiting"] != "waitingFailure" {
|
||||
t.Fatal("restart restored stale error")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetrySkipAndLimitDoNotRetainAnOverallFailure(t *testing.T) {
|
||||
for _, status := range []string{"SKIPPED", "LIMIT_REACHED"} {
|
||||
previous := Execution{ObjectStatuses: map[string]string{"a": "RETRYING"}, ObjectErrors: map[string]string{"a": "oldFailure"}, Error: "oldFailure"}
|
||||
retry := Execution{ObjectStatuses: map[string]string{"a": status}}
|
||||
merged, _ := mergeRetryResult(previous, retry, false)
|
||||
if merged.Status != "PARTIAL" || merged.Error != "" || merged.ObjectErrors["a"] != "" {
|
||||
t.Fatalf("new nonfailure result retained old failure: %+v", merged)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"reflect"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type retryOrderAdapter struct {
|
||||
tableSkipAdapter
|
||||
order *[]string
|
||||
unavailable bool
|
||||
}
|
||||
|
||||
func TestRetrySubmissionOrdersConfiguredTablesBeforeHistoricalTables(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
s.runtimeCtx = ctx
|
||||
s.jobs = make(map[string]context.CancelFunc)
|
||||
s.slots = make(chan struct{}, 1)
|
||||
t.Cleanup(func() { cancel(); s.workers.Wait() })
|
||||
var order []string
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
return &retryOrderAdapter{order: &order}, nil
|
||||
}
|
||||
task := Task{ID: "task", Name: "retry task", SourceID: "source", TargetID: "target", SourceDatabase: "app", TargetDatabase: "app", Objects: []string{"a", "b", "c"}, Mode: "full", Conflict: "upsert", MaxRows: -1, BatchSize: 10}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["original"] = Execution{ID: "original", TaskID: task.ID, Status: "PARTIAL", ObjectStatuses: map[string]string{"a": "COMPLETED", "b": "FAILED", "c": "FAILED", "historical": "FAILED"}, SkippedObjects: []string{"b", "c", "historical"}}
|
||||
retry, err := s.retrySkippedObjects("original", []string{"historical", "c", "b"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
done := make(chan struct{})
|
||||
go func() { s.workers.Wait(); close(done) }()
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("submitted retry did not finish")
|
||||
}
|
||||
if !reflect.DeepEqual(order, []string{"b", "c", "historical"}) || !reflect.DeepEqual(s.store.Tasks[task.ID].Objects, task.Objects) || s.store.Executions[retry.ID].Status != "COMPLETED" || s.store.Executions["original"].Status != "COMPLETED" {
|
||||
t.Fatalf("submission reordered/configuration changed: reads=%v task=%+v execution=%+v", order, s.store.Tasks[task.ID], s.store.Executions[retry.ID])
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryValidatesEffectiveObjectsBeforeMutatingState(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name, mode, code string
|
||||
objects []string
|
||||
targets map[string]string
|
||||
}{
|
||||
{"multiple historical tables in incremental mode", "incremental", "invalidIncremental", []string{"old_a", "old_b"}, nil},
|
||||
{"invalid historical target name", "full", "invalidObject", []string{"old_a"}, map[string]string{"old_a": "bad\nname"}},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
task := Task{ID: "task", Name: "task", SourceID: "source", TargetID: "target", SourceDatabase: "app", TargetDatabase: "app", Objects: []string{"current"}, Mode: test.mode, Conflict: "upsert", IncrementalField: "id", IncrementalValue: 0, TargetObjects: test.targets}
|
||||
s.store.Tasks[task.ID] = task
|
||||
original := Execution{ID: "original", TaskID: task.ID, Status: "PARTIAL", ObjectStatuses: map[string]string{"old_a": "FAILED", "old_b": "FAILED"}, SkippedObjects: []string{"old_a", "old_b"}}
|
||||
s.store.Executions["original"] = original
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
t.Fatal("invalid retry connected to database")
|
||||
return nil, errors.New("unexpected connection")
|
||||
}
|
||||
_, err := s.retrySkippedObjects("original", test.objects)
|
||||
if err == nil || err.Error() != test.code || len(s.store.Executions) != 1 || !reflect.DeepEqual(s.store.Executions["original"], original) || !reflect.DeepEqual(s.store.Tasks[task.ID], task) {
|
||||
t.Fatalf("invalid retry mutated state: %v %+v", err, s.store)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func (a *retryOrderAdapter) EstimatedRows(_ context.Context, name string) (int64, error) {
|
||||
if a.unavailable && name == "small" {
|
||||
return 0, errors.New("rowEstimateUnavailable")
|
||||
}
|
||||
return map[string]int64{"large": 100000, "small": 10, "medium": 1000}[name], nil
|
||||
}
|
||||
func (a *retryOrderAdapter) Read(_ context.Context, name string, _ ReadOptions, consume func([]Record) error) error {
|
||||
*a.order = append(*a.order, name)
|
||||
return consume([]Record{{"id": 1}})
|
||||
}
|
||||
func TestRetryPreservesConfiguredOrder(t *testing.T) {
|
||||
for _, unavailable := range []bool{false, true} {
|
||||
t.Run(map[bool]string{false: "estimated order", true: "estimate unavailable"}[unavailable], func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
var order []string
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
return &retryOrderAdapter{order: &order, unavailable: unavailable}, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"large", "small", "medium"}, MaxRows: 100, BatchSize: 10}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["original"] = Execution{ID: "original", SkippedObjects: append([]string(nil), task.Objects...)}
|
||||
s.store.Executions["retry"] = Execution{ID: "retry", TaskID: task.ID, RetryOf: "original", Status: "RUNNING"}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "retry")
|
||||
want := task.Objects
|
||||
if unavailable {
|
||||
want = task.Objects
|
||||
}
|
||||
if !reflect.DeepEqual(order, want) {
|
||||
t.Fatalf("retry order=%v want=%v", order, want)
|
||||
}
|
||||
if s.store.Executions["retry"].Status != "COMPLETED" {
|
||||
t.Fatal("retry did not complete")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,189 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestRestartReconcilesRetryMetadataAndCountsOnce(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
old := time.Now().UTC().Add(-time.Hour)
|
||||
started := old.Add(time.Minute)
|
||||
finished := started.Add(time.Minute)
|
||||
s.store.Tasks["task"] = Task{ID: "task", SourceID: "source", TargetID: "target", Mode: "incremental", IncrementalValue: float64(10), Status: "RUNNING"}
|
||||
s.store.Executions["exec-1"] = Execution{
|
||||
ID: "exec-1", TaskID: "task", Status: "FAILED", Rows: 2, Written: 2, StartedAt: old, FinishedAt: &old,
|
||||
SkippedObjects: []string{"a", "b", "c"}, ObjectStatuses: map[string]string{"a": "RETRYING", "b": "RETRYING", "c": "RETRYING"},
|
||||
ObjectErrors: map[string]string{"a": "oldError", "b": "oldError", "c": "oldError"}, SchemaStatuses: map[string]string{"a": "FAILED", "b": "FAILED", "c": "FAILED"},
|
||||
ObjectStartedAt: map[string]time.Time{"a": old, "b": old, "c": old}, ObjectFinishedAt: map[string]time.Time{"a": old, "b": old, "c": old},
|
||||
}
|
||||
s.store.Executions["exec-3"] = Execution{
|
||||
ID: "exec-3", TaskID: "task", RetryOf: "exec-1", Status: "RUNNING", Rows: 3, Written: 3, Skipped: 1, StartedAt: started,
|
||||
ObjectStatuses: map[string]string{"a": "COMPLETED", "b": "RUNNING", "c": "PENDING"}, RetryOriginalStatuses: map[string]string{"a": "FAILED", "b": "FAILED", "c": "SKIPPED"},
|
||||
SchemaStatuses: map[string]string{"a": "COMPLETED", "b": "COMPLETED", "c": "PENDING"}, ObjectStartedAt: map[string]time.Time{"a": started, "b": finished}, ObjectFinishedAt: map[string]time.Time{"a": finished},
|
||||
}
|
||||
// An older attempt must neither override the active attempt nor add its
|
||||
// already-accounted totals to the original on every restart.
|
||||
s.store.Executions["exec-2"] = Execution{ID: "exec-2", TaskID: "task", RetryOf: "exec-1", Status: "FAILED", StartedAt: old, FinishedAt: &old, Rows: 99, Written: 99, ObjectStatuses: map[string]string{"a": "FAILED", "b": "COMPLETED", "c": "FAILED"}, RetryOriginalStatuses: map[string]string{"a": "FAILED", "b": "FAILED", "c": "FAILED"}}
|
||||
if err := s.persist(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
restored, key, err := loadStore(s.dataDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
original := restored.Executions["exec-1"]
|
||||
if restored.Tasks["task"].Status != "PARTIAL" {
|
||||
t.Fatalf("task disagrees with recovered original: %+v", restored.Tasks["task"])
|
||||
}
|
||||
if original.Status != "PARTIAL" || original.ObjectStatuses["a"] != "COMPLETED" || original.ObjectStatuses["b"] != "FAILED" || original.ObjectStatuses["c"] != "SKIPPED" || original.Rows != 5 || original.Written != 5 || original.Skipped != 1 {
|
||||
t.Fatalf("retry results not recovered: %+v", original)
|
||||
}
|
||||
if original.ObjectErrors["a"] != "" || original.ObjectErrors["b"] != "interruptedOnRestart" || original.Error != "interruptedOnRestart" || original.SchemaStatuses["a"] != "COMPLETED" || original.SchemaStatuses["b"] != "COMPLETED" || original.SchemaStatuses["c"] != "FAILED" {
|
||||
t.Fatalf("retry errors/schema not recovered: %+v", original)
|
||||
}
|
||||
if !original.ObjectStartedAt["a"].Equal(started) || !original.ObjectFinishedAt["a"].Equal(finished) || !original.ObjectFinishedAt["b"].After(finished) || !original.ObjectFinishedAt["c"].Equal(old) || original.FinishedAt == nil || !original.FinishedAt.Equal(*restored.Executions["exec-3"].FinishedAt) {
|
||||
t.Fatalf("retry timestamps not recovered: %+v", original)
|
||||
}
|
||||
if len(original.SkippedObjects) != 2 || original.SkippedObjects[0] != "b" || original.SkippedObjects[1] != "c" || restored.Tasks["task"].IncrementalValue != json.Number("10") {
|
||||
t.Fatalf("completed table remains retryable or recovery advanced the watermark: skipped=%v cursor=%v (%T)", original.SkippedObjects, restored.Tasks["task"].IncrementalValue, restored.Tasks["task"].IncrementalValue)
|
||||
}
|
||||
s.store, s.persistenceKey = restored, key
|
||||
if err := s.persist(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
again, _, err := loadStore(s.dataDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if again.Executions["exec-1"].Rows != 5 || again.Executions["exec-1"].Written != 5 || again.Executions["exec-1"].ObjectStatuses["b"] != "FAILED" {
|
||||
t.Fatal("repeated restart replayed historical retry results")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryCompletionCopiesTableMetadata(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
started := time.Now().UTC().Add(-time.Minute)
|
||||
finished := started.Add(time.Second)
|
||||
s.store.Executions["original"] = Execution{ID: "original", Status: "FAILED", ObjectStatuses: map[string]string{"a": "RETRYING"}, ObjectErrors: map[string]string{"a": "previousError"}}
|
||||
s.store.Executions["retry"] = Execution{ID: "retry", RetryOf: "original", Status: "RUNNING", ObjectStatuses: map[string]string{"a": "COMPLETED"}, RetryOriginalStatuses: map[string]string{"a": "FAILED"}, SchemaStatuses: map[string]string{"a": "COMPLETED"}, ObjectStartedAt: map[string]time.Time{"a": started}, ObjectFinishedAt: map[string]time.Time{"a": finished}}
|
||||
s.finishExecution("retry", "COMPLETED", "", nil)
|
||||
original := s.store.Executions["original"]
|
||||
if original.Status != "COMPLETED" || original.ObjectErrors["a"] != "" || !original.ObjectStartedAt["a"].Equal(started) || !original.ObjectFinishedAt["a"].Equal(finished) || original.SchemaStatuses["a"] != "COMPLETED" {
|
||||
t.Fatalf("original metadata not updated: %+v", original)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRecoveredFailureRemainsIndependentlyRetryable(t *testing.T) {
|
||||
previous := Execution{Status: "COMPLETED", ObjectStatuses: map[string]string{"a": "RETRYING"}}
|
||||
retry := Execution{ObjectStatuses: map[string]string{"a": "FAILED"}, ObjectErrors: map[string]string{"a": "interruptedOnRestart"}}
|
||||
merged, applied := mergeRetryResult(previous, retry, true)
|
||||
if !applied || merged.Status != "FAILED" || len(merged.SkippedObjects) != 1 || merged.SkippedObjects[0] != "a" || merged.Error != "interruptedOnRestart" {
|
||||
t.Fatalf("recovered failure has no retry action: %+v", merged)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetrySummaryUsesAllOriginalTables(t *testing.T) {
|
||||
for _, recovering := range []bool{false, true} {
|
||||
for _, test := range []struct {
|
||||
name, other, result, want string
|
||||
}{
|
||||
{"successful subset still leaves failure", "FAILED", "COMPLETED", "PARTIAL"},
|
||||
{"failed retry preserves previous success", "COMPLETED", "FAILED", "PARTIAL"},
|
||||
{"no successful table", "FAILED", "FAILED", "FAILED"},
|
||||
{"all tables repaired", "COMPLETED", "COMPLETED", "COMPLETED"},
|
||||
{"deliberate skips remain incomplete", "SKIPPED", "SKIPPED", "PARTIAL"},
|
||||
{"row limit remains incomplete", "LIMIT_REACHED", "LIMIT_REACHED", "PARTIAL"},
|
||||
{"failure with queued table", "PENDING", "FAILED", "PARTIAL"},
|
||||
} {
|
||||
t.Run(test.name+map[bool]string{false: "/live", true: "/restart"}[recovering], func(t *testing.T) {
|
||||
previous := Execution{Status: "FAILED", Rows: 10, Written: 8, ObjectStatuses: map[string]string{"other": test.other, "retry": "RETRYING"}, SkippedObjects: []string{"other", "retry"}, ObjectErrors: map[string]string{"other": "oldFailure", "retry": "oldRetryFailure"}}
|
||||
retry := Execution{Rows: 2, Written: 1, ObjectStatuses: map[string]string{"retry": test.result}, ObjectErrors: map[string]string{"retry": "mysqlOperationFailed"}}
|
||||
merged, applied := mergeRetryResult(previous, retry, recovering)
|
||||
if !applied || merged.Status != test.want || merged.Rows != 12 || merged.Written != 9 || merged.ObjectStatuses["other"] != test.other {
|
||||
t.Fatalf("incorrect original summary: %+v", merged)
|
||||
}
|
||||
if test.want == "COMPLETED" && (len(merged.SkippedObjects) != 0 || merged.Error != "") {
|
||||
t.Fatalf("repaired tables retain failures: %+v", merged)
|
||||
}
|
||||
if test.want != "COMPLETED" && (len(merged.SkippedObjects) == 0 || merged.Error == "") {
|
||||
t.Fatalf("remaining failure is hidden: %+v", merged)
|
||||
}
|
||||
if recovering {
|
||||
again, replayed := mergeRetryResult(merged, retry, true)
|
||||
if replayed || again.Rows != merged.Rows || again.Status != test.want {
|
||||
t.Fatal("restart replay changed settled summary")
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryTargetSnapshotOnlyChangesForAttemptedTables(t *testing.T) {
|
||||
for _, recovering := range []bool{false, true} {
|
||||
previous := Execution{ObjectStatuses: map[string]string{"attempted": "RETRYING", "waiting": "RETRYING"}, TargetObjects: map[string]string{"attempted": "old_a", "waiting": "old_b"}}
|
||||
retry := Execution{ObjectStatuses: map[string]string{"attempted": "FAILED", "waiting": "PENDING"}, RetryOriginalStatuses: map[string]string{"attempted": "FAILED", "waiting": "FAILED"}, TargetObjects: map[string]string{"attempted": "new_a", "waiting": "new_b"}}
|
||||
merged, applied := mergeRetryResult(previous, retry, recovering)
|
||||
if !applied || merged.TargetObjects["attempted"] != "new_a" || merged.TargetObjects["waiting"] != "old_b" {
|
||||
t.Fatalf("audit target incorrectly merged: %+v", merged)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestartRecoversRetryMarkerWithoutInternalRecord(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Executions["original"] = Execution{ID: "original", Status: "PARTIAL", Rows: 7, Written: 5, ObjectStatuses: map[string]string{"a": "RETRYING", "b": "COMPLETED"}, SchemaStatuses: map[string]string{"a": "RUNNING", "b": "COMPLETED"}}
|
||||
if err := s.persist(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
store, _, err := loadStore(s.dataDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
original := store.Executions["original"]
|
||||
if original.Status != "PARTIAL" || original.ObjectStatuses["a"] != "FAILED" || original.SchemaStatuses["a"] != "FAILED" || original.ObjectErrors["a"] != "interruptedOnRestart" || len(original.SkippedObjects) != 1 || original.ObjectFinishedAt["a"].IsZero() || original.Rows != 7 || original.Written != 5 || original.ObjectStatuses["b"] != "COMPLETED" {
|
||||
t.Fatalf("orphan retry marker remained stuck: %+v", original)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryAttemptScopePersistsWithoutInternalHistory(t *testing.T) {
|
||||
for _, recovering := range []bool{false, true} {
|
||||
previous := Execution{ID: "original", Status: "PARTIAL", ObjectStatuses: map[string]string{"attempted": "RETRYING", "waiting": "RETRYING"}, ObjectAttemptIDs: map[string]string{"attempted": "older", "waiting": "older"}}
|
||||
retry := Execution{ID: "latest", ObjectStatuses: map[string]string{"attempted": "FAILED", "waiting": "PENDING"}, ObjectErrors: map[string]string{"attempted": "writeConflict"}, RetryOriginalStatuses: map[string]string{"attempted": "FAILED", "waiting": "FAILED"}}
|
||||
merged, applied := mergeRetryResult(previous, retry, recovering)
|
||||
if !applied || merged.ObjectAttemptIDs["attempted"] != "latest" || merged.ObjectAttemptIDs["waiting"] != "older" {
|
||||
t.Fatalf("attempt scope wrong: %+v", merged.ObjectAttemptIDs)
|
||||
}
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Executions[merged.ID] = merged // Internal retry has already been pruned.
|
||||
if err := s.persist(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
restored, _, err := loadStore(s.dataDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if restored.Executions[merged.ID].ObjectAttemptIDs["attempted"] != "latest" {
|
||||
t.Fatal("restart lost audit scope")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryTableCountsUseLatestAttemptAndPreserveUnattemptedTables(t *testing.T) {
|
||||
for _, recovering := range []bool{false, true} {
|
||||
previous := Execution{Rows: 10, Written: 8, ObjectStatuses: map[string]string{"a": "RETRYING", "waiting": "RETRYING"}, ObjectRows: map[string]int64{"a": 10, "waiting": 4}, ObjectWritten: map[string]int64{"a": 8, "waiting": 3}, ObjectSkipped: map[string]int64{"a": 2, "waiting": 1}}
|
||||
retry := Execution{ID: "retry", Rows: 2, Written: 1, Skipped: 1, ObjectStatuses: map[string]string{"a": "COMPLETED", "waiting": "PENDING"}, RetryOriginalStatuses: map[string]string{"waiting": "FAILED"}, ObjectRows: map[string]int64{"a": 2, "waiting": 0}, ObjectWritten: map[string]int64{"a": 1, "waiting": 0}, ObjectSkipped: map[string]int64{"a": 1, "waiting": 0}}
|
||||
merged, applied := mergeRetryResult(previous, retry, recovering)
|
||||
if !applied || merged.Rows != 12 || merged.Written != 9 || merged.ObjectRows["a"] != 2 || merged.ObjectWritten["a"] != 1 || merged.ObjectSkipped["a"] != 1 || merged.ObjectRows["waiting"] != 4 || merged.ObjectWritten["waiting"] != 3 {
|
||||
t.Fatalf("latest table counts incorrect: %+v", merged)
|
||||
}
|
||||
if recovering {
|
||||
again, replayed := mergeRetryResult(merged, retry, true)
|
||||
if replayed || again.Rows != 12 || again.ObjectRows["a"] != 2 {
|
||||
t.Fatal("recovery replay changed table counts")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"time"
|
||||
)
|
||||
|
||||
// mergeRetryResult updates the original execution. During restart recovery only
|
||||
// tables still marked RETRYING are eligible, so a previously applied result is
|
||||
// never counted a second time.
|
||||
func mergeRetryResult(previous, retry Execution, recovering bool) (Execution, bool) {
|
||||
objects := make(map[string]bool)
|
||||
for object := range retry.RetryOriginalStatuses {
|
||||
objects[object] = true
|
||||
}
|
||||
for object := range retry.ObjectStatuses {
|
||||
objects[object] = true
|
||||
}
|
||||
if previous.ObjectStatuses == nil {
|
||||
previous.ObjectStatuses = make(map[string]string)
|
||||
}
|
||||
applied := false
|
||||
names := make([]string, 0, len(objects))
|
||||
for object := range objects {
|
||||
names = append(names, object)
|
||||
}
|
||||
sort.Strings(names)
|
||||
for _, object := range names {
|
||||
if recovering && previous.ObjectStatuses[object] != "RETRYING" {
|
||||
continue
|
||||
}
|
||||
applied = true
|
||||
result := retry.ObjectStatuses[object]
|
||||
attempted := result != "" && result != "PENDING"
|
||||
if result == "" || result == "PENDING" {
|
||||
result = retry.RetryOriginalStatuses[object]
|
||||
if result == "" {
|
||||
result = "SKIPPED"
|
||||
}
|
||||
}
|
||||
previous.ObjectStatuses[object] = result
|
||||
if attempted {
|
||||
copyCount := func(counts map[string]int64, destination *map[string]int64) {
|
||||
if count, present := counts[object]; present {
|
||||
if *destination == nil {
|
||||
*destination = make(map[string]int64)
|
||||
}
|
||||
(*destination)[object] = count
|
||||
} else {
|
||||
delete(*destination, object)
|
||||
}
|
||||
}
|
||||
copyCount(retry.ObjectRows, &previous.ObjectRows)
|
||||
copyCount(retry.ObjectWritten, &previous.ObjectWritten)
|
||||
copyCount(retry.ObjectSkipped, &previous.ObjectSkipped)
|
||||
}
|
||||
if retry.ID != "" && attempted {
|
||||
if previous.ObjectAttemptIDs == nil {
|
||||
previous.ObjectAttemptIDs = make(map[string]string)
|
||||
}
|
||||
previous.ObjectAttemptIDs[object] = retry.ID
|
||||
}
|
||||
if targetName := retry.TargetObjects[object]; targetName != "" && attempted {
|
||||
if previous.TargetObjects == nil {
|
||||
previous.TargetObjects = make(map[string]string)
|
||||
}
|
||||
previous.TargetObjects[object] = targetName
|
||||
}
|
||||
if result != "COMPLETED" {
|
||||
found := false
|
||||
for _, skipped := range previous.SkippedObjects {
|
||||
if skipped == object {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
previous.SkippedObjects = append(previous.SkippedObjects, object)
|
||||
}
|
||||
}
|
||||
if schema := retry.SchemaStatuses[object]; schema != "" && schema != "PENDING" {
|
||||
if previous.SchemaStatuses == nil {
|
||||
previous.SchemaStatuses = make(map[string]string)
|
||||
}
|
||||
previous.SchemaStatuses[object] = schema
|
||||
}
|
||||
if started := retry.ObjectStartedAt[object]; attempted && !started.IsZero() {
|
||||
if previous.ObjectStartedAt == nil {
|
||||
previous.ObjectStartedAt = make(map[string]time.Time)
|
||||
}
|
||||
previous.ObjectStartedAt[object] = started
|
||||
} else if attempted {
|
||||
delete(previous.ObjectStartedAt, object)
|
||||
}
|
||||
if finished := retry.ObjectFinishedAt[object]; attempted && !finished.IsZero() {
|
||||
if previous.ObjectFinishedAt == nil {
|
||||
previous.ObjectFinishedAt = make(map[string]time.Time)
|
||||
}
|
||||
previous.ObjectFinishedAt[object] = finished
|
||||
} else if attempted {
|
||||
delete(previous.ObjectFinishedAt, object)
|
||||
}
|
||||
if result == "COMPLETED" {
|
||||
delete(previous.ObjectErrors, object)
|
||||
} else if attempted && retry.ObjectErrors[object] != "" {
|
||||
if previous.ObjectErrors == nil {
|
||||
previous.ObjectErrors = make(map[string]string)
|
||||
}
|
||||
previous.ObjectErrors[object] = retry.ObjectErrors[object]
|
||||
} else if attempted {
|
||||
delete(previous.ObjectErrors, object)
|
||||
}
|
||||
}
|
||||
if recovering && !applied {
|
||||
return previous, false
|
||||
}
|
||||
filtered := make([]string, 0, len(previous.SkippedObjects))
|
||||
for _, object := range previous.SkippedObjects {
|
||||
if previous.ObjectStatuses[object] != "COMPLETED" {
|
||||
filtered = append(filtered, object)
|
||||
}
|
||||
}
|
||||
previous.SkippedObjects = filtered
|
||||
previous.Rows += retry.Rows
|
||||
previous.Written += retry.Written
|
||||
previous.Skipped += retry.Skipped
|
||||
if retry.FinishedAt != nil {
|
||||
previous.FinishedAt = retry.FinishedAt
|
||||
}
|
||||
previous.Status, previous.Error = executionTableSummary(previous)
|
||||
return previous, true
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestRetryAPIRejectsChangedReviewedTaskWithoutChangingResults(t *testing.T) {
|
||||
for _, change := range []string{"mapping", "watermark", "run-state"} {
|
||||
t.Run(change, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
task := revisionTaskFixture()
|
||||
revision := taskForAPI(task).Revision
|
||||
switch change {
|
||||
case "mapping":
|
||||
task.Mapping = map[string]string{"id": "other_id"}
|
||||
case "watermark":
|
||||
task.IncrementalValue = json.Number("9007199254740993")
|
||||
case "run-state":
|
||||
now := time.Now().UTC()
|
||||
task.LastRunAt = &now
|
||||
}
|
||||
s.store.Tasks[task.ID] = task
|
||||
original := Execution{ID: "original", TaskID: task.ID, Status: "PARTIAL", ObjectStatuses: map[string]string{"table": "FAILED"}, SkippedObjects: []string{"table"}, Written: 12}
|
||||
s.store.Executions[original.ID] = original
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
t.Error("stale retry opened a database connection")
|
||||
return &tableSkipAdapter{}, nil
|
||||
}
|
||||
for _, endpoint := range []string{"retry", "retry-object"} {
|
||||
input := map[string]any{"revision": revision}
|
||||
if endpoint == "retry-object" {
|
||||
input["object"] = "table"
|
||||
} else {
|
||||
input["objects"] = []string{"table"}
|
||||
}
|
||||
data, err := json.Marshal(input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
s.execution(rec, httptest.NewRequest(http.MethodPost, "/api/executions/original/"+endpoint, bytes.NewReader(data)))
|
||||
if rec.Code != http.StatusConflict || !strings.Contains(rec.Body.String(), "retryReviewChanged") {
|
||||
t.Fatalf("stale retry accepted: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if len(s.store.Executions) != 1 || !reflect.DeepEqual(s.store.Tasks[task.ID], task) || !reflect.DeepEqual(s.store.Executions[original.ID], original) {
|
||||
t.Fatal("rejected retry changed task or original execution")
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryAPIAcceptsCurrentRevisionAndLegacyRequests(t *testing.T) {
|
||||
for _, reviewed := range []bool{true, false} {
|
||||
t.Run(map[bool]string{true: "reviewed", false: "legacy"}[reviewed], func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
s.runtimeCtx = ctx
|
||||
s.jobs = make(map[string]context.CancelFunc)
|
||||
s.slots = make(chan struct{}, 4)
|
||||
t.Cleanup(func() { cancel(); s.workers.Wait() })
|
||||
reader := &schemaParallelReader{gate: make(chan struct{}), entered: make(chan struct{}, 4)}
|
||||
t.Cleanup(func() { close(reader.gate) })
|
||||
s.adapterFactory = func(_ context.Context, source DataSource, _ func(SQLAudit)) (Adapter, error) {
|
||||
if source.ID == "source" {
|
||||
return reader, nil
|
||||
}
|
||||
return &tableSkipAdapter{}, nil
|
||||
}
|
||||
task := Task{ID: "task", Name: "task", SourceID: "source", TargetID: "target", SourceDatabase: "app", TargetDatabase: "app", Objects: []string{"a"}, Status: "PARTIAL", MaxRows: -1, BatchSize: 10}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["original"] = Execution{ID: "original", TaskID: task.ID, Status: "PARTIAL", ObjectStatuses: map[string]string{"a": "FAILED"}, SkippedObjects: []string{"a"}}
|
||||
input := map[string]any{"objects": []string{"a"}}
|
||||
if reviewed {
|
||||
input["revision"] = taskForAPI(task).Revision
|
||||
}
|
||||
data, err := json.Marshal(input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
s.execution(rec, httptest.NewRequest(http.MethodPost, "/api/executions/original/retry", bytes.NewReader(data)))
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("current retry rejected: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
s.store.RLock()
|
||||
defer s.store.RUnlock()
|
||||
if len(s.store.Executions) != 2 || s.store.Executions["original"].ObjectStatuses["a"] != "RETRYING" {
|
||||
t.Fatal("accepted retry did not mark original table as retrying")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -14,6 +14,7 @@ func (s *server) handler() http.Handler {
|
||||
mux.HandleFunc("/api/executions/", s.execution)
|
||||
mux.HandleFunc("GET /api/events", s.streamEvents)
|
||||
mux.HandleFunc("/api/exports/sql", s.createExport)
|
||||
mux.HandleFunc("/api/exports/wes", s.createExport)
|
||||
mux.HandleFunc("/api/exports/csv", s.createCSVExport)
|
||||
mux.HandleFunc("/api/exports", s.exports)
|
||||
mux.HandleFunc("/api/exports/", s.export)
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestRunAPIRejectsChangedReviewedTaskWithoutStartingExecution(t *testing.T) {
|
||||
for _, change := range []string{"mapping", "watermark", "run-state", "dry-run", "target-database"} {
|
||||
t.Run(change, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
task := revisionTaskFixture()
|
||||
revision := taskForAPI(task).Revision
|
||||
switch change {
|
||||
case "mapping":
|
||||
task.Mapping = map[string]string{"id": "other_id"}
|
||||
case "watermark":
|
||||
task.IncrementalValue = json.Number("9007199254740993")
|
||||
case "run-state":
|
||||
now := time.Now().UTC()
|
||||
task.LastRunAt = &now
|
||||
case "dry-run":
|
||||
task.DryRun = !task.DryRun
|
||||
case "target-database":
|
||||
task.TargetDatabase = "another_database"
|
||||
}
|
||||
s.store.Tasks[task.ID] = task
|
||||
data, err := json.Marshal(map[string]any{"revision": revision, "confirm": true})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
s.task(rec, httptest.NewRequest(http.MethodPost, "/api/tasks/task/run", bytes.NewReader(data)))
|
||||
if rec.Code != http.StatusConflict || !strings.Contains(rec.Body.String(), "runReviewChanged") {
|
||||
t.Fatalf("stale run accepted: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if len(s.store.Executions) != 0 || !reflect.DeepEqual(s.store.Tasks[task.ID], task) {
|
||||
t.Fatal("rejected run changed task or created execution")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunAPIAcceptsCurrentRevisionAndLegacyRequests(t *testing.T) {
|
||||
for _, reviewed := range []bool{true, false} {
|
||||
for _, dryRun := range []bool{true, false} {
|
||||
t.Run(map[bool]string{true: "reviewed", false: "legacy"}[reviewed]+"/"+map[bool]string{true: "dry", false: "write"}[dryRun], func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
s.runtimeCtx = ctx
|
||||
s.jobs = make(map[string]context.CancelFunc)
|
||||
s.slots = make(chan struct{}, 4)
|
||||
t.Cleanup(func() { cancel(); s.workers.Wait() })
|
||||
reader := &schemaParallelReader{gate: make(chan struct{}), entered: make(chan struct{}, 4)}
|
||||
t.Cleanup(func() { close(reader.gate) })
|
||||
s.adapterFactory = func(_ context.Context, source DataSource, _ func(SQLAudit)) (Adapter, error) {
|
||||
if source.ID == "source" {
|
||||
return reader, nil
|
||||
}
|
||||
return &tableSkipAdapter{}, nil
|
||||
}
|
||||
task := Task{ID: "task", Name: "task", SourceID: "source", TargetID: "target", SourceDatabase: "app", TargetDatabase: "app", Objects: []string{"a"}, Status: "PARTIAL", MaxRows: -1, BatchSize: 10, DryRun: dryRun}
|
||||
s.store.Tasks[task.ID] = task
|
||||
input := map[string]any{"confirm": !dryRun}
|
||||
if reviewed {
|
||||
input["revision"] = taskForAPI(task).Revision
|
||||
}
|
||||
data, err := json.Marshal(input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
s.task(rec, httptest.NewRequest(http.MethodPost, "/api/tasks/task/run", bytes.NewReader(data)))
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("current run rejected: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
s.store.RLock()
|
||||
defer s.store.RUnlock()
|
||||
if len(s.store.Executions) != 1 || s.store.Tasks[task.ID].Status != "RUNNING" {
|
||||
t.Fatal("accepted run did not start execution")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunCurrentRevisionStillRequiresWriteConfirmation(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
task := revisionTaskFixture()
|
||||
s.store.Tasks[task.ID] = task
|
||||
data, err := json.Marshal(map[string]any{"revision": taskForAPI(task).Revision, "confirm": false})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
s.task(rec, httptest.NewRequest(http.MethodPost, "/api/tasks/task/run", bytes.NewReader(data)))
|
||||
if rec.Code != http.StatusConflict || !strings.Contains(rec.Body.String(), "confirmationRequired") || len(s.store.Executions) != 0 || !reflect.DeepEqual(s.store.Tasks[task.ID], task) {
|
||||
t.Fatalf("version bypassed confirmation: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"database/sql/driver"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/go-sql-driver/mysql"
|
||||
)
|
||||
|
||||
type schemaAuditConnector struct {
|
||||
scopeNamesConnector
|
||||
execute func(context.Context, string, []driver.NamedValue) (driver.Result, error)
|
||||
}
|
||||
|
||||
func (c schemaAuditConnector) Connect(context.Context) (driver.Conn, error) {
|
||||
return &schemaAuditConn{scopeNamesConn{c.scopeNamesConnector}, c.execute}, nil
|
||||
}
|
||||
|
||||
type schemaAuditConn struct {
|
||||
scopeNamesConn
|
||||
execute func(context.Context, string, []driver.NamedValue) (driver.Result, error)
|
||||
}
|
||||
|
||||
func (c *schemaAuditConn) ExecContext(ctx context.Context, statement string, args []driver.NamedValue) (driver.Result, error) {
|
||||
return c.execute(ctx, statement, args)
|
||||
}
|
||||
|
||||
func TestAutomaticMySQLSchemaCreationAuditsActualStatementAndOutcome(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name, code string
|
||||
failure error
|
||||
cancel bool
|
||||
}{
|
||||
{"success", "", nil, false},
|
||||
{"permissions", "mysqlOperationFailed", &mysql.MySQLError{Number: 1142, Message: "private server details"}, false},
|
||||
{"cancelled", "operationCancelled", nil, true},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
var statement string
|
||||
connector := schemaAuditConnector{scopeNamesConnector: scopeNamesConnector{query: func(query string, _ []driver.NamedValue) (driver.Rows, error) {
|
||||
if query != "SHOW CREATE TABLE `target_%``table`" {
|
||||
t.Fatalf("unexpected preflight query: %s", query)
|
||||
}
|
||||
return nil, &mysql.MySQLError{Number: 1146}
|
||||
}}, execute: func(ctx context.Context, sql string, args []driver.NamedValue) (driver.Result, error) {
|
||||
statement = sql
|
||||
if len(args) != 0 {
|
||||
t.Fatal("unexpected CREATE arguments")
|
||||
}
|
||||
if test.cancel {
|
||||
cancel()
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
return driver.RowsAffected(0), test.failure
|
||||
}}
|
||||
db := sql.OpenDB(connector)
|
||||
defer db.Close()
|
||||
var audits []SQLAudit
|
||||
writer := &mysqlAdapter{database: db, source: DataSource{ID: "target", Type: "mysql"}, audit: func(entry SQLAudit) { audits = append(audits, entry) }}
|
||||
info := syncColumnFixture("id")
|
||||
info.Name = "source"
|
||||
info.CreateSQL = "CREATE TABLE `source` (`id` int NOT NULL) ENGINE=InnoDB"
|
||||
err := prepareSyncSchema(ctx, writer, DataSource{Type: "mysql"}, writer.source, info, "target_%`table", Task{})
|
||||
if AdapterErrorCode(err) != test.code || len(audits) != 1 {
|
||||
t.Fatalf("audit missing or wrong failure: %v %+v", err, audits)
|
||||
}
|
||||
entry := audits[0]
|
||||
wantStatus := "SUCCESS"
|
||||
if test.code != "" {
|
||||
wantStatus = "FAILED"
|
||||
}
|
||||
if statement != "CREATE TABLE `target_%``table` (`id` int NOT NULL) ENGINE=InnoDB" || entry.SQL != statement || entry.SourceID != "target" || entry.Object != "target_%`table" || entry.Operation != "SCHEMA" || entry.Status != wantStatus || entry.Error != test.code || entry.StartedAt.IsZero() || entry.Rows != 0 {
|
||||
t.Fatalf("incorrect statement/outcome audit: %+v", entry)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestAutomaticSchemaPreflightRejectionDoesNotInventSQLAudit(t *testing.T) {
|
||||
db := sql.OpenDB(schemaAuditConnector{scopeNamesConnector: scopeNamesConnector{query: func(string, []driver.NamedValue) (driver.Rows, error) { return nil, &mysql.MySQLError{Number: 1146} }}, execute: func(context.Context, string, []driver.NamedValue) (driver.Result, error) {
|
||||
t.Fatal("rejected schema was executed")
|
||||
return nil, errors.New("unexpected execution")
|
||||
}})
|
||||
defer db.Close()
|
||||
writer := &mysqlAdapter{database: db, audit: func(SQLAudit) { t.Fatal("SQL was audited without being attempted") }}
|
||||
info := syncColumnFixture("id")
|
||||
info.Name = "source"
|
||||
info.CreateSQL = "CREATE TABLE `source` (`id` int) ENGINE=InnoDB"
|
||||
for _, task := range []Task{{DryRun: true}, {Mapping: map[string]string{"id": "renamed"}}} {
|
||||
if err := prepareSyncSchema(context.Background(), writer, DataSource{Type: "mysql"}, DataSource{Type: "mysql"}, info, "target", task); err == nil {
|
||||
t.Fatal("unsafe or dry-run schema was accepted")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAutomaticSchemaFailureAuditKeepsTaskExecutionAndTargetScope(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
info := syncColumnFixture("id")
|
||||
info.CreateSQL = "CREATE TABLE `source_table` (`id` int NOT NULL) ENGINE=InnoDB"
|
||||
reader := &syncColumnsAdapter{objects: map[string]ObjectInfo{"source_table": info}}
|
||||
db := sql.OpenDB(schemaAuditConnector{scopeNamesConnector: scopeNamesConnector{query: func(string, []driver.NamedValue) (driver.Rows, error) { return nil, &mysql.MySQLError{Number: 1146} }}, execute: func(context.Context, string, []driver.NamedValue) (driver.Result, error) {
|
||||
return nil, &mysql.MySQLError{Number: 1142}
|
||||
}})
|
||||
t.Cleanup(func() { db.Close() })
|
||||
s.adapterFactory = func(_ context.Context, source DataSource, audit func(SQLAudit)) (Adapter, error) {
|
||||
if source.ID == "source" {
|
||||
return reader, nil
|
||||
}
|
||||
return &mysqlAdapter{database: db, source: source, audit: audit}, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"source_table"}, TargetObjects: map[string]string{"source_table": "target_table"}, MaxRows: -1, BatchSize: 10}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
if len(reader.reads) != 0 || s.store.Executions["exec"].SchemaStatuses["source_table"] != "FAILED" {
|
||||
t.Fatal("failed CREATE did not stop data for its table")
|
||||
}
|
||||
result := s.store.Executions["exec"]
|
||||
if result.Status != "FAILED" || result.Error != "mysqlOperationFailed" || len(result.SkippedObjects) != 1 || result.SkippedObjects[0] != "source_table" {
|
||||
t.Fatalf("all-table failure was not reported as failed and retryable: %+v", result)
|
||||
}
|
||||
restored, _, err := loadStore(s.dataDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(restored.Audits) != 1 {
|
||||
t.Fatalf("DDL audit was not persisted: %v", restored.Audits)
|
||||
}
|
||||
if restored.Executions["exec"].Status != "FAILED" || restored.Executions["exec"].Error != "mysqlOperationFailed" {
|
||||
t.Fatal("failed execution summary was not persisted")
|
||||
}
|
||||
for _, entry := range restored.Audits {
|
||||
if entry.TaskID != "task" || entry.ExecutionID != "exec" || entry.SourceID != "target" || entry.Object != "target_table" || entry.Status != "FAILED" || entry.Error != "mysqlOperationFailed" || entry.SQL != "CREATE TABLE `target_table` (`id` int NOT NULL) ENGINE=InnoDB" || entry.SQLHash == "" || entry.FinishedAt == nil {
|
||||
t.Fatalf("failure detail cannot locate persisted DDL: %+v", entry)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,121 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"reflect"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type dependencySchemaReader struct {
|
||||
tableSkipAdapter
|
||||
parents map[string]string
|
||||
reads []string
|
||||
}
|
||||
|
||||
func (a *dependencySchemaReader) Inspect(_ context.Context, name string) (ObjectInfo, error) {
|
||||
info := syncColumnFixture("id")
|
||||
info.Name = name
|
||||
if a.parents[name] != "" {
|
||||
info.CreateSQL = "CREATE TABLE " + QuoteIdentifier(name) + " (id INT, FOREIGN KEY (id) REFERENCES " + QuoteIdentifier(a.parents[name]) + " (id))"
|
||||
}
|
||||
return info, nil
|
||||
}
|
||||
func (a *dependencySchemaReader) Read(_ context.Context, name string, _ ReadOptions, consume func([]Record) error) error {
|
||||
a.reads = append(a.reads, name)
|
||||
return consume([]Record{{"id": 1}})
|
||||
}
|
||||
|
||||
// Model target availability: a dependent target becomes usable only after
|
||||
// its parent structure has completed. No real database is touched.
|
||||
type dependencySchemaWriter struct {
|
||||
tableSkipAdapter
|
||||
mu sync.Mutex
|
||||
parents map[string]string
|
||||
ready map[string]bool
|
||||
attempts map[string]int
|
||||
started chan struct{}
|
||||
}
|
||||
|
||||
func (a *dependencySchemaWriter) Inspect(ctx context.Context, name string) (ObjectInfo, error) {
|
||||
if a.started != nil {
|
||||
close(a.started)
|
||||
<-ctx.Done()
|
||||
return ObjectInfo{}, ctx.Err()
|
||||
}
|
||||
a.mu.Lock()
|
||||
defer a.mu.Unlock()
|
||||
a.attempts[name]++
|
||||
if parent := a.parents[name]; parent != "" && !a.ready[parent] {
|
||||
return ObjectInfo{}, errors.New("mysqlOperationFailed")
|
||||
}
|
||||
a.ready[name] = true
|
||||
return syncColumnFixture("id"), nil
|
||||
}
|
||||
|
||||
func TestSchemaDependencyPassesResolveReverseChainAndBoundCycles(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
parents := map[string]string{"leaf": "middle", "middle": "root", "cycle_a": "cycle_b", "cycle_b": "cycle_a"}
|
||||
reader := &dependencySchemaReader{parents: parents}
|
||||
writer := &dependencySchemaWriter{parents: parents, ready: map[string]bool{}, attempts: map[string]int{}}
|
||||
s.adapterFactory = func(_ context.Context, source DataSource, _ func(SQLAudit)) (Adapter, error) {
|
||||
if source.ID == "source" {
|
||||
return reader, nil
|
||||
}
|
||||
return writer, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"leaf", "middle", "root", "cycle_a", "cycle_b"}, SchemaWorkers: 4, MaxRows: -1, BatchSize: 10}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
|
||||
defer cancel()
|
||||
s.runTask(ctx, task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
result := s.store.Executions["exec"]
|
||||
if result.Status != "PARTIAL" || !reflect.DeepEqual(reader.reads, []string{"leaf", "middle", "root"}) || writer.attempts["leaf"] != 2 || writer.attempts["middle"] != 1 || writer.attempts["root"] != 1 || writer.attempts["cycle_a"] != 3 || writer.attempts["cycle_b"] != 3 {
|
||||
t.Fatalf("dependency processing did not progress/bound retries: %+v reads=%v attempts=%v", result, reader.reads, writer.attempts)
|
||||
}
|
||||
for _, name := range []string{"cycle_a", "cycle_b"} {
|
||||
if result.SchemaStatuses[name] != "FAILED" || result.ObjectStatuses[name] != "FAILED" || result.ObjectErrors[name] != "mysqlOperationFailed" {
|
||||
t.Fatalf("cycle was not independently retryable: %+v", result)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCancelledDependencyTableSettlesFailureWithoutStartingData(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
reader := &dependencySchemaReader{parents: map[string]string{"active": "parent", "waiting": "parent"}}
|
||||
writer := &dependencySchemaWriter{started: make(chan struct{})}
|
||||
s.adapterFactory = func(_ context.Context, source DataSource, _ func(SQLAudit)) (Adapter, error) {
|
||||
if source.ID == "source" {
|
||||
return reader, nil
|
||||
}
|
||||
return writer, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"active", "waiting"}, SchemaWorkers: 2, MaxRows: -1, BatchSize: 10}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
s.runTask(ctx, task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
}()
|
||||
select {
|
||||
case <-writer.started:
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("dependency phase did not start")
|
||||
}
|
||||
cancel()
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("dependency phase did not stop")
|
||||
}
|
||||
result := s.store.Executions["exec"]
|
||||
if result.Status != "CANCELLED" || result.SchemaStatuses["active"] != "FAILED" || result.ObjectStatuses["active"] != "FAILED" || result.ObjectErrors["active"] != "operationCancelled" || result.ObjectFinishedAt["active"].IsZero() || result.SchemaStatuses["waiting"] != "PENDING" || !reflect.DeepEqual(result.SkippedObjects, []string{"active", "waiting"}) || len(reader.reads) != 0 {
|
||||
t.Fatalf("cancelled dependency appears unexecuted or starts data: %+v reads=%v", result, reader.reads)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,193 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"reflect"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestCancelSchemaPhaseSettlesTablesWithoutRestart(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
reader := &schemaParallelReader{gate: make(chan struct{}), entered: make(chan struct{}, 2)}
|
||||
var connections atomic.Int32
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
if connections.Add(1) == 1 {
|
||||
return reader, nil
|
||||
}
|
||||
return &tableSkipAdapter{}, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"active", "waiting"}, SchemaWorkers: 1, MaxRows: -1, BatchSize: 10, Mode: "incremental", IncrementalValue: 42}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
s.runTask(ctx, task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
}()
|
||||
select {
|
||||
case <-reader.entered:
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("schema worker did not start")
|
||||
}
|
||||
cancel()
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("cancelled schema worker did not finish")
|
||||
}
|
||||
execution := s.store.Executions["exec"]
|
||||
if execution.Status != "CANCELLED" || execution.SchemaStatuses["active"] != "FAILED" || execution.ObjectStatuses["active"] != "FAILED" || execution.ObjectErrors["active"] != "operationCancelled" || execution.ObjectFinishedAt["active"].IsZero() {
|
||||
t.Fatalf("active schema table remained unfinished: %+v", execution)
|
||||
}
|
||||
if execution.SchemaStatuses["waiting"] != "PENDING" || execution.ObjectStatuses["waiting"] != "PENDING" || !reflect.DeepEqual(execution.SkippedObjects, []string{"active", "waiting"}) || reader.reads.Load() != 0 || s.store.Tasks[task.ID].IncrementalValue != 42 {
|
||||
t.Fatalf("waiting table was not retryable or cancellation read/advanced data: %+v", execution)
|
||||
}
|
||||
s.finishExecution("exec", "FAILED", "adapterOperationFailed", nil)
|
||||
if !reflect.DeepEqual(s.store.Executions["exec"], execution) {
|
||||
t.Fatal("repeated finish changed cancelled tables")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFinishCancelledRetryRestoresWaitingTableAndSettlesSchema(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Executions["original"] = Execution{ID: "original", Status: "PARTIAL", ObjectStatuses: map[string]string{"active": "RETRYING", "waiting": "RETRYING", "done": "COMPLETED"}, SchemaStatuses: map[string]string{"active": "COMPLETED", "waiting": "COMPLETED", "done": "COMPLETED"}, SkippedObjects: []string{"active", "waiting"}}
|
||||
s.store.Executions["retry"] = Execution{ID: "retry", RetryOf: "original", Status: "RUNNING", ObjectStatuses: map[string]string{"active": "PENDING", "waiting": "PENDING"}, SchemaStatuses: map[string]string{"active": "RUNNING", "waiting": "PENDING"}, RetryOriginalStatuses: map[string]string{"active": "FAILED", "waiting": "SKIPPED"}}
|
||||
s.finishExecution("retry", "CANCELLED", "operationCancelled", nil)
|
||||
original := s.store.Executions["original"]
|
||||
if original.ObjectStatuses["active"] != "FAILED" || original.SchemaStatuses["active"] != "FAILED" || original.ObjectErrors["active"] != "operationCancelled" || original.ObjectStatuses["waiting"] != "SKIPPED" || original.SchemaStatuses["waiting"] != "COMPLETED" || original.ObjectStatuses["done"] != "COMPLETED" || !reflect.DeepEqual(original.SkippedObjects, []string{"active", "waiting"}) {
|
||||
t.Fatalf("cancelled retry left stale states in original: %+v", original)
|
||||
}
|
||||
}
|
||||
|
||||
type schemaParallelReader struct {
|
||||
tableSkipAdapter
|
||||
gate chan struct{}
|
||||
entered chan struct{}
|
||||
active atomic.Int32
|
||||
maximum atomic.Int32
|
||||
reads atomic.Int32
|
||||
}
|
||||
|
||||
func (a *schemaParallelReader) Inspect(ctx context.Context, name string) (ObjectInfo, error) {
|
||||
n := a.active.Add(1)
|
||||
defer a.active.Add(-1)
|
||||
for {
|
||||
old := a.maximum.Load()
|
||||
if n <= old || a.maximum.CompareAndSwap(old, n) {
|
||||
break
|
||||
}
|
||||
}
|
||||
select {
|
||||
case a.entered <- struct{}{}:
|
||||
default:
|
||||
}
|
||||
select {
|
||||
case <-a.gate:
|
||||
case <-ctx.Done():
|
||||
return ObjectInfo{}, ctx.Err()
|
||||
}
|
||||
return a.tableSkipAdapter.Inspect(ctx, name)
|
||||
}
|
||||
func (a *schemaParallelReader) Read(_ context.Context, _ string, _ ReadOptions, consume func([]Record) error) error {
|
||||
a.reads.Add(1)
|
||||
return consume([]Record{{"id": 1}})
|
||||
}
|
||||
func TestSchemaParallelismAndDataBarrier(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
reader := &schemaParallelReader{gate: make(chan struct{}), entered: make(chan struct{}, 10)}
|
||||
var connections atomic.Int32
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
if connections.Add(1) == 1 {
|
||||
return reader, nil
|
||||
}
|
||||
return &tableSkipAdapter{}, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"a", "b", "c"}, SchemaWorkers: 2, MaxRows: 100, BatchSize: 10}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
}()
|
||||
for i := 0; i < 2; i++ {
|
||||
select {
|
||||
case <-reader.entered:
|
||||
case <-time.After(3 * time.Second):
|
||||
close(reader.gate)
|
||||
t.Fatal("two structure workers did not start")
|
||||
}
|
||||
}
|
||||
if reader.reads.Load() != 0 {
|
||||
close(reader.gate)
|
||||
t.Fatal("data started before schema workers finished")
|
||||
}
|
||||
close(reader.gate)
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("task did not finish")
|
||||
}
|
||||
if reader.maximum.Load() != 2 || reader.reads.Load() != 3 {
|
||||
t.Fatalf("max=%d reads=%d", reader.maximum.Load(), reader.reads.Load())
|
||||
}
|
||||
execution := s.store.Executions["exec"]
|
||||
for _, object := range task.Objects {
|
||||
if execution.SchemaStatuses[object] != "COMPLETED" || execution.ObjectStatuses[object] != "COMPLETED" {
|
||||
t.Fatalf("unexpected stage states: %+v", execution)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type liveSchemaReader struct {
|
||||
tableSkipAdapter
|
||||
slowGate chan struct{}
|
||||
}
|
||||
|
||||
func (a *liveSchemaReader) Inspect(ctx context.Context, name string) (ObjectInfo, error) {
|
||||
if name == "slow" {
|
||||
select {
|
||||
case <-a.slowGate:
|
||||
case <-ctx.Done():
|
||||
return ObjectInfo{}, ctx.Err()
|
||||
}
|
||||
}
|
||||
return a.tableSkipAdapter.Inspect(ctx, name)
|
||||
}
|
||||
func TestSchemaCompletionVisibleWhileOtherWorkerRuns(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", ObjectStatuses: map[string]string{"fast": "PENDING", "slow": "PENDING"}}
|
||||
reader := &liveSchemaReader{slowGate: make(chan struct{})}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 4*time.Second)
|
||||
defer cancel()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- s.runSchemaPhase(ctx, Task{SchemaWorkers: 2}, reader, &tableSkipAdapter{}, DataSource{}, DataSource{}, []string{"fast", "slow"}, "exec")
|
||||
}()
|
||||
deadline := time.After(3 * time.Second)
|
||||
timer := time.NewTicker(10 * time.Millisecond)
|
||||
defer timer.Stop()
|
||||
for {
|
||||
s.store.RLock()
|
||||
fast, slow := s.store.Executions["exec"].SchemaStatuses["fast"], s.store.Executions["exec"].SchemaStatuses["slow"]
|
||||
s.store.RUnlock()
|
||||
if fast == "COMPLETED" && slow == "RUNNING" {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-deadline:
|
||||
cancel()
|
||||
<-done
|
||||
t.Fatal("completed table stayed RUNNING until every worker finished")
|
||||
case <-timer.C:
|
||||
}
|
||||
}
|
||||
close(reader.slowGate)
|
||||
if err := <-done; err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type lateSchemaSuccessAdapter struct {
|
||||
tableSkipAdapter
|
||||
block bool
|
||||
blockedInspections atomic.Int32
|
||||
blockedReads atomic.Int32
|
||||
goodReads atomic.Int32
|
||||
}
|
||||
|
||||
func (a *lateSchemaSuccessAdapter) Inspect(ctx context.Context, object string) (ObjectInfo, error) {
|
||||
if object == "blocked" {
|
||||
a.blockedInspections.Add(1)
|
||||
if a.block {
|
||||
<-ctx.Done()
|
||||
}
|
||||
}
|
||||
// Simulate an adapter which returns successfully even after cancellation.
|
||||
return a.databaseSchemaAdapter.Inspect(context.Background(), object)
|
||||
}
|
||||
|
||||
func (a *lateSchemaSuccessAdapter) Read(ctx context.Context, object string, options ReadOptions, consume func([]Record) error) error {
|
||||
if object == "blocked" {
|
||||
a.blockedReads.Add(1)
|
||||
} else {
|
||||
a.goodReads.Add(1)
|
||||
}
|
||||
return a.tableSkipAdapter.Read(ctx, object, options, consume)
|
||||
}
|
||||
|
||||
func TestSchemaTimeoutRejectsLateSuccessAndContinuesOtherTables(t *testing.T) {
|
||||
for _, stage := range []string{"source", "target"} {
|
||||
t.Run(stage, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
reader := &lateSchemaSuccessAdapter{block: stage == "source"}
|
||||
writer := &lateSchemaSuccessAdapter{block: stage == "target"}
|
||||
var connections atomic.Int32
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
if connections.Add(1) == 1 {
|
||||
return reader, nil
|
||||
}
|
||||
return writer, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"blocked", "good"}, SchemaWorkers: 2, MaxRows: 100, BatchSize: 10, TimeoutSeconds: 1}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
execution := s.store.Executions["exec"]
|
||||
if execution.Status != "PARTIAL" || execution.SchemaStatuses["blocked"] != "FAILED" || execution.ObjectStatuses["blocked"] != "FAILED" || execution.ObjectErrors["blocked"] != "operationTimeout" {
|
||||
t.Fatalf("late schema success was accepted: %+v", execution)
|
||||
}
|
||||
if execution.SchemaStatuses["good"] != "COMPLETED" || execution.ObjectStatuses["good"] != "COMPLETED" || execution.Written != 1 || reader.goodReads.Load() != 1 || reader.blockedReads.Load() != 0 {
|
||||
t.Fatalf("schema timeout did not safely continue data sync: %+v", execution)
|
||||
}
|
||||
if stage == "source" && writer.blockedInspections.Load() != 0 {
|
||||
t.Fatal("expired source inspection reached the target")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
+21
-3
@@ -42,7 +42,9 @@ func (s *server) auth(next http.Handler) http.Handler {
|
||||
}
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("X-Frame-Options", "DENY")
|
||||
w.Header().Set("Content-Security-Policy", "default-src 'self'; connect-src 'self'; script-src 'self'; style-src 'self'; img-src 'self' data:; base-uri 'none'; frame-ancestors 'none'; form-action 'self'")
|
||||
// Ant Design injects component styles at runtime. Scripts remain restricted
|
||||
// to same-origin files; the inline allowance applies only to styles.
|
||||
w.Header().Set("Content-Security-Policy", "default-src 'self'; connect-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; base-uri 'none'; frame-ancestors 'none'; form-action 'self'")
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
w.Header().Set("Referrer-Policy", "no-referrer")
|
||||
requestPath := path.Clean(r.URL.Path)
|
||||
@@ -62,7 +64,7 @@ func (s *server) auth(next http.Handler) http.Handler {
|
||||
next.ServeHTTP(w, r)
|
||||
return
|
||||
}
|
||||
if tokenErr != nil || s.sessionToken == "" {
|
||||
if !localRequest(r) && (tokenErr != nil || s.sessionToken == "") {
|
||||
jsonOut(w, http.StatusServiceUnavailable, map[string]string{"code": "authenticationUnavailable"})
|
||||
return
|
||||
}
|
||||
@@ -87,7 +89,7 @@ func (s *server) session(w http.ResponseWriter, r *http.Request) {
|
||||
jsonOut(w, http.StatusForbidden, map[string]string{"code": "invalidOrigin"})
|
||||
return
|
||||
}
|
||||
if s.sessionToken == "" {
|
||||
if !localRequest(r) && s.sessionToken == "" {
|
||||
jsonOut(w, http.StatusServiceUnavailable, map[string]string{"code": "authenticationUnavailable"})
|
||||
return
|
||||
}
|
||||
@@ -160,6 +162,9 @@ func constantTimeToken(provided, expected string) bool {
|
||||
}
|
||||
|
||||
func (s *server) authenticated(r *http.Request) bool {
|
||||
if localRequest(r) {
|
||||
return true
|
||||
}
|
||||
if s.sessionToken == "" {
|
||||
return false
|
||||
}
|
||||
@@ -184,6 +189,19 @@ func (s *server) authenticated(r *http.Request) bool {
|
||||
return expires > uint64(now) && expires <= uint64(now+int64(sessionLifetime/time.Second))
|
||||
}
|
||||
|
||||
func localRequest(r *http.Request) bool {
|
||||
peer, _, err := net.SplitHostPort(r.RemoteAddr)
|
||||
if err != nil || !net.ParseIP(peer).IsLoopback() {
|
||||
return false
|
||||
}
|
||||
host := r.Host
|
||||
if hostname, _, err := net.SplitHostPort(host); err == nil {
|
||||
host = hostname
|
||||
}
|
||||
host = strings.Trim(host, "[]")
|
||||
return strings.EqualFold(host, "localhost") || net.ParseIP(host).IsLoopback()
|
||||
}
|
||||
|
||||
func (s *server) issueSession(expires time.Time) (string, error) {
|
||||
payload := make([]byte, 32)
|
||||
binary.BigEndian.PutUint64(payload[:8], uint64(expires.Unix()))
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestSourceMutationsProtectQueuedTasksAndActiveExecutions(t *testing.T) {
|
||||
for _, active := range []string{"task-pending", "task-running", "execution-running", "execution-pending", "retry-running", "retry-pending"} {
|
||||
for _, sourceID := range []string{"source", "target"} {
|
||||
for _, method := range []string{http.MethodPut, http.MethodDelete} {
|
||||
t.Run(active+"/"+sourceID+"/"+method, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
task := Task{ID: "task", SourceID: "source", TargetID: "target", Status: "PARTIAL"}
|
||||
if active == "task-pending" {
|
||||
task.Status = "PENDING"
|
||||
}
|
||||
if active == "task-running" {
|
||||
task.Status = "RUNNING"
|
||||
}
|
||||
s.store.Tasks[task.ID] = task
|
||||
if !strings.HasPrefix(active, "task-") {
|
||||
state := "RUNNING"
|
||||
if strings.HasSuffix(active, "pending") {
|
||||
state = "PENDING"
|
||||
}
|
||||
retryOf := ""
|
||||
if strings.HasPrefix(active, "retry-") {
|
||||
retryOf = "original"
|
||||
}
|
||||
s.store.Executions["active"] = Execution{ID: "active", TaskID: task.ID, Status: state, RetryOf: retryOf}
|
||||
}
|
||||
before := s.store.Sources[sourceID]
|
||||
rec := httptest.NewRecorder()
|
||||
s.datasource(rec, httptest.NewRequest(method, "/api/datasources/"+sourceID, strings.NewReader(`{"name":"edited","type":"mysql","endpoint":"changed:3306","username":"reader","password":"changed"}`)))
|
||||
if rec.Code != 409 || !strings.Contains(rec.Body.String(), "sourceBusy") || !reflect.DeepEqual(s.store.Sources[sourceID], before) {
|
||||
t.Fatalf("active source changed: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSourceDeletionKeepsCompletedRestoreTargetReference(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Exports["restore"] = Export{ID: "restore", SourceID: "source", TargetID: "target", Status: "COMPLETED", Mode: "restore"}
|
||||
rec := httptest.NewRecorder()
|
||||
s.datasource(rec, httptest.NewRequest(http.MethodDelete, "/api/datasources/target", nil))
|
||||
if rec.Code != 409 || !strings.Contains(rec.Body.String(), "sourceInUse") {
|
||||
t.Fatalf("restore target reference was lost: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if _, exists := s.store.Sources["target"]; !exists {
|
||||
t.Fatal("referenced target deleted")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"database/sql/driver"
|
||||
"errors"
|
||||
"sync"
|
||||
|
||||
"modernc.org/sqlite"
|
||||
)
|
||||
|
||||
// database/sql may return from DB.Close while a cancelled transaction or Rows
|
||||
// is still releasing its driver connection. Track the driver lifetime so local
|
||||
// database files are no longer open when the store's Close returns.
|
||||
type localSQLiteDB struct {
|
||||
*sql.DB
|
||||
connector *sqliteLifetimeConnector
|
||||
once sync.Once
|
||||
closeErr error
|
||||
}
|
||||
|
||||
type sqliteDriverConnection interface {
|
||||
driver.Conn
|
||||
driver.ConnBeginTx
|
||||
driver.ConnPrepareContext
|
||||
driver.ExecerContext
|
||||
driver.QueryerContext
|
||||
driver.Pinger
|
||||
driver.SessionResetter
|
||||
driver.Validator
|
||||
}
|
||||
|
||||
type sqliteLifetimeConnector struct {
|
||||
driver driver.Driver
|
||||
dsn string
|
||||
mu sync.Mutex
|
||||
closed bool
|
||||
err error
|
||||
live sync.WaitGroup
|
||||
}
|
||||
|
||||
type sqliteLifetimeConnection struct {
|
||||
sqliteDriverConnection
|
||||
owner *sqliteLifetimeConnector
|
||||
once sync.Once
|
||||
closeErr error
|
||||
}
|
||||
|
||||
func openLocalSQLite(dsn string) *localSQLiteDB {
|
||||
return newLocalSQLite(&sqlite.Driver{}, dsn)
|
||||
}
|
||||
|
||||
func newLocalSQLite(sqlDriver driver.Driver, dsn string) *localSQLiteDB {
|
||||
connector := &sqliteLifetimeConnector{driver: sqlDriver, dsn: dsn}
|
||||
return &localSQLiteDB{DB: sql.OpenDB(connector), connector: connector}
|
||||
}
|
||||
|
||||
func (connector *sqliteLifetimeConnector) Driver() driver.Driver { return connector.driver }
|
||||
|
||||
func (connector *sqliteLifetimeConnector) Connect(ctx context.Context) (driver.Conn, error) {
|
||||
connector.mu.Lock()
|
||||
if connector.closed {
|
||||
connector.mu.Unlock()
|
||||
return nil, sql.ErrConnDone
|
||||
}
|
||||
connector.live.Add(1)
|
||||
connector.mu.Unlock()
|
||||
if err := ctx.Err(); err != nil {
|
||||
connector.live.Done()
|
||||
return nil, err
|
||||
}
|
||||
conn, err := connector.driver.Open(connector.dsn)
|
||||
if err != nil {
|
||||
connector.live.Done()
|
||||
return nil, err
|
||||
}
|
||||
native, ok := conn.(sqliteDriverConnection)
|
||||
if !ok {
|
||||
err := conn.Close()
|
||||
connector.live.Done()
|
||||
return nil, errors.Join(errors.New("unsupported SQLite driver connection"), err)
|
||||
}
|
||||
return &sqliteLifetimeConnection{sqliteDriverConnection: native, owner: connector}, nil
|
||||
}
|
||||
|
||||
func (conn *sqliteLifetimeConnection) Close() error {
|
||||
conn.once.Do(func() {
|
||||
conn.closeErr = conn.sqliteDriverConnection.Close()
|
||||
conn.owner.mu.Lock()
|
||||
conn.owner.err = errors.Join(conn.owner.err, conn.closeErr)
|
||||
conn.owner.mu.Unlock()
|
||||
conn.owner.live.Done()
|
||||
})
|
||||
return conn.closeErr
|
||||
}
|
||||
|
||||
func (database *localSQLiteDB) Close() error {
|
||||
database.once.Do(func() {
|
||||
// Seal registration before waiting, including connections whose Open
|
||||
// is still running when database/sql starts shutting down.
|
||||
database.connector.mu.Lock()
|
||||
database.connector.closed = true
|
||||
database.connector.mu.Unlock()
|
||||
database.closeErr = database.DB.Close()
|
||||
database.connector.live.Wait()
|
||||
database.connector.mu.Lock()
|
||||
if database.closeErr == nil {
|
||||
database.closeErr = database.connector.err
|
||||
}
|
||||
database.connector.mu.Unlock()
|
||||
})
|
||||
return database.closeErr
|
||||
}
|
||||
@@ -0,0 +1,254 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql/driver"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"modernc.org/sqlite"
|
||||
)
|
||||
|
||||
type gatedSQLiteDriver struct {
|
||||
driver.Driver
|
||||
openEntered, releaseOpen chan struct{}
|
||||
rollbackEntered, releaseRollback chan struct{}
|
||||
rowsEntered, releaseRows chan struct{}
|
||||
closed chan struct{}
|
||||
closeErr error
|
||||
}
|
||||
|
||||
func (gate *gatedSQLiteDriver) Open(dsn string) (driver.Conn, error) {
|
||||
if gate.openEntered != nil {
|
||||
close(gate.openEntered)
|
||||
<-gate.releaseOpen
|
||||
}
|
||||
conn, err := gate.Driver.Open(dsn)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &gatedSQLiteConnection{sqliteDriverConnection: conn.(sqliteDriverConnection), gate: gate}, nil
|
||||
}
|
||||
|
||||
type gatedSQLiteConnection struct {
|
||||
sqliteDriverConnection
|
||||
gate *gatedSQLiteDriver
|
||||
}
|
||||
|
||||
func (conn *gatedSQLiteConnection) QueryContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Rows, error) {
|
||||
rows, err := conn.sqliteDriverConnection.QueryContext(ctx, query, args)
|
||||
if err != nil || conn.gate.rowsEntered == nil {
|
||||
return rows, err
|
||||
}
|
||||
return &gatedSQLiteRows{Rows: rows, gate: conn.gate}, nil
|
||||
}
|
||||
|
||||
type gatedSQLiteRows struct {
|
||||
driver.Rows
|
||||
gate *gatedSQLiteDriver
|
||||
}
|
||||
|
||||
func (rows *gatedSQLiteRows) Close() error {
|
||||
close(rows.gate.rowsEntered)
|
||||
<-rows.gate.releaseRows
|
||||
return rows.Rows.Close()
|
||||
}
|
||||
|
||||
func (conn *gatedSQLiteConnection) BeginTx(ctx context.Context, options driver.TxOptions) (driver.Tx, error) {
|
||||
tx, err := conn.sqliteDriverConnection.BeginTx(ctx, options)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &gatedSQLiteTransaction{Tx: tx, gate: conn.gate}, nil
|
||||
}
|
||||
|
||||
func (conn *gatedSQLiteConnection) Close() error {
|
||||
err := conn.sqliteDriverConnection.Close()
|
||||
close(conn.gate.closed)
|
||||
return errors.Join(err, conn.gate.closeErr)
|
||||
}
|
||||
|
||||
type gatedSQLiteTransaction struct {
|
||||
driver.Tx
|
||||
gate *gatedSQLiteDriver
|
||||
}
|
||||
|
||||
func (tx *gatedSQLiteTransaction) Rollback() error {
|
||||
close(tx.gate.rollbackEntered)
|
||||
<-tx.gate.releaseRollback
|
||||
return tx.Tx.Rollback()
|
||||
}
|
||||
|
||||
func awaitSQLiteSignal(t *testing.T, signal <-chan struct{}) {
|
||||
t.Helper()
|
||||
select {
|
||||
case <-signal:
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("SQLite lifecycle operation did not reach the expected boundary")
|
||||
}
|
||||
}
|
||||
|
||||
func assertSQLiteClosingWaits(t *testing.T, database *localSQLiteDB, finished <-chan error) {
|
||||
t.Helper()
|
||||
deadline := time.NewTimer(3 * time.Second)
|
||||
defer deadline.Stop()
|
||||
tick := time.NewTicker(time.Millisecond)
|
||||
defer tick.Stop()
|
||||
for {
|
||||
database.connector.mu.Lock()
|
||||
closing := database.connector.closed
|
||||
database.connector.mu.Unlock()
|
||||
if closing {
|
||||
break
|
||||
}
|
||||
select {
|
||||
case <-tick.C:
|
||||
case <-deadline.C:
|
||||
t.Fatal("SQLite close did not start")
|
||||
}
|
||||
}
|
||||
select {
|
||||
case err := <-finished:
|
||||
t.Fatalf("SQLite close returned before the driver released its connection: %v", err)
|
||||
case <-time.After(20 * time.Millisecond):
|
||||
}
|
||||
}
|
||||
|
||||
func TestLocalSQLiteCloseWaitsForCancelledRollback(t *testing.T) {
|
||||
for _, closeFailure := range []bool{false, true} {
|
||||
t.Run(map[bool]string{false: "normal close", true: "late close error"}[closeFailure], func(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "index.sqlite")
|
||||
gate := &gatedSQLiteDriver{Driver: &sqlite.Driver{}, rollbackEntered: make(chan struct{}), releaseRollback: make(chan struct{}), closed: make(chan struct{})}
|
||||
if closeFailure {
|
||||
gate.closeErr = errors.New("driver close error")
|
||||
}
|
||||
database := newLocalSQLite(gate, path)
|
||||
var release sync.Once
|
||||
releaseRollback := func() { release.Do(func() { close(gate.releaseRollback) }) }
|
||||
t.Cleanup(func() { releaseRollback(); database.Close() })
|
||||
if _, err := database.Exec("CREATE TABLE items(id INTEGER)"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
tx, err := database.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := tx.Exec("INSERT INTO items VALUES(1)"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cancel()
|
||||
awaitSQLiteSignal(t, gate.rollbackEntered)
|
||||
// Reproduce the standard pool's early return while the background
|
||||
// transaction still owns the underlying connection and file.
|
||||
if err := database.DB.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
finished := make(chan error, 1)
|
||||
go func() { finished <- database.Close() }()
|
||||
assertSQLiteClosingWaits(t, database, finished)
|
||||
releaseRollback()
|
||||
select {
|
||||
case err := <-finished:
|
||||
if !errors.Is(err, gate.closeErr) {
|
||||
t.Fatalf("lost driver close result: %v", err)
|
||||
}
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("close did not finish after rollback")
|
||||
}
|
||||
awaitSQLiteSignal(t, gate.closed)
|
||||
if !errors.Is(database.Close(), gate.closeErr) {
|
||||
t.Fatal("repeated close lost its result")
|
||||
}
|
||||
reopened := openLocalSQLite(path)
|
||||
var count int
|
||||
err = reopened.QueryRow("SELECT COUNT(*) FROM items").Scan(&count)
|
||||
closeErr := reopened.Close()
|
||||
if err != nil || closeErr != nil || count != 0 {
|
||||
t.Fatalf("cancelled transaction was not rolled back: count=%d query=%v close=%v", count, err, closeErr)
|
||||
}
|
||||
if err := os.Remove(path); err != nil {
|
||||
t.Fatalf("closed SQLite file is still held: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLocalSQLiteCloseWaitsForOpeningConnection(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "index.sqlite")
|
||||
gate := &gatedSQLiteDriver{Driver: &sqlite.Driver{}, openEntered: make(chan struct{}), releaseOpen: make(chan struct{}), closed: make(chan struct{})}
|
||||
database := newLocalSQLite(gate, path)
|
||||
var release sync.Once
|
||||
releaseOpen := func() { release.Do(func() { close(gate.releaseOpen) }) }
|
||||
t.Cleanup(func() { releaseOpen(); database.Close() })
|
||||
pinged := make(chan struct{})
|
||||
go func() { _ = database.Ping(); close(pinged) }()
|
||||
awaitSQLiteSignal(t, gate.openEntered)
|
||||
finished := make(chan error, 1)
|
||||
go func() { finished <- database.Close() }()
|
||||
assertSQLiteClosingWaits(t, database, finished)
|
||||
releaseOpen()
|
||||
awaitSQLiteSignal(t, pinged)
|
||||
select {
|
||||
case err := <-finished:
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("close did not finish after connection opened")
|
||||
}
|
||||
awaitSQLiteSignal(t, gate.closed)
|
||||
if _, err := database.connector.Connect(context.Background()); err == nil {
|
||||
t.Fatal("closed connector allowed a new connection")
|
||||
}
|
||||
if err := os.Remove(path); err != nil {
|
||||
t.Fatalf("connection opened during close still holds the file: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLocalSQLiteCloseWaitsForCancelledRows(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "index.sqlite")
|
||||
gate := &gatedSQLiteDriver{Driver: &sqlite.Driver{}, rowsEntered: make(chan struct{}), releaseRows: make(chan struct{}), closed: make(chan struct{})}
|
||||
database := newLocalSQLite(gate, path)
|
||||
var release sync.Once
|
||||
releaseRows := func() { release.Do(func() { close(gate.releaseRows) }) }
|
||||
t.Cleanup(func() { releaseRows(); database.Close() })
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
rows, err := database.QueryContext(ctx, "SELECT 1")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cancel()
|
||||
awaitSQLiteSignal(t, gate.rowsEntered)
|
||||
// DB.Close can return while the cancellation goroutine is still inside
|
||||
// driver Rows.Close and holds the underlying connection.
|
||||
if err := database.DB.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
finished := make(chan error, 2)
|
||||
go func() { finished <- database.Close() }()
|
||||
assertSQLiteClosingWaits(t, database, finished)
|
||||
go func() { finished <- database.Close() }()
|
||||
releaseRows()
|
||||
for range 2 {
|
||||
select {
|
||||
case err := <-finished:
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("concurrent close did not finish after rows were released")
|
||||
}
|
||||
}
|
||||
awaitSQLiteSignal(t, gate.closed)
|
||||
_ = rows.Close()
|
||||
if err := os.Remove(path); err != nil {
|
||||
t.Fatalf("cancelled read still holds the file after close: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,92 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type tableCleanupAdapter struct {
|
||||
tableSkipAdapter
|
||||
test *testing.T
|
||||
panicStage string
|
||||
inspections map[string]int
|
||||
contexts []context.Context
|
||||
}
|
||||
|
||||
func (a *tableCleanupAdapter) Inspect(ctx context.Context, object string) (ObjectInfo, error) {
|
||||
a.inspections[object]++
|
||||
if a.inspections[object] == 2 {
|
||||
for _, previous := range a.contexts {
|
||||
if previous.Err() == nil {
|
||||
a.test.Error("previous table context remained live when the next table began")
|
||||
}
|
||||
}
|
||||
a.contexts = append(a.contexts, ctx)
|
||||
if object == "bad" && a.panicStage == "inspect" {
|
||||
panic("inspection failure")
|
||||
}
|
||||
}
|
||||
return a.databaseSchemaAdapter.Inspect(ctx, object)
|
||||
}
|
||||
|
||||
func (a *tableCleanupAdapter) Read(ctx context.Context, object string, options ReadOptions, consume func([]Record) error) error {
|
||||
if object == "bad" && a.panicStage == "read" {
|
||||
panic("read failure")
|
||||
}
|
||||
return a.tableSkipAdapter.Read(ctx, object, options, consume)
|
||||
}
|
||||
|
||||
type tableCleanupWriter struct {
|
||||
tableSkipAdapter
|
||||
panicStage string
|
||||
}
|
||||
|
||||
func (a *tableCleanupWriter) Write(ctx context.Context, object string, records []Record, options WriteOptions) (WriteResult, error) {
|
||||
if object == "bad" && a.panicStage == "write" {
|
||||
panic("write failure")
|
||||
}
|
||||
return a.tableSkipAdapter.Write(ctx, object, records, options)
|
||||
}
|
||||
|
||||
func TestTableContextsAreReleasedOnSuccessAndAdapterPanic(t *testing.T) {
|
||||
for _, stage := range []string{"success", "inspect", "read", "write"} {
|
||||
t.Run(stage, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
reader := &tableCleanupAdapter{test: t, panicStage: stage, inspections: map[string]int{}}
|
||||
var connections atomic.Int32
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
if connections.Add(1) == 1 {
|
||||
return reader, nil
|
||||
}
|
||||
return &tableCleanupWriter{panicStage: stage}, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"good", "bad", "later"}, SchemaWorkers: 1, MaxRows: 100, BatchSize: 10}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
for _, ctx := range reader.contexts {
|
||||
if ctx.Err() != context.Canceled {
|
||||
t.Fatal("finished or panicked table context was not cancelled")
|
||||
}
|
||||
}
|
||||
s.jobsMu.Lock()
|
||||
active := len(s.tableRuns)
|
||||
s.jobsMu.Unlock()
|
||||
if active != 0 || s.skipTable("exec", "bad") {
|
||||
t.Fatal("finished table retained an active skip handle")
|
||||
}
|
||||
execution := s.store.Executions["exec"]
|
||||
if execution.ObjectStatuses["good"] != "COMPLETED" {
|
||||
t.Fatal("an already completed table lost its result")
|
||||
}
|
||||
if stage == "success" {
|
||||
if execution.Status != "COMPLETED" || execution.Written != 3 || len(reader.contexts) != 3 {
|
||||
t.Fatalf("success path changed: %+v", execution)
|
||||
}
|
||||
} else if execution.Status != "FAILED" || execution.Error != "adapterOperationFailed" || execution.Written != 1 || execution.ObjectStatuses["bad"] != "FAILED" || execution.ObjectStatuses["later"] != "PENDING" || len(reader.contexts) != 2 {
|
||||
t.Fatalf("panic path changed: %+v", execution)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type blockedDataInspection struct {
|
||||
tableSkipAdapter
|
||||
inspections atomic.Int32
|
||||
blockedReads atomic.Int32
|
||||
entered chan struct{}
|
||||
}
|
||||
|
||||
func (a *blockedDataInspection) Inspect(ctx context.Context, object string) (ObjectInfo, error) {
|
||||
if object == "blocked" && a.inspections.Add(1) == 2 {
|
||||
close(a.entered)
|
||||
<-ctx.Done()
|
||||
return ObjectInfo{}, ctx.Err()
|
||||
}
|
||||
return a.databaseSchemaAdapter.Inspect(ctx, object)
|
||||
}
|
||||
|
||||
func (a *blockedDataInspection) Read(ctx context.Context, object string, options ReadOptions, consume func([]Record) error) error {
|
||||
if object == "blocked" {
|
||||
a.blockedReads.Add(1)
|
||||
}
|
||||
return a.tableSkipAdapter.Read(ctx, object, options, consume)
|
||||
}
|
||||
|
||||
func TestDataInspectionHonorsTableSkipTimeoutAndTaskCancellation(t *testing.T) {
|
||||
for _, action := range []string{"skip", "timeout", "cancel"} {
|
||||
t.Run(action, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
reader := &blockedDataInspection{entered: make(chan struct{})}
|
||||
var connections atomic.Int32
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
if connections.Add(1) == 1 {
|
||||
return reader, nil
|
||||
}
|
||||
return &tableSkipAdapter{}, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"blocked", "good"}, SchemaWorkers: 1, MaxRows: 100, BatchSize: 10, TimeoutSeconds: 1}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
s.runTask(ctx, task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
}()
|
||||
select {
|
||||
case <-reader.entered:
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("data inspection did not start")
|
||||
}
|
||||
if action == "skip" {
|
||||
if s.skipTable("exec", "good") || !s.skipTable("exec", "blocked") || s.skipTable("exec", "blocked") {
|
||||
t.Fatal("active inspection was not skippable or duplicate skip accepted")
|
||||
}
|
||||
} else if action == "cancel" {
|
||||
cancel()
|
||||
}
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(3 * time.Second):
|
||||
t.Fatal("data inspection did not terminate")
|
||||
}
|
||||
execution := s.store.Executions["exec"]
|
||||
if reader.blockedReads.Load() != 0 {
|
||||
t.Fatal("failed inspection reached a data reader")
|
||||
}
|
||||
if execution.SchemaStatuses["blocked"] != "COMPLETED" || execution.SchemaStatuses["good"] != "COMPLETED" {
|
||||
t.Fatalf("data cancellation changed schema result: %+v", execution)
|
||||
}
|
||||
s.jobsMu.Lock()
|
||||
active := len(s.tableRuns)
|
||||
s.jobsMu.Unlock()
|
||||
if active != 0 || s.skipTable("exec", "blocked") {
|
||||
t.Fatal("finished inspection remained active")
|
||||
}
|
||||
if action == "cancel" {
|
||||
if execution.Status != "CANCELLED" || execution.ObjectStatuses["blocked"] != "FAILED" || execution.ObjectErrors["blocked"] != "operationCancelled" || execution.ObjectStatuses["good"] != "PENDING" || execution.Written != 0 {
|
||||
t.Fatalf("cancelled inspection did not preserve waiting table: %+v", execution)
|
||||
}
|
||||
} else {
|
||||
if execution.Status != "PARTIAL" || execution.ObjectStatuses["good"] != "COMPLETED" || execution.Written != 1 || execution.ObjectFinishedAt["blocked"].IsZero() {
|
||||
t.Fatalf("inspection did not continue to next table: %+v", execution)
|
||||
}
|
||||
if action == "skip" && execution.ObjectStatuses["blocked"] != "SKIPPED" {
|
||||
t.Fatalf("inspection skip result missing: %+v", execution)
|
||||
}
|
||||
if action == "timeout" && (execution.ObjectStatuses["blocked"] != "FAILED" || execution.ObjectErrors["blocked"] != "operationTimeout") {
|
||||
t.Fatalf("inspection timeout result missing: %+v", execution)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type tableSkipAdapter struct {
|
||||
databaseSchemaAdapter
|
||||
started chan struct{}
|
||||
}
|
||||
|
||||
func (a *tableSkipAdapter) Ping(context.Context) error { return nil }
|
||||
|
||||
func (a *tableSkipAdapter) Read(ctx context.Context, object string, _ ReadOptions, consume func([]Record) error) error {
|
||||
if object == "large" {
|
||||
close(a.started)
|
||||
<-ctx.Done()
|
||||
return ctx.Err()
|
||||
}
|
||||
return consume([]Record{{"id": 1}})
|
||||
}
|
||||
|
||||
func (a *tableSkipAdapter) Write(_ context.Context, _ string, records []Record, _ WriteOptions) (WriteResult, error) {
|
||||
return WriteResult{Rows: int64(len(records)), Written: int64(len(records))}, nil
|
||||
}
|
||||
|
||||
func TestSkipActiveTableContinuesTask(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
started := make(chan struct{})
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
return &tableSkipAdapter{started: started}, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"large", "small"}, MaxRows: 100, BatchSize: 10}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
}()
|
||||
select {
|
||||
case <-started:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("table never started")
|
||||
}
|
||||
if s.skipTable("exec", "small") || !s.skipTable("exec", "large") || s.skipTable("exec", "large") {
|
||||
t.Fatal("skip did not validate the active table or repeated request")
|
||||
}
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("skip did not interrupt the table")
|
||||
}
|
||||
execution := s.store.Executions["exec"]
|
||||
if execution.Status != "PARTIAL" || execution.Written != 1 || len(execution.SkippedObjects) != 1 || execution.SkippedObjects[0] != "large" {
|
||||
t.Fatalf("unexpected execution: %+v", execution)
|
||||
}
|
||||
if s.skipTable("exec", "small") {
|
||||
t.Fatal("finished table can be skipped")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
package main
|
||||
|
||||
// The caller holds the store lock. Execution state remains authoritative when
|
||||
// a task's summary is stale, including internal retries and queued executions.
|
||||
func (s *server) taskBusyLocked(taskID string) bool {
|
||||
if unfinishedStatus(s.store.Tasks[taskID].Status) {
|
||||
return true
|
||||
}
|
||||
for _, execution := range s.store.Executions {
|
||||
if execution.TaskID == taskID && unfinishedStatus(execution.Status) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestTaskMutationsRejectActiveTasksAndExecutions(t *testing.T) {
|
||||
for _, state := range []string{"task-running", "task-pending", "execution-running", "execution-pending", "retry-running", "retry-pending"} {
|
||||
for _, action := range []string{"run", "edit", "delete"} {
|
||||
t.Run(state+"/"+action, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
task := Task{ID: "task", Name: "task", Status: "PARTIAL", SourceID: "source", TargetID: "target", SourceDatabase: "app", TargetDatabase: "app", Objects: []string{"table"}, DryRun: true}
|
||||
if state == "task-running" {
|
||||
task.Status = "RUNNING"
|
||||
}
|
||||
if state == "task-pending" {
|
||||
task.Status = "PENDING"
|
||||
}
|
||||
s.store.Tasks[task.ID] = task
|
||||
if !strings.HasPrefix(state, "task-") {
|
||||
execution := Execution{ID: "active", TaskID: task.ID, Status: "RUNNING"}
|
||||
if strings.HasSuffix(state, "pending") {
|
||||
execution.Status = "PENDING"
|
||||
}
|
||||
if strings.HasPrefix(state, "retry-") {
|
||||
execution.RetryOf = "original"
|
||||
}
|
||||
s.store.Executions[execution.ID] = execution
|
||||
}
|
||||
before := len(s.store.Executions)
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
t.Fatal("busy task opened database connection")
|
||||
return nil, nil
|
||||
}
|
||||
method, path, body := http.MethodPost, "/api/tasks/task/run", `{"confirm":true}`
|
||||
if action == "edit" {
|
||||
method, path, body = http.MethodPut, "/api/tasks/task", `{"name":"edited"}`
|
||||
}
|
||||
if action == "delete" {
|
||||
method, path, body = http.MethodDelete, "/api/tasks/task", ""
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
s.task(rec, httptest.NewRequest(method, path, strings.NewReader(body)))
|
||||
if rec.Code != 409 || !strings.Contains(rec.Body.String(), "taskBusy") || !reflect.DeepEqual(s.store.Tasks[task.ID], task) || len(s.store.Executions) != before {
|
||||
t.Fatalf("busy task mutated: %d %s %+v", rec.Code, rec.Body.String(), s.store.Tasks[task.ID])
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTaskBusyIgnoresSettledAndUnrelatedExecutions(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Tasks["task"] = Task{ID: "task", Status: "PARTIAL"}
|
||||
s.store.Executions["old"] = Execution{ID: "old", TaskID: "task", Status: "COMPLETED"}
|
||||
s.store.Executions["other"] = Execution{ID: "other", TaskID: "other", Status: "RUNNING"}
|
||||
s.store.RLock()
|
||||
busy := s.taskBusyLocked("task")
|
||||
s.store.RUnlock()
|
||||
if busy {
|
||||
t.Fatal("settled or unrelated execution locked the task")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTaskAndRetryProtectTargetOfQueuedOrStaleTask(t *testing.T) {
|
||||
for _, state := range []string{"task-pending", "stale-running", "stale-pending"} {
|
||||
for _, retry := range []bool{false, true} {
|
||||
t.Run(state+map[bool]string{false: "/run", true: "/retry"}[retry], func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
task := Task{ID: "task", Name: "task", SourceID: "source", TargetID: "target", SourceDatabase: "app", TargetDatabase: "app", Objects: []string{"table"}, DryRun: true}
|
||||
s.store.Tasks[task.ID] = task
|
||||
other := task
|
||||
other.ID = "other"
|
||||
other.Status = "PARTIAL"
|
||||
if state == "task-pending" {
|
||||
other.Status = "PENDING"
|
||||
} else {
|
||||
status := "RUNNING"
|
||||
if state == "stale-pending" {
|
||||
status = "PENDING"
|
||||
}
|
||||
s.store.Executions["active"] = Execution{ID: "active", TaskID: other.ID, Status: status}
|
||||
}
|
||||
s.store.Tasks[other.ID] = other
|
||||
s.store.Executions["original"] = Execution{ID: "original", TaskID: task.ID, Status: "PARTIAL", ObjectStatuses: map[string]string{"table": "FAILED"}, SkippedObjects: []string{"table"}}
|
||||
before := len(s.store.Executions)
|
||||
var err error
|
||||
if retry {
|
||||
_, err = s.retrySkippedObjects("original", []string{"table"})
|
||||
} else {
|
||||
_, err = s.startTask(task.ID, true)
|
||||
}
|
||||
if err == nil || err.Error() != "targetBusy" || len(s.store.Executions) != before || !reflect.DeepEqual(s.store.Tasks[task.ID], task) {
|
||||
t.Fatalf("busy target accepted: %v", err)
|
||||
}
|
||||
s.store.RLock()
|
||||
target, _ := resolveSourceDatabase(s.store.Sources["target"], "app")
|
||||
busy := s.sourceDatabaseBusyLocked(target)
|
||||
s.store.RUnlock()
|
||||
if !busy {
|
||||
t.Fatal("busy database exposed to structural operations")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestTaskCancelTargetsReviewedExecution(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name, body string
|
||||
status int
|
||||
cancel bool
|
||||
}{
|
||||
{"current", `{"execution_id":"current"}`, 202, true},
|
||||
{"finished execution", `{"execution_id":"old"}`, 409, false},
|
||||
{"other task", `{"execution_id":"foreign"}`, 409, false},
|
||||
{"missing execution", `{"execution_id":"missing"}`, 409, false},
|
||||
{"malformed", `{"execution_id":`, 400, false},
|
||||
{"legacy empty object", `{}`, 202, true},
|
||||
{"legacy empty body", ``, 202, true},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
s.store.Tasks["task"] = Task{ID: "task", Status: "RUNNING"}
|
||||
s.store.Executions["old"] = Execution{ID: "old", TaskID: "task", Status: "COMPLETED"}
|
||||
s.store.Executions["current"] = Execution{ID: "current", TaskID: "task", Status: "RUNNING"}
|
||||
s.store.Executions["foreign"] = Execution{ID: "foreign", TaskID: "other", Status: "RUNNING"}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
foreign, cancelForeign := context.WithCancel(context.Background())
|
||||
defer cancelForeign()
|
||||
s.jobs = map[string]context.CancelFunc{"current": cancel, "foreign": cancelForeign}
|
||||
rec := httptest.NewRecorder()
|
||||
s.task(rec, httptest.NewRequest(http.MethodPost, "/api/tasks/task/cancel", strings.NewReader(test.body)))
|
||||
if rec.Code != test.status || (ctx.Err() != nil) != test.cancel || foreign.Err() != nil {
|
||||
t.Fatalf("cancel scope wrong: status=%d body=%s current=%v foreign=%v", rec.Code, rec.Body.String(), ctx.Err(), foreign.Err())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func revisionTaskFixture() Task {
|
||||
return Task{ID: "task", Name: "task", SourceID: "source", TargetID: "target", SourceDatabase: "app", TargetDatabase: "app", Objects: []string{"table"}, Mode: "incremental", Conflict: "upsert", IncrementalField: "id", IncrementalValue: json.Number("10"), Status: "COMPLETED"}
|
||||
}
|
||||
|
||||
func TestTaskRevisionRejectsStaleWatermarkConfigurationAndRunState(t *testing.T) {
|
||||
for _, change := range []string{"watermark", "configuration", "run-state"} {
|
||||
t.Run(change, func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
original := revisionTaskFixture()
|
||||
input := taskForAPI(original)
|
||||
input.Name = "edited in old window"
|
||||
current := original
|
||||
switch change {
|
||||
case "watermark":
|
||||
current.IncrementalValue = json.Number("11")
|
||||
case "configuration":
|
||||
current.Mapping = map[string]string{"id": "new_id"}
|
||||
case "run-state":
|
||||
now := time.Now().UTC()
|
||||
current.LastRunAt = &now
|
||||
current.Status = "PARTIAL"
|
||||
}
|
||||
s.store.Tasks[current.ID] = current
|
||||
data, err := json.Marshal(input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
s.task(rec, httptest.NewRequest(http.MethodPut, "/api/tasks/task", bytes.NewReader(data)))
|
||||
if rec.Code != 409 || !strings.Contains(rec.Body.String(), "taskReviewChanged") || !reflect.DeepEqual(s.store.Tasks[current.ID], current) {
|
||||
t.Fatalf("stale editor overwrote current task: %d %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestTaskRevisionAcceptsCurrentEditorAndRejectsSecondStaleEditor(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
original := revisionTaskFixture()
|
||||
s.store.Tasks[original.ID] = original
|
||||
input := taskForAPI(original)
|
||||
input.Name = "first edit"
|
||||
for _, want := range []int{200, 409} {
|
||||
data, err := json.Marshal(input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
s.task(rec, httptest.NewRequest(http.MethodPut, "/api/tasks/task", bytes.NewReader(data)))
|
||||
if rec.Code != want {
|
||||
t.Fatalf("revision update: got %d, want %d: %s", rec.Code, want, rec.Body.String())
|
||||
}
|
||||
}
|
||||
if s.store.Tasks[original.ID].Revision != "" {
|
||||
t.Fatal("transport revision saved as mutable task state")
|
||||
}
|
||||
stored := s.store.Tasks[original.ID]
|
||||
before := taskRevision(stored)
|
||||
restored, _, err := loadStore(s.dataDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if taskRevision(restored.Tasks[original.ID]) != before {
|
||||
t.Fatal("restart changed unchanged task revision")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,311 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
func (s *server) setSchemaStatus(executionID, object, status string) {
|
||||
s.store.Lock()
|
||||
execution := s.store.Executions[executionID]
|
||||
if execution.SchemaStatuses == nil {
|
||||
execution.SchemaStatuses = make(map[string]string)
|
||||
}
|
||||
execution.SchemaStatuses[object] = status
|
||||
s.store.Executions[executionID] = execution
|
||||
s.store.Unlock()
|
||||
}
|
||||
|
||||
func validateSyncTargetColumns(task Task, sourceInfo, targetInfo ObjectInfo) error {
|
||||
if targetInfo.Type != "table" {
|
||||
return errors.New("schemaIncompatible")
|
||||
}
|
||||
// MySQL reads every source column. Follow field names through mapping,
|
||||
// defaults and omission without guessing whether row values convert safely.
|
||||
original := make(map[string]bool, len(sourceInfo.Columns))
|
||||
columns := make(map[string]ColumnInfo, len(sourceInfo.Columns))
|
||||
fields := make(map[string]bool, len(sourceInfo.Columns))
|
||||
for _, column := range sourceInfo.Columns {
|
||||
original[column.Name], fields[column.Name] = true, true
|
||||
columns[column.Name] = column
|
||||
}
|
||||
for name := range task.Mapping {
|
||||
delete(fields, name)
|
||||
}
|
||||
for name, destination := range task.Mapping {
|
||||
root := strings.SplitN(name, ".", 2)[0]
|
||||
if original[name] || strings.EqualFold(columns[root].Type, "json") {
|
||||
fields[strings.SplitN(destination, ".", 2)[0]] = true
|
||||
}
|
||||
}
|
||||
for name := range task.Defaults {
|
||||
// A non-null literal dotted column is found directly by recordPath;
|
||||
// its default never creates a separate nested object.
|
||||
if strings.Contains(name, ".") && original[name] && fields[name] && !columns[name].Nullable {
|
||||
continue
|
||||
}
|
||||
fields[strings.SplitN(name, ".", 2)[0]] = true
|
||||
}
|
||||
// recordPath reads a literal dotted column first, whereas setRecordPath
|
||||
// writes its converted value into a nested object. Keep both top-level
|
||||
// fields, following the same conversion/omit/masking order as row writes.
|
||||
addDottedRoots := func(paths map[string]string) {
|
||||
for name := range paths {
|
||||
if strings.Contains(name, ".") && fields[name] {
|
||||
fields[strings.SplitN(name, ".", 2)[0]] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
addDottedRoots(task.Conversions)
|
||||
for _, name := range task.Omit {
|
||||
delete(fields, name)
|
||||
}
|
||||
addDottedRoots(task.Masking)
|
||||
known := make(map[string]bool, len(targetInfo.Columns))
|
||||
for _, column := range targetInfo.Columns {
|
||||
known[column.Name] = true
|
||||
}
|
||||
for name := range fields {
|
||||
if !known[name] {
|
||||
return errors.New("syncTargetColumnsMissing")
|
||||
}
|
||||
}
|
||||
// Update only changes supplied fields. Insert-capable strategies (and
|
||||
// replace resetting omitted fields to DEFAULT) need every mandatory column.
|
||||
if task.Conflict != "update" {
|
||||
for _, column := range targetInfo.Columns {
|
||||
if column.Required && !fields[column.Name] {
|
||||
return errors.New("syncTargetRequiredColumnsMissing")
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func prepareSyncSchema(ctx context.Context, writer Adapter, source, target DataSource, info ObjectInfo, targetName string, task Task) error {
|
||||
validate := func(targetInfo ObjectInfo) error {
|
||||
if source.Type == "mysql" && target.Type == "mysql" {
|
||||
return validateSyncTargetColumns(task, info, targetInfo)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if targetInfo, err := inspectSyncObject(ctx, writer, targetName); err == nil {
|
||||
return validate(targetInfo)
|
||||
} else if AdapterErrorCode(err) != "objectNotFound" {
|
||||
return err
|
||||
}
|
||||
if task.DryRun || source.Type != target.Type {
|
||||
return errors.New("schemaIncompatible")
|
||||
}
|
||||
// A source-cloned table cannot accept newly named/default fields. Check
|
||||
// before executing CREATE, rather than leaving an unusable target table.
|
||||
if err := validate(info); err != nil {
|
||||
return err
|
||||
}
|
||||
if mysql, ok := writer.(*mysqlAdapter); ok {
|
||||
prefix := "CREATE TABLE " + QuoteIdentifier(info.Name) + " "
|
||||
if info.Type != "table" || !strings.HasPrefix(info.CreateSQL, prefix) {
|
||||
return errors.New("schemaIncompatible")
|
||||
}
|
||||
upper := strings.ToUpper(info.CreateSQL)
|
||||
for _, unsafe := range []string{"DATA DIRECTORY", "INDEX DIRECTORY", "ENGINE=FEDERATED", "CONNECTION=", "TABLESPACE"} {
|
||||
if strings.Contains(upper, unsafe) {
|
||||
return errors.New("schemaIncompatible")
|
||||
}
|
||||
}
|
||||
statement := "CREATE TABLE " + QuoteIdentifier(targetName) + " " + strings.TrimPrefix(info.CreateSQL, prefix)
|
||||
started := time.Now()
|
||||
_, err := mysql.database.ExecContext(ctx, statement)
|
||||
err = mysqlOperationError(err)
|
||||
adapterAudit(mysql.audit, mysql.source.ID, targetName, "SCHEMA", statement, started, 0, false, err)
|
||||
return err
|
||||
}
|
||||
if mongo, ok := writer.(*mongoAdapter); ok {
|
||||
if info.Type != "collection" {
|
||||
return errors.New("schemaIncompatible")
|
||||
}
|
||||
// Collection validators and indexes need an explicit schema plan.
|
||||
for key, value := range info.Options {
|
||||
// Inspect always adds read_only, even for ordinary writable collections.
|
||||
if key != "read_only" || value != false {
|
||||
return errors.New("schemaIncompatible")
|
||||
}
|
||||
}
|
||||
for _, index := range info.Indexes {
|
||||
if index["name"] != "_id_" {
|
||||
return errors.New("schemaIncompatible")
|
||||
}
|
||||
}
|
||||
statement, _ := json.Marshal(map[string]string{"create": targetName})
|
||||
started := time.Now()
|
||||
err := mongoOperationError(mongo.database.CreateCollection(ctx, targetName))
|
||||
adapterAudit(mongo.audit, mongo.source.ID, targetName, "SCHEMA", string(statement), started, 0, false, err)
|
||||
return err
|
||||
}
|
||||
return errors.New("schemaIncompatible")
|
||||
}
|
||||
|
||||
func inspectSyncObject(ctx context.Context, adapter Adapter, name string) (ObjectInfo, error) {
|
||||
if mysql, ok := adapter.(*mysqlAdapter); ok {
|
||||
return mysql.writeSchema(ctx, name)
|
||||
}
|
||||
return adapter.Inspect(ctx, name)
|
||||
}
|
||||
|
||||
// runSchemaPhase joins every structure worker before any data reader starts.
|
||||
func (s *server) runSchemaPhase(ctx context.Context, task Task, reader, writer Adapter, source, target DataSource, objects []string, executionID string) error {
|
||||
workers := task.SchemaWorkers
|
||||
if workers == 0 {
|
||||
workers = 4
|
||||
}
|
||||
workers = min(max(workers, 1), 16, len(objects))
|
||||
if workers == 0 {
|
||||
return nil
|
||||
}
|
||||
phaseCtx, cancel := context.WithCancel(ctx)
|
||||
defer cancel()
|
||||
type result struct {
|
||||
object string
|
||||
dependent bool
|
||||
err error
|
||||
}
|
||||
jobs := make(chan string, len(objects))
|
||||
results := make(chan result, len(objects))
|
||||
for _, object := range objects {
|
||||
jobs <- object
|
||||
}
|
||||
close(jobs)
|
||||
execute := func(object string, allowDependency bool) (out result) {
|
||||
out.object = object
|
||||
defer func() {
|
||||
if recover() != nil {
|
||||
out.err = errors.New("adapterOperationFailed")
|
||||
}
|
||||
}()
|
||||
s.setSchemaStatus(executionID, object, "RUNNING")
|
||||
opCtx := phaseCtx
|
||||
finish := func() {}
|
||||
if task.TimeoutSeconds > 0 {
|
||||
opCtx, finish = context.WithTimeout(phaseCtx, time.Duration(task.TimeoutSeconds)*time.Second)
|
||||
}
|
||||
defer finish()
|
||||
info, err := inspectSyncObject(opCtx, reader, object)
|
||||
if err == nil {
|
||||
err = opCtx.Err()
|
||||
}
|
||||
if err != nil {
|
||||
out.err = err
|
||||
return
|
||||
}
|
||||
if task.Mode == "incremental" && source.Type == "mysql" {
|
||||
found := false
|
||||
for _, column := range info.Columns {
|
||||
if column.Name == task.IncrementalField {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
out.err = errors.New("incrementalSourceFieldMissing")
|
||||
return
|
||||
}
|
||||
}
|
||||
if !allowDependency && strings.Contains(strings.ToUpper(info.CreateSQL), "FOREIGN KEY") {
|
||||
s.setSchemaStatus(executionID, object, "PENDING")
|
||||
out.dependent = true
|
||||
return
|
||||
}
|
||||
targetName := task.TargetObjects[object]
|
||||
if targetName == "" {
|
||||
targetName = object
|
||||
}
|
||||
out.err = prepareSyncSchema(opCtx, writer, source, target, info, targetName, task)
|
||||
if out.err == nil {
|
||||
out.err = opCtx.Err()
|
||||
}
|
||||
return
|
||||
}
|
||||
var group sync.WaitGroup
|
||||
for i := 0; i < workers; i++ {
|
||||
group.Add(1)
|
||||
go func() {
|
||||
defer group.Done()
|
||||
for object := range jobs {
|
||||
if phaseCtx.Err() != nil {
|
||||
return
|
||||
}
|
||||
results <- execute(object, false)
|
||||
}
|
||||
}()
|
||||
}
|
||||
done := make(chan struct{})
|
||||
go func() { group.Wait(); close(results); close(done) }()
|
||||
// Even when persistence fails, join workers before their adapters close.
|
||||
defer func() { cancel(); <-done }()
|
||||
deferred := make(map[string]bool)
|
||||
record := func(out result) error {
|
||||
if out.err == nil {
|
||||
s.setSchemaStatus(executionID, out.object, "COMPLETED")
|
||||
return nil
|
||||
}
|
||||
s.setSchemaStatus(executionID, out.object, "FAILED")
|
||||
if ctx.Err() != nil {
|
||||
return ctx.Err()
|
||||
}
|
||||
return s.markSkippedObject(executionID, out.object, out.err)
|
||||
}
|
||||
for out := range results {
|
||||
if out.dependent {
|
||||
deferred[out.object] = true
|
||||
continue
|
||||
}
|
||||
if err := record(out); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
return ctx.Err()
|
||||
}
|
||||
// Foreign-key tables run after independent tables. Each successful pass
|
||||
// unlocks further dependencies; no progress ends the bounded retry loop.
|
||||
for len(deferred) > 0 {
|
||||
progress := false
|
||||
failed := make([]result, 0, len(deferred))
|
||||
for _, object := range objects {
|
||||
if !deferred[object] {
|
||||
continue
|
||||
}
|
||||
out := execute(object, true)
|
||||
if out.err == nil {
|
||||
delete(deferred, object)
|
||||
progress = true
|
||||
s.setSchemaStatus(executionID, object, "COMPLETED")
|
||||
} else {
|
||||
if ctx.Err() != nil {
|
||||
return record(out)
|
||||
}
|
||||
failed = append(failed, out)
|
||||
s.setSchemaStatus(executionID, object, "PENDING")
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
return ctx.Err()
|
||||
}
|
||||
}
|
||||
if !progress {
|
||||
for _, out := range failed {
|
||||
if err := record(out); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
if err := s.persist(); err != nil {
|
||||
return errors.New("persistenceFailed")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,224 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"reflect"
|
||||
"sync"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func syncColumnFixture(names ...string) ObjectInfo {
|
||||
info := ObjectInfo{Type: "table"}
|
||||
for _, name := range names {
|
||||
info.Columns = append(info.Columns, ColumnInfo{Name: name, Type: "int"})
|
||||
}
|
||||
return info
|
||||
}
|
||||
|
||||
func TestSyncTargetColumnsFollowTransforms(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
source ObjectInfo
|
||||
target ObjectInfo
|
||||
task Task
|
||||
failure string
|
||||
}{
|
||||
{name: "unchanged", source: syncColumnFixture("id", "name"), target: syncColumnFixture("id", "name", "extra")},
|
||||
{name: "missing source column", source: syncColumnFixture("id", "name"), target: syncColumnFixture("id"), failure: "syncTargetColumnsMissing"},
|
||||
{name: "renamed", source: syncColumnFixture("id"), target: syncColumnFixture("user_id"), task: Task{Mapping: map[string]string{"id": "user_id"}}},
|
||||
{name: "mapping needs destination", source: syncColumnFixture("id"), target: syncColumnFixture("id"), task: Task{Mapping: map[string]string{"id": "user_id"}}, failure: "syncTargetColumnsMissing"},
|
||||
{name: "swapped", source: syncColumnFixture("id", "name"), target: syncColumnFixture("id", "name"), task: Task{Mapping: map[string]string{"id": "name", "name": "id"}}},
|
||||
{name: "omitted", source: syncColumnFixture("id", "name"), target: syncColumnFixture("id"), task: Task{Omit: []string{"name"}}},
|
||||
{name: "default adds field", source: syncColumnFixture("id"), target: syncColumnFixture("id"), task: Task{Defaults: map[string]any{"active": true}}, failure: "syncTargetColumnsMissing"},
|
||||
{name: "default omitted", source: syncColumnFixture("id"), target: syncColumnFixture("id"), task: Task{Defaults: map[string]any{"active": true}, Omit: []string{"active"}}},
|
||||
{name: "nested target is root JSON field", source: syncColumnFixture("id"), target: syncColumnFixture("document"), task: Task{Mapping: map[string]string{"id": "document.id"}}},
|
||||
{name: "nested source JSON", source: ObjectInfo{Type: "table", Columns: []ColumnInfo{{Name: "payload", Type: "json"}}}, target: syncColumnFixture("payload", "name"), task: Task{Mapping: map[string]string{"payload.name": "name"}}},
|
||||
{name: "nested scalar source is absent", source: syncColumnFixture("id"), target: syncColumnFixture("id"), task: Task{Mapping: map[string]string{"id.name": "name"}}},
|
||||
{name: "literal dotted column default is already present", source: syncColumnFixture("user.id"), target: syncColumnFixture("user.id"), task: Task{Defaults: map[string]any{"user.id": 1}}},
|
||||
{name: "nested default keeps root", source: syncColumnFixture("id"), target: syncColumnFixture("id"), task: Task{Defaults: map[string]any{"document.id": 1}, Omit: []string{"document.id"}}, failure: "syncTargetColumnsMissing"},
|
||||
{name: "view target", source: syncColumnFixture("id"), target: ObjectInfo{Type: "view"}, failure: "schemaIncompatible"},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
err := validateSyncTargetColumns(test.task, test.source, test.target)
|
||||
if err == nil && test.failure != "" || err != nil && err.Error() != test.failure {
|
||||
t.Fatalf("expected %q, got %v", test.failure, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDottedColumnPreflightMatchesActualTransformOrder(t *testing.T) {
|
||||
source := syncColumnFixture("user.id")
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
task Task
|
||||
target []string
|
||||
failure bool
|
||||
}{
|
||||
{"conversion creates root", Task{Conversions: map[string]string{"user.id": "string"}}, []string{"user.id"}, true},
|
||||
{"conversion with nested target", Task{Conversions: map[string]string{"user.id": "string"}}, []string{"user.id", "user"}, false},
|
||||
{"conversion root omitted", Task{Conversions: map[string]string{"user.id": "string"}, Omit: []string{"user"}}, []string{"user.id"}, false},
|
||||
{"masking creates root", Task{Masking: map[string]string{"user.id": "null"}}, []string{"user.id"}, true},
|
||||
{"omitted literal is not masked", Task{Omit: []string{"user.id"}, Masking: map[string]string{"user.id": "null"}}, nil, false},
|
||||
{"masking recreates root after omission", Task{Omit: []string{"user"}, Masking: map[string]string{"user.id": "null"}}, []string{"user.id"}, true},
|
||||
{"mapped literal no longer converted", Task{Mapping: map[string]string{"user.id": "id"}, Conversions: map[string]string{"user.id": "string"}}, []string{"id"}, false},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
result, err := transformRecord(Record{"user.id": 42}, test.task.Mapping, test.task.Defaults, test.task.Conversions, test.task.Omit, test.task.Masking)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
target := syncColumnFixture(test.target...)
|
||||
missing := false
|
||||
for name := range result {
|
||||
found := false
|
||||
for _, column := range target.Columns {
|
||||
if column.Name == name {
|
||||
found = true
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
missing = true
|
||||
}
|
||||
}
|
||||
if missing != test.failure {
|
||||
t.Fatalf("actual transform disagrees with fixture: %+v", result)
|
||||
}
|
||||
err = validateSyncTargetColumns(test.task, source, target)
|
||||
if test.failure && (err == nil || err.Error() != "syncTargetColumnsMissing") || !test.failure && err != nil {
|
||||
t.Fatalf("preflight disagrees with actual fields %v: %v", result, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type syncColumnsAdapter struct {
|
||||
tableSkipAdapter
|
||||
mu sync.Mutex
|
||||
objects map[string]ObjectInfo
|
||||
missing bool
|
||||
reads []string
|
||||
inspects map[string]int
|
||||
}
|
||||
|
||||
func (adapter *syncColumnsAdapter) Inspect(_ context.Context, name string) (ObjectInfo, error) {
|
||||
adapter.mu.Lock()
|
||||
defer adapter.mu.Unlock()
|
||||
if adapter.inspects == nil {
|
||||
adapter.inspects = map[string]int{}
|
||||
}
|
||||
adapter.inspects[name]++
|
||||
if adapter.missing {
|
||||
return ObjectInfo{}, errors.New("objectNotFound")
|
||||
}
|
||||
info := adapter.objects[name]
|
||||
info.Name = name
|
||||
return info, nil
|
||||
}
|
||||
|
||||
func (adapter *syncColumnsAdapter) Read(_ context.Context, name string, _ ReadOptions, consume func([]Record) error) error {
|
||||
adapter.mu.Lock()
|
||||
adapter.reads = append(adapter.reads, name)
|
||||
adapter.mu.Unlock()
|
||||
return consume([]Record{{"id": 1}})
|
||||
}
|
||||
|
||||
func TestSchemaMissingColumnsSkipsDataAndContinuesOtherTables(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
reader := &syncColumnsAdapter{objects: map[string]ObjectInfo{"bad": syncColumnFixture("id", "name"), "good": syncColumnFixture("id")}}
|
||||
writer := &syncColumnsAdapter{objects: map[string]ObjectInfo{"bad": syncColumnFixture("id"), "good": syncColumnFixture("id")}}
|
||||
s.adapterFactory = func(_ context.Context, source DataSource, _ func(SQLAudit)) (Adapter, error) {
|
||||
if source.ID == "source" {
|
||||
return reader, nil
|
||||
}
|
||||
return writer, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"bad", "good"}, MaxRows: -1, BatchSize: 10, SchemaWorkers: 2}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
result := s.store.Executions["exec"]
|
||||
if result.Status != "PARTIAL" || result.SchemaStatuses["bad"] != "FAILED" || result.ObjectStatuses["bad"] != "FAILED" || result.ObjectErrors["bad"] != "syncTargetColumnsMissing" || result.ObjectStatuses["good"] != "COMPLETED" {
|
||||
t.Fatalf("unexpected structure/data results: %+v", result)
|
||||
}
|
||||
if !reflect.DeepEqual(reader.reads, []string{"good"}) {
|
||||
t.Fatalf("read data before schema was compatible: %v", reader.reads)
|
||||
}
|
||||
if reader.inspects["bad"] != 1 {
|
||||
t.Fatal("structure phase repeated source metadata inspection")
|
||||
}
|
||||
if !reflect.DeepEqual(result.SkippedObjects, []string{"bad"}) {
|
||||
t.Fatalf("failed structure cannot be retried independently: %v", result.SkippedObjects)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMissingTargetRejectsIncompatibleCloneBeforeCreate(t *testing.T) {
|
||||
writer := &syncColumnsAdapter{missing: true}
|
||||
err := prepareSyncSchema(context.Background(), writer, DataSource{Type: "mysql"}, DataSource{Type: "mysql"}, syncColumnFixture("id"), "users", Task{Mapping: map[string]string{"id": "user_id"}})
|
||||
if err == nil || err.Error() != "syncTargetColumnsMissing" {
|
||||
t.Fatalf("attempted to clone a schema that cannot accept transformed rows: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSyncRequiredTargetColumnsRespectStrategyAndTransforms(t *testing.T) {
|
||||
source := syncColumnFixture("id", "name")
|
||||
target := syncColumnFixture("id", "name", "tenant_id")
|
||||
target.Columns[2].Required = true
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
task Task
|
||||
fails bool
|
||||
}{
|
||||
{"insert", Task{Conflict: "insert"}, true},
|
||||
{"skip duplicates still inserts", Task{Conflict: "skip"}, true},
|
||||
{"upsert", Task{Conflict: "upsert"}, true},
|
||||
{"replace resets omitted fields", Task{Conflict: "replace"}, true},
|
||||
{"update preserves existing fields", Task{Conflict: "update"}, false},
|
||||
{"task default supplies mandatory field", Task{Conflict: "insert", Defaults: map[string]any{"tenant_id": 0}}, false},
|
||||
{"mapping supplies mandatory field", Task{Conflict: "insert", Mapping: map[string]string{"name": "tenant_id"}}, false},
|
||||
{"omitting mandatory default", Task{Conflict: "insert", Defaults: map[string]any{"tenant_id": 1}, Omit: []string{"tenant_id"}}, true},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
err := validateSyncTargetColumns(test.task, source, target)
|
||||
if test.fails && (err == nil || AdapterErrorCode(err) != "syncTargetRequiredColumnsMissing") || !test.fails && err != nil {
|
||||
t.Fatalf("unexpected preflight result: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRequiredTargetFieldFailsSchemaBeforeReadingData(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
reader := &syncColumnsAdapter{objects: map[string]ObjectInfo{"bad": syncColumnFixture("id"), "good": syncColumnFixture("id")}}
|
||||
bad := syncColumnFixture("id", "tenant_id")
|
||||
bad.Columns[1].Required = true
|
||||
writer := &syncColumnsAdapter{objects: map[string]ObjectInfo{"renamed_bad": bad, "good": syncColumnFixture("id")}}
|
||||
s.adapterFactory = func(_ context.Context, source DataSource, _ func(SQLAudit)) (Adapter, error) {
|
||||
if source.ID == "source" {
|
||||
return reader, nil
|
||||
}
|
||||
return writer, nil
|
||||
}
|
||||
task := Task{ID: "task", Objects: []string{"bad", "good"}, TargetObjects: map[string]string{"bad": "renamed_bad"}, Conflict: "upsert", MaxRows: -1, BatchSize: 10, SchemaWorkers: 2}
|
||||
s.store.Tasks[task.ID] = task
|
||||
s.store.Executions["exec"] = Execution{ID: "exec", TaskID: task.ID, Status: "RUNNING"}
|
||||
s.runTask(context.Background(), task, s.store.Sources["source"], s.store.Sources["target"], "exec")
|
||||
result := s.store.Executions["exec"]
|
||||
if result.Status != "PARTIAL" || result.SchemaStatuses["bad"] != "FAILED" || result.ObjectErrors["bad"] != "syncTargetRequiredColumnsMissing" || result.ObjectStatuses["good"] != "COMPLETED" || !reflect.DeepEqual(reader.reads, []string{"good"}) || !reflect.DeepEqual(result.SkippedObjects, []string{"bad"}) {
|
||||
t.Fatalf("required-field preflight failed to skip/continue: %+v reads=%v", result, reader.reads)
|
||||
}
|
||||
if !reflect.DeepEqual(result.TargetObjects, map[string]string{"bad": "renamed_bad", "good": "good"}) {
|
||||
t.Fatalf("target names not recorded: %v", result.TargetObjects)
|
||||
}
|
||||
// Editing the task must not change the audit scope of its old execution.
|
||||
task.TargetObjects["bad"] = "changed_later"
|
||||
s.store.Tasks[task.ID] = task
|
||||
if err := s.persist(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
restored, _, err := loadStore(s.dataDir)
|
||||
if err != nil || restored.Executions["exec"].TargetObjects["bad"] != "renamed_bad" {
|
||||
t.Fatalf("historical target lost after restart: %v %v", restored, err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,78 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestTaskStatusFollowsOriginalAfterSubsetRetry(t *testing.T) {
|
||||
for _, restart := range []bool{false, true} {
|
||||
for _, test := range []struct{ name, other, result, attempt, want string }{
|
||||
{"repaired subset leaves failure", "FAILED", "COMPLETED", "COMPLETED", "PARTIAL"},
|
||||
{"failed retry preserves success", "COMPLETED", "FAILED", "FAILED", "PARTIAL"},
|
||||
{"all failed", "FAILED", "FAILED", "FAILED", "FAILED"},
|
||||
{"all repaired", "COMPLETED", "COMPLETED", "COMPLETED", "COMPLETED"},
|
||||
{"cancelled retry keeps incomplete tables", "COMPLETED", "PENDING", "CANCELLED", "PARTIAL"},
|
||||
} {
|
||||
t.Run(test.name+map[bool]string{false: "/live", true: "/restart"}[restart], func(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
old := time.Now().UTC().Add(-time.Hour)
|
||||
started := old.Add(time.Minute)
|
||||
s.store.Tasks["task"] = Task{ID: "task", Status: "RUNNING", Mode: "incremental", IncrementalValue: 42, LastRunAt: &started}
|
||||
s.store.Executions["exec-1"] = Execution{ID: "exec-1", TaskID: "task", Status: "FAILED", StartedAt: old, ObjectStatuses: map[string]string{"other": test.other, "retry": "RETRYING"}}
|
||||
s.store.Executions["exec-2"] = Execution{ID: "exec-2", TaskID: "task", RetryOf: "exec-1", Status: "RUNNING", StartedAt: started, ObjectStatuses: map[string]string{"retry": test.result}, RetryOriginalStatuses: map[string]string{"retry": "FAILED"}}
|
||||
if restart {
|
||||
// Crash after the worker persisted its outcome, before the root
|
||||
// and task were updated. Recovery must reconcile both.
|
||||
retry := s.store.Executions["exec-2"]
|
||||
retry.Status = test.attempt
|
||||
s.store.Executions[retry.ID] = retry
|
||||
if err := s.persist(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
restored, _, err := loadStore(s.dataDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
s.store = restored
|
||||
} else {
|
||||
s.finishExecution("exec-2", test.attempt, "", 999)
|
||||
}
|
||||
if s.store.Tasks["task"].Status != test.want || s.store.Executions["exec-1"].Status != test.want || s.store.Executions["exec-2"].Status != test.attempt {
|
||||
t.Fatalf("task/root/retry differ: %s / %s / %s", s.store.Tasks["task"].Status, s.store.Executions["exec-1"].Status, s.store.Executions["exec-2"].Status)
|
||||
}
|
||||
cursor := s.store.Tasks["task"].IncrementalValue
|
||||
if cursor != 42 && cursor != json.Number("42") {
|
||||
t.Fatalf("retry advanced watermark: %v", cursor)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestartTaskStatusUsesLatestExecutionAndPreservesDrafts(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
old := time.Now().UTC().Add(-time.Hour)
|
||||
for _, taskID := range []string{"active", "draft", "settled", "missing"} {
|
||||
status := map[string]string{"active": "RUNNING", "draft": "DRAFT", "settled": "COMPLETED", "missing": "PENDING"}[taskID]
|
||||
s.store.Tasks[taskID] = Task{ID: taskID, Status: status}
|
||||
if taskID != "missing" {
|
||||
executionID := taskID + "-1"
|
||||
s.store.Executions[executionID] = Execution{ID: executionID, TaskID: taskID, Status: "COMPLETED", StartedAt: old}
|
||||
}
|
||||
}
|
||||
s.store.Executions["active-2"] = Execution{ID: "active-2", TaskID: "active", Status: "RUNNING", StartedAt: old.Add(time.Minute), ObjectStatuses: map[string]string{"a": "RUNNING"}}
|
||||
if err := s.persist(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
restored, _, err := loadStore(s.dataDir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for taskID, want := range map[string]string{"active": "FAILED", "draft": "DRAFT", "settled": "COMPLETED", "missing": "DRAFT"} {
|
||||
if restored.Tasks[taskID].Status != want {
|
||||
t.Fatalf("%s: got %s, want %s", taskID, restored.Tasks[taskID].Status, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestTaskAcceptsAllTablesInLargeDatabase(t *testing.T) {
|
||||
source := DataSource{Type: "mysql", Endpoint: "source:3306", Database: "db"}
|
||||
target := DataSource{Type: "mysql", Endpoint: "target:3306", Database: "db"}
|
||||
task := Task{Name: "all tables"}
|
||||
for i := 0; i < 388; i++ {
|
||||
task.Objects = append(task.Objects, fmt.Sprintf("table_%d", i))
|
||||
}
|
||||
if err := validateTask(&task, source, target); err != nil {
|
||||
t.Fatalf("full database selection rejected: %v", err)
|
||||
}
|
||||
task.Objects = append(task.Objects, task.Objects[0])
|
||||
if err := validateTask(&task, source, target); err == nil || err.Error() != "invalidObject" {
|
||||
t.Fatalf("duplicate table validation lost: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Keep editable cursor, filter and default numbers exact through a browser.
|
||||
// JSON text fields are transport values, not separately saved configuration.
|
||||
func taskForAPI(task Task) Task {
|
||||
task.Revision = taskRevision(task)
|
||||
task.IncrementalValueJSON = ""
|
||||
task.FiltersJSON = ""
|
||||
task.DefaultsJSON = ""
|
||||
if len(task.Defaults) > 0 {
|
||||
if data, err := json.Marshal(task.Defaults); err == nil {
|
||||
task.DefaultsJSON = string(data)
|
||||
}
|
||||
}
|
||||
if len(task.Filters) > 0 {
|
||||
if data, err := json.Marshal(task.Filters); err == nil {
|
||||
task.FiltersJSON = string(data)
|
||||
}
|
||||
}
|
||||
if task.IncrementalValue != nil {
|
||||
if data, err := json.Marshal(task.IncrementalValue); err == nil {
|
||||
task.IncrementalValueJSON = string(data)
|
||||
}
|
||||
}
|
||||
return task
|
||||
}
|
||||
|
||||
func taskRevision(task Task) string {
|
||||
task.Revision, task.IncrementalValueJSON = "", ""
|
||||
task.FiltersJSON = ""
|
||||
task.DefaultsJSON = ""
|
||||
data, err := json.Marshal(task)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
hash := sha256.Sum256(data)
|
||||
return hex.EncodeToString(hash[:])
|
||||
}
|
||||
|
||||
func parseExactFilters(raw string) ([]Filter, error) {
|
||||
var filters []Filter
|
||||
if !strings.HasPrefix(strings.TrimSpace(raw), "[") || decodePersistenceJSON([]byte(raw), &filters) != nil {
|
||||
return nil, errors.New("invalidFilter")
|
||||
}
|
||||
if err := ValidateFilters(filters); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return filters, nil
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestTaskAPIKeepsExactWatermarkOnBrowserRoundTrip(t *testing.T) {
|
||||
s := databaseOperationServer(t)
|
||||
task := Task{ID: "task", Name: "task", SourceID: "source", TargetID: "target", SourceDatabase: "app", TargetDatabase: "app", Mode: "incremental", Objects: []string{"table"}, IncrementalField: "id", IncrementalValue: json.Number("9007199254740993")}
|
||||
s.store.Tasks[task.ID] = task
|
||||
for _, path := range []string{"/api/tasks", "/api/tasks/task"} {
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodGet, path, nil)
|
||||
if path == "/api/tasks" {
|
||||
s.tasks(rec, req)
|
||||
} else {
|
||||
s.task(rec, req)
|
||||
}
|
||||
if rec.Code != 200 || !bytes.Contains(rec.Body.Bytes(), []byte(`"incremental_value_json":"9007199254740993"`)) {
|
||||
t.Fatalf("exact watermark missing: %s", rec.Body.String())
|
||||
}
|
||||
}
|
||||
input := taskForAPI(task)
|
||||
input.IncrementalValue = float64(9007199254740992) // Browser's rounded number.
|
||||
data, err := json.Marshal(input)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
s.task(rec, httptest.NewRequest(http.MethodPut, "/api/tasks/task", bytes.NewReader(data)))
|
||||
if rec.Code != 200 {
|
||||
t.Fatalf("edit rejected: %s", rec.Body.String())
|
||||
}
|
||||
stored := s.store.Tasks[task.ID]
|
||||
if stored.IncrementalValue != json.Number("9007199254740993") || stored.IncrementalValueJSON != "" {
|
||||
t.Fatalf("rounded or duplicate cursor saved: %+v", stored)
|
||||
}
|
||||
stored.IncrementalValue = json.Number("9007199254740995")
|
||||
if taskForAPI(stored).IncrementalValueJSON != "9007199254740995" {
|
||||
t.Fatal("API retained stale cursor text after progress")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTaskRejectsMalformedExactWatermark(t *testing.T) {
|
||||
for _, raw := range []string{"1 trailing", "{}", "[1]", "null"} {
|
||||
task := Task{Name: "task", Objects: []string{"table"}, Mode: "incremental", IncrementalField: "id", IncrementalValue: 1, IncrementalValueJSON: raw}
|
||||
err := validateTask(&task, DataSource{Type: "mysql", Endpoint: "source:3306", Database: "app"}, DataSource{Type: "mysql", Endpoint: "target:3306", Database: "app"})
|
||||
if err == nil || err.Error() != "invalidIncremental" {
|
||||
t.Fatalf("bad exact watermark %q accepted: %v", raw, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
+58
-1
@@ -17,9 +17,23 @@ import (
|
||||
"time"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/bson/bsontype"
|
||||
"go.mongodb.org/mongo-driver/bson/primitive"
|
||||
)
|
||||
|
||||
// A decoded JSON string needs quoting when written into a MySQL JSON column.
|
||||
// Keep it distinguishable from raw JSON text read from the source; its string
|
||||
// kind still encodes as a normal string in BSON and ordinary text columns.
|
||||
type convertedJSONString string
|
||||
|
||||
// Preserve explicit JSON null independently of an absent/SQL-null value.
|
||||
type convertedJSONNull struct{}
|
||||
|
||||
func (convertedJSONNull) MarshalJSON() ([]byte, error) { return []byte("null"), nil }
|
||||
func (convertedJSONNull) MarshalBSONValue() (bsontype.Type, []byte, error) {
|
||||
return bsontype.Null, nil, nil
|
||||
}
|
||||
|
||||
func recordMap(value any) (map[string]any, bool) {
|
||||
switch typed := value.(type) {
|
||||
case Record:
|
||||
@@ -217,7 +231,7 @@ func transformRecord(record Record, mapping map[string]string, defaults map[stri
|
||||
}
|
||||
for key, kind := range masking {
|
||||
value, ok := recordPath(result, key)
|
||||
if !ok || value == nil {
|
||||
if !ok || adapterNil(value) {
|
||||
continue
|
||||
}
|
||||
masked, err := maskValue(value, kind)
|
||||
@@ -245,6 +259,10 @@ func scalarText(value any) string {
|
||||
return typed.String()
|
||||
case string:
|
||||
return typed
|
||||
case convertedJSONString:
|
||||
return string(typed)
|
||||
case convertedJSONNull:
|
||||
return "null"
|
||||
case json.Number:
|
||||
return typed.String()
|
||||
}
|
||||
@@ -274,6 +292,10 @@ func convertValue(value any, kind string) (any, error) {
|
||||
if _, ok := recordMap(value); ok {
|
||||
return value, nil
|
||||
}
|
||||
switch value.(type) {
|
||||
case []any, bson.A:
|
||||
return value, nil
|
||||
}
|
||||
var result any
|
||||
decoder := json.NewDecoder(strings.NewReader(text))
|
||||
decoder.UseNumber()
|
||||
@@ -282,6 +304,12 @@ func convertValue(value any, kind string) (any, error) {
|
||||
if trailing := decoder.Decode(new(any)); trailing != io.EOF {
|
||||
err = errors.New("invalidConversion")
|
||||
}
|
||||
if text, ok := result.(string); ok {
|
||||
result = convertedJSONString(text)
|
||||
}
|
||||
if err == nil && result == nil {
|
||||
result = convertedJSONNull{}
|
||||
}
|
||||
}
|
||||
return result, err
|
||||
case "datetime":
|
||||
@@ -373,6 +401,11 @@ func automaticMask(record Record) Record {
|
||||
}
|
||||
|
||||
func compareCursor(first, second any) int {
|
||||
leftTime, leftIsTime := cursorTime(first)
|
||||
rightTime, rightIsTime := cursorTime(second)
|
||||
if leftIsTime && rightIsTime {
|
||||
return leftTime.Compare(rightTime)
|
||||
}
|
||||
left, right := scalarText(first), scalarText(second)
|
||||
leftNumber, leftOK := new(big.Rat).SetString(left)
|
||||
rightNumber, rightOK := new(big.Rat).SetString(right)
|
||||
@@ -381,3 +414,27 @@ func compareCursor(first, second any) int {
|
||||
}
|
||||
return strings.Compare(left, right)
|
||||
}
|
||||
|
||||
func incrementalCursorValid(value any) bool {
|
||||
if adapterNil(value) || !adapterScalar(value) {
|
||||
return false
|
||||
}
|
||||
if decimal, ok := value.(primitive.Decimal128); ok {
|
||||
_, finite := new(big.Rat).SetString(decimal.String())
|
||||
return finite
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func cursorTime(value any) (time.Time, bool) {
|
||||
switch typed := value.(type) {
|
||||
case time.Time:
|
||||
return typed, true
|
||||
case primitive.DateTime:
|
||||
return typed.Time(), true
|
||||
case string:
|
||||
parsed, err := time.Parse(time.RFC3339Nano, typed)
|
||||
return parsed, err == nil
|
||||
}
|
||||
return time.Time{}, false
|
||||
}
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestUnlockedSnapshotUsesOneReadViewWithoutGTID(t *testing.T) {
|
||||
fixture := &seedFixture{allowUnlocked: true}
|
||||
db := seedTestDB(t, fixture, true)
|
||||
snapshot, err := acquireUnlockedSnapshot(context.Background(), db, "shopee-wes-test")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if fixture.snapshotStarts != 1 || snapshot.GTID != "" || db.Stats().InUse != 1 || snapshot.Readers[0] != snapshot.Readers[1] {
|
||||
t.Fatal("unlocked snapshot does not share a single read view")
|
||||
}
|
||||
for _, statement := range fixture.executed {
|
||||
if strings.Contains(statement, "LOCK INSTANCE") || strings.Contains(statement, "FLUSH TABLES") {
|
||||
t.Fatalf("unlocked snapshot issued a lock: %s", statement)
|
||||
}
|
||||
}
|
||||
snapshot.close()
|
||||
if db.Stats().InUse != 0 {
|
||||
t.Fatal("snapshot connection leaked")
|
||||
}
|
||||
}
|
||||
+776
-93
File diff suppressed because it is too large
Load Diff
@@ -30,6 +30,7 @@ function renderReplicationSteps(steps, saved = false) {
|
||||
|
||||
function closeAutoReplication(force = false) {
|
||||
if (replicationBusy && !force) return;
|
||||
window.syncdbReact?.closeReplicationWizard?.();
|
||||
replicationVersion++;
|
||||
replicationStatusVersion++;
|
||||
replicationStatusPending = null;
|
||||
@@ -77,6 +78,31 @@ function updateReplicationControls() {
|
||||
function openAutoReplication(saved = null) {
|
||||
closeAutoReplication(true);
|
||||
const sources = state.sources.filter(source => source.type === "mysql");
|
||||
if (window.syncdbReact?.replicationWizard) {
|
||||
const session=authVersion;
|
||||
const current=()=>{if(!state.authenticated || session!==authVersion)throw new Error("sessionExpired");};
|
||||
window.syncdbReact.replicationWizard({sources,saved:saved ? {...saved,steps:normalizeReplicationSteps(saved.steps)} : null,translate:text,userError,language,validPassword:value=>validAccountPassword(value) && value.length<=32,
|
||||
loadDatabases:async sourceID=>{
|
||||
current();const result=await request(`/datasources/${encoded(sourceID)}/databases`);current();
|
||||
if(!result || !Array.isArray(result.databases) || result.databases.some(value=>typeof value!=="string"))throw new Error("invalidResponse");
|
||||
return [...new Set(result.databases)].filter(Boolean);
|
||||
},
|
||||
check:async values=>{current();const result=await send("/replications/auto/check","POST",values);current();return normalizeReplicationCheck(result);},
|
||||
apply:async values=>{
|
||||
current();if(isProduction(values.target_id) || !values.confirm || !values.plan_hash)throw new Error("productionBlocked");
|
||||
const result=await send("/replications/auto/apply","POST",values);current();
|
||||
if(!result || !["COMPLETED","FAILED"].includes(result.status))throw new Error("invalidResponse");
|
||||
return {status:result.status,error:result.error ? errorKey(result.error) : "",steps:normalizeReplicationSteps(result.steps)};
|
||||
},
|
||||
loadStatus:async (sourceID,targetID)=>{
|
||||
current();const params=new URLSearchParams({target_id:targetID});if(sourceID)params.set("source_id",sourceID);
|
||||
const result=await request(`/replications/auto/status?${params}`);current();
|
||||
if(!result || typeof result.configured!=="boolean")throw new Error("invalidResponse");
|
||||
return result;
|
||||
},onApplied:()=>toast("replicationApplied"),onFinished:()=>{void loadPageData().catch(showPageError);}
|
||||
});
|
||||
return;
|
||||
}
|
||||
const targets = sources.filter(source => !["production", "prod"].includes(source.environment));
|
||||
const options = items => [["", text("choose")], ...sourceOptions(items)];
|
||||
byId("replication-fields").innerHTML = notice("replicationScope") + notice("replicationRequirements")
|
||||
@@ -106,6 +132,17 @@ function openAutoReplication(saved = null) {
|
||||
}, 5000);
|
||||
}
|
||||
|
||||
function normalizeReplicationSteps(steps) {
|
||||
if(!Array.isArray(steps) || !steps.every(step=>step && typeof step.id==="string" && typeof step.status==="string"))throw new Error("invalidResponse");
|
||||
const redact=sql=>accountReviewSQL(sql).replace(/(\bSOURCE_PASSWORD\s*=\s*)'(?:''|\\.|[^'])*'/gi,"$1'<replication-password>'");
|
||||
return steps.map(step=>({id:step.id,status:step.status,...(step.error ? {error:errorKey(step.error)} : {}),...(typeof step.sql==="string" ? {sql:redact(step.sql)} : {})}));
|
||||
}
|
||||
|
||||
function normalizeReplicationCheck(result) {
|
||||
if(!result || !Array.isArray(result.statements) || !result.statements.every(item=>typeof item==="string") || typeof result.can_apply!=="boolean" || typeof result.needs_password!=="boolean" || (!result.error && !/^[a-f0-9]{64}$/.test(result.plan_hash)))throw new Error("invalidResponse");
|
||||
return {plan_hash:result.plan_hash,can_apply:result.can_apply,needs_password:result.needs_password,error:result.error ? errorKey(result.error) : "",steps:normalizeReplicationSteps(result.steps)};
|
||||
}
|
||||
|
||||
async function loadReplicationDatabases(sourceID, preferred = "") {
|
||||
const form = byId("replication-form");
|
||||
const control = form.elements.database;
|
||||
|
||||
@@ -6,11 +6,13 @@
|
||||
<meta name="referrer" content="same-origin">
|
||||
<title>syncdb</title>
|
||||
<link rel="stylesheet" href="/style.css">
|
||||
<script src="/react-console.js" defer></script>
|
||||
<script src="/app.js" defer></script>
|
||||
<script src="/auto-replication.js" defer></script>
|
||||
<script src="/replication-operations.js" defer></script>
|
||||
</head>
|
||||
<body>
|
||||
<div id="legacy-app" class="legacy-ui">
|
||||
<div class="app" id="app" hidden>
|
||||
<aside class="sidebar">
|
||||
<a class="brand" href="#overview"><span class="brand-icon" aria-hidden="true">⇄</span>syncdb</a>
|
||||
@@ -37,6 +39,8 @@
|
||||
<footer><span>syncdb</span><span id="updated"></span></footer>
|
||||
</main>
|
||||
</div>
|
||||
</div>
|
||||
<div class="legacy-ui">
|
||||
<dialog id="dialog" aria-labelledby="dialog-title">
|
||||
<form id="editor">
|
||||
<header><h2 id="dialog-title"></h2><button type="button" id="close-dialog" data-i18n="close"></button></header>
|
||||
@@ -80,5 +84,6 @@
|
||||
</form>
|
||||
</dialog>
|
||||
<div id="toast" role="status" hidden></div>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
Vendored
+682
File diff suppressed because one or more lines are too long
@@ -287,6 +287,7 @@ function replicationOperationsHTML() {
|
||||
}
|
||||
|
||||
function renderReplicationOperations() {
|
||||
if (typeof window !== "undefined" && window.syncdbReact?.replicationWorkspace && page() === "replication") {render();return;}
|
||||
const root = typeof document === "undefined" ? null : document.getElementById("replication-operations");
|
||||
if (root) root.innerHTML = replicationOperationsBodyHTML();
|
||||
}
|
||||
@@ -302,6 +303,7 @@ function clearReplicationOperationsTargetData(topology) {
|
||||
operations.reconnect = null;
|
||||
operations.seedPlan = null;
|
||||
operations.seedPlanTarget = "";
|
||||
operations.seedDraft = null;
|
||||
operations.switchoverPlan = null;
|
||||
operations.switchoverPlanTarget = "";
|
||||
operations.errors = {};
|
||||
|
||||
+113
-25
@@ -1,19 +1,24 @@
|
||||
:root {
|
||||
body { margin: 0; }
|
||||
[hidden] { display: none !important; }
|
||||
|
||||
/* Legacy dialogs and the pre-bundle fallback only. */
|
||||
@scope (.legacy-ui) {
|
||||
:scope {
|
||||
font-family: "Segoe UI", "Microsoft YaHei", sans-serif;
|
||||
color: #19283f;
|
||||
background: #f4f7fb;
|
||||
font-synthesis: none;
|
||||
--muted: #61718a;
|
||||
--line: #e0e7f0;
|
||||
--blue: #3867ed;
|
||||
--blue: #087f8c;
|
||||
}
|
||||
|
||||
* { box-sizing: border-box; }
|
||||
[hidden] { display: none !important; }
|
||||
body { margin: 0; }
|
||||
button, input, select, textarea { font: inherit; }
|
||||
button, a, input, select, textarea { -webkit-tap-highlight-color: transparent; }
|
||||
button, .button {
|
||||
button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *)), input:not(:where(.ant-input, .ant-input-number-input, .ant-select *, .ant-pagination *)), select, textarea { font: inherit; }
|
||||
button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *)), a, input:not(:where(.ant-input, .ant-input-number-input, .ant-select *, .ant-pagination *)), select, textarea { -webkit-tap-highlight-color: transparent; }
|
||||
button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *)), .button {
|
||||
cursor: pointer;
|
||||
border: 1px solid var(--line);
|
||||
border-radius: 8px;
|
||||
@@ -28,12 +33,12 @@ button, .button {
|
||||
font-size: 13px;
|
||||
font-weight: 600;
|
||||
}
|
||||
button:hover, .button:hover { background: #edf2ff; border-color: #b8c9fc; }
|
||||
button:disabled { opacity: .5; cursor: not-allowed; }
|
||||
button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *)):hover, .button:hover { background: #edf2ff; border-color: #b8c9fc; }
|
||||
button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *)):disabled { opacity: .5; cursor: not-allowed; }
|
||||
.primary { background: var(--blue); border-color: var(--blue); color: #fff; }
|
||||
.primary:hover { background: #2855d3; color: #fff; }
|
||||
button:focus-visible, a:focus-visible, summary:focus-visible { outline: 3px solid #8daafa; outline-offset: 3px; }
|
||||
input, select, textarea {
|
||||
button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *)):focus-visible, a:focus-visible, summary:focus-visible { outline: 3px solid #8daafa; outline-offset: 3px; }
|
||||
input:not(:where(.ant-input, .ant-input-number-input, .ant-select *, .ant-pagination *)), select, textarea {
|
||||
border: 1px solid #cbd5e2;
|
||||
border-radius: 7px;
|
||||
padding: 10px 12px;
|
||||
@@ -42,10 +47,10 @@ input, select, textarea {
|
||||
width: 100%;
|
||||
min-width: 0;
|
||||
}
|
||||
input:focus, select:focus, textarea:focus { outline: 2px solid #a9bffc; outline-offset: 1px; }
|
||||
input:disabled, select:disabled { background: #f0f3f7; }
|
||||
input:not(:where(.ant-input, .ant-input-number-input, .ant-select *, .ant-pagination *)):focus, select:focus, textarea:focus { outline: 2px solid #a9bffc; outline-offset: 1px; }
|
||||
input:not(:where(.ant-input, .ant-input-number-input, .ant-select *, .ant-pagination *)):disabled, select:disabled { background: #f0f3f7; }
|
||||
textarea { resize: vertical; min-height: 88px; }
|
||||
input[type="checkbox"] { width: 18px; height: 18px; flex-shrink: 0; accent-color: var(--blue); }
|
||||
input:not(:where(.ant-input, .ant-input-number-input, .ant-select *, .ant-pagination *))[type="checkbox"] { width: 18px; height: 18px; flex-shrink: 0; accent-color: var(--blue); }
|
||||
.app { display: flex; min-height: 100vh; }
|
||||
.sidebar {
|
||||
width: 222px;
|
||||
@@ -81,7 +86,7 @@ h3 { font-size: 14px; }
|
||||
.section-toolbar p { color: var(--muted); font-size: 13px; line-height: 1.8; max-width: 75ch; margin: 0; }
|
||||
#page-actions { display: flex; gap: 8px; flex-shrink: 0; flex-wrap: wrap; }
|
||||
.search { display: flex; gap: 15px; align-items: center; font-size: 13px; color: var(--muted); margin-bottom: 20px; }
|
||||
.search input { max-width: 400px; }
|
||||
.search input:not(:where(.ant-input, .ant-input-number-input, .ant-select *, .ant-pagination *)) { max-width: 400px; }
|
||||
.stats { display: grid; grid-template-columns: repeat(4, minmax(0, 1fr)); gap: 18px; margin-bottom: 24px; }
|
||||
.stat { border: 1px solid var(--line); border-radius: 12px; background: #fff; padding: 21px; text-decoration: none; color: inherit; }
|
||||
.stat:hover { border-color: var(--blue); }
|
||||
@@ -95,7 +100,7 @@ th { font-size: 12px; font-weight: 600; color: var(--muted); background: #fafbfd
|
||||
td { padding: 15px 18px; border-top: 1px solid #eff2f7; max-width: 320px; min-width: 85px; overflow-wrap: anywhere; }
|
||||
td small { display: block; color: var(--muted); font-size: 11px; margin-top: 5px; }
|
||||
.actions { display: flex; gap: 6px; flex-wrap: wrap; min-width: 180px; }
|
||||
.actions button, .actions .button { padding: 6px 9px; font-size: 12px; }
|
||||
.actions button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *)), .actions .button { padding: 6px 9px; font-size: 12px; }
|
||||
[data-action^="delete"] { color: #a33147; }
|
||||
.badge { display: inline-block; border-radius: 5px; background: #edf2fa; color: #52647f; font-size: 11px; padding: 5px 8px; white-space: nowrap; }
|
||||
.badge.good { background: #e7f7f0; color: #107450; }
|
||||
@@ -113,9 +118,10 @@ td small { display: block; color: var(--muted); font-size: 11px; margin-top: 5px
|
||||
.csv-download-link a { font-size: 13px; text-decoration: underline; }
|
||||
#toast a { color: inherit; text-decoration: underline; font-weight: 600; }
|
||||
footer { display: flex; justify-content: space-between; margin-top: 28px; font-size: 11px; color: var(--muted); }
|
||||
dialog { width: min(880px, calc(100vw - 32px)); padding: 26px; border: 0; border-radius: 14px; box-shadow: 0 24px 90px #17243d40; color: #19283f; max-height: 90vh; }
|
||||
dialog::backdrop { background: #0f1c366e; backdrop-filter: blur(3px); }
|
||||
dialog header { display: flex; align-items: center; justify-content: space-between; gap: 16px; margin-bottom: 24px; }
|
||||
dialog { width: 100vw; max-width: none; height: 100vh; max-height: none; min-height: 100vh; margin: 0; padding: 32px max(32px, calc((100vw - 1120px) / 2)); border: 0; border-radius: 0; box-shadow: none; color: #19283f; }
|
||||
dialog::backdrop { background: #f4f7fb; }
|
||||
dialog > form { width: min(100%, 1120px); margin: 0 auto; }
|
||||
dialog header { display: flex; align-items: center; justify-content: space-between; gap: 16px; margin-bottom: 28px; padding-bottom: 18px; border-bottom: 1px solid var(--line); }
|
||||
.form-grid { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); gap: 18px; }
|
||||
.field { display: grid; align-content: start; gap: 8px; font-size: 13px; color: #52627b; min-width: 0; }
|
||||
.full, .form-grid .notice { grid-column: 1 / -1; min-width: 0; }
|
||||
@@ -124,21 +130,22 @@ dialog header { display: flex; align-items: center; justify-content: space-betwe
|
||||
.form-error { font-size: 13px; color: #a33147; white-space: pre-wrap; line-height: 1.6; }
|
||||
.form-error:empty { display: none; }
|
||||
.check { display: flex; align-items: flex-start; gap: 9px; font-size: 13px; line-height: 1.6; cursor: pointer; }
|
||||
.check input { margin: 2px 0; }
|
||||
.check input:not(:where(.ant-input, .ant-input-number-input, .ant-select *, .ant-pagination *)) { margin: 2px 0; }
|
||||
.advanced { border: 1px solid var(--line); border-radius: 8px; padding: 14px; }
|
||||
.advanced summary { cursor: pointer; color: #355cbd; font-size: 14px; font-weight: 600; }
|
||||
.advanced[open] summary { margin-bottom: 16px; }
|
||||
.json-input { font: 12px/1.7 Consolas, monospace; min-height: 100px; }
|
||||
.object-picker { max-height: 240px; overflow-y: auto; display: grid; gap: 8px; margin-top: 12px; }
|
||||
.query-form { display: grid; grid-template-columns: repeat(4, minmax(0, 1fr)); align-items: end; gap: 16px; padding: 18px; background: #fff; border: 1px solid var(--line); border-radius: 10px; margin-bottom: 22px; }
|
||||
.query-form button { justify-self: start; }
|
||||
.query-form button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *)) { justify-self: start; }
|
||||
.binlog-picker-controls { display: flex; gap: 6px; flex-wrap: wrap; }
|
||||
.binlog-picker-controls select { flex: 1; min-width: 100px; width: 0; }
|
||||
.binlog-picker-controls button { padding: 8px; font-size: 12px; }
|
||||
.binlog-picker-controls button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *)) { padding: 8px; font-size: 12px; }
|
||||
.record-detail { display: grid; grid-template-columns: minmax(100px, 1fr) minmax(0, 3fr); gap: 1px; margin: 0; background: var(--line); border: 1px solid var(--line); }
|
||||
.record-detail dt, .record-detail dd { margin: 0; padding: 12px; background: #fff; overflow-wrap: anywhere; }
|
||||
.record-detail dt { font-size: 12px; color: var(--muted); }
|
||||
#login-dialog { width: min(560px, calc(100vw - 32px)); }
|
||||
#login-dialog { width: min(560px, calc(100vw - 32px)); height: auto; min-height: 0; max-height: 90vh; margin: auto; padding: 26px; border-radius: 14px; box-shadow: 0 24px 90px #17243d40; }
|
||||
#login-dialog::backdrop { background: #0f1c366e; backdrop-filter: blur(3px); }
|
||||
#account-plan { display: grid; gap: 14px; margin-top: 20px; }
|
||||
#account-plan h3 { margin: 8px 0 0; font-size: 15px; }
|
||||
#account-plan .code { max-height: 280px; overflow: auto; white-space: pre-wrap; overflow-wrap: anywhere; }
|
||||
@@ -159,9 +166,9 @@ dialog header { display: flex; align-items: center; justify-content: space-betwe
|
||||
.replication-operations-toolbar > label { display: grid; gap: 7px; color: var(--muted); font-size: 12px; min-width: min(320px, 100%); }
|
||||
.replication-operations-toolbar .actions { justify-content: flex-end; }
|
||||
.tabs { display: flex; gap: 3px; overflow-x: auto; border-bottom: 1px solid var(--line); margin-bottom: 20px; }
|
||||
.tabs button { border: 0; border-bottom: 2px solid transparent; border-radius: 0; background: transparent; white-space: nowrap; padding: 10px 13px; }
|
||||
.tabs button:hover { border-color: #b8c9fc; }
|
||||
.tabs button[aria-selected="true"] { color: #244fb8; border-bottom-color: var(--blue); background: #edf2ff; }
|
||||
.tabs button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *)) { border: 0; border-bottom: 2px solid transparent; border-radius: 0; background: transparent; white-space: nowrap; padding: 10px 13px; }
|
||||
.tabs button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *)):hover { border-color: #b8c9fc; }
|
||||
.tabs button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *))[aria-selected="true"] { color: #244fb8; border-bottom-color: var(--blue); background: #edf2ff; }
|
||||
.replication-operations-panel { min-width: 0; }
|
||||
.replication-topology-group { background: #fff; border: 1px solid var(--line); border-radius: 8px; margin-bottom: 16px; overflow: hidden; }
|
||||
.replication-operations .section-header { display: flex; align-items: center; justify-content: space-between; gap: 14px; padding: 14px 18px; border-bottom: 1px solid var(--line); }
|
||||
@@ -228,9 +235,90 @@ dialog header { display: flex; align-items: center; justify-content: space-betwe
|
||||
.form-grid, .query-form { grid-template-columns: 1fr; }
|
||||
td, th { padding: 12px; }
|
||||
.card-header { padding: 16px; }
|
||||
dialog { padding: 18px; }
|
||||
dialog { padding: 20px 16px; }
|
||||
#login-dialog { padding: 18px; }
|
||||
.record-detail { grid-template-columns: 1fr; }
|
||||
.replication-operations-toolbar { align-items: stretch; flex-direction: column; }
|
||||
.replication-operations-toolbar .actions { justify-content: flex-start; }
|
||||
.replication-operations .section-header { align-items: flex-start; flex-direction: column; }
|
||||
}
|
||||
|
||||
/* Unified workspace layout. */
|
||||
:scope { color: #20323a; background: #f1f5f5; --muted: #677b83; --line: #dce6e8; }
|
||||
.sidebar { width: 240px; padding: 28px 16px; background: #122d35; border-right: 1px solid #26434b; }
|
||||
.brand { font-size: 27px; letter-spacing: -.8px; padding-bottom: 28px; }
|
||||
.brand-icon { color: #64d4cb; background: #23434b; border-radius: 12px; width: 44px; height: 44px; display: grid; place-items: center; }
|
||||
.sidebar-caption { color: #83a4ae; letter-spacing: 1px; }
|
||||
nav { gap: 5px; }
|
||||
nav a { padding: 11px 14px; color: #aec5cc; border: 1px solid transparent; }
|
||||
nav a:hover { background: #1c3b44; }
|
||||
nav a.active { background: #234750; border-color: #36606a; color: #8eebe2; box-shadow: inset 3px 0 #64d4cb; }
|
||||
.sidebar-footer { border-top: 1px solid #2c4850; color: #83a4ae; }
|
||||
.status-dot { background: #64d4cb; }
|
||||
main { padding: 0 30px 24px; }
|
||||
.topbar { margin: 0 -30px 24px; padding: 24px 30px; background: #fff; border-bottom: 1px solid var(--line); }
|
||||
h1 { font-size: 26px; font-weight: 700; letter-spacing: -.6px; }
|
||||
.eyebrow { text-transform: uppercase; letter-spacing: 1.5px; font-size: 10px; }
|
||||
.toolbar { gap: 8px; flex-wrap: wrap; }
|
||||
button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *)), .button { border-radius: 7px; transition: background .15s, border-color .15s; }
|
||||
button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *)):hover, .button:hover { background: #eaf5f4; border-color: #91c9c7; }
|
||||
.primary { background: var(--blue); border-color: var(--blue); color: #fff; }
|
||||
.primary:hover { background: #066772; border-color: #066772; }
|
||||
input:not(:where(.ant-input, .ant-input-number-input, .ant-select *, .ant-pagination *)):focus, select:focus, textarea:focus { outline-color: #90cfca; }
|
||||
button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *)):focus-visible, a:focus-visible, summary:focus-visible { outline-color: #70bdb7; }
|
||||
.section-toolbar { margin: 18px 0; }
|
||||
.search { padding: 12px 16px; background: #fff; border: 1px solid var(--line); border-radius: 10px; }
|
||||
.search input:not(:where(.ant-input, .ant-input-number-input, .ant-select *, .ant-pagination *)) { max-width: 480px; padding: 8px 12px; background: #f7fafa; }
|
||||
.stats { gap: 14px; }
|
||||
.stat { padding: 20px 22px; border-radius: 10px; border-top: 3px solid #60b9b1; box-shadow: 0 3px 12px #183c4405; }
|
||||
.stat-value { font-size: 34px; color: #173f47; font-variant-numeric: tabular-nums; }
|
||||
.card { border-radius: 10px; box-shadow: 0 3px 12px #183c4405; }
|
||||
.card-header { padding: 16px 20px; }
|
||||
th { background: #f3f7f7; padding: 11px 16px; color: #5d747d; }
|
||||
td { padding: 12px 16px; border-color: #e9eff0; font-variant-numeric: tabular-nums; }
|
||||
tbody tr:hover { background: #f6fbfa; }
|
||||
.badge { border-radius: 20px; padding: 5px 10px; }
|
||||
.badge.blue { background: #e1f3f2; color: #087580; }
|
||||
.query-form { border-radius: 10px; padding: 20px; gap: 16px; }
|
||||
.code { background: #f4f8f8; color: #355963; }
|
||||
.advanced summary { color: #087580; }
|
||||
.tabs button:not(:where(.ant-btn, .ant-pagination *, .ant-select *, .ant-table *))[aria-selected="true"] { color: #087580; background: #eaf6f4; }
|
||||
dialog:not(#login-dialog) { background: #f1f5f5; padding-top: 28px; }
|
||||
dialog:not(#login-dialog) > form { background: #fff; border: 1px solid var(--line); border-radius: 12px; padding: 28px; }
|
||||
dialog header { margin-bottom: 24px; }
|
||||
.dialog-actions { padding-top: 18px; border-top: 1px solid var(--line); }
|
||||
#login-dialog { border: 1px solid var(--line); border-radius: 16px; }
|
||||
#toast { background: #163b43; box-shadow: 0 8px 30px #163b4326; }
|
||||
@media (min-width: 1500px) { main { padding: 0 40px 28px; } .topbar { margin: 0 -40px 26px; padding: 26px 40px; } }
|
||||
@media (max-width: 1050px) { .sidebar { width: 190px; padding: 24px 12px; } main { padding: 0 22px 22px; } .topbar { margin: 0 -22px 22px; padding: 22px; gap: 16px; } }
|
||||
@media (max-width: 700px) { .app { flex-direction: column; } .sidebar { width: 100%; height: auto; position: static; padding: 16px; } .brand { padding-bottom: 12px; font-size: 23px; } .sidebar-caption, .sidebar-footer { display: none; } nav { display: flex; overflow-x: auto; padding-bottom: 4px; } nav a { flex: 0 0 auto; white-space: nowrap; } main { padding: 0 14px 18px; } .topbar { margin: 0 -14px 18px; padding: 18px 14px; } .toolbar { width: 100%; } .search { flex-wrap: wrap; gap: 8px; } .search input:not(:where(.ant-input, .ant-input-number-input, .ant-select *, .ant-pagination *)) { max-width: none; } dialog:not(#login-dialog) { padding: 12px; } dialog:not(#login-dialog) > form { padding: 18px; } .query-form { grid-template-columns: 1fr; } .actions { min-width: 140px; } }
|
||||
.sidebar .ant-menu { background: transparent; }
|
||||
.sidebar .ant-menu-item a { padding: 0; border: 0; }
|
||||
.sidebar .ant-menu-item-selected { background: #234750 !important; }
|
||||
.table-wrap .ant-table th, .table-wrap .ant-table td { max-width: 360px; }
|
||||
.table-wrap .ant-pagination { padding: 0 16px; }
|
||||
@media (max-width:700px) { .sidebar .ant-menu { display: flex; width: max-content; } .sidebar .ant-menu-item { width: auto; } }
|
||||
.execution-sql { white-space: pre-wrap; overflow-wrap: anywhere; min-width: 320px; margin: 0; font: 12px/1.7 Consolas, monospace; }
|
||||
|
||||
/* Light navigation theme matches the Ant Design menu. */
|
||||
.sidebar { background: #fff; color: #595959; border-right: 1px solid #f0f0f0; }
|
||||
.sidebar .brand { color: #262626; }
|
||||
.sidebar .brand-icon { color: #1677ff; background: #e6f4ff; }
|
||||
.sidebar-caption, .sidebar-footer { color: #8c8c8c; }
|
||||
.sidebar-footer { border-top-color: #f0f0f0; }
|
||||
.sidebar .ant-menu { background: #fff; }
|
||||
.sidebar .ant-menu-item a, .sidebar .ant-menu-item a:hover { color: inherit; background: transparent; box-shadow: none; }
|
||||
.sidebar .ant-menu-item-selected { background: #e6f4ff !important; }
|
||||
|
||||
/* Navigation spacing; colors and selection come from Ant Design. */
|
||||
.sidebar { padding: 24px 12px 18px; overflow-y: auto; }
|
||||
.sidebar .brand { font-size: 24px; padding: 0 12px 20px; gap: 10px; }
|
||||
.sidebar .brand-icon { width: 38px; height: 38px; font-size: 28px; }
|
||||
.sidebar-caption { padding: 0 12px 16px; font-size: 12px; letter-spacing: 0; }
|
||||
#navigation { display: block; }
|
||||
.sidebar .ant-menu-item-group-title { font-size: 11px; padding-top: 18px; }
|
||||
.sidebar .ant-menu-item { margin-block: 4px; }
|
||||
.sidebar-footer { padding: 16px 12px 0; margin-top: auto; }
|
||||
@media (max-width:700px) { #navigation { overflow-x: auto; } .sidebar .ant-menu-item-group-list { display: flex; } .sidebar .ant-menu-item-group-title { display: none; } .sidebar .ant-menu-item-group { flex-shrink: 0; } }
|
||||
|
||||
}
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func wesDictionaryTable(name string) bool {
|
||||
return name == "config_header" || name == "config_detail"
|
||||
}
|
||||
|
||||
// A dictionary selection always includes both sides of its relationship.
|
||||
func wesDictionaryObjects(objects []string) []string {
|
||||
header, detail := false, false
|
||||
for _, name := range objects {
|
||||
header = header || name == "config_header"
|
||||
detail = detail || name == "config_detail"
|
||||
}
|
||||
if header == detail {
|
||||
return objects
|
||||
}
|
||||
objects = append([]string(nil), objects...)
|
||||
if header {
|
||||
return append(objects, "config_detail")
|
||||
}
|
||||
return append(objects, "config_header")
|
||||
}
|
||||
|
||||
type wesDictionaryWarehouse struct {
|
||||
WarehouseCode string `json:"warehouseCode"`
|
||||
Header json.RawMessage `json:"header"`
|
||||
Details []json.RawMessage `json:"details"`
|
||||
}
|
||||
|
||||
type wesDictionaryDocument struct {
|
||||
RecordType string `json:"recordType"`
|
||||
Warehouses []*wesDictionaryWarehouse `json:"warehouses"`
|
||||
}
|
||||
|
||||
type wesDictionaryGroup struct {
|
||||
warehouses map[string]*wesDictionaryWarehouse
|
||||
bytes int
|
||||
}
|
||||
|
||||
type wesDictionaryExport struct {
|
||||
groups map[string]*wesDictionaryGroup
|
||||
bytes int64
|
||||
}
|
||||
|
||||
func (d *wesDictionaryExport) add(table string, record Record) error {
|
||||
recordType, ok := wesString(record["recordType"])
|
||||
if !ok || strings.TrimSpace(recordType) == "" {
|
||||
return wesRecordError(table, record, "recordType", "wesInvalidConfigRelation")
|
||||
}
|
||||
warehouse, ok := wesString(record["warehouseCode"])
|
||||
if !ok {
|
||||
return wesRecordError(table, record, "warehouseCode", "wesInvalidConfigRelation")
|
||||
}
|
||||
if !wesFileComponent(wesJSONFileName(recordType)) {
|
||||
return wesRecordError(table, record, "recordType", "wesInvalidFileName")
|
||||
}
|
||||
// The MySQL adapter returns text as strings, numbers without float conversion,
|
||||
// and SQL NULL as nil. Store encoded rows so later batches cannot mutate them.
|
||||
payload, err := json.Marshal(record)
|
||||
if err != nil {
|
||||
return wesRecordError(table, record, "row", "exportValueUnsupported")
|
||||
}
|
||||
if d.groups == nil {
|
||||
d.groups = map[string]*wesDictionaryGroup{}
|
||||
}
|
||||
group := d.groups[recordType]
|
||||
if group == nil {
|
||||
group = &wesDictionaryGroup{warehouses: map[string]*wesDictionaryWarehouse{}}
|
||||
d.groups[recordType] = group
|
||||
}
|
||||
if group.bytes+len(payload) > exportChunkBytes || d.bytes+int64(len(payload)) > exportTotalBytes {
|
||||
return exportFailure("exportSizeLimit")
|
||||
}
|
||||
item := group.warehouses[warehouse]
|
||||
if item == nil {
|
||||
item = &wesDictionaryWarehouse{WarehouseCode: warehouse, Details: []json.RawMessage{}}
|
||||
group.warehouses[warehouse] = item
|
||||
}
|
||||
if table == "config_header" {
|
||||
if item.Header != nil {
|
||||
return wesRecordError(table, record, "duplicate recordType/warehouseCode", "wesInvalidConfigRelation")
|
||||
}
|
||||
item.Header = payload
|
||||
} else {
|
||||
item.Details = append(item.Details, payload)
|
||||
}
|
||||
group.bytes += len(payload)
|
||||
d.bytes += int64(len(payload))
|
||||
return nil
|
||||
}
|
||||
|
||||
func (d *wesDictionaryExport) write(ctx context.Context, add func(string, []byte) error, manifest *exportManifest, entries map[string]int) error {
|
||||
types := make([]string, 0, len(d.groups))
|
||||
for recordType := range d.groups {
|
||||
types = append(types, recordType)
|
||||
}
|
||||
sort.Strings(types)
|
||||
for _, recordType := range types {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
group := d.groups[recordType]
|
||||
warehouses := make([]string, 0, len(group.warehouses))
|
||||
for warehouse := range group.warehouses {
|
||||
warehouses = append(warehouses, warehouse)
|
||||
}
|
||||
sort.Strings(warehouses)
|
||||
document := wesDictionaryDocument{RecordType: recordType}
|
||||
hasHeader, hasDetails := false, false
|
||||
for _, warehouse := range warehouses {
|
||||
item := group.warehouses[warehouse]
|
||||
document.Warehouses = append(document.Warehouses, item)
|
||||
hasHeader = hasHeader || item.Header != nil
|
||||
hasDetails = hasDetails || len(item.Details) > 0
|
||||
}
|
||||
payload, err := json.MarshalIndent(document, "", " ")
|
||||
if err != nil {
|
||||
return exportFailure("exportValueUnsupported")
|
||||
}
|
||||
filename := "wes/config/" + wesJSONFileName(recordType)
|
||||
if err := add(filename, append(payload, '\n')); err != nil {
|
||||
return err
|
||||
}
|
||||
for table, contains := range map[string]bool{"config_header": hasHeader, "config_detail": hasDetails} {
|
||||
if contains {
|
||||
entry := &manifest.Tables[entries[table]]
|
||||
entry.Files = append(entry.Files, filename)
|
||||
}
|
||||
}
|
||||
delete(d.groups, recordType)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,173 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func wesDictionaryFixture() *wesExportFixture {
|
||||
f := wesFixture()
|
||||
f.rows["config_header"] = []Record{
|
||||
{"id": int64(9007199254740993), "recordType": "STATUS", "warehouseCode": "*", "description": "全局状态"},
|
||||
{"id": int64(2), "recordType": "STATUS", "warehouseCode": "WH1", "description": "仓库状态"},
|
||||
{"id": int64(3), "recordType": "EMPTY", "warehouseCode": "WH1", "description": nil},
|
||||
}
|
||||
f.rows["config_detail"] = []Record{
|
||||
// Matching uses recordType and warehouseCode, even if headerId is stale.
|
||||
{"id": int64(10), "headerId": int64(999), "recordType": "STATUS", "warehouseCode": "*", "identifier": "OPEN", "value1": nil},
|
||||
{"id": int64(11), "headerId": nil, "recordType": "STATUS", "warehouseCode": "WH1", "identifier": "CLOSED", "value1": ""},
|
||||
{"id": int64(12), "headerId": int64(3), "recordType": "ORPHAN", "warehouseCode": "WH2", "identifier": "KEEP", "value1": "第一行\n第二行\t\"quoted\""},
|
||||
}
|
||||
for _, name := range []string{"config_header", "config_detail"} {
|
||||
info := ObjectInfo{Name: name, Type: "table", CreateSQL: "CREATE TABLE " + QuoteIdentifier(name) + " (`id` BIGINT)"}
|
||||
for _, column := range []string{"id", "recordType", "warehouseCode", "description", "headerId", "identifier", "value1"} {
|
||||
if _, ok := f.rows[name][0][column]; ok {
|
||||
info.Columns = append(info.Columns, ColumnInfo{Name: column, Type: "text"})
|
||||
}
|
||||
}
|
||||
f.objects[name] = info
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
func TestWESDictionaryExportAssociatesRecordTypeWithinWarehouse(t *testing.T) {
|
||||
for _, selected := range []string{"config_detail", "config_header"} {
|
||||
t.Run(selected, func(t *testing.T) {
|
||||
s := newCSVJobTestServer(t)
|
||||
f := wesDictionaryFixture()
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) { return f, nil }
|
||||
job := wesStart(t, s, fmt.Sprintf(`{"source_id":"source","database":"wes_db","objects":[%q],"max_rows":6}`, selected))
|
||||
if job.Status != "COMPLETED" || job.Rows != 6 || len(job.Objects) != 2 || len(f.read) != 2 {
|
||||
t.Fatalf("pair not completely exported: %+v, reads=%v", job, f.read)
|
||||
}
|
||||
files := wesReadPackage(t, s, job)
|
||||
for _, name := range []string{"config_header", "config_detail"} {
|
||||
if _, exists := files["wes/data/"+name+".csv"]; exists {
|
||||
t.Fatal("related dictionary exported as CSV")
|
||||
}
|
||||
}
|
||||
var document wesDictionaryDocument
|
||||
if err := json.Unmarshal(files["wes/config/STATUS.json"], &document); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if document.RecordType != "STATUS" || len(document.Warehouses) != 2 {
|
||||
t.Fatalf("incorrect grouping: %+v", document)
|
||||
}
|
||||
decode := func(raw json.RawMessage) Record {
|
||||
t.Helper()
|
||||
var row Record
|
||||
decoder := json.NewDecoder(bytes.NewReader(raw))
|
||||
decoder.UseNumber()
|
||||
if err := decoder.Decode(&row); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return row
|
||||
}
|
||||
global, local := document.Warehouses[0], document.Warehouses[1]
|
||||
if global.WarehouseCode != "*" || local.WarehouseCode != "WH1" || len(global.Details) != 1 || len(local.Details) != 1 {
|
||||
t.Fatal("warehouse configurations were mixed")
|
||||
}
|
||||
if header := decode(global.Header); header["id"] != json.Number("9007199254740993") || header["description"] != "全局状态" {
|
||||
t.Fatalf("header changed: %v", header)
|
||||
}
|
||||
if detail := decode(global.Details[0]); detail["headerId"] != json.Number("999") || detail["identifier"] != "OPEN" || detail["value1"] != nil {
|
||||
t.Fatalf("detail changed or associated by headerId: %v", detail)
|
||||
}
|
||||
if detail := decode(local.Details[0]); detail["identifier"] != "CLOSED" || detail["value1"] != "" {
|
||||
t.Fatalf("local detail changed: %v", detail)
|
||||
}
|
||||
for _, recordType := range []string{"EMPTY", "ORPHAN"} {
|
||||
var other wesDictionaryDocument
|
||||
if json.Unmarshal(files["wes/config/"+recordType+".json"], &other) != nil || len(other.Warehouses) != 1 {
|
||||
t.Fatalf("missing unmatched rows for %s", recordType)
|
||||
}
|
||||
item := other.Warehouses[0]
|
||||
if recordType == "EMPTY" && (string(item.Header) == "null" || len(item.Details) != 0 || item.Details == nil) {
|
||||
t.Fatal("header without details not retained")
|
||||
}
|
||||
if recordType == "ORPHAN" && (string(item.Header) != "null" || len(item.Details) != 1 || decode(item.Details[0])["value1"] != f.rows["config_detail"][2]["value1"]) {
|
||||
t.Fatal("orphan detail lost or attached to wrong header")
|
||||
}
|
||||
}
|
||||
var manifest exportManifest
|
||||
if json.Unmarshal(files["manifest.json"], &manifest) != nil || manifest.Rows != 6 {
|
||||
t.Fatal("incorrect manifest")
|
||||
}
|
||||
for _, entry := range manifest.Tables {
|
||||
if wesDictionaryTable(entry.Name) {
|
||||
extra := "EMPTY"
|
||||
if entry.Name == "config_detail" {
|
||||
extra = "ORPHAN"
|
||||
}
|
||||
if entry.Rows != 3 || !reflect.DeepEqual(entry.Files, []string{"schema/" + entry.Name + ".sql", "wes/config/" + extra + ".json", "wes/config/STATUS.json"}) {
|
||||
t.Fatalf("incorrect table manifest: %+v", entry)
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(manifest.Checksums) != len(files)-1 {
|
||||
t.Fatal("missing dictionary checksums")
|
||||
}
|
||||
for name, checksum := range manifest.Checksums {
|
||||
hash := sha256.Sum256(files[name])
|
||||
if hex.EncodeToString(hash[:]) != checksum {
|
||||
t.Fatalf("checksum mismatch for %s", name)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestWESDictionaryExportRejectsIncompleteOrAmbiguousData(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name, code string
|
||||
change func(*wesExportFixture)
|
||||
}{
|
||||
{"missing related table", "invalidExportObjects", func(f *wesExportFixture) { delete(f.objects, "config_header") }},
|
||||
{"missing recordType column", "exportSchemaUnavailable", func(f *wesExportFixture) {
|
||||
info := f.objects["config_detail"]
|
||||
info.Columns = []ColumnInfo{{Name: "warehouseCode"}}
|
||||
f.objects["config_detail"] = info
|
||||
}},
|
||||
{"null recordType", "wesInvalidConfigRelation", func(f *wesExportFixture) { f.rows["config_detail"][0]["recordType"] = nil }},
|
||||
{"null warehouse", "wesInvalidConfigRelation", func(f *wesExportFixture) { f.rows["config_header"][0]["warehouseCode"] = nil }},
|
||||
{"duplicate header", "wesInvalidConfigRelation", func(f *wesExportFixture) {
|
||||
f.rows["config_header"] = append(f.rows["config_header"], f.rows["config_header"][0])
|
||||
}},
|
||||
{"unsafe name", "wesInvalidFileName", func(f *wesExportFixture) { f.rows["config_detail"][0]["recordType"] = "../escape" }},
|
||||
{"case collision", "wesDuplicateFile", func(f *wesExportFixture) { f.rows["config_detail"][0]["recordType"] = "status" }},
|
||||
{"size limit", "exportSizeLimit", func(f *wesExportFixture) {
|
||||
f.rows["config_detail"][0]["value1"] = strings.Repeat("x", exportChunkBytes)
|
||||
}},
|
||||
{"row limit", "wesRowLimitReached", func(f *wesExportFixture) {}},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
s := newCSVJobTestServer(t)
|
||||
f := wesDictionaryFixture()
|
||||
test.change(f)
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) { return f, nil }
|
||||
limit := 20
|
||||
if test.name == "row limit" {
|
||||
limit = 5
|
||||
}
|
||||
job := wesStart(t, s, fmt.Sprintf(`{"source_id":"source","database":"wes_db","objects":["config_detail"],"max_rows":%d}`, limit))
|
||||
if job.Status != "FAILED" || job.Error != test.code || job.File != "" {
|
||||
t.Fatalf("invalid dictionary was published: %+v", job)
|
||||
}
|
||||
if test.code == "wesInvalidConfigRelation" && (!strings.Contains(job.ErrorDetail, "recordType=") || !strings.Contains(job.ErrorDetail, "warehouseCode=")) {
|
||||
t.Fatalf("missing relation error context: %s", job.ErrorDetail)
|
||||
}
|
||||
files, err := os.ReadDir(s.artifactDir)
|
||||
if err != nil || len(files) != 0 {
|
||||
t.Fatalf("partial archive retained: %v %v", files, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,427 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"encoding/csv"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// WES loads the structure JSON itself, not the database row containing it.
|
||||
var wesConfigFolders = map[string]string{
|
||||
"ttx_table_model": "table", "ttx_bill_model": "bill",
|
||||
"ttx_wso_data": "view", "ttx_navigator": "navigator",
|
||||
}
|
||||
|
||||
// File names must survive extraction on Windows as well as Unix. Reject
|
||||
// ambiguous identities instead of renaming them and changing WES lookups.
|
||||
func wesFileComponent(name string) bool {
|
||||
if name == "" || len(name) > 255 || !utf8.ValidString(name) || strings.ContainsAny(name, `/\:*?"<>|`) || strings.HasSuffix(name, ".") || strings.HasSuffix(name, " ") || strings.ContainsFunc(name, unicode.IsControl) {
|
||||
return false
|
||||
}
|
||||
base := strings.ToUpper(strings.SplitN(name, ".", 2)[0])
|
||||
switch base {
|
||||
case "CON", "PRN", "AUX", "NUL", "CONIN$", "CONOUT$":
|
||||
return false
|
||||
}
|
||||
if len([]rune(base)) == 4 && (strings.HasPrefix(base, "COM") || strings.HasPrefix(base, "LPT")) && strings.ContainsRune("123456789¹²³", []rune(base)[3]) {
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func wesString(value any) (string, bool) {
|
||||
switch value := value.(type) {
|
||||
case string:
|
||||
return value, utf8.ValidString(value)
|
||||
case []byte:
|
||||
return string(value), utf8.Valid(value)
|
||||
default:
|
||||
return "", false
|
||||
}
|
||||
}
|
||||
|
||||
// WES permits wildcard identities such as ClearRedis*. Encode wildcard and
|
||||
// percent characters in the file name, keeping the JSON identity unchanged.
|
||||
func wesJSONFileName(identity string) string {
|
||||
return strings.NewReplacer("%", "%25", "*", "%2A", "?", "%3F").Replace(identity) + ".json"
|
||||
}
|
||||
|
||||
type wesConfigError struct {
|
||||
code exportFailure
|
||||
detail string
|
||||
}
|
||||
|
||||
func (err *wesConfigError) Error() string { return string(err.code) + ": " + err.detail }
|
||||
func (err *wesConfigError) Unwrap() error { return err.code }
|
||||
|
||||
func wesRecordError(table string, record Record, field, code string) error {
|
||||
keys := []string{"code"}
|
||||
if table == "ttx_wso_data" {
|
||||
keys = []string{"tid", "oid"}
|
||||
} else if wesDictionaryTable(table) {
|
||||
keys = []string{"recordType", "warehouseCode"}
|
||||
} else if kind, _ := wesMongoCollection(table); kind != "" {
|
||||
keys = []string{"_id", "path"}
|
||||
}
|
||||
var identity []string
|
||||
for _, key := range keys {
|
||||
value, _ := wesString(record[key])
|
||||
if key == "_id" && record[key] != nil {
|
||||
value = scalarText(record[key])
|
||||
}
|
||||
runes := []rune(value)
|
||||
if len(runes) > 128 {
|
||||
value = string(runes[:128]) + "…"
|
||||
}
|
||||
identity = append(identity, fmt.Sprintf("%s=%q", key, value))
|
||||
}
|
||||
return &wesConfigError{code: exportFailure(code), detail: table + " (" + strings.Join(identity, ", ") + "): " + field}
|
||||
}
|
||||
|
||||
// Legacy WES descriptions can contain literal tabs inside JSON strings.
|
||||
// Remove those tabs for export; leave valid escapes, whitespace outside strings,
|
||||
// and malformed escape sequences untouched so normal validation still applies.
|
||||
func wesRemoveLiteralTabs(raw []byte) []byte {
|
||||
if !bytes.ContainsRune(raw, '\t') {
|
||||
return raw
|
||||
}
|
||||
cleaned := make([]byte, 0, len(raw))
|
||||
inString, escaped := false, false
|
||||
for _, ch := range raw {
|
||||
if inString && !escaped && ch == '\t' {
|
||||
continue
|
||||
}
|
||||
cleaned = append(cleaned, ch)
|
||||
if escaped {
|
||||
escaped = false
|
||||
} else if inString && ch == '\\' {
|
||||
escaped = true
|
||||
} else if ch == '"' {
|
||||
inString = !inString
|
||||
}
|
||||
}
|
||||
return cleaned
|
||||
}
|
||||
|
||||
func wesConfigFile(table string, record Record) (string, []byte, error) {
|
||||
failure := func(field string) (string, []byte, error) {
|
||||
return "", nil, wesRecordError(table, record, field, "wesInvalidStructure")
|
||||
}
|
||||
var raw []byte
|
||||
switch value := record["structure"].(type) {
|
||||
case string:
|
||||
raw = []byte(value)
|
||||
case []byte:
|
||||
raw = value
|
||||
default:
|
||||
var err error
|
||||
raw, err = json.Marshal(value)
|
||||
if err != nil {
|
||||
return failure("structure")
|
||||
}
|
||||
}
|
||||
if !utf8.Valid(raw) || len(raw) > exportChunkBytes {
|
||||
return failure("structure")
|
||||
}
|
||||
raw = wesRemoveLiteralTabs(raw)
|
||||
decoder := json.NewDecoder(bytes.NewReader(raw))
|
||||
decoder.UseNumber()
|
||||
var document any
|
||||
if decoder.Decode(&document) != nil || decoder.Decode(new(any)) != io.EOF {
|
||||
return failure("structure")
|
||||
}
|
||||
object, isObject := document.(map[string]any)
|
||||
if table == "ttx_navigator" {
|
||||
if _, ok := document.([]any); !ok {
|
||||
return failure("structure (array)")
|
||||
}
|
||||
} else if !isObject {
|
||||
return failure("structure (object)")
|
||||
}
|
||||
name, ok := wesString(record["code"])
|
||||
if table == "ttx_wso_data" {
|
||||
tid, tidOK := wesString(record["tid"])
|
||||
oid, oidOK := wesString(record["oid"])
|
||||
if !tidOK || object["tid"] != tid {
|
||||
return failure("tid")
|
||||
}
|
||||
if !oidOK || object["oid"] != oid {
|
||||
return failure("oid")
|
||||
}
|
||||
filename := wesJSONFileName(oid)
|
||||
if !wesFileComponent(tid) || !wesFileComponent(filename) {
|
||||
return "", nil, wesRecordError(table, record, "tid/oid", "wesInvalidFileName")
|
||||
}
|
||||
name = "wes/view/" + tid + "/" + filename
|
||||
} else {
|
||||
if !ok || isObject && object["code"] != name {
|
||||
return failure("code")
|
||||
}
|
||||
filename := wesJSONFileName(name)
|
||||
if !wesFileComponent(filename) {
|
||||
return "", nil, wesRecordError(table, record, "code", "wesInvalidFileName")
|
||||
}
|
||||
name = "wes/" + wesConfigFolders[table] + "/" + filename
|
||||
}
|
||||
var formatted bytes.Buffer
|
||||
if json.Indent(&formatted, raw, "", " ") != nil {
|
||||
return failure("structure")
|
||||
}
|
||||
formatted.WriteByte('\n')
|
||||
return name, formatted.Bytes(), nil
|
||||
}
|
||||
|
||||
func (s *server) runWESExport(ctx context.Context, source DataSource, job Export) {
|
||||
adapter, err := s.connect(ctx, source, s.recordAudit)
|
||||
var rows int64
|
||||
if err == nil {
|
||||
defer adapter.Close(context.Background())
|
||||
rows, err = s.writeWESExport(ctx, source, job, adapter)
|
||||
}
|
||||
s.store.RLock()
|
||||
job.Warnings = append([]string(nil), s.store.Exports[job.ID].Warnings...)
|
||||
s.store.RUnlock()
|
||||
s.finishExport(ctx, job, rows, err)
|
||||
}
|
||||
|
||||
func (s *server) writeWESExport(ctx context.Context, source DataSource, job Export, adapter Adapter) (int64, error) {
|
||||
if strings.EqualFold(source.Type, "mongodb") {
|
||||
return s.writeWESMongoExport(ctx, source, job, adapter)
|
||||
}
|
||||
objects, err := adapter.ListObjects(ctx)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
available := map[string]bool{}
|
||||
var tables []string
|
||||
for _, object := range objects {
|
||||
if exportIsTable(object) && !available[object.Name] {
|
||||
available[object.Name] = true
|
||||
tables = append(tables, object.Name)
|
||||
}
|
||||
}
|
||||
for _, name := range job.Objects {
|
||||
if !available[name] {
|
||||
return 0, exportFailure("invalidExportObjects")
|
||||
}
|
||||
}
|
||||
sort.Strings(tables)
|
||||
if err := os.MkdirAll(s.artifactDir, 0700); err != nil {
|
||||
return 0, exportFailure("exportArtifactFailed")
|
||||
}
|
||||
path, err := s.sqlExportPath(job.ID, true)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
file, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600)
|
||||
if err != nil {
|
||||
return 0, exportFailure("exportArtifactFailed")
|
||||
}
|
||||
defer file.Close()
|
||||
compressed := gzip.NewWriter(exportContextWriter{ctx: ctx, writer: file})
|
||||
defer compressed.Close()
|
||||
tw := tar.NewWriter(compressed)
|
||||
defer tw.Close()
|
||||
archive := exportArchive{ctx: ctx, writer: tw, created: job.CreatedAt}
|
||||
seen := map[string]bool{}
|
||||
add := func(name string, data []byte) error {
|
||||
key := strings.ToLower(name)
|
||||
if seen[key] {
|
||||
return exportFailure("wesDuplicateFile")
|
||||
}
|
||||
seen[key] = true
|
||||
return archive.add(name, data)
|
||||
}
|
||||
manifest := exportManifest{Version: 1, ID: job.ID, SourceID: job.SourceID, Mode: "wes", CreatedAt: job.CreatedAt, MaxRows: job.MaxRows, ProductionMasking: exportProduction(source)}
|
||||
inspected := map[string]ObjectInfo{}
|
||||
entries := map[string]int{}
|
||||
// Include physical DDL for the whole database, even for business tables
|
||||
// whose data was not selected for the configuration package.
|
||||
for _, name := range tables {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if !wesFileComponent(name + ".sql") {
|
||||
return 0, exportFailure("wesInvalidFileName")
|
||||
}
|
||||
info, err := adapter.Inspect(ctx, name)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if info.Name != name || !exportIsTable(info) || strings.TrimSpace(info.CreateSQL) == "" || len(info.Columns) == 0 {
|
||||
return 0, exportFailure("exportSchemaUnavailable")
|
||||
}
|
||||
filename := "schema/" + name + ".sql"
|
||||
if err := add(filename, []byte(strings.TrimRight(strings.TrimSpace(info.CreateSQL), ";")+";\n")); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
inspected[name] = info
|
||||
entries[name] = len(manifest.Tables)
|
||||
manifest.Tables = append(manifest.Tables, exportManifestTable{Name: name, Files: []string{filename}})
|
||||
}
|
||||
dictionary := &wesDictionaryExport{}
|
||||
for _, name := range job.Objects {
|
||||
entry := &manifest.Tables[entries[name]]
|
||||
columns := exportWritableColumns(inspected[name])
|
||||
related := wesDictionaryTable(name)
|
||||
if wesConfigFolders[name] != "" || related {
|
||||
required := []string{"code", "structure"}
|
||||
if related {
|
||||
required = []string{"recordType", "warehouseCode"}
|
||||
} else if name == "ttx_wso_data" {
|
||||
required = []string{"tid", "oid", "structure"}
|
||||
}
|
||||
for _, field := range required {
|
||||
found := false
|
||||
for _, column := range columns {
|
||||
found = found || column.Name == field
|
||||
}
|
||||
if !found {
|
||||
return manifest.Rows, exportFailure("exportSchemaUnavailable")
|
||||
}
|
||||
}
|
||||
}
|
||||
var data bytes.Buffer
|
||||
csvWriter := csv.NewWriter(&data)
|
||||
config := wesConfigFolders[name] != ""
|
||||
if !config && !related && name != "ttx_system" {
|
||||
header := make([]string, len(columns))
|
||||
for i, column := range columns {
|
||||
header[i] = column.Name
|
||||
}
|
||||
if err := csvWriter.Write(header); err != nil {
|
||||
return manifest.Rows, err
|
||||
}
|
||||
}
|
||||
var consumeErr error
|
||||
err := adapter.Read(ctx, name, ReadOptions{BatchSize: job.BatchSize, MaxRows: job.MaxRows - manifest.Rows + 1}, func(records []Record) (callbackErr error) {
|
||||
defer func() {
|
||||
if callbackErr != nil {
|
||||
consumeErr = callbackErr
|
||||
}
|
||||
}()
|
||||
for _, record := range records {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
if manifest.Rows >= job.MaxRows {
|
||||
return exportFailure("wesRowLimitReached")
|
||||
}
|
||||
if exportProduction(source) {
|
||||
record = automaticMask(record)
|
||||
}
|
||||
if related {
|
||||
if err := dictionary.add(name, record); err != nil {
|
||||
return err
|
||||
}
|
||||
} else if config {
|
||||
filename, payload, err := wesConfigFile(name, record)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if err := add(filename, payload); err != nil {
|
||||
if err == exportFailure("wesDuplicateFile") {
|
||||
return wesRecordError(name, record, filename, "wesDuplicateFile")
|
||||
}
|
||||
return err
|
||||
}
|
||||
entry.Files = append(entry.Files, filename)
|
||||
} else if name == "ttx_system" {
|
||||
statement, err := exportInsertSQL(name, columns, record)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
data.WriteString(statement)
|
||||
} else {
|
||||
values := make([]string, len(columns))
|
||||
for i, column := range columns {
|
||||
value, exists := record[column.Name]
|
||||
if !exists {
|
||||
return exportFailure("exportValueUnsupported")
|
||||
}
|
||||
if value != nil {
|
||||
values[i] = scalarText(value)
|
||||
switch value.(type) {
|
||||
case map[string]any, Record, []any:
|
||||
encoded, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
return exportFailure("exportValueUnsupported")
|
||||
}
|
||||
values[i] = string(encoded)
|
||||
}
|
||||
if !utf8.ValidString(values[i]) || strings.ContainsRune(values[i], 0) {
|
||||
return exportFailure("exportValueUnsupported")
|
||||
}
|
||||
}
|
||||
}
|
||||
if err := csvWriter.Write(values); err != nil {
|
||||
return err
|
||||
}
|
||||
csvWriter.Flush()
|
||||
if err := csvWriter.Error(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if data.Len() > exportChunkBytes {
|
||||
return exportFailure("exportSizeLimit")
|
||||
}
|
||||
manifest.Rows++
|
||||
entry.Rows++
|
||||
}
|
||||
s.store.Lock()
|
||||
if current, exists := s.store.Exports[job.ID]; exists {
|
||||
current.Rows = manifest.Rows
|
||||
s.store.Exports[job.ID] = current
|
||||
}
|
||||
s.store.Unlock()
|
||||
return nil
|
||||
})
|
||||
if consumeErr != nil {
|
||||
err = consumeErr
|
||||
}
|
||||
if err != nil {
|
||||
return manifest.Rows, err
|
||||
}
|
||||
if !config && !related {
|
||||
csvWriter.Flush()
|
||||
if err := csvWriter.Error(); err != nil {
|
||||
return manifest.Rows, err
|
||||
}
|
||||
filename := "wes/data/" + name + ".csv"
|
||||
if name == "ttx_system" {
|
||||
filename = "wes/sql/ttx_system.sql"
|
||||
}
|
||||
if err := add(filename, data.Bytes()); err != nil {
|
||||
return manifest.Rows, err
|
||||
}
|
||||
entry.Files = append(entry.Files, filename)
|
||||
}
|
||||
}
|
||||
if err := dictionary.write(ctx, add, &manifest, entries); err != nil {
|
||||
return manifest.Rows, err
|
||||
}
|
||||
manifest.Checksums = archive.checksums
|
||||
data, err := json.MarshalIndent(manifest, "", " ")
|
||||
if err != nil {
|
||||
return manifest.Rows, err
|
||||
}
|
||||
if err := archive.add("manifest.json", data); err != nil {
|
||||
return manifest.Rows, err
|
||||
}
|
||||
for _, close := range []func() error{tw.Close, compressed.Close, file.Sync, file.Close} {
|
||||
if err := close(); err != nil {
|
||||
return manifest.Rows, exportFailure("exportArtifactFailed")
|
||||
}
|
||||
}
|
||||
return manifest.Rows, ctx.Err()
|
||||
}
|
||||
@@ -0,0 +1,315 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/csv"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type wesExportFixture struct {
|
||||
Adapter
|
||||
objects map[string]ObjectInfo
|
||||
rows map[string][]Record
|
||||
read []string
|
||||
}
|
||||
|
||||
func (f *wesExportFixture) ListObjects(context.Context) ([]ObjectInfo, error) {
|
||||
var result []ObjectInfo
|
||||
for _, object := range f.objects {
|
||||
result = append(result, object)
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
func (f *wesExportFixture) Inspect(_ context.Context, name string) (ObjectInfo, error) {
|
||||
object, ok := f.objects[name]
|
||||
if !ok {
|
||||
return ObjectInfo{}, errors.New("notFound")
|
||||
}
|
||||
return object, nil
|
||||
}
|
||||
func (f *wesExportFixture) Close(context.Context) error { return nil }
|
||||
func (f *wesExportFixture) Read(ctx context.Context, name string, options ReadOptions, consume func([]Record) error) error {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
f.read = append(f.read, name)
|
||||
rows := f.rows[name]
|
||||
if options.MaxRows > 0 && int64(len(rows)) > options.MaxRows {
|
||||
rows = rows[:options.MaxRows]
|
||||
}
|
||||
return consume(rows)
|
||||
}
|
||||
|
||||
func wesFixture() *wesExportFixture {
|
||||
f := &wesExportFixture{objects: map[string]ObjectInfo{}, rows: map[string][]Record{
|
||||
"ttx_table_model": {{"code": "orders", "structure": "{\"code\":\"orders\",\"fields\":[{\"name\":\"订\t单\",\"default\":9007199254740993}]}"}},
|
||||
"ttx_bill_model": {{"code": "orders", "structure": `{"code":"orders","header":"orders"}`}},
|
||||
"ttx_wso_data": {{"tid": "ttx.wso.Bill", "oid": "orders", "structure": []byte(`{"tid":"ttx.wso.Bill","oid":"orders","vm":{"title":"订单"}}`)}},
|
||||
"ttx_navigator": {{"code": "admin", "structure": `[{"id":"orders","tid":"ttx.wso.Bill","oid":"orders"}]`}},
|
||||
"settings": {{"code": "a", "value": "a,\"b\"\n第二行", "number": json.Number("9007199254740993")}},
|
||||
"ttx_system": {{"code": "password_regex", "value": `^\d+'$`}},
|
||||
"orders": {{"id": 1}},
|
||||
}}
|
||||
for name, rows := range f.rows {
|
||||
info := ObjectInfo{Name: name, Type: "table", CreateSQL: "CREATE TABLE " + QuoteIdentifier(name) + " (`id` BIGINT)"}
|
||||
// A fixed column order makes the expected CSV independent of map iteration.
|
||||
for _, column := range []string{"id", "code", "tid", "oid", "structure", "value", "number"} {
|
||||
if _, ok := rows[0][column]; ok {
|
||||
info.Columns = append(info.Columns, ColumnInfo{Name: column, Type: "text"})
|
||||
}
|
||||
}
|
||||
f.objects[name] = info
|
||||
}
|
||||
f.objects["orders_view"] = ObjectInfo{Name: "orders_view", Type: "view"}
|
||||
return f
|
||||
}
|
||||
|
||||
func wesStart(t *testing.T, s *server, body string) Export {
|
||||
t.Helper()
|
||||
response := httptest.NewRecorder()
|
||||
request := httptest.NewRequest(http.MethodPost, "http://127.0.0.1/api/exports/wes", strings.NewReader(body))
|
||||
request.RemoteAddr = "127.0.0.1:12345"
|
||||
s.handler().ServeHTTP(response, request)
|
||||
var job Export
|
||||
if response.Code != http.StatusAccepted || json.Unmarshal(response.Body.Bytes(), &job) != nil {
|
||||
t.Fatalf("export not accepted: %d %s", response.Code, response.Body.String())
|
||||
}
|
||||
s.workers.Wait()
|
||||
return csvJobSnapshot(s, job.ID)
|
||||
}
|
||||
|
||||
func TestWESExportPackageAndWholeDatabaseDDL(t *testing.T) {
|
||||
s := newCSVJobTestServer(t)
|
||||
f := wesFixture()
|
||||
s.adapterFactory = func(_ context.Context, source DataSource, _ func(SQLAudit)) (Adapter, error) {
|
||||
if source.Database != "wes_db" {
|
||||
t.Errorf("wrong database: %q", source.Database)
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
job := wesStart(t, s, `{"source_id":"source","database":"wes_db","objects":["ttx_table_model","ttx_bill_model","ttx_wso_data","ttx_navigator","settings","ttx_system"],"max_rows":6}`)
|
||||
if job.Status != "COMPLETED" || job.Mode != "wes" || job.Rows != 6 {
|
||||
t.Fatalf("unexpected job: %+v", job)
|
||||
}
|
||||
response := httptest.NewRecorder()
|
||||
request := httptest.NewRequest(http.MethodGet, "http://127.0.0.1/api/exports/"+job.ID+"/download", nil)
|
||||
request.RemoteAddr = "127.0.0.1:12345"
|
||||
s.handler().ServeHTTP(response, request)
|
||||
if response.Code != 200 || response.Header().Get("Content-Type") != "application/gzip" {
|
||||
t.Fatal("package not downloadable")
|
||||
}
|
||||
zr, err := gzip.NewReader(bytes.NewReader(response.Body.Bytes()))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer zr.Close()
|
||||
tr := tar.NewReader(zr)
|
||||
files := map[string][]byte{}
|
||||
for {
|
||||
header, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, exists := files[header.Name]; exists {
|
||||
t.Fatal("duplicate archive path")
|
||||
}
|
||||
files[header.Name], err = io.ReadAll(tr)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
for _, name := range []string{"wes/table/orders.json", "wes/bill/orders.json", "wes/view/ttx.wso.Bill/orders.json", "wes/navigator/admin.json"} {
|
||||
if !json.Valid(files[name]) {
|
||||
t.Fatalf("missing/invalid config %s", name)
|
||||
}
|
||||
}
|
||||
if !bytes.Contains(files["wes/table/orders.json"], []byte("9007199254740993")) || !bytes.Contains(files["wes/table/orders.json"], []byte("订单")) {
|
||||
t.Fatal("configuration values changed")
|
||||
}
|
||||
for name, info := range f.objects {
|
||||
if info.Type == "table" && string(files["schema/"+name+".sql"]) != info.CreateSQL+";\n" {
|
||||
t.Fatalf("DDL missing for %s", name)
|
||||
}
|
||||
}
|
||||
if _, exists := files["schema/orders_view.sql"]; exists {
|
||||
t.Fatal("view exported as table")
|
||||
}
|
||||
for _, name := range f.read {
|
||||
if name == "orders" {
|
||||
t.Fatal("business table data read for schema-only export")
|
||||
}
|
||||
}
|
||||
rows, err := csv.NewReader(bytes.NewReader(files["wes/data/settings.csv"])).ReadAll()
|
||||
if err != nil || !reflect.DeepEqual(rows, [][]string{{"code", "value", "number"}, {"a", "a,\"b\"\n第二行", "9007199254740993"}}) {
|
||||
t.Fatalf("CSV changed data: %v %v", rows, err)
|
||||
}
|
||||
if !strings.HasPrefix(string(files["wes/sql/ttx_system.sql"]), "INSERT INTO `ttx_system`") {
|
||||
t.Fatal("missing system SQL")
|
||||
}
|
||||
var manifest exportManifest
|
||||
if json.Unmarshal(files["manifest.json"], &manifest) != nil || manifest.Rows != 6 || manifest.LimitReached || manifest.ConsistentSnapshot || len(manifest.Tables) != 7 {
|
||||
t.Fatal("incorrect manifest")
|
||||
}
|
||||
for name, expected := range manifest.Checksums {
|
||||
checksum := sha256.Sum256(files[name])
|
||||
if expected != hex.EncodeToString(checksum[:]) {
|
||||
t.Fatalf("invalid checksum for %s", name)
|
||||
}
|
||||
}
|
||||
if len(manifest.Checksums) != len(files)-1 {
|
||||
t.Fatal("manifest omitted files")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWESExportNeverPublishesIncompletePackage(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name, code string
|
||||
change func(*wesExportFixture)
|
||||
}{
|
||||
{"row limit", "wesRowLimitReached", func(f *wesExportFixture) {
|
||||
f.rows["ttx_table_model"] = append(f.rows["ttx_table_model"], Record{"code": "second", "structure": `{"code":"second"}`})
|
||||
}},
|
||||
{"malformed JSON", "wesInvalidStructure", func(f *wesExportFixture) { f.rows["ttx_table_model"][0]["structure"] = "not JSON" }},
|
||||
{"identity mismatch", "wesInvalidStructure", func(f *wesExportFixture) { f.rows["ttx_table_model"][0]["structure"] = `{"code":"wrong"}` }},
|
||||
{"traversal", "wesInvalidFileName", func(f *wesExportFixture) {
|
||||
f.rows["ttx_table_model"][0] = Record{"code": "../escape", "structure": `{"code":"../escape"}`}
|
||||
}},
|
||||
{"case collision", "wesDuplicateFile", func(f *wesExportFixture) {
|
||||
f.rows["ttx_table_model"] = append(f.rows["ttx_table_model"], Record{"code": "ORDERS", "structure": `{"code":"ORDERS"}`})
|
||||
}},
|
||||
{"DDL unavailable", "exportSchemaUnavailable", func(f *wesExportFixture) {
|
||||
info := f.objects["orders"]
|
||||
info.CreateSQL = ""
|
||||
f.objects["orders"] = info
|
||||
}},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
s := newCSVJobTestServer(t)
|
||||
f := wesFixture()
|
||||
test.change(f)
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) { return f, nil }
|
||||
limit := 10
|
||||
if test.name == "row limit" {
|
||||
limit = 1
|
||||
}
|
||||
body, _ := json.Marshal(map[string]any{"source_id": "source", "database": "wes_db", "objects": []string{"ttx_table_model"}, "max_rows": limit})
|
||||
job := wesStart(t, s, string(body))
|
||||
if job.Status != "FAILED" || job.Error != test.code || job.File != "" {
|
||||
t.Fatalf("unexpected job: %+v", job)
|
||||
}
|
||||
if test.code == "wesInvalidStructure" && (!strings.Contains(job.ErrorDetail, `ttx_table_model (code="orders")`) || !strings.HasSuffix(job.ErrorDetail, map[string]string{"malformed JSON": ": structure", "identity mismatch": ": code"}[test.name])) {
|
||||
t.Fatalf("missing configuration error context: %q", job.ErrorDetail)
|
||||
}
|
||||
files, err := os.ReadDir(s.artifactDir)
|
||||
if err != nil || len(files) != 0 {
|
||||
t.Fatalf("incomplete files retained: %v %v", files, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestWESExportValidatesRequestsAndCancellation(t *testing.T) {
|
||||
s := newCSVJobTestServer(t)
|
||||
for _, test := range []struct{ path, body string }{
|
||||
{"/api/exports/sql", `{"source_id":"source","database":"wes_db","mode":"wes","objects":["ttx_table_model"]}`},
|
||||
{"/api/exports/wes", `{"source_id":"source","database":"wes_db","mode":"data","objects":["ttx_table_model"]}`},
|
||||
{"/api/exports/wes", `{"source_id":"source","database":"wes_db","objects":["ttx_table_model"],"filters":[{"field":"code","operator":"eq","value":"orders"}]}`},
|
||||
} {
|
||||
response := httptest.NewRecorder()
|
||||
s.createExport(response, httptest.NewRequest("POST", test.path, strings.NewReader(test.body)))
|
||||
if response.Code != 400 || len(s.store.Exports) != 0 {
|
||||
t.Fatalf("invalid request accepted: %d", response.Code)
|
||||
}
|
||||
}
|
||||
source := s.store.Sources["source"]
|
||||
source.Environment = "production"
|
||||
s.store.Sources["source"] = source
|
||||
response := httptest.NewRecorder()
|
||||
s.createExport(response, httptest.NewRequest("POST", "/api/exports/wes", strings.NewReader(`{"source_id":"source","database":"wes_db","objects":["ttx_table_model"]}`)))
|
||||
if response.Code != 400 || !strings.Contains(response.Body.String(), "productionConfirmationRequired") {
|
||||
t.Fatal("production confirmation not enforced")
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
_, err := s.writeWESExport(ctx, source, Export{ID: "export-cancel", Objects: []string{"ttx_table_model"}}, wesFixture())
|
||||
if !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("cancel ignored: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWESFileNamesAndEmptyConfigurationCode(t *testing.T) {
|
||||
for _, name := range []string{"../x", "a/b.json", `a\b.json`, "C:x", "CON.json", "LPT1.json", "COM¹.json", "trailing.", "trailing ", "control\x00.json"} {
|
||||
if wesFileComponent(name) {
|
||||
t.Fatalf("unsafe filename accepted: %q", name)
|
||||
}
|
||||
}
|
||||
name, payload, err := wesConfigFile("ttx_bill_model", Record{"code": "", "structure": `{"code":""}`})
|
||||
if err != nil || name != "wes/bill/.json" || !json.Valid(payload) {
|
||||
t.Fatalf("empty code from reference format rejected: %s %v", name, err)
|
||||
}
|
||||
for oid, filename := range map[string]string{"ClearRedis*": "ClearRedis%2A.json", "ClearRedis%2A": "ClearRedis%252A.json", "ClearRedis?": "ClearRedis%3F.json"} {
|
||||
structure, _ := json.Marshal(map[string]string{"tid": "wms.views.BaseWso", "oid": oid})
|
||||
name, payload, err := wesConfigFile("ttx_wso_data", Record{"tid": "wms.views.BaseWso", "oid": oid, "structure": structure})
|
||||
var view map[string]any
|
||||
if err != nil || name != "wes/view/wms.views.BaseWso/"+filename || json.Unmarshal(payload, &view) != nil || view["oid"] != oid {
|
||||
t.Fatalf("wildcard view export changed identity: %s %v", name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWESConfigRemovesLiteralTabsWithoutChangingEscapes(t *testing.T) {
|
||||
raw := "{\n\t\"code\":\"item\",\"name\":\"入库质检方式(\t不启用:0)\",\"quoted\":\"a\\\"\tb\",\"slash\":\"a\\\\\tb\",\"escaped\":\"a\\tb\\u0009c\",\"number\":9007199254740993}"
|
||||
for _, structure := range []any{raw, []byte(raw)} {
|
||||
name, payload, err := wesConfigFile("ttx_table_model", Record{"code": "item", "structure": structure})
|
||||
if err != nil || name != "wes/table/item.json" {
|
||||
t.Fatalf("literal tab rejected: %s %v", name, err)
|
||||
}
|
||||
var got map[string]any
|
||||
decoder := json.NewDecoder(bytes.NewReader(payload))
|
||||
decoder.UseNumber()
|
||||
if err := decoder.Decode(&got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := map[string]any{"code": "item", "name": "入库质检方式(不启用:0)", "quoted": "a\"b", "slash": `a\b`, "escaped": "a\tb\tc", "number": json.Number("9007199254740993")}
|
||||
if !reflect.DeepEqual(got, want) || bytes.ContainsRune(payload, '\t') {
|
||||
t.Fatalf("configuration content changed unexpectedly: %s", payload)
|
||||
}
|
||||
if text, ok := structure.([]byte); ok && string(text) != raw {
|
||||
t.Fatal("input buffer modified")
|
||||
}
|
||||
}
|
||||
// Removing whitespace outside strings or a tab after an escape could turn
|
||||
// invalid syntax into a different valid document; both must remain errors.
|
||||
for _, raw := range []string{
|
||||
"{\"code\":\"item\",\"number\":1\t2}",
|
||||
"{\"code\":\"item\",\"name\":\"a\\\tt\"}",
|
||||
"{\"code\":\"item\",\"name\":\"a\nb\"}",
|
||||
`{"code":"item"} {"other":true}`,
|
||||
} {
|
||||
_, _, err := wesConfigFile("ttx_table_model", Record{"code": "item", "structure": raw})
|
||||
if !errors.Is(err, exportFailure("wesInvalidStructure")) {
|
||||
t.Fatalf("malformed JSON accepted: %q (%v)", raw, err)
|
||||
}
|
||||
}
|
||||
_, _, err := wesConfigFile("ttx_wso_data", Record{"tid": "ttx.wso.Bill", "oid": "orders", "structure": `{"tid":"ttx.wso.Bill","oid":"wrong"}`})
|
||||
var detail *wesConfigError
|
||||
if !errors.As(err, &detail) || detail.detail != `ttx_wso_data (tid="ttx.wso.Bill", oid="orders"): oid` {
|
||||
t.Fatalf("missing view identity: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,237 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/mongo/options"
|
||||
)
|
||||
|
||||
type wesMongoDatabases interface {
|
||||
ListDatabases(context.Context) ([]string, error)
|
||||
DatabaseAdapter(string) Adapter
|
||||
}
|
||||
|
||||
type wesMongoDuplicate struct {
|
||||
Database string `json:"database"`
|
||||
Collection string `json:"collection"`
|
||||
KeptID string `json:"kept_id"`
|
||||
DuplicateID string `json:"duplicate_id"`
|
||||
File string `json:"file"`
|
||||
DuplicateFile string `json:"duplicate_file"`
|
||||
}
|
||||
|
||||
func (adapter *mongoAdapter) ListDatabases(ctx context.Context) ([]string, error) {
|
||||
// This export must not use the UI's bounded database picker (512 names).
|
||||
names, err := adapter.client.ListDatabaseNames(ctx, bson.D{}, options.ListDatabases().SetAuthorizedDatabases(true))
|
||||
return names, mongoOperationError(err)
|
||||
}
|
||||
|
||||
// Database adapters borrow the root client's connection pool. Only the root
|
||||
// adapter is closed, after all databases have been exported.
|
||||
func (adapter *mongoAdapter) DatabaseAdapter(database string) Adapter {
|
||||
source := adapter.source
|
||||
source.Database = database
|
||||
return &mongoAdapter{client: adapter.client, database: adapter.client.Database(database), source: source, audit: adapter.audit}
|
||||
}
|
||||
|
||||
func wesMongoDatabaseError(database string, err error) error {
|
||||
code := exportFailure(AdapterErrorCode(err))
|
||||
var failure exportFailure
|
||||
if errors.As(err, &failure) {
|
||||
code = failure
|
||||
}
|
||||
detail := fmt.Sprintf("database=%q", database)
|
||||
var recordError *wesConfigError
|
||||
if errors.As(err, &recordError) {
|
||||
detail += ": " + recordError.detail
|
||||
}
|
||||
return &wesConfigError{code: code, detail: detail}
|
||||
}
|
||||
|
||||
func (s *server) writeWESMongoExport(ctx context.Context, source DataSource, job Export, adapter Adapter) (int64, error) {
|
||||
databases, ok := adapter.(wesMongoDatabases)
|
||||
if !ok {
|
||||
return 0, exportFailure("invalidSource")
|
||||
}
|
||||
names, err := databases.ListDatabases(ctx)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
sort.Strings(names)
|
||||
if err := os.MkdirAll(s.artifactDir, 0700); err != nil {
|
||||
return 0, exportFailure("exportArtifactFailed")
|
||||
}
|
||||
path, err := s.sqlExportPath(job.ID, true)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
file, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600)
|
||||
if err != nil {
|
||||
return 0, exportFailure("exportArtifactFailed")
|
||||
}
|
||||
defer file.Close()
|
||||
compressed := gzip.NewWriter(exportContextWriter{ctx: ctx, writer: file})
|
||||
defer compressed.Close()
|
||||
tw := tar.NewWriter(compressed)
|
||||
defer tw.Close()
|
||||
archive := exportArchive{ctx: ctx, writer: tw, created: job.CreatedAt}
|
||||
manifest := exportManifest{Version: 1, ID: job.ID, SourceID: job.SourceID, Mode: "wes", CreatedAt: job.CreatedAt, MaxRows: job.MaxRows, ProductionMasking: exportProduction(source)}
|
||||
selected := map[string]bool{}
|
||||
for _, name := range job.Objects {
|
||||
selected[name] = true
|
||||
}
|
||||
seenDB := map[string]bool{}
|
||||
type resource struct{ filename, collection, id string }
|
||||
seenFiles := map[string]resource{}
|
||||
for _, database := range names {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return manifest.Rows, err
|
||||
}
|
||||
if seenDB[database] {
|
||||
continue
|
||||
}
|
||||
seenDB[database] = true
|
||||
if validateDatabaseName("mongodb", database) != nil || !wesFileComponent(database) {
|
||||
return manifest.Rows, wesMongoDatabaseError(database, exportFailure("wesInvalidFileName"))
|
||||
}
|
||||
current := databases.DatabaseAdapter(database)
|
||||
objects, err := current.ListObjects(ctx)
|
||||
if err != nil {
|
||||
return manifest.Rows, wesMongoDatabaseError(database, err)
|
||||
}
|
||||
manifest.Databases = append(manifest.Databases, database)
|
||||
sort.Slice(objects, func(i, j int) bool { return objects[i].Name < objects[j].Name })
|
||||
for _, object := range objects {
|
||||
kind, _ := wesMongoCollection(object.Name)
|
||||
if object.Type != "collection" || kind == "" || len(selected) > 0 && !selected[object.Name] {
|
||||
continue
|
||||
}
|
||||
entry := exportManifestTable{Name: object.Name, Database: database, Files: []string{}}
|
||||
var documents bytes.Buffer
|
||||
if kind == "ttx_scheduler" {
|
||||
documents.WriteString("[\n")
|
||||
}
|
||||
var consumeErr error
|
||||
err = current.Read(ctx, object.Name, ReadOptions{BatchSize: job.BatchSize, MaxRows: job.MaxRows - manifest.Rows + 1}, func(records []Record) (callbackErr error) {
|
||||
defer func() {
|
||||
if callbackErr != nil {
|
||||
consumeErr = callbackErr
|
||||
}
|
||||
}()
|
||||
for _, record := range records {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
if manifest.Rows >= job.MaxRows {
|
||||
return exportFailure("wesRowLimitReached")
|
||||
}
|
||||
if exportProduction(source) {
|
||||
if kind != "ttx_scheduler" {
|
||||
value, err := wesMongoJSONValue(record)
|
||||
if err != nil {
|
||||
return wesRecordError(object.Name, record, "document", "wesInvalidStructure")
|
||||
}
|
||||
record = Record(value.(map[string]any))
|
||||
}
|
||||
record = automaticMask(record)
|
||||
}
|
||||
if kind == "ttx_scheduler" {
|
||||
payload, err := bson.MarshalExtJSON(bson.M(record), true, false)
|
||||
if err != nil {
|
||||
return wesRecordError(object.Name, record, "document", "wesInvalidStructure")
|
||||
}
|
||||
if entry.Rows > 0 {
|
||||
documents.WriteString(",\n")
|
||||
}
|
||||
documents.Write(payload)
|
||||
if documents.Len()+3 > exportChunkBytes {
|
||||
return exportFailure("exportSizeLimit")
|
||||
}
|
||||
entry.Rows++
|
||||
manifest.Rows++
|
||||
continue
|
||||
}
|
||||
filename, payload, err := wesMongoConfigFile(object.Name, record)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
filename = "edi/" + database + "/" + strings.TrimPrefix(filename, "edi/")
|
||||
key := strings.ToLower(filename)
|
||||
if existing, exists := seenFiles[key]; exists {
|
||||
if kind != "xslt" || existing.filename != filename || existing.collection != object.Name {
|
||||
return wesRecordError(object.Name, record, filename, "wesDuplicateFile")
|
||||
}
|
||||
duplicate := fmt.Sprintf("duplicates/%s/%s/%d.xslt", database, object.Name, entry.Rows+1)
|
||||
manifest.Duplicates = append(manifest.Duplicates, wesMongoDuplicate{Database: database, Collection: object.Name, KeptID: existing.id, DuplicateID: scalarText(record["_id"]), File: filename, DuplicateFile: duplicate})
|
||||
filename = duplicate
|
||||
} else {
|
||||
seenFiles[key] = resource{filename: filename, collection: object.Name, id: scalarText(record["_id"])}
|
||||
}
|
||||
if err := archive.add(filename, payload); err != nil {
|
||||
return err
|
||||
}
|
||||
entry.Files = append(entry.Files, filename)
|
||||
entry.Rows++
|
||||
manifest.Rows++
|
||||
}
|
||||
s.store.Lock()
|
||||
if current, exists := s.store.Exports[job.ID]; exists {
|
||||
current.Rows = manifest.Rows
|
||||
s.store.Exports[job.ID] = current
|
||||
}
|
||||
s.store.Unlock()
|
||||
return nil
|
||||
})
|
||||
if consumeErr != nil {
|
||||
err = consumeErr
|
||||
}
|
||||
if err != nil {
|
||||
return manifest.Rows, wesMongoDatabaseError(database, err)
|
||||
}
|
||||
if kind == "ttx_scheduler" {
|
||||
documents.WriteString("\n]\n")
|
||||
filename := "mongo/" + database + "/ttx_scheduler.json"
|
||||
if err := archive.add(filename, documents.Bytes()); err != nil {
|
||||
return manifest.Rows, wesMongoDatabaseError(database, err)
|
||||
}
|
||||
entry.Files = append(entry.Files, filename)
|
||||
}
|
||||
manifest.Tables = append(manifest.Tables, entry)
|
||||
}
|
||||
}
|
||||
if len(manifest.Tables) == 0 {
|
||||
return 0, exportFailure("wesMongoConfigNotFound")
|
||||
}
|
||||
manifest.Checksums = archive.checksums
|
||||
data, err := json.MarshalIndent(manifest, "", " ")
|
||||
if err != nil {
|
||||
return manifest.Rows, err
|
||||
}
|
||||
if err := archive.add("manifest.json", data); err != nil {
|
||||
return manifest.Rows, err
|
||||
}
|
||||
for _, close := range []func() error{tw.Close, compressed.Close, file.Sync, file.Close} {
|
||||
if err := close(); err != nil {
|
||||
return manifest.Rows, exportFailure("exportArtifactFailed")
|
||||
}
|
||||
}
|
||||
if len(manifest.Duplicates) > 0 {
|
||||
s.store.Lock()
|
||||
if current, exists := s.store.Exports[job.ID]; exists {
|
||||
current.Warnings = append(current.Warnings, "wesMongoDuplicateResources")
|
||||
s.store.Exports[job.ID] = current
|
||||
}
|
||||
s.store.Unlock()
|
||||
}
|
||||
return manifest.Rows, ctx.Err()
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/bson/primitive"
|
||||
)
|
||||
|
||||
// EDI installations use either the resource names or one collection pair per
|
||||
// ERP. Keep each ERP's resources together without mixing their configurations.
|
||||
func wesMongoCollection(name string) (kind, scope string) {
|
||||
switch name {
|
||||
case "edijson", "xslt", "ttx_scheduler":
|
||||
return name, ""
|
||||
}
|
||||
for prefix, kind := range map[string]string{"edi_erp_setting_": "edijson", "edi_erp_transform_file_": "xslt"} {
|
||||
if strings.HasPrefix(name, prefix) && len(name) > len(prefix) {
|
||||
return kind, strings.TrimPrefix(name, prefix)
|
||||
}
|
||||
}
|
||||
return "", ""
|
||||
}
|
||||
|
||||
// bson.D must become JSON objects, not arrays of Key/Value pairs. Preserve
|
||||
// numeric precision while emitting ordinary JSON usable by the EDI loader.
|
||||
func wesMongoJSONValue(value any) (any, error) {
|
||||
if object, ok := recordMap(value); ok {
|
||||
result := make(map[string]any, len(object))
|
||||
for key, value := range object {
|
||||
converted, err := wesMongoJSONValue(value)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result[key] = converted
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
if array, ok := value.(bson.A); ok {
|
||||
value = []any(array)
|
||||
}
|
||||
switch value := value.(type) {
|
||||
case []any:
|
||||
result := make([]any, len(value))
|
||||
for i, item := range value {
|
||||
converted, err := wesMongoJSONValue(item)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result[i] = converted
|
||||
}
|
||||
return result, nil
|
||||
case primitive.ObjectID:
|
||||
return value.Hex(), nil
|
||||
case primitive.DateTime:
|
||||
return value.Time().UTC().Format(time.RFC3339Nano), nil
|
||||
case primitive.Decimal128:
|
||||
return json.Number(value.String()), nil
|
||||
case nil, string, bool, int, int32, int64, float32, float64, json.Number, time.Time:
|
||||
return value, nil
|
||||
default:
|
||||
return nil, exportFailure("wesInvalidStructure")
|
||||
}
|
||||
}
|
||||
|
||||
func wesMongoConfigFile(collection string, record Record) (string, []byte, error) {
|
||||
failure := func(field, code string) (string, []byte, error) {
|
||||
return "", nil, wesRecordError(collection, record, field, code)
|
||||
}
|
||||
kind, scope := wesMongoCollection(collection)
|
||||
base := "edi/"
|
||||
if scope != "" {
|
||||
if !wesFileComponent(scope) {
|
||||
return failure("collection", "wesInvalidFileName")
|
||||
}
|
||||
base += scope + "/"
|
||||
}
|
||||
switch kind {
|
||||
case "edijson":
|
||||
config := make(Record, len(record))
|
||||
for key, value := range record {
|
||||
if key != "_id" && key != "_class" {
|
||||
config[key] = value
|
||||
}
|
||||
}
|
||||
value, err := wesMongoJSONValue(config)
|
||||
if err != nil {
|
||||
return failure("document", "wesInvalidStructure")
|
||||
}
|
||||
payload, err := json.MarshalIndent(value, "", " ")
|
||||
if err != nil {
|
||||
return failure("document", "wesInvalidStructure")
|
||||
}
|
||||
return base + "edijson.json", append(payload, '\n'), nil
|
||||
case "xslt":
|
||||
path, ok := wesString(record["path"])
|
||||
if !ok || path == "" {
|
||||
return failure("path", "wesInvalidStructure")
|
||||
}
|
||||
typeName := "xslt"
|
||||
if value, exists := record["type"]; exists {
|
||||
typeName, ok = wesString(value)
|
||||
if !ok || typeName != "xslt" {
|
||||
return failure("type (xslt)", "wesInvalidStructure")
|
||||
}
|
||||
}
|
||||
path = strings.ReplaceAll(path, `\`, "/")
|
||||
// EDI classpath resources may start at the resource root ("/xslt/..."
|
||||
// or "/a"). Place them under the archive's EDI directory. A second
|
||||
// leading slash, drive prefix, or traversal remains invalid below.
|
||||
path = strings.TrimPrefix(path, "/")
|
||||
// Validate each original segment before adding extensions or prefixes;
|
||||
// cleaning a traversal would silently change the referenced resource.
|
||||
for _, part := range strings.Split(path, "/") {
|
||||
if !wesFileComponent(part) {
|
||||
return failure("path", "wesInvalidFileName")
|
||||
}
|
||||
}
|
||||
if !strings.HasSuffix(path, "."+typeName) {
|
||||
path += "." + typeName
|
||||
}
|
||||
if !strings.HasPrefix(path, "xslt/") {
|
||||
path = "xslt/" + path
|
||||
}
|
||||
for _, part := range strings.Split(path, "/") {
|
||||
if !wesFileComponent(part) {
|
||||
return failure("path", "wesInvalidFileName")
|
||||
}
|
||||
}
|
||||
content, ok := wesString(record["text"])
|
||||
if !ok || !utf8.ValidString(content) {
|
||||
return failure("text", "wesInvalidStructure")
|
||||
}
|
||||
return base + path, []byte(content), nil
|
||||
default:
|
||||
return "", nil, exportFailure("wesMongoCollectionsOnly")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,368 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"archive/tar"
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"go.mongodb.org/mongo-driver/bson"
|
||||
"go.mongodb.org/mongo-driver/bson/primitive"
|
||||
)
|
||||
|
||||
type wesMongoServerFixture struct {
|
||||
Adapter
|
||||
databases map[string]*wesExportFixture
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *wesMongoServerFixture) ListDatabases(ctx context.Context) ([]string, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var names []string
|
||||
for name := range f.databases {
|
||||
names = append(names, name)
|
||||
}
|
||||
return names, f.err
|
||||
}
|
||||
func (f *wesMongoServerFixture) DatabaseAdapter(name string) Adapter { return f.databases[name] }
|
||||
func (f *wesMongoServerFixture) Close(context.Context) error { return nil }
|
||||
|
||||
func wesMongoFixture() *wesExportFixture {
|
||||
f := &wesExportFixture{objects: map[string]ObjectInfo{}, rows: map[string][]Record{
|
||||
"edi_erp_setting_shopee": {{"_id": primitive.NewObjectID(), "_class": "EdiSetting", "erpName": "shopee", "apiMap": bson.D{{Key: "query", Value: bson.D{{Key: "retries", Value: int64(9007199254740993)}}}}, "customerMap": bson.A{bson.D{{Key: "appSecret", Value: "private-value"}}}}},
|
||||
"edi_erp_transform_file_shopee": {{"_id": primitive.NewObjectID(), "path": "xslt/in/query", "type": "xslt", "text": "<?xml version=\"1.0\"?>\r\n<xsl:stylesheet xmlns:xsl=\"http://www.w3.org/1999/XSL/Transform\" version=\"1.0\">\n\t<!--查询-->\n</xsl:stylesheet>"}},
|
||||
"edijson": {{"_id": "config", "appKey": "example", "enabled": true}},
|
||||
"xslt": {{"_id": "transform", "path": `out\orders.xslt`, "text": "<transform/>"}},
|
||||
"interface_log": {{"body": "not configuration"}},
|
||||
}}
|
||||
for name := range f.rows {
|
||||
f.objects[name] = ObjectInfo{Name: name, Type: "collection"}
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
func wesReadPackage(t *testing.T, s *server, job Export) map[string][]byte {
|
||||
t.Helper()
|
||||
response := httptest.NewRecorder()
|
||||
request := httptest.NewRequest(http.MethodGet, "http://127.0.0.1/api/exports/"+job.ID+"/download", nil)
|
||||
request.RemoteAddr = "127.0.0.1:12345"
|
||||
s.handler().ServeHTTP(response, request)
|
||||
if response.Code != http.StatusOK {
|
||||
t.Fatalf("download failed: %d", response.Code)
|
||||
}
|
||||
zr, err := gzip.NewReader(bytes.NewReader(response.Body.Bytes()))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer zr.Close()
|
||||
tr := tar.NewReader(zr)
|
||||
files := map[string][]byte{}
|
||||
for {
|
||||
header, err := tr.Next()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
files[header.Name], err = io.ReadAll(tr)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return files
|
||||
}
|
||||
|
||||
func TestWESMongoExportResources(t *testing.T) {
|
||||
for _, production := range []bool{false, true} {
|
||||
s := newCSVJobTestServer(t)
|
||||
source := s.store.Sources["source"]
|
||||
source.Type = "mongodb"
|
||||
if production {
|
||||
source.Environment = "production"
|
||||
}
|
||||
s.store.Sources["source"] = source
|
||||
f := wesMongoFixture()
|
||||
s.adapterFactory = func(_ context.Context, source DataSource, _ func(SQLAudit)) (Adapter, error) {
|
||||
if source.Database != "" || source.Type != "mongodb" {
|
||||
t.Fatal("wrong MongoDB scope")
|
||||
}
|
||||
return &wesMongoServerFixture{databases: map[string]*wesExportFixture{"edi_db": f}}, nil
|
||||
}
|
||||
job := wesStart(t, s, `{"source_id":"source","database":"edi_db","objects":["edi_erp_setting_shopee","edi_erp_transform_file_shopee","edijson","xslt"],"max_rows":4,"confirm_production":true}`)
|
||||
if job.Status != "COMPLETED" || job.Rows != 4 {
|
||||
t.Fatalf("unexpected export: %+v", job)
|
||||
}
|
||||
files := wesReadPackage(t, s, job)
|
||||
if len(files) != 5 {
|
||||
t.Fatalf("unexpected files: %d", len(files))
|
||||
}
|
||||
config := files["edi/edi_db/shopee/edijson.json"]
|
||||
var parsed map[string]any
|
||||
if json.Unmarshal(config, &parsed) != nil || parsed["_id"] != nil || parsed["_class"] != nil {
|
||||
t.Fatal("invalid EDI JSON")
|
||||
}
|
||||
if _, ok := parsed["apiMap"].(map[string]any); !ok {
|
||||
t.Fatal("BSON document emitted as array")
|
||||
}
|
||||
if !bytes.Contains(config, []byte("9007199254740993")) {
|
||||
t.Fatal("number precision lost")
|
||||
}
|
||||
if bytes.Contains(config, []byte("private-value")) == production {
|
||||
t.Fatal("unexpected production masking")
|
||||
}
|
||||
if !json.Valid(files["edi/edi_db/edijson.json"]) || string(files["edi/edi_db/xslt/out/orders.xslt"]) != "<transform/>" {
|
||||
t.Fatal("literal collection resources missing")
|
||||
}
|
||||
if string(files["edi/edi_db/shopee/xslt/in/query.xslt"]) != f.rows["edi_erp_transform_file_shopee"][0]["text"] {
|
||||
t.Fatal("XSLT content changed")
|
||||
}
|
||||
var manifest exportManifest
|
||||
if json.Unmarshal(files["manifest.json"], &manifest) != nil || manifest.Rows != 4 || len(manifest.Tables) != 4 || len(manifest.Checksums) != 4 {
|
||||
t.Fatal("invalid Mongo manifest")
|
||||
}
|
||||
for name, sum := range manifest.Checksums {
|
||||
actual := sha256.Sum256(files[name])
|
||||
if sum != hex.EncodeToString(actual[:]) {
|
||||
t.Fatal("invalid checksum")
|
||||
}
|
||||
}
|
||||
for _, name := range f.read {
|
||||
if name == "interface_log" {
|
||||
t.Fatal("unselected collection read")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestWESMongoRejectsUnsafePathsAndInvalidDocuments(t *testing.T) {
|
||||
for _, path := range []string{"../escape", "xslt/../../escape", "//absolute", "/../escape", `C:\escape`, `xslt\..\escape`, "xslt//empty", "xslt/CON", "xslt/trailing.", "xslt/trailing "} {
|
||||
_, _, err := wesMongoConfigFile("xslt", Record{"path": path, "text": "x"})
|
||||
if !errors.Is(err, exportFailure("wesInvalidFileName")) {
|
||||
t.Fatalf("unsafe path accepted: %q %v", path, err)
|
||||
}
|
||||
}
|
||||
for input, expected := range map[string]string{"/a": "edi/xslt/a.xslt", "/xslt/in/orders": "edi/xslt/in/orders.xslt"} {
|
||||
name, content, err := wesMongoConfigFile("xslt", Record{"path": input, "type": "xslt", "text": "<stylesheet/>"})
|
||||
if err != nil || name != expected || string(content) != "<stylesheet/>" {
|
||||
t.Fatalf("rooted resource failed: %s %v", name, err)
|
||||
}
|
||||
}
|
||||
for _, record := range []Record{{"path": "xslt/a"}, {"path": "xslt/a", "text": 42}, {"path": "xslt/a", "text": "x", "type": "../xslt"}} {
|
||||
_, _, err := wesMongoConfigFile("xslt", record)
|
||||
if !errors.Is(err, exportFailure("wesInvalidStructure")) {
|
||||
t.Fatalf("invalid record accepted: %v", err)
|
||||
}
|
||||
}
|
||||
_, _, err := wesMongoConfigFile("edi_erp_setting_..", Record{"appKey": "x"})
|
||||
if !errors.Is(err, exportFailure("wesInvalidFileName")) {
|
||||
t.Fatal("unsafe ERP scope accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWESMongoFailureDoesNotPublishPartialPackage(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name, code string
|
||||
change func(*wesExportFixture)
|
||||
}{
|
||||
{"duplicate", "wesDuplicateFile", func(f *wesExportFixture) {
|
||||
f.rows["edijson"] = append(f.rows["edijson"], Record{"_id": "other", "appKey": "other"})
|
||||
}},
|
||||
{"path collision", "wesDuplicateFile", func(f *wesExportFixture) {
|
||||
f.rows["xslt"] = append(f.rows["xslt"], Record{"path": "xslt/OUT/orders", "type": "xslt", "text": "other"})
|
||||
}},
|
||||
{"size", "exportSizeLimit", func(f *wesExportFixture) { f.rows["xslt"][0]["text"] = strings.Repeat("x", exportChunkBytes+1) }},
|
||||
{"row limit", "wesRowLimitReached", func(f *wesExportFixture) {
|
||||
for i := 0; i < 3; i++ {
|
||||
f.rows["xslt"] = append(f.rows["xslt"], Record{"path": string(rune('a' + i)), "text": "x"})
|
||||
}
|
||||
}},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
s := newCSVJobTestServer(t)
|
||||
source := s.store.Sources["source"]
|
||||
source.Type = "mongodb"
|
||||
s.store.Sources["source"] = source
|
||||
f := wesMongoFixture()
|
||||
test.change(f)
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
return &wesMongoServerFixture{databases: map[string]*wesExportFixture{"edi_db": f}}, nil
|
||||
}
|
||||
job := wesStart(t, s, `{"source_id":"source","database":"edi_db","objects":["edijson","xslt"],"max_rows":4}`)
|
||||
if job.Status != "FAILED" || job.Error != test.code || job.File != "" {
|
||||
t.Fatalf("unexpected failure: %+v", job)
|
||||
}
|
||||
files, err := os.ReadDir(s.artifactDir)
|
||||
if err != nil || len(files) != 0 {
|
||||
t.Fatal("partial package retained")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestWESMongoDiscoversAllDatabasesAndERPGroups(t *testing.T) {
|
||||
s := newCSVJobTestServer(t)
|
||||
source := s.store.Sources["source"]
|
||||
source.Type = "mongodb"
|
||||
source.Database = "auth_db"
|
||||
s.store.Sources["source"] = source
|
||||
root := &wesMongoServerFixture{databases: map[string]*wesExportFixture{}}
|
||||
// Discovery must not inherit the database picker's 512-name cap.
|
||||
for i := 0; i < 513; i++ {
|
||||
root.databases[fmt.Sprintf("empty_%03d", i)] = &wesExportFixture{objects: map[string]ObjectInfo{"logs": {Name: "logs", Type: "collection"}}}
|
||||
}
|
||||
for _, database := range []string{"warehouse_a", "warehouse_b"} {
|
||||
f := wesMongoFixture()
|
||||
for name, rows := range map[string][]Record{
|
||||
"edi_erp_setting_ess": {{"_id": "ess", "appKey": database}},
|
||||
"edi_erp_transform_file_ess": {{"path": "xslt/in/query", "type": "xslt", "text": database}},
|
||||
} {
|
||||
f.rows[name] = rows
|
||||
f.objects[name] = ObjectInfo{Name: name, Type: "collection"}
|
||||
}
|
||||
root.databases[database] = f
|
||||
}
|
||||
s.adapterFactory = func(_ context.Context, source DataSource, _ func(SQLAudit)) (Adapter, error) {
|
||||
if source.Database != "" || source.AuthSource != "auth_db" {
|
||||
t.Error("business/authentication scope mixed")
|
||||
}
|
||||
return root, nil
|
||||
}
|
||||
job := wesStart(t, s, `{"source_id":"source","max_rows":12}`)
|
||||
if job.Status != "COMPLETED" || job.Database != "" || job.Rows != 12 {
|
||||
t.Fatalf("unexpected export: %+v", job)
|
||||
}
|
||||
files := wesReadPackage(t, s, job)
|
||||
for _, db := range []string{"warehouse_a", "warehouse_b"} {
|
||||
for _, erp := range []string{"ess", "shopee"} {
|
||||
if !json.Valid(files["edi/"+db+"/"+erp+"/edijson.json"]) || len(files["edi/"+db+"/"+erp+"/xslt/in/query.xslt"]) == 0 {
|
||||
t.Fatalf("missing %s/%s resources", db, erp)
|
||||
}
|
||||
}
|
||||
if string(files["edi/"+db+"/ess/xslt/in/query.xslt"]) != db {
|
||||
t.Fatal("cross-database overwrite")
|
||||
}
|
||||
}
|
||||
var manifest exportManifest
|
||||
if json.Unmarshal(files["manifest.json"], &manifest) != nil || len(manifest.Databases) != 515 || len(manifest.Tables) != 12 {
|
||||
t.Fatal("incomplete database/collection inventory")
|
||||
}
|
||||
for _, entry := range manifest.Tables {
|
||||
if entry.Database == "" {
|
||||
t.Fatal("missing collection database")
|
||||
}
|
||||
}
|
||||
// The row limit is shared by all databases, not reset for each database.
|
||||
job = wesStart(t, s, `{"source_id":"source","max_rows":11}`)
|
||||
if job.Status != "FAILED" || job.Error != "wesRowLimitReached" || job.Rows != 11 || !strings.Contains(job.ErrorDetail, "warehouse_b") {
|
||||
t.Fatalf("global limit ignored: %+v", job)
|
||||
}
|
||||
if _, err := os.Stat(filepathJoinExportForTest(t, s, job.ID)); !os.IsNotExist(err) {
|
||||
t.Fatal("partial package published")
|
||||
}
|
||||
}
|
||||
|
||||
func filepathJoinExportForTest(t *testing.T, s *server, id string) string {
|
||||
t.Helper()
|
||||
path, err := s.sqlExportPath(id, false)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func TestWESMongoDiscoveryFailureAndMissingConfiguration(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name, code string
|
||||
err error
|
||||
}{
|
||||
{"no configuration", "wesMongoConfigNotFound", nil},
|
||||
{"discovery denied", "mongoDatabaseDenied", errors.New("mongoDatabaseDenied")},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
s := newCSVJobTestServer(t)
|
||||
source := s.store.Sources["source"]
|
||||
source.Type = "mongodb"
|
||||
s.store.Sources["source"] = source
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
return &wesMongoServerFixture{databases: map[string]*wesExportFixture{}, err: test.err}, nil
|
||||
}
|
||||
job := wesStart(t, s, `{"source_id":"source"}`)
|
||||
if job.Status != "FAILED" || job.Error != test.code || job.File != "" {
|
||||
t.Fatalf("unexpected export: %+v", job)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestWESMongoSchedulerPreservesBSONAndDuplicateXSLT(t *testing.T) {
|
||||
s := newCSVJobTestServer(t)
|
||||
source := s.store.Sources["source"]
|
||||
source.Type = "mongodb"
|
||||
s.store.Sources["source"] = source
|
||||
f := wesMongoFixture()
|
||||
id := primitive.NewObjectID()
|
||||
date := primitive.DateTime(1791777600123)
|
||||
f.rows["ttx_scheduler"] = []Record{{"_id": id, "createdAt": date, "counter": int64(9007199254740993), "config": bson.D{{Key: "enabled", Value: true}}}}
|
||||
f.objects["ttx_scheduler"] = ObjectInfo{Name: "ttx_scheduler", Type: "collection"}
|
||||
f.rows["xslt"] = append(f.rows["xslt"], Record{"_id": "second", "path": "xslt/out/orders", "type": "xslt", "text": "<second/>"})
|
||||
s.adapterFactory = func(context.Context, DataSource, func(SQLAudit)) (Adapter, error) {
|
||||
return &wesMongoServerFixture{databases: map[string]*wesExportFixture{"app": f}}, nil
|
||||
}
|
||||
job := wesStart(t, s, `{"source_id":"source","objects":["ttx_scheduler","xslt"]}`)
|
||||
if job.Status != "COMPLETED" || job.Rows != 3 || len(job.Warnings) != 1 || job.Warnings[0] != "wesMongoDuplicateResources" {
|
||||
t.Fatalf("unexpected export: %+v", job)
|
||||
}
|
||||
files := wesReadPackage(t, s, job)
|
||||
var documents []json.RawMessage
|
||||
if json.Unmarshal(files["mongo/app/ttx_scheduler.json"], &documents) != nil || len(documents) != 1 {
|
||||
t.Fatal("scheduler documents missing")
|
||||
}
|
||||
var document bson.M
|
||||
if bson.UnmarshalExtJSON(documents[0], true, &document) != nil || document["_id"] != id || document["createdAt"] != date || document["counter"] != int64(9007199254740993) {
|
||||
t.Fatalf("scheduler BSON types changed: %v", document)
|
||||
}
|
||||
var manifest exportManifest
|
||||
if json.Unmarshal(files["manifest.json"], &manifest) != nil || len(manifest.Duplicates) != 1 {
|
||||
t.Fatal("duplicate inventory missing")
|
||||
}
|
||||
duplicate := manifest.Duplicates[0]
|
||||
if duplicate.KeptID != "transform" || duplicate.DuplicateID != "second" || string(files[duplicate.File]) != "<transform/>" || string(files[duplicate.DuplicateFile]) != "<second/>" {
|
||||
t.Fatal("duplicate XSLT not preserved")
|
||||
}
|
||||
if f.rows["ttx_scheduler"][0]["_id"] != id {
|
||||
t.Fatal("source record changed")
|
||||
}
|
||||
}
|
||||
|
||||
func TestWESMongoRequestRestrictions(t *testing.T) {
|
||||
s := newCSVJobTestServer(t)
|
||||
source := s.store.Sources["source"]
|
||||
source.Type = "mongodb"
|
||||
s.store.Sources["source"] = source
|
||||
for _, test := range []struct{ path, body, code string }{
|
||||
{"/api/exports/sql", `{"source_id":"source","database":"edi_db","objects":["edijson"]}`, "exportMySQLOnly"},
|
||||
{"/api/exports/wes", `{"source_id":"source","database":"edi_db","objects":["interface_log"]}`, "wesMongoCollectionsOnly"},
|
||||
} {
|
||||
response := httptest.NewRecorder()
|
||||
request := httptest.NewRequest(http.MethodPost, "http://127.0.0.1"+test.path, strings.NewReader(test.body))
|
||||
request.RemoteAddr = "127.0.0.1:12345"
|
||||
s.handler().ServeHTTP(response, request)
|
||||
if response.Code != http.StatusBadRequest || !strings.Contains(response.Body.String(), test.code) || len(s.store.Exports) != 0 {
|
||||
t.Fatalf("unexpected response: %d %s", response.Code, response.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
server:
|
||||
addr: ":0"
|
||||
data_dir: ./data
|
||||
# Default data directory: ~/.syncdb (omit data_dir to use it).
|
||||
workers: 2
|
||||
export:
|
||||
retention_hours: 24
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import {readFileSync} from 'node:fs';
|
||||
import {runInNewContext} from 'node:vm';
|
||||
import {webcrypto} from 'node:crypto';
|
||||
import {JSDOM,VirtualConsole} from 'jsdom';
|
||||
|
||||
const app=readFileSync('../cmd/syncdb/web/app.js','utf8');
|
||||
const sources=[{id:'db',name:'db',type:'mysql',database:'',environment:'testing',username:'admin'}];
|
||||
function helpers() {
|
||||
const context={state:{sources},isProduction:id=>id==='prod',errorKey:key=>key,crypto:webcrypto,accountInputNames:['source_id','action','role','username','host','database']};
|
||||
const section=(start,end)=>app.slice(app.indexOf(start),app.indexOf(end,app.indexOf(start)));
|
||||
runInNewContext(section('const accountPasswordCharacters =','let accountPlan =')+section('function validAccountHost(','function clearAccountPassword(')+section('function accountBlockers(','function updateAccountControls(')+section('function accountReviewSQL(','function renderAccountPlan('),context);
|
||||
return context;
|
||||
}
|
||||
const response=values=>({...values,plan_hash:'reviewed',can_apply:true,account_exists:values.action==='drop',server_version:'8.0.36',server_role:'primary',active_connections:0,replication_channels:0,statements:[values.action==='drop'?"DROP USER 'syncdb_reader';":"CREATE USER 'syncdb_reader' IDENTIFIED BY 'must-not-display';"],warnings:[]});
|
||||
|
||||
test('account plan normalizes scope, removes secrets and allows explicit database for an unbound source',()=>{
|
||||
const h=helpers(),values={source_id:'db',action:'create',role:'reader',username:'syncdb_reader',host:'10.0.0.1',database:'app'};
|
||||
const plan=h.normalizeAccountPlan({...response(values),password:'hidden-secret'},values);
|
||||
assert.ok(!JSON.stringify(plan).includes('must-not-display'));assert.ok(!JSON.stringify(plan).includes('hidden-secret'));
|
||||
assert.equal(h.accountBlockers(plan).length,0);
|
||||
assert.throws(()=>h.normalizeAccountPlan({...response(values),database:'other'},values),/invalidResponse/);
|
||||
assert.throws(()=>h.normalizeAccountPlan({...response(values),can_apply:'true'},values),/invalidResponse/);
|
||||
for(const overrides of [{can_apply:false},{server_version:'5.7.44'},{source_id:'prod'},{action:'drop',role:'repl'},{action:'drop',username:'admin'}]) assert.ok(h.accountBlockers({...plan,...overrides}).length>0);
|
||||
for(let i=0;i<10;i++){const password=h.generateAccountPassword();assert.equal(password.length,24);assert.ok(h.validAccountPassword(password));}
|
||||
});
|
||||
|
||||
function setup() {
|
||||
const dom=new JSDOM('<!doctype html><html><body></body></html>',{url:'http://localhost/#accounts',runScripts:'outside-only',pretendToBeVisual:true,virtualConsole:new VirtualConsole()});
|
||||
const {window}=dom;
|
||||
window.matchMedia=()=>({matches:false,addListener(){},removeListener(){},addEventListener(){},removeEventListener(){}});
|
||||
window.ResizeObserver=class {observe(){} unobserve(){} disconnect(){}};
|
||||
window.eval(readFileSync('../cmd/syncdb/web/react-console.js','utf8'));
|
||||
const h=helpers(),props={sources,translate:key=>key,userError:error=>error.message,language:'zh',validateInput:h.validateAccountInput,validPassword:h.validAccountPassword,getBlockers:h.accountBlockers,generatePassword:h.generateAccountPassword,inspect:async values=>h.normalizeAccountPlan(response(values),values),apply:async()=>({status:'COMPLETED',applied_statements:1})};
|
||||
return {window,props,h,open:()=>window.syncdbReact.accountWizard(props),close:()=>{window.syncdbReact.closeDetails();window.close();}};
|
||||
}
|
||||
async function until(check){for(let i=0;i<100;i++){if(check())return;await new Promise(resolve=>setTimeout(resolve,25));}assert.ok(check(),'UI did not update');}
|
||||
const button=(window,label)=>[...window.document.querySelectorAll('button')].find(item=>item.textContent===label);
|
||||
function type(window,name,value) {
|
||||
const input=window.document.getElementById(`native-account_${name}`);
|
||||
Object.getOwnPropertyDescriptor(window.HTMLInputElement.prototype,'value').set.call(input,value);input.dispatchEvent(new window.Event('input',{bubbles:true}));
|
||||
}
|
||||
|
||||
test('native account creation uses reviewed plan, keeps secrets out of preview and clears password after apply',async()=>{
|
||||
const {window,props,h,open,close}=setup();
|
||||
try {
|
||||
let inspected,applied;props.inspect=async values=>{inspected=values;return h.normalizeAccountPlan(response(values),values);};props.apply=async values=>{applied=values;return {status:'COMPLETED',applied_statements:2};};open();
|
||||
assert.equal(window.document.querySelector('#native-account_database').readOnly,false);
|
||||
type(window,'database','app');type(window,'host','10.0.0.1');
|
||||
button(window,'accountGeneratePassword').click();
|
||||
await until(()=>window.document.querySelector('#native-account_password').value.length===24);
|
||||
const password=window.document.querySelector('#native-account_password').value;assert.ok(h.validAccountPassword(password));
|
||||
button(window,'accountInspect').click();await until(()=>window.document.querySelector('#native-account_confirm'));
|
||||
assert.equal(Object.hasOwn(inspected,'password'),false);
|
||||
assert.ok(window.document.body.textContent.includes('<account-password>'));
|
||||
assert.ok(!window.document.body.textContent.includes('must-not-display'));
|
||||
assert.ok(button(window,'accountApply').disabled);
|
||||
window.document.querySelector('#native-account_confirm').click();await until(()=>!button(window,'accountApply').disabled);
|
||||
button(window,'accountApply').click();await until(()=>applied && window.document.body.textContent.includes('COMPLETED'));
|
||||
assert.equal(applied.plan_hash,'reviewed');assert.equal(applied.password,password);assert.equal(applied.confirm,true);
|
||||
assert.equal(window.document.querySelector('#native-account_password').value,'');assert.ok(button(window,'accountApply').disabled);
|
||||
} finally {close();}
|
||||
});
|
||||
|
||||
test('editing account scope invalidates pending inspection and an existing reviewed plan',async()=>{
|
||||
const {window,props,h,open,close}=setup();
|
||||
try {
|
||||
let resolvePlan,values;
|
||||
props.inspect=input=>{values=input;return new Promise(resolve=>{resolvePlan=resolve;});};open();type(window,'database','app');type(window,'host','10.0.0.1');
|
||||
button(window,'accountInspect').click();await until(()=>resolvePlan);
|
||||
type(window,'host','10.0.0.2');resolvePlan(h.normalizeAccountPlan(response(values),values));
|
||||
await until(()=>!button(window,'accountInspect').disabled);
|
||||
assert.equal(window.document.querySelector('#native-account_confirm'),null);
|
||||
button(window,'accountInspect').click();await until(()=>values.host==='10.0.0.2');resolvePlan(h.normalizeAccountPlan(response(values),values));
|
||||
await until(()=>window.document.querySelector('#native-account_confirm'));
|
||||
type(window,'username','syncdb_changed');await until(()=>!window.document.querySelector('#native-account_confirm'));
|
||||
assert.ok(button(window,'accountApply').disabled);
|
||||
} finally {close();}
|
||||
});
|
||||
|
||||
test('blocked account plans disable confirmation and closing discards password',async()=>{
|
||||
const {window,props,h,open,close}=setup();
|
||||
try {
|
||||
props.inspect=async values=>h.normalizeAccountPlan({...response(values),can_apply:false,warnings:['accountProductionReadOnly']},values);
|
||||
open();type(window,'database','app');type(window,'host','10.0.0.1');button(window,'accountGeneratePassword').click();button(window,'accountInspect').click();
|
||||
await until(()=>window.document.querySelector('#native-account_confirm'));
|
||||
assert.ok(window.document.querySelector('#native-account_confirm').disabled);assert.ok(button(window,'accountApply').disabled);
|
||||
window.syncdbReact.closeAccountWizard();assert.equal(window.document.querySelector('#native-account_password'),null);
|
||||
open();assert.equal(window.document.querySelector('#native-account_password').value,'');
|
||||
} finally {close();}
|
||||
});
|
||||
|
||||
test('account deletion needs both confirmations, omits password and reports partial application',async()=>{
|
||||
const {window,props,open,close}=setup();
|
||||
try {
|
||||
let applied;props.apply=async values=>{applied=values;return {status:'FAILED',applied_statements:1,error:'permissionDenied'};};open();
|
||||
type(window,'database','app');type(window,'host','10.0.0.1');button(window,'accountGeneratePassword').click();
|
||||
await until(()=>window.document.querySelector('#native-account_password').value.length===24);
|
||||
window.document.querySelector('#native-account_action').closest('.ant-select').querySelector('.ant-select-selector').dispatchEvent(new window.MouseEvent('mousedown',{bubbles:true}));
|
||||
await until(()=>[...window.document.querySelectorAll('.ant-select-item-option-content')].some(item=>item.textContent==='accountDrop'));
|
||||
[...window.document.querySelectorAll('.ant-select-item-option-content')].find(item=>item.textContent==='accountDrop').closest('.ant-select-item-option').click();
|
||||
await until(()=>!window.document.querySelector('#native-account_password'));
|
||||
button(window,'accountInspect').click();await until(()=>window.document.querySelector('#native-account_drop_confirm'));
|
||||
window.document.querySelector('#native-account_confirm').click();await new Promise(resolve=>setTimeout(resolve,25));
|
||||
assert.ok(button(window,'accountApply').disabled);
|
||||
window.document.querySelector('#native-account_drop_confirm').click();await until(()=>!button(window,'accountApply').disabled);
|
||||
button(window,'accountApply').click();await until(()=>window.document.body.textContent.includes('accountPartialApply'));
|
||||
assert.equal(applied.action,'drop');assert.equal(Object.hasOwn(applied,'password'),false);
|
||||
assert.ok(button(window,'accountApply').disabled);
|
||||
} finally {close();}
|
||||
});
|
||||
|
||||
test('application account callback validates plans and rejects changed permissions and sessions',async()=>{
|
||||
const h=helpers();let wizard;const requests=[];
|
||||
Object.assign(h,{state:{sources,authenticated:true},authVersion:1,window:{syncdbReact:{accountWizard:props=>{wizard=props;}}},text:key=>key,userError:error=>error.message,language:'zh',resetAccountWizard(){},send:async(path,method,values)=>{requests.push({path,values});return path.endsWith('/plan')?response(values):{status:'COMPLETED',applied_statements:1};}});
|
||||
runInNewContext(app.slice(app.indexOf('function openAccountWizard('),app.indexOf('function accountReviewSQL(')),h);
|
||||
h.openAccountWizard();const values={source_id:'db',action:'create',role:'reader',username:'syncdb_reader',host:'10.0.0.1',database:'app'};
|
||||
const plan=await wizard.inspect(values);assert.equal(requests[0].path,'/replications/accounts/plan');
|
||||
await wizard.apply({...values,plan_hash:plan.plan_hash,confirm:true,password:h.generateAccountPassword()},plan);assert.equal(requests[1].path,'/replications/accounts/apply');
|
||||
h.state.sources=[{...sources[0],database:'other'}];await assert.rejects(wizard.apply({...values,confirm:true},plan),/accountInvalidInput/);
|
||||
h.state.sources=sources;h.state.authenticated=false;await assert.rejects(wizard.inspect(values),/sessionExpired/);
|
||||
});
|
||||
@@ -0,0 +1,108 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import {readFileSync} from 'node:fs';
|
||||
import {runInNewContext} from 'node:vm';
|
||||
import {JSDOM,VirtualConsole} from 'jsdom';
|
||||
import {translate} from './test-translations.mjs';
|
||||
|
||||
async function until(check){for(let i=0;i<100;i++){if(check())return;await new Promise(resolve=>setTimeout(resolve,25));}assert.ok(check(),'confirmation did not update');}
|
||||
|
||||
test('native action confirmation localizes controls, renders scope as text and resolves cancellation',async()=>{
|
||||
const dom=new JSDOM('<!doctype html><html><body></body></html>',{url:'http://localhost/',runScripts:'outside-only',pretendToBeVisual:true,virtualConsole:new VirtualConsole()}),w=dom.window;
|
||||
w.matchMedia=()=>({matches:false,addListener(){},removeListener(){}});w.ResizeObserver=class {observe(){} unobserve(){} disconnect(){}};
|
||||
w.eval(readFileSync('../cmd/syncdb/web/react-console.js','utf8'));
|
||||
try {
|
||||
const t=translate('en'),scope='<script>unsafe</script> · execution-1';
|
||||
const pending=w.syncdbReact.confirmAction({title:'deleteConfirm',content:scope,danger:true,confirmKey:'remove',translate:t,language:'en'});
|
||||
await until(()=>w.document.querySelector('.ant-modal-confirm-title')?.textContent===t('deleteConfirm'));
|
||||
const modal=w.document.querySelector('.ant-modal-confirm');
|
||||
assert.ok(modal.textContent.includes(scope));assert.equal(modal.querySelector('script'),null);
|
||||
const confirm=[...modal.querySelectorAll('button')].find(button=>button.textContent===t('remove'));assert.ok(confirm);
|
||||
confirm.click();assert.equal(await pending,true);
|
||||
const cancelled=w.syncdbReact.confirmAction({title:'cancelConfirm',translate:t,language:'en'});
|
||||
w.syncdbReact.cancelActionConfirmations();assert.equal(await cancelled,false,'leaving the page cancels confirmation');
|
||||
const sessionEnded=w.syncdbReact.confirmAction({title:'skipTableConfirm',translate:t,language:'en'});
|
||||
w.syncdbReact.closeDetails();assert.equal(await sessionEnded,false,'closing the session settles pending operations');
|
||||
}finally{w.syncdbReact.closeDetails();w.close();}
|
||||
});
|
||||
|
||||
function controller() {
|
||||
const app=readFileSync('../cmd/syncdb/web/app.js','utf8');
|
||||
let finish,confirmations=0;const sent=[],errors=[];
|
||||
const context={window:{syncdbReact:{confirmAction:()=>{confirmations++;return new Promise(resolve=>{finish=resolve;});}}},state:{authenticated:true,executions:[{id:'execution',task_id:'task',status:'RUNNING',stage:'DATA',object:'table_a'}]},authVersion:1,loggingOut:false,location:{hash:'#executions'},text:translate('en'),language:'en',taskName:id=>id,encoded:encodeURIComponent,send:async(...args)=>sent.push(args),toast(){},reload:async()=>{},showPageError:error=>errors.push(error.message)};
|
||||
runInNewContext(app.slice(app.indexOf('async function confirmUIAction('),app.indexOf('window.syncdbTableError =')),context);
|
||||
return {context,sent,errors,confirm:value=>finish(value),get confirmations(){return confirmations;}};
|
||||
}
|
||||
|
||||
test('confirmation rejects changed session, route and logout before submitting',async()=>{
|
||||
for(const change of ['session','route','logout','cancel','unchanged']) {
|
||||
const h=controller(),pending=h.context.confirmUIAction('deleteConfirm');
|
||||
if(change==='session')h.context.authVersion++;
|
||||
if(change==='route')h.context.location.hash='#tasks';
|
||||
if(change==='logout')h.context.loggingOut=true;
|
||||
h.confirm(change!=='cancel');assert.equal(await pending,change==='unchanged',change);
|
||||
}
|
||||
const h=controller();delete h.context.window.syncdbReact;
|
||||
assert.equal(await h.context.confirmUIAction('deleteConfirm'),false);
|
||||
assert.deepEqual(h.errors,['uiUnavailable'],'missing framework must not authorize the action');
|
||||
});
|
||||
|
||||
test('skip confirmation rechecks the active table after a live update',async()=>{
|
||||
for(const change of ['next_table','finished','in_place_table','in_place_task','unchanged']) {
|
||||
const h=controller(),record=h.context.state.executions[0];
|
||||
const pending=h.context.window.syncdbSkipCurrent(record);
|
||||
if(change==='next_table')h.context.state.executions=[{...record,object:'table_b'}];
|
||||
if(change==='finished')h.context.state.executions=[{...record,status:'COMPLETED'}];
|
||||
if(change==='in_place_table')record.object='table_b';
|
||||
if(change==='in_place_task')record.task_id='another_task';
|
||||
h.confirm(true);await pending;
|
||||
assert.equal(h.sent.length,change==='unchanged'?1:0,change);
|
||||
if(change==='unchanged')assert.equal(h.sent[0][2].object,'table_a');
|
||||
}
|
||||
});
|
||||
|
||||
test('skip uses the active retry execution and ignores schema or waiting executions',async()=>{
|
||||
for(const stage of ['DATA','SCHEMA','PENDING']) {
|
||||
const h=controller();
|
||||
const retry={id:'retry',retry_of:'original',task_id:'task',object:'table_a',stage:stage==='PENDING'?'DATA':stage,status:stage==='PENDING'?'PENDING':'RUNNING'};
|
||||
h.context.state.executions=[{id:'original',task_id:'task',status:'PARTIAL'},retry];
|
||||
const pending=h.context.window.syncdbSkipCurrent(retry);
|
||||
if(stage==='DATA')h.confirm(true);
|
||||
await pending;
|
||||
assert.equal(h.sent.length,stage==='DATA'?1:0);
|
||||
if(stage==='DATA')assert.equal(h.sent[0][2].execution_id,'retry');
|
||||
}
|
||||
});
|
||||
|
||||
test('skip shares a pending guard across entry points and releases it after cancellation or request failure',async()=>{
|
||||
const h=controller(),record=h.context.state.executions[0];
|
||||
const first=h.context.window.syncdbSkipCurrent(record);
|
||||
await h.context.window.syncdbSkipCurrent({...record});
|
||||
assert.equal(h.confirmations,1,'a second entry point must not open another confirmation');
|
||||
h.confirm(false);await first;assert.equal(h.sent.length,0);
|
||||
h.context.send=async()=>{throw new Error('requestError');};
|
||||
const failed=h.context.window.syncdbSkipCurrent(record);
|
||||
assert.equal(h.confirmations,2);h.confirm(true);await failed;
|
||||
assert.deepEqual(h.errors,['requestError']);
|
||||
let completeRequest;
|
||||
h.context.send=async(...args)=>{h.sent.push(args);await new Promise(resolve=>{completeRequest=resolve;});};
|
||||
const next=h.context.window.syncdbSkipCurrent(record);h.confirm(true);
|
||||
await new Promise(resolve=>setTimeout(resolve,0));
|
||||
assert.equal(h.sent.length,1);
|
||||
await h.context.window.syncdbSkipCurrent(record);assert.equal(h.confirmations,3,'pending HTTP request also holds the guard');
|
||||
completeRequest();await next;
|
||||
const final=h.context.window.syncdbSkipCurrent(record);assert.equal(h.confirmations,4);
|
||||
h.confirm(false);await final;
|
||||
});
|
||||
|
||||
test('late skip failures after navigation or session changes do not affect the new workspace',async()=>{
|
||||
for(const change of ['route','session']) {
|
||||
const h=controller();let rejectRequest;
|
||||
h.context.send=()=>new Promise((_,reject)=>{rejectRequest=reject;});
|
||||
const pending=h.context.window.syncdbSkipCurrent(h.context.state.executions[0]);h.confirm(true);
|
||||
await new Promise(resolve=>setTimeout(resolve,0));
|
||||
if(change==='route')h.context.location.hash='#tasks';else h.context.authVersion++;
|
||||
rejectRequest(new Error('requestError'));await pending;
|
||||
assert.deepEqual(h.errors,[]);
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,68 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import {validateAdvanced,taskInitial,taskValues} from './src/task-values.js';
|
||||
import {operationValues} from './src/operation-values.js';
|
||||
|
||||
const rejects=(values,field)=>assert.throws(()=>validateAdvanced(values),error=>error.message==='invalidConfig' && error.field===field);
|
||||
|
||||
test('advanced field paths and mapping destinations match synchronization constraints',()=>{
|
||||
validateAdvanced({mapping:{'old.name':'new.name'},defaults:{'new.age':0},conversions:{'new.age':'int'},masking:{'new.name':'hash'},omit:['old.age'],key_fields:['id'],target_objects:{'old.table':'new.table'}});
|
||||
rejects({mapping:{first:'same',second:'same'}},'mapping');
|
||||
for(const field of ['mapping','conversions','masking','defaults'])for(const path of ['a..b','$reserved','a.$reserved','a\0b','中'.repeat(86),'\ud800'])rejects({[field]:{[path]:field==='conversions'?'string':field==='masking'?'hash':'valid'}},field);
|
||||
for(const field of ['omit','key_fields'])for(const path of ['a..b','$reserved','a\nb'])rejects({[field]:[path]},field);
|
||||
rejects({mapping:{source:'a..b'}},'mapping');
|
||||
rejects({target_objects:{source:'bad\0name'}},'target_objects');
|
||||
validateAdvanced({mapping:{['中'.repeat(85)]:'a'},key_fields:['a'.repeat(255)]});
|
||||
});
|
||||
|
||||
test('filter operations accept backend aliases, null tests and typed scalar values',()=>{
|
||||
for(const operator of ['eq','=','==','ne','!=','<>','gt','>','gte','>=','lt','<','lte','<=',' GTE '])validateAdvanced({filters:[{field:'a.b',operator,value:1}]});
|
||||
for(const operator of ['in','nin','not_in','not in'])validateAdvanced({filters:[{field:'id',operator,value:[0,false,{$numberLong:'9007199254740993'},{$oid:'0123456789abcdef01234567'}]}]});
|
||||
for(const operator of ['is_null','is null','not_null','is_not_null','is not null',' IS NULL ']){
|
||||
validateAdvanced({filters:[{field:'id',operator}]});
|
||||
validateAdvanced({filters:[{field:'id',operator,value:null}]});
|
||||
rejects({filters:[{field:'id',operator,value:1}]},'filters');
|
||||
}
|
||||
for(const value of [null,true,'text',0,{$date:'2026-10-05T01:02:03Z'},{$numberDecimal:'123456789.0123'}])validateAdvanced({filters:[{field:'id',operator:'eq',value}]});
|
||||
});
|
||||
|
||||
test('invalid filter shapes, list sizes and non-scalar comparisons are rejected before submission',()=>{
|
||||
for(const filter of [{field:'id',operator:'gt'}, {field:'a..b',operator:'eq',value:1},{field:'id',operator:'eq',value:{unknown:1}},{field:'id',operator:'eq',value:[1]},{field:'id',operator:'in',value:[]},{field:'id',operator:'nin',value:[null]},{field:'id',operator:'in',value:[[1]]},{field:'id',operator:'in',value:[{$numberLong:'9223372036854775808'}]},{field:'id',operator:'__proto__',value:1},{field:'id',operator:'eq',value:1,unknown:true}])rejects({filters:[filter]},'filters');
|
||||
rejects({filters:[{field:'id',operator:'in',value:Array(1001).fill(1)}]},'filters');
|
||||
rejects({filters:Array(101).fill({field:'id',operator:'eq',value:1})},'filters');
|
||||
validateAdvanced({filters:Array(100).fill({field:'id',operator:'in',value:Array(1000).fill(1)})});
|
||||
});
|
||||
|
||||
test('SQL export shares typed and null filter validation with task editing',()=>{
|
||||
const source={id:'source',type:'mysql',environment:'development'};
|
||||
const input={source_id:'source',database:'db',objects:'table',batch_size:200,max_rows:100,filters:'[{"field":"deleted_at","operator":"is_null"},{"field":"id","operator":"not_in","value":[{"$numberLong":"9007199254740993"}]}]',masking:'{}'};
|
||||
const result=operationValues('newExport',input,{},[source]);
|
||||
assert.equal(result.filters[0].operator,'is_null');
|
||||
assert.equal(result.filters[1].value[0].$numberLong,'9007199254740993');
|
||||
assert.throws(()=>operationValues('newExport',{...input,filters:'[{"field":"id","operator":"in","value":[]}]'},{},[source]),error=>error.field==='filters');
|
||||
});
|
||||
|
||||
test('task and export payloads preserve original filter JSON instead of browser-rounded numbers',()=>{
|
||||
const raw='[{"field":"id","operator":"gt","value":9007199254740993},{"field":"amount","operator":"lte","value":12345678901234567890.0001}]';
|
||||
const sources=[{id:'source',type:'mysql',environment:'development',database:'db'},{id:'target',type:'mysql',environment:'development',database:'db'}];
|
||||
const record={name:'task',mode:'range',objects:['table'],filters:JSON.parse(raw),filters_json:raw};
|
||||
const initial=taskInitial(record,sources);
|
||||
assert.equal(initial.filters,raw);
|
||||
assert.equal(taskValues(initial,sources).filters_json,raw);
|
||||
const changed='[{"field":"id","operator":"gt","value":9007199254740995}]';
|
||||
assert.equal(taskValues({...initial,filters:changed},sources).filters_json,changed);
|
||||
assert.equal(taskValues({...initial,mode:'full',filters:''},sources).filters_json,'[]');
|
||||
const input={source_id:'source',database:'db',objects:'table',batch_size:200,max_rows:100,filters:raw,masking:'{}'};
|
||||
assert.equal(operationValues('newExport',input,{},sources).filters_json,raw);
|
||||
});
|
||||
|
||||
test('default JSON stays exact in editable task configuration, including nested arrays',()=>{
|
||||
const raw='{"amount":12345678901234567890.0001,"large":9007199254740993,"nested":{"values":[9007199254740995]}}';
|
||||
const sources=[{id:'source',database:'db'},{id:'target',database:'db'}];
|
||||
const input=taskInitial({name:'task',objects:['table'],defaults:JSON.parse(raw),defaults_json:raw},sources);
|
||||
assert.equal(input.defaults,raw);
|
||||
assert.equal(taskValues(input,sources).defaults_json,raw);
|
||||
assert.equal(taskValues({...input,defaults:''},sources).defaults_json,'{}');
|
||||
const changed='{"large":9007199254740995}';
|
||||
assert.equal(taskValues({...input,defaults:changed},sources).defaults_json,changed);
|
||||
});
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user