feat(import): stream SQL script parsing and drop the 100 MB cap
This commit is contained in:
4 files changed
+737
-76
No files matched your search
@@ -31,7 +31,7 @@ MongoDB Collection 不走这条关系型导入流程。请在 Collection 文档
|
||||
|
||||
SQL 导入只接受字面量 `INSERT INTO ... VALUES` 行;表达式、`REPLACE`、`INSERT IGNORE`、`INSERT ... SELECT` 等无法无损改写的内容会被拒绝,以保留数据和语句语义。
|
||||
|
||||
SQL 导入通过非流式路径读取整个脚本,单个 `.sql` 文件不能超过 100 MB,超出上限会显式报错。`TO_DATE`、`TO_TIMESTAMP` 等日期函数的值会按第一个参数的文本导入,目标字段必须能解析该格式。
|
||||
SQL 导入按分块流式读取脚本,不把整个文件读进内存,因此没有文件体积上限,超大文件的内存占用也保持平稳。总行数要等脚本扫描到结尾才能确定,所以导入过程中的进度按已读取字节数显示,总行数在导入完成后给出。脚本是边扫描边写入的,中途某条语句失败时,已经写入的行会保留在目标表里;预览步骤会先完整解析一遍脚本,所以格式有问题的脚本通常会在写入前就被拦下。`TO_DATE`、`TO_TIMESTAMP` 等日期函数的值会按第一个参数的文本导入,目标字段必须能解析该格式。
|
||||
|
||||
Desktop 直接读取所选本地文件;Docker/Web 会先把浏览器选择的文件上传到服务器临时目录,预览和导入结束后释放临时源文件。
|
||||
|
||||
|
||||
@@ -31,7 +31,7 @@ Create-table mode accepts multiple files at once. Excel workbooks are expanded i
|
||||
|
||||
SQL import accepts only literal `INSERT INTO ... VALUES` rows. Expressions, `REPLACE`, `INSERT IGNORE`, `INSERT ... SELECT`, and other constructs that cannot be rewritten losslessly are rejected so data and statement semantics are preserved.
|
||||
|
||||
SQL import reads the whole script through the non-streaming path, so each `.sql` file is limited to 100 MB; larger files are rejected with an explicit error. Values written with date functions such as `TO_DATE` and `TO_TIMESTAMP` are imported as the text of their first argument, and the target column must be able to parse that format.
|
||||
SQL import streams the script in chunks instead of loading it into memory, so there is no file-size limit and memory use stays flat on large dumps. The total row count is only known once the script has been scanned to the end, so progress during import is reported by bytes read and the row count appears when the import finishes. Because rows are written while the script is still being scanned, a statement that fails half-way leaves the rows already written in the target table; the preview step parses the entire script first, so a malformed dump is normally rejected before any data is written. Values written with date functions such as `TO_DATE` and `TO_TIMESTAMP` are imported as the text of their first argument, and the target column must be able to parse that format.
|
||||
|
||||
Desktop reads selected local files directly. Docker/Web uploads browser-selected files to temporary server storage, then releases the prepared source after preview and import finish.
|
||||
|
||||
|
||||
Reference in new issue
Block a user