Integration test processes now get their own work directory, config file
and random ports, so several runs can execute in parallel on one
machine. The slowest tests also lose fixed sleeps and needlessly
expensive setup, with their assertions kept.
- Batch PostgreSQL sequence resets, the main per-test fixture cost there
- Cheap hashing params in hash and OAuth2 fixture tests, full-cost
vectors stay
- Single fetch for negative Actions runner assertions instead of 1s
polls
- Queue flushes and backdated timestamps instead of sleeps
- In-memory SQLite test journal, tmpfs for PostgreSQL in CI
- E2E instance writes SSH keys into its work directory instead of
`~/.ssh/authorized_keys`
Bound the memory used when parsing uploaded packages, so instances with
low memory limits survive non-standard packages.
1. Metadata files are read through a size-limited reader, archives
themselves stay unbounded so large payloads keep working
2. zstd decoding caps the window at 128 MiB, the zstd CLI default
3. RPM headers go through a first-party reader with rpmlib's size and
array count limits that allocates only what the upload contains, until
https://github.com/sassoftware/go-rpmutils/pull/44 is released. Signing
still verifies the payload digest, and its signature header matches
rpmsign's byte for byte
4. Swift and NuGet read the zip entry they size-checked, a duplicate
name could stand in for it before
5. Terraform states check `LIMIT_SIZE_TERRAFORM_STATE` before parsing,
NuGet symbol PDBs share a 256 MiB budget
6. Oversized container configs are rejected instead of stored without
metadata, invalid Vagrant boxes answer 400 instead of 500
Signed-off-by: techknowlogick <git@tklk.dev>
Co-authored-by: techknowlogick <git@tklk.dev>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: techknowlogick <techknowlogick@gitea.com>
Upgrade `github.com/SaveTheRbtz/zstd-seekable-format-go/pkg` from
`v0.8.3` to `v0.10.0`:
https://github.com/SaveTheRbtz/zstd-seekable-format-go/releases/tag/pkg%2Fv0.10.0
This keeps Gitea's seekable zstd wrapper on the stable v0.10 API while
preserving the existing public `modules/zstd` API.
API migration:
- update `SeekableWriter` and `SeekableReader` internals for the
concrete `*seekable.Writer` and `*seekable.Reader` types introduced by
SaveTheRbtz/zstd-seekable-format-go#264
- update generated dependency metadata after `go mod tidy` removed the
now-unused `github.com/google/btree` transitive dependency
- no Gitea call sites needed changes because `modules/zstd` still
exposes the same constructors and interfaces
Validation:
- `go test ./modules/zstd`
- `make --always-make checks-backend`
---------
Co-authored-by: Giteabot <teabot@gitea.io>
Benefits:
1. smaller binary size (reduces more than 1MB)
2. better control of the assets details
3. fewer unmaintained dependencies
4. faster startup if the assets are not needed
5. won't hang up editors when open "bindata.go" by accident
Support compression for Actions logs to save storage space and
bandwidth. Inspired by
https://github.com/go-gitea/gitea/issues/24256#issuecomment-1521153015
The biggest challenge is that the compression format should support
[seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md).
So when users are viewing a part of the log lines, Gitea doesn't need to
download the whole compressed file and decompress it.
That means gzip cannot help here. And I did research, there aren't too
many choices, like bgzip and xz, but I think zstd is the most popular
one. It has an implementation in Golang with
[zstd](https://github.com/klauspost/compress/tree/master/zstd) and
[zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go),
and what is better is that it has good compatibility: a seekable format
zstd file can be read by a regular zstd reader.
This PR introduces a new package `zstd` to combine and wrap the two
packages, to provide a unified and easy-to-use API.
And a new setting `LOG_COMPRESSION` is added to the config, although I
don't see any reason why not to use compression, I think's it's a good
idea to keep the default with `none` to be consistent with old versions.
`LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as
an extension to the file name, so Gitea can determine if it needs
decompression according to the file name when reading. Old files will
keep the format since it's not worth converting them, as they will be
cleared after #31735.
<img width="541" alt="image"
src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9">
silverwind