Bound the memory used when parsing uploaded packages, so instances with
low memory limits survive non-standard packages.
1. Metadata files are read through a size-limited reader, archives
themselves stay unbounded so large payloads keep working
2. zstd decoding caps the window at 128 MiB, the zstd CLI default
3. RPM headers go through a first-party reader with rpmlib's size and
array count limits that allocates only what the upload contains, until
https://github.com/sassoftware/go-rpmutils/pull/44 is released. Signing
still verifies the payload digest, and its signature header matches
rpmsign's byte for byte
4. Swift and NuGet read the zip entry they size-checked, a duplicate
name could stand in for it before
5. Terraform states check `LIMIT_SIZE_TERRAFORM_STATE` before parsing,
NuGet symbol PDBs share a 256 MiB budget
6. Oversized container configs are rejected instead of stored without
metadata, invalid Vagrant boxes answer 400 instead of 500
Signed-off-by: techknowlogick <git@tklk.dev>
Co-authored-by: techknowlogick <git@tklk.dev>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: techknowlogick <techknowlogick@gitea.com>
Upgrade `github.com/SaveTheRbtz/zstd-seekable-format-go/pkg` from
`v0.8.3` to `v0.10.0`:
https://github.com/SaveTheRbtz/zstd-seekable-format-go/releases/tag/pkg%2Fv0.10.0
This keeps Gitea's seekable zstd wrapper on the stable v0.10 API while
preserving the existing public `modules/zstd` API.
API migration:
- update `SeekableWriter` and `SeekableReader` internals for the
concrete `*seekable.Writer` and `*seekable.Reader` types introduced by
SaveTheRbtz/zstd-seekable-format-go#264
- update generated dependency metadata after `go mod tidy` removed the
now-unused `github.com/google/btree` transitive dependency
- no Gitea call sites needed changes because `modules/zstd` still
exposes the same constructors and interfaces
Validation:
- `go test ./modules/zstd`
- `make --always-make checks-backend`
---------
Co-authored-by: Giteabot <teabot@gitea.io>
Support compression for Actions logs to save storage space and
bandwidth. Inspired by
https://github.com/go-gitea/gitea/issues/24256#issuecomment-1521153015
The biggest challenge is that the compression format should support
[seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md).
So when users are viewing a part of the log lines, Gitea doesn't need to
download the whole compressed file and decompress it.
That means gzip cannot help here. And I did research, there aren't too
many choices, like bgzip and xz, but I think zstd is the most popular
one. It has an implementation in Golang with
[zstd](https://github.com/klauspost/compress/tree/master/zstd) and
[zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go),
and what is better is that it has good compatibility: a seekable format
zstd file can be read by a regular zstd reader.
This PR introduces a new package `zstd` to combine and wrap the two
packages, to provide a unified and easy-to-use API.
And a new setting `LOG_COMPRESSION` is added to the config, although I
don't see any reason why not to use compression, I think's it's a good
idea to keep the default with `none` to be consistent with old versions.
`LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as
an extension to the file name, so Gitea can determine if it needs
decompression according to the file name when reading. Old files will
keep the format since it's not worth converting them, as they will be
cleared after #31735.
<img width="541" alt="image"
src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9">
