mirror of
https://github.com/wweir/sower.git
synced 2026-10-19 16:14:33 +00:00
fix(sowerd): give the service a soft heap ceiling on every install path
The 2026-10-12 OOM killed the host, not the service: sowerd had no memory
ceiling anywhere while the sower client has had SOWER_MEMORY_LIMIT_MB since
d0031e4. Put a GOMEMLIMIT on the unit the installer writes and on the
packaged unit, so a runaway heap degrades sowerd instead of OOM-killing the
machine it runs on. kr already carries the same value through a drop-in.
This commit is contained in:
@@ -7,6 +7,9 @@ Type=simple
|
||||
ExecStart=/usr/local/bin/sowerd
|
||||
Environment="FAKE_SITE=127.0.0.1:8080"
|
||||
Environment="PASSWORD=I_am_Passw0rd"
|
||||
# Soft heap ceiling: a runaway heap degrades the service instead of
|
||||
# OOM-killing the host (docs/security.md, Connection Resource Bounds).
|
||||
Environment="GOMEMLIMIT=512MiB"
|
||||
Restart=on-failure
|
||||
RestartSec=1min
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ jobs:
|
||||
- uses: actions/checkout@v7
|
||||
- uses: actions/setup-go@v7
|
||||
with:
|
||||
go-version: ^1.26
|
||||
go-version: ^1.27
|
||||
- uses: oven-sh/setup-bun@v2
|
||||
|
||||
- name: test and build
|
||||
|
||||
@@ -92,7 +92,7 @@ jobs:
|
||||
- uses: actions/checkout@v7
|
||||
- uses: actions/setup-go@v7
|
||||
with:
|
||||
go-version: ^1.26
|
||||
go-version: ^1.27
|
||||
- uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: web-dist
|
||||
|
||||
@@ -1,6 +1,20 @@
|
||||
# Plan: sowerd 连接资源上界(2026-10-12 OOM 复盘)
|
||||
|
||||
状态:实施中(2026-10-12)。本文为非权威记录;定稿后的长期规则回写 [security.md](../security.md)。
|
||||
状态:已定稿并回写(2026-10-12)。本文为非权威历史记录,权威规则见 [security.md](../security.md) 的
|
||||
"Connection Resource Bounds"。
|
||||
|
||||
实施结果摘要:
|
||||
|
||||
- 落地 commit `28c4f9e`:`newRedirectServer` 补齐 :80 的四个超时;`serve443` 加
|
||||
`maxInboundConns = 2048` 并发上界;`writeStallListener`/`writeStallConn` 位于 TLS
|
||||
之下,为每次写入武装 `writeStallTimeout`;`dumpHeapOnSignal` 在 SIGUSR1 时写 heap profile。
|
||||
- kr 已部署 `v1.7.1-4-g28c4f9e`(sha256 `9ec44038190b...`,备份
|
||||
`/usr/local/bin/sowerd.bak-20261012175623`),并以 drop-in
|
||||
`/etc/systemd/system/sowerd.service.d/10-memory.conf` 设 `GOMEMLIMIT=512MiB`。
|
||||
- 验证:`:80` 仍 308 跳转;`:443` 回退反代 200;SIGUSR1 产出
|
||||
`/var/cache/sower/sowerd-heap-*.pprof`;用 `transport/sower` 从本机经 kr 中继
|
||||
`www.baidu.com:80` 得到 HTTP/1.0 200(证明新增的写超时不误伤中继);r6c 客户端
|
||||
socks5 到 google 仍 204,其日志自 17:31 起无握手失败。
|
||||
|
||||
## 事件
|
||||
|
||||
|
||||
@@ -207,6 +207,36 @@ sower clients should set `[remote.tls] client_hello = "chrome"` (uTLS) in
|
||||
deployment configs. This is a client configuration knob, not a protocol
|
||||
property, and has no server-side effect.
|
||||
|
||||
## Connection Resource Bounds
|
||||
|
||||
Both listeners are internet-facing, so an inbound connection must have a
|
||||
bounded cost and a bounded lifetime. These bounds are normative:
|
||||
|
||||
- The plain-HTTP listener (`:80`) sets `ReadHeaderTimeout`, `ReadTimeout`,
|
||||
`WriteTimeout` and `IdleTimeout`. It only ever serves a redirect or an ACME
|
||||
challenge, so a connection that is slow, idle or half-sent has nothing to
|
||||
wait for; without the timeouts a peer can pin a goroutine, a socket and its
|
||||
buffers indefinitely, and nothing about it is logged.
|
||||
- The TLS listener (`:443`) serves at most `maxInboundConns` connections at
|
||||
once; a connection that finds no slot is closed on accept instead of queued.
|
||||
Per-connection state (TLS record buffers, relay copy buffers, goroutine
|
||||
stacks) is what turns an arrival-rate burst into resident memory.
|
||||
- Every write on a `:443` connection carries a deadline (`writeStallTimeout`,
|
||||
armed below the TLS listener, so handshake, reverse-proxied response and
|
||||
hijacked tunnel are all covered). The relay library only arms read
|
||||
deadlines, so a peer that stops reading would otherwise block its writer
|
||||
forever, leaking the connection pair, its goroutines and its descriptors.
|
||||
Idle tunnels issue no writes and are not affected.
|
||||
- A heap profile is written per `SIGUSR1` (`systemctl kill -s USR1 sowerd`,
|
||||
read with `go tool pprof`). It is the only way to attribute a grown heap to
|
||||
an allocation site, and unlike an HTTP endpoint it adds no network surface.
|
||||
|
||||
The service unit also carries a soft heap ceiling (`GOMEMLIMIT`), so an
|
||||
unbounded heap degrades this service instead of OOM-killing the host: the
|
||||
2026-10-12 OOM on `kr` (970 MiB anon-rss on a 1.6 GiB host) killed the machine,
|
||||
not the service. Historical analysis and the still-unproven trigger live in `docs/plans/`
|
||||
(2026-10-12, sowerd connection bounds; non-authoritative).
|
||||
|
||||
## Rejected Directions
|
||||
|
||||
Decisions recorded here so they are not re-litigated without new evidence:
|
||||
|
||||
@@ -191,6 +191,9 @@ Wants=network-online.target
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart="%s" -c "%s"
|
||||
# Soft heap ceiling: a runaway heap degrades this service instead of
|
||||
# OOM-killing the host (see docs/security.md, Connection Resource Bounds).
|
||||
Environment=GOMEMLIMIT=512MiB
|
||||
Restart=always
|
||||
RestartSec=5s
|
||||
|
||||
|
||||
Reference in New Issue
Block a user