fix(sowerd): give the service a soft heap ceiling on every install path

The 2026-10-12 OOM killed the host, not the service: sowerd had no memory
ceiling anywhere while the sower client has had SOWER_MEMORY_LIMIT_MB since
d0031e4. Put a GOMEMLIMIT on the unit the installer writes and on the
packaged unit, so a runaway heap degrades sowerd instead of OOM-killing the
machine it runs on. kr already carries the same value through a drop-in.
This commit is contained in:
wweir
2026-10-12 17:58:08 +08:00
parent 28c4f9e8ca
commit 3c869cd32d
6 changed files with 53 additions and 3 deletions
+3
View File
@@ -7,6 +7,9 @@ Type=simple
ExecStart=/usr/local/bin/sowerd
Environment="FAKE_SITE=127.0.0.1:8080"
Environment="PASSWORD=I_am_Passw0rd"
# Soft heap ceiling: a runaway heap degrades the service instead of
# OOM-killing the host (docs/security.md, Connection Resource Bounds).
Environment="GOMEMLIMIT=512MiB"
Restart=on-failure
RestartSec=1min
+1 -1
View File
@@ -11,7 +11,7 @@ jobs:
- uses: actions/checkout@v7
- uses: actions/setup-go@v7
with:
go-version: ^1.26
go-version: ^1.27
- uses: oven-sh/setup-bun@v2
- name: test and build
+1 -1
View File
@@ -92,7 +92,7 @@ jobs:
- uses: actions/checkout@v7
- uses: actions/setup-go@v7
with:
go-version: ^1.26
go-version: ^1.27
- uses: actions/download-artifact@v8
with:
name: web-dist
@@ -1,6 +1,20 @@
# Plan: sowerd 连接资源上界(2026-10-12 OOM 复盘)
状态:实施中(2026-10-12)。本文为非权威记录;定稿后的长期规则回写 [security.md](../security.md)。
状态:已定稿并回写(2026-10-12)。本文为非权威历史记录,权威规则见 [security.md](../security.md) 的
"Connection Resource Bounds"。
实施结果摘要:
- 落地 commit `28c4f9e`:`newRedirectServer` 补齐 :80 的四个超时;`serve443` 加
`maxInboundConns = 2048` 并发上界;`writeStallListener`/`writeStallConn` 位于 TLS
之下,为每次写入武装 `writeStallTimeout`;`dumpHeapOnSignal` 在 SIGUSR1 时写 heap profile。
- kr 已部署 `v1.7.1-4-g28c4f9e`(sha256 `9ec44038190b...`,备份
`/usr/local/bin/sowerd.bak-20261012175623`),并以 drop-in
`/etc/systemd/system/sowerd.service.d/10-memory.conf` 设 `GOMEMLIMIT=512MiB`。
- 验证:`:80` 仍 308 跳转;`:443` 回退反代 200;SIGUSR1 产出
`/var/cache/sower/sowerd-heap-*.pprof`;用 `transport/sower` 从本机经 kr 中继
`www.baidu.com:80` 得到 HTTP/1.0 200(证明新增的写超时不误伤中继);r6c 客户端
socks5 到 google 仍 204,其日志自 17:31 起无握手失败。
## 事件
+30
View File
@@ -207,6 +207,36 @@ sower clients should set `[remote.tls] client_hello = "chrome"` (uTLS) in
deployment configs. This is a client configuration knob, not a protocol
property, and has no server-side effect.
## Connection Resource Bounds
Both listeners are internet-facing, so an inbound connection must have a
bounded cost and a bounded lifetime. These bounds are normative:
- The plain-HTTP listener (`:80`) sets `ReadHeaderTimeout`, `ReadTimeout`,
`WriteTimeout` and `IdleTimeout`. It only ever serves a redirect or an ACME
challenge, so a connection that is slow, idle or half-sent has nothing to
wait for; without the timeouts a peer can pin a goroutine, a socket and its
buffers indefinitely, and nothing about it is logged.
- The TLS listener (`:443`) serves at most `maxInboundConns` connections at
once; a connection that finds no slot is closed on accept instead of queued.
Per-connection state (TLS record buffers, relay copy buffers, goroutine
stacks) is what turns an arrival-rate burst into resident memory.
- Every write on a `:443` connection carries a deadline (`writeStallTimeout`,
armed below the TLS listener, so handshake, reverse-proxied response and
hijacked tunnel are all covered). The relay library only arms read
deadlines, so a peer that stops reading would otherwise block its writer
forever, leaking the connection pair, its goroutines and its descriptors.
Idle tunnels issue no writes and are not affected.
- A heap profile is written per `SIGUSR1` (`systemctl kill -s USR1 sowerd`,
read with `go tool pprof`). It is the only way to attribute a grown heap to
an allocation site, and unlike an HTTP endpoint it adds no network surface.
The service unit also carries a soft heap ceiling (`GOMEMLIMIT`), so an
unbounded heap degrades this service instead of OOM-killing the host: the
2026-10-12 OOM on `kr` (970 MiB anon-rss on a 1.6 GiB host) killed the machine,
not the service. Historical analysis and the still-unproven trigger live in `docs/plans/`
(2026-10-12, sowerd connection bounds; non-authoritative).
## Rejected Directions
Decisions recorded here so they are not re-litigated without new evidence:
+3
View File
@@ -191,6 +191,9 @@ Wants=network-online.target
[Service]
Type=simple
ExecStart="%s" -c "%s"
# Soft heap ceiling: a runaway heap degrades this service instead of
# OOM-killing the host (see docs/security.md, Connection Resource Bounds).
Environment=GOMEMLIMIT=512MiB
Restart=always
RestartSec=5s