The 2026-10-12 OOM killed the host, not the service: sowerd had no memory
ceiling anywhere while the sower client has had SOWER_MEMORY_LIMIT_MB since
d0031e4. Put a GOMEMLIMIT on the unit the installer writes and on the
packaged unit, so a runaway heap degrades sowerd instead of OOM-killing the
machine it runs on. kr already carries the same value through a drop-in.
sowerd was OOM-killed on kr with 970 MiB anon-rss on a 1.6 GiB host: a
connection burst in one window ballooned the heap and never came back.
Three unbounded holds made that possible, all on paths the log does not
show:
- the :80 listener had no timeouts, so a connection that never finished a
request kept a goroutine, a socket and its buffers for as long as the
peer stayed connected;
- the :443 accept loop had no concurrency cap, so resident memory grew with
the arrival rate (a failed handshake returns silently);
- relay.Relay only arms read deadlines, so a peer that stopped reading
blocked its writer forever, leaking the connection pair, its goroutines
and its descriptors (and the relay conn log never fired, because the call
never returned).
Arm a write deadline per write below the TLS listener, cap served
connections, set every plain-HTTP timeout, and dump a heap profile on
SIGUSR1 so the next occurrence can be attributed to an allocation site.
Evidence and the open questions (the trigger is still unproven, and r6c's
sower client shows a different, fd-independent growth) are recorded in
docs/plans/2026-10-12-sowerd-connection-bounds.md.
512515d renamed "## 实现决策" to "## 已推翻的实现决策" and kept the
struck-through upgrade claim, but the other two decisions were merged into one
line that stayed under the renamed header although neither was overturned.
Both halves of that line — the outbound Host rewrite, and the four reverse
proxy timeouts — were already stated in ARCHITECTURE.md at the time of that
trim (lines 143 and 146), and this plan's own scope line defers runtime facts
to the code and ARCHITECTURE.md. So the bullet was a duplicate as well as a
misfiling: drop it rather than move it, which also retires the forward note
dab041b had added to patch the stale claim.
The surviving bullet's evidence still exists: TestReverseProxyConnWebSocket and
TestReverseProxyConnPathRoutedWebSocket, cmd/sowerd/main_test.go:777 and :709.
Second half of the review: the rationale for `authority` had been written into
every comment that touched it, which is three copies to keep in sync. Keep the
argument in docs/security.md and leave the config example, the struct comment
and the constants stating only what the value does.
- docs/security.md: record "an upstream trusting X-Forwarded-Host" under
Rejected Directions, where the document already collects declined designs,
instead of inside the new rule section; merge the per-route bullet into the
client-controlled one they overlapped with.
- ARCHITECTURE.md: the sentence added last commit left the fake-site rationale
dangling after the site-route clause; split it back into one claim per
sentence.
- docs/plans/site-routing.md: the plan states the unconditional Host rewrite as
a decision of its time, so it gets a forward note to the authoritative
document rather than a rewrite of the record.
- config/sowerd.go: clearer validation error ("unknown value", naming both
accepted values) and a tighter comment; no behavior change beyond the
message.
A site route always rewrote the outbound Host to the upstream host, so an
upstream that validates the authority it is visited under could not sit behind
one: pi-web compares Origin's host against the Host header, answers 403 to
every browser-originated /api request, and the page loads while its WebSocket
never connects.
Add `authority = "upstream" | "client"` to a site route, defaulting to
"upstream" so existing routes keep the current behavior byte for byte.
"client" forwards the authority the client asked for, which is what such an
upstream needs; its own allowlist must still name the public domain (pi-web's
PI_WEB_ALLOWED_HOSTS), because the forwarded name is client-controlled.
The key is one word on purpose: aconfig v0.19.0 title-cases the keys of an
array-of-tables element, so `preserve_host` is rejected at load time — probed
before the spelling was chosen.
- config: SiteRoute.Authority, the AuthorityUpstream/AuthorityClient values,
and SiteRoute.PreserveHost(). Validate rejects an unknown value instead of
silently defaulting, since the two values differ in what the upstream sees.
- cmd/sowerd: newSiteRouter feeds that flag into the siteEntry field the
fake-site fallback already used.
- docs: ARCHITECTURE.md and security.md record the rule, why the adaptation
belongs to the edge rather than to an upstream trusting X-Forwarded-Host,
and what preserving a client-controlled authority does not protect.
- tests: TOML round-trip, unknown-value rejection, router plumbing, and an
end-to-end handleConn case asserting the upstream sees the client's name.
`sowerd -i` wrote config.ExampleSowerdConfigTOML verbatim, so a fresh
install started with `password = "change_me"` — a publicly known credential
— while Validate rejects only an empty password. The admin console already
generates a random password when none is configured; do the same here.
- renderDefaultConfig replaces the template's top-level `password = ` line
with a random 128-bit hex value (the commented `# password = ...` sample
inside site_routes.auth is left alone) and errors when the template has no
such line, instead of silently keeping a known value.
- The password is printed once during install for the client configs, and an
existing /etc/sower/sowerd.toml is still never rewritten.
- docs: security.md states this under "What the Transport Password Gates"
and moves the risk out of Residual Accepted Signals — a fixable
operational defect is not an inherent signal — recording the declined
alternative (a placeholder blocklist in Validate) with its rationale;
ARCHITECTURE/README describe the generated password.
Gate: gofmt, go vet ./..., go test ./... green; install_e2e_test PASS.
The threat model scoped the adversary as "does not know the sower
password" but never said what holding it grants, which is the question an
operator asks before pointing a public domain at an intranet service:
- one shared secret, no per-client identity and no revocation short of
rotating it for everyone (transport/sower's Sower holds only the password);
- a passing frame reaches any target the host can dial — Sower.Unwrap
rejects only empty hosts and control characters, so there is no allowlist
and a leaked password exposes the whole network position;
- no attempt throttling on the transport path (a wrong checksum just falls
back to the website path); only the site_routes gate is throttled;
- password handling is the operator's: validation rejects only an empty
value, the HMAC key is the raw password, and the shipped default config
still carries the change_me placeholder (now flagged in Residual Accepted
Signals and marked in the example config as requiring replacement).
ARCHITECTURE.md step 11 points at the authority for the no-allowlist fact.
Site-route domains become public once autocert issues for them
(Certificate Transparency), and the edge answers any SNI. An upstream that
cannot authenticate its own visitors is therefore exposed, and sowerd owns
80/443 on the host, so nothing can be placed in front of it.
Add an opt-in per-route visitor gate:
- config: site_routes[].auth {username, password}; both fields required,
empty means ungated, ':' rejected in the username (RFC 7617 splits there).
- runtime: the gate runs in the reverse-proxy handler, so it covers path
overrides and WebSocket upgrades too; the sower transport path is
dispatched earlier and stays untouched. Credentials are compared in
constant time, failed attempts accumulate per source IP and lock it out for
15 minutes (bounded map, mirroring the admin login throttle), and a passing
request has its Authorization header removed so the edge credential never
reaches the upstream. The fake-site decoy is deliberately never gated.
- fix: upstream URL userinfo was silently dropped. ReverseProxy sends
through a Transport, and only net/http.Client turns userinfo into Basic
credentials, so sowerd now sets that header itself; the upstream password
is redacted in proxy error logs (url.URL.Redacted).
- docs: security.md records the mechanism and its limits (not a substitute
for upstream auth, no session/logout, CORS preflight blind spot, and the
browser WebSocket handshake: script cannot set Authorization, cached-
credential replay is browser-specific and unverified, and a reconnecting
client's failed handshakes feed the same per-IP counter, so it can lock its
own source IP out of the whole route) plus the rejected alternatives
(session login, gating the fake site).
Gate: gofmt, go vet ./..., go test ./... green; install_e2e_test PASS.
The rewrite kept the pre-rewrite claim that leaving site_routes unset restores
the old behaviour exactly. It does not: the HTTP-looking fallback is
reverse-proxied with the original Host and edge forwarding headers
(ARCHITECTURE.md, sowerd Data Flow steps 13-14) and NextProtos is http/1.1-only,
which this same file already calls a deliberate decision. Say only what holds:
fake_site stays the single fallback target.
admin-console.md and site-routing.md still carried the full design and
implementation checklists for work that shipped and was written back to
ARCHITECTURE.md. Keep only what cannot be recovered from the code -- the two
frontend constraints and the site-routing trade-offs -- and point at the
authoritative documents. Drop the plan-to-plan "Supersedes" reference from
admin-persistence-config.md so plans stop sourcing rules from other plans.
security.md gains the rejected directions those rewrites now point at:
wildcard site_routes domains (rejected in validateSiteRoutes, and the SNI
dispatch is an exact-match siteRouter.lookup), additional listen addresses or
configurable 80/443 ports, per-site certificate binding, and domain routing on
plain HTTP.
Review of the anti-probing hardening found no code defects; the fixes
are documentation:
- docs/security.md: authoritative threat model, the three camouflage
layers, unknown-SNI certificate fallback rules (whitelist source,
error surfacing, capability propagation, concrete first entry), the
client_hello fingerprint knob, rejected directions (QUIC, h2) with
evidence, and residual accepted signals.
- Promote the plan doc to a non-authoritative historical record per
AGENTS.md docs rules; index security.md from docs/README.md and
ARCHITECTURE.md.
- config/sowerd.toml: document cert.domains; the first entry answers
unknown-SNI handshakes.
A handshake failure on unknown SNI is an active-probing signal: real
sites present their default vhost certificate. Whitelisted names keep
the plain autocert path with errors surfaced as-is; unknown or empty
SNI falls back to the primary domain's certificate, carrying the
client hello over so autocert derives the same ECDSA/RSA cache key.
The h2-on-fallback companion idea was investigated and rejected:
httputil.ReverseProxy has no RFC 8441 support and x/net gates
extended CONNECT behind GODEBUG only, which would break WingGate WSS.
ARCHITECTURE.md and README.md now describe editable config fields,
in-place restart, rule hit/miss stats, and session persistence; the
admin console plan is marked implemented.
- StateStore persists rule deltas and whitelisted config overrides atomically
- Config display/edit API with optimistic concurrency (revision)
- Traffic charts with time-axis linking across all views
- Split server.go into auth/rules/traffic/config by responsibility
- internal/admin: session-cookie auth server, rule CRUD API, bounded
per-domain traffic stats with pre-parse byte accounting
- cmd/sower: wrap proxied connections and DNS queries into stats, serve the
admin console on its own listener
- web/: Svelte 5 + Vite admin console with shadcn-svelte UI and an advanced
breadcrumb header (section/context dropdowns, favorites), embedded via
go:embed and committed to web/dist
- Makefile: sower build depends on the frontend build (bun); sowerd does not
- docs: ARCHITECTURE, README, and plan updated for the admin surface
wweir