A running tun entrypoint cannot tell a dead device fd from an idle link: both look like frozen stats. The probe writes a self-addressed IPv4/UDP packet (spokeIP->spokeIP, magic+seq) into the device every 30s and reads it back on its own bound socket; sent/acked deltas go to the caller through the metadata probeReport callback. Enabled by metadata probe (default off, gost path unchanged); new probeSent/probeAcked counters (101/102) in x stats.
IsReference no longer uses reflect.DeepEqual; ResolveConfig trims the name before registry lookup; cloneConfig deep-copies all nested fields so hop parsing never mutates shared globals. Also documents the intentional fail-fast vs warn-and-skip error semantics.
A 20546-byte spoke datagram hit wbuf[:writeOffset+len(p)] (cap 16400)
and panicked the whole hub process. Three parts:
- tunDevice.Write rejects packets bigger than its buffers with the new
exported ErrPacketTooLarge instead of reslicing past the end. The
kernel would have refused the write anyway (tun MTU 1420), so nothing
deliverable is lost.
- Buffers follow the device MTU (max 16KB): newTunDevice(ifce, mtu),
so raising tun.mtu for legitimate jumbo UDP automatically makes such
packets writable. The relay/KCP/stream path already passes 64KB.
- fromSpoke drops an oversized datagram with a windowed count and
keeps the peer stream up; only a real device failure still tears
the stream down.
Tests: tun write guard, normal passthrough, large-MTU accept, and
stream survival with rate-limited warnings; all written first and
watched fail.
An unrouted destination under load produced one warn line per packet,
burying the event that caused the routing gap. Count misses and report
the total once per window instead; the first miss is still logged at
once.
Also log how long a peer stream lived and how much arrived on it when
it ends, so a dying peer's story (failed to connect vs failed mid-
transfer) is visible without correlating route drops.
Also ignore .memory/.
The tun link needs the tunnel network "ip" rather than the "udp" its dialer
would otherwise derive: "ip" is the datagram link, marked session-scoped on
the host so it ends on a lost peer session and the consumer re-dials (see
the p2p companion change). Add WithTunnelNetwork to override the derived
network and MDKeyP2PNetwork ("p2p.network") for a chain node to set it, and
have streamconn frame "ip" like "udp" so the GOST-side conn owns the 2-byte
framing on both carriers. An empty network keeps the derived value, so every
existing node is unchanged.
Gate: go build ./... and go vet ./... clean.
When no live session exists, each request dials a conn of its own and
registers it in Dial, then builds the session on it in Handshake, each
step under sessionMutex. A Dial that finds a registration still waiting
for its handshake takes it for dead (IsClosed reports true while
session is nil) and replaces it, so the earlier request's Handshake no
longer finds its own conn and fails with "unrecognized connection".
Since c210b74 the replacing Dial also closes that conn.
Once requests queue for the lock, another request's Dial can sit
between every Dial and its Handshake, and no session is completed for
as long as requests keep arriving. Seen on a three-hop relay chain
whose last hop is mtls: after the client host woke from sleep, nearly
every request on a chain with ~28 concurrent clients failed this way
for 80 minutes, until the load happened to drop.
Dial now closes the base conn only of a session that was established
and died; a session still waiting for its handshake belongs to the
request that dialed it, whose Handshake either builds the session on
it or closes it. Handshake no longer fails on a mismatch: if the
registered session is live it uses that and closes its own conn,
otherwise it builds the session on its own conn and registers it, and
the other request's Handshake then finds a live session to use.
The tests dial twice before either handshake, in both orders, against
the package's own listener. On master they fail with "unrecognized
connection", and with "use of closed network connection" when only
Handshake is changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A registration handshake or keepalive echo is protocol, not a packet,
and must never reach the device. The client read loop only special-cased
the 20-byte bare echo, so a 20+N*16-byte registration frame from the
peer was misread as IPv4 ('G' is 0x47) and injected into the kernel
device as junk. The hub (p2pHub.fromSpoke) and the socket server already
filter this shape; the client now does too.
Covered by TestTransportClientDropsKeepaliveFrames, which fails against
the old shape (the device receives the frame verbatim).
The comment above newPeerTable claimed "the authorizer is an option rather
than an argument", written when NewP2PHandler did not take one. It now does,
so the comment contradicted the signature directly above it, and it still
referred to an "auther argument" that no longer exists.
Both facts are true but at different levels, which is what made the old
wording hard to read: the authorizer is a variadic option on newPeerTable
(whose other callers all mean "no address authorization"), while
NewP2PHandler takes it as an argument so a hub's owner has one place to
pass it. The auther is nil throughout, since the p2p peer allowlist has
already decided who may connect.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The bypass was evaluated once per connection, which authorizes the first
request and nothing after it. A keep-alive request naming another host
re-dialed without a check, a request served from the HTTP cache needed no
dial and so passed no check, a request arriving after a reload was still
judged by the policy that had been in force when the connection opened,
and an HTTP/2 :authority was never checked at all. With MITM enabled the
same holes applied to HTTPS, since the decrypted stream is handed to the
HTTP path.
State the invariant instead of patching the sites one by one: every
request forwarded to an upstream is authorized against the bypass in
effect at the time of that request. CheckBypass is that one check, called
from the four request-handling paths in the two sniffer copies. In the
HTTP loop it replaces the pre-loop check rather than adding to it, so the
file ends up with one check site instead of two.
A reload now reaches established connections at their next request. That
is a behaviour change, and it is what closes the cache and reload paths.
Tunnels stay authorized at establishment: a TLS connection without MITM,
a websocket after upgrade and a CONNECT tunnel have no later request to
check.
CheckBypass is nil-receiver safe because h2Handler can be built without
handle options, and a data path should not panic over that.
Measured cost against a 269-pattern bypass: 645 ns for a host that
matches nothing (full wildcard scan), 149 ns for an exact hit.
A tun client registers its device's addresses with the far side's tun
server by sending the handshake frame. That frame is what tells the server
which addresses this device holds, and it is the only thing that can: a
server routes by destination address, so without it the link is one-way —
traffic leaves and nothing comes back.
keepalive() has always sent it unconditionally and returned early when no
period was configured, so the period only ever decided whether it
repeated. The call site did not: it was gated on the udp network or a
configured period, which on a p2p link ("ip") meant no period, no
handshake, no registration at all. That is the shape a p2p hub needs
least — its route table has no TTL to refresh, so the repeat buys nothing
there and the one-shot is the whole of it.
The gate is gone; the handshake always runs. The period keeps its meaning
where it has one: a server reached over UDP is connectionless, so silence
is the only signal that a peer has left, and the repeat is what lets its
route expire.
Found by a user asking why the keepalive switch was needed at all. It was
not, on a p2p hub — and turning it off silently produced the one-way link
the entrypoint's own comment warns about.
gost#911: two concurrent UDP source sockets over one RUDP reverse tunnel
over relay+TLS corrupted echo responses and made the host tear the tunnel
down and rebind it every second.
The tunnel is a single stream shared by every endpoint multiplexed over it.
A datagram frame is not written to that stream atomically —
UDPDatagram.WriteTo issues three separate Write calls (RSV/FRAG, SOCKS5
address, payload). Each Write is concurrency-safe on its own, but with
several endpoints writing, frames interleave between those calls. A torn
frame also leaves half a header in the stream, which the far end reads as
io.ErrUnexpectedEOF and answers by rebuilding the whole tunnel.
Hold a mutex for the whole frame instead of just the writes the datagram
happens to make. Same defect in both udpTunConn copies.
Reproduced with the reporter's script: relay+tls concurrent phase went from
44% loss with corrupted payloads and 1.0s gaps to 0.02% loss, no corruption,
0.003s gaps. Plain relay likewise. Single-endpoint high-rate loss (~35% at
14k pps) is unaffected UDP backpressure, not this bug.
p2p.go was a complete engine with nothing driving it: no handler.Handler,
so no GOST service could reach the hub and a spoke's registration frame had
no path to one at all.
NewP2PHandler takes the tun device conn and an auther, builds the peerTable
(no TTL — a stream close says a peer is gone outright, where a socket peer
has only silence to guess from), the peerRouter and the p2pHub, and starts
the device reader. Handle installs one accepted stream under its remote
address — the peer key the p2p host stamped on it, replacing whatever a
reconnect left behind — and feeds its datagrams to the hub until the read
fails. Teardown is peerGone and nothing else: it reclaims a peer's routes
only while that stream still holds its key, so a stale stream ending cannot
cut off the live connection its peer already replaced it with.
ownNets comes off the device conn's context, the two steps server.go's Handle
already uses, because a nil answer silently disables the self-route guard.
Handle returns the read error unwrapped. io.EOF, net.ErrClosed and
context.Canceled are what isServerError calls an ordinary end, so a peer
reconnecting would otherwise land in someone's error log.
The p2p half of the compat suite now runs against the real handler and the
real hub rather than reporting itself unavailable: the same six assertions,
the same two cases skipped without an auther. A second p2p case covers the
reconnect, which is the only way the suite would notice a peerGone that
dropped routes unconditionally — a hub that did so passes every shared case.
The socket server and the p2p hub are two answers to the same question, and
the design rests on a spoke being unable to tell which one it reached: the
same frame, the same magic, the same passphrase, the same registered
addresses, the same reply it refreshes its read deadline from. A spoke
configured for one is supposed to work against the other.
That claim is only worth something if the *same* assertions run against
both. Two separately-written suites would be allowed to drift, and nobody
would notice until someone deployed one against the other — which is a
field failure with nothing in any log to explain it. So the assertions are
written once here, against a compatHub that names no transport, and each
case is run once per implementation. The duplication that is left is
structural: how to build a hub and how it delivers, which is the part that
genuinely differs.
The socket half drives server.go's own transportServer over a real UDP link
rather than a copy of its loops; a compat suite that asserted against a
reimplementation would only prove the reimplementation agreed with itself.
The p2p half skips, and says why: nothing yet turns an inbound gRPC Tunnel
stream into a peerStream and feeds it to p2pHub.fromSpoke, so a spoke's
registration frame has no path to that hub. The skip is visible and the
six shared cases are named in it, because an absent assertion is not a
skipped one.
Three deliberate divergences turn the suite red, each from a real behavior
rather than a stub: registering a route for a frame the auther refused,
delivering to whichever peer spoke last, and echoing nothing after a
registration that took.
The route table answers a destination with a name, and the socket server
turns that name into a UDP address. A p2p hub needs the other answer: the
name is a peer key, and the thing it resolves to is a stream, not an
address. That is the whole of the second delivery — resolveUDPAddr is
unreachable here by construction, and putting it back on this path would be
a design error rather than a bug.
The shape is forced by the device, not chosen. tunDevice shares one read
buffer and one write buffer across every call and takes no lock: two
readers would have the second overwrite the first's target mid-call, and two
writers would copy through the same write buffer at once. It is safe today
only because the socket server happens to run one goroutine per side. A p2p
hub runs one per peer, so it has to reach the same guarantee a different
way: run() is the only goroutine that reads, and wmu is the only way to
write. This is the socket server's shape arrived at from the other side —
not a bridge per stream. Which goroutine read a packet carries no meaning,
because the route is what says where it goes.
Two locks, for two different hazards. The device's wmu guards the shared
device buffer. peerStream's own mu guards one peer's stream, whose conn
frames every write onto a byte stream and serializes only its read buffer:
two writers would put one frame's length prefix against the other's
payload, and every frame length would then be read from the wrong offset.
The -race run is what proves both are doing their job.
withdraw removes a stream only while it still holds its key. A peer that
reconnected already installed a replacement under that name, so reclaiming
routes for it on the old stream's teardown would cut off the live
connection — which is what makes a flapping peer unreachable rather than
merely briefly absent.
The keepalive is answered on the stream it arrived on, because the table
registers but never writes. Drop the echo and every keepalive:true spoke
misses the reply its read deadline is refreshed from and redials on its own
timeout, with nothing in any log: client.go's registration gate tests for
"udp" and a p2p link is "ip", so a spoke that never registers is already
indistinguishable from silence. Hence the warning naming the destination.
The concurrency tests are mutation-checked: with the device lock removed the
first fails on overlapping writes in round 0, and with the stream lock
removed the second fails on a spliced frame. Both were caught by the
assertions, not only by the race detector — the first harness was a
datagram pipe, which is atomic per message and so passed with the lock gone
until it was replaced by a byte stream, which is what the p2p conn really
is.
Its keepalive handling, route lookups and both delivery loops move onto
peerTable unchanged: the same magic-header reply, the same gateway
fallback, the same discard for an unrouted packet. Only the route's
value is now a name, resolved at the point of delivery. A test that
needed updating to pass would mean the socket path moved when it must
not have, so the line-by-line diff is the evidence here — no test in
this package exercises server.go.
The route table's reverse index is gone. It was a second copy of a fact
already in the route's name, and two maps cannot be made consistent:
concurrent set calls could leave the index naming one peer while the
route belonged to another, and dropPeer would then delete the winner's
route on the loser's teardown. dropPeer now scans the routes themselves
and deletes only entries still owned by the peer it is dropping.
A lock would have been the wrong fix: lookup runs per delivered packet
and dropPeer once per stream close, so the cost would have landed on the
hot path to settle a race that only matters at reclamation.
The keepalive protocol, the auther check, the TTL and destination lookup
are the same whichever way datagrams arrive, so they belong together and
know nothing about transport. A route's value becomes a string: a UDP
address for the socket server, a peer key for the p2p hub. net.Addr fit
neither — it is an endpoint, and one of these is not.
dropPeer is the p2p half of reclamation. The socket server expires a
route by TTL, guessing from silence that a peer is gone; a stream close
says so exactly, so the hub drops a peer's routes when its stream ends
and keeps no timer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
streamConn embeds the session's net.Conn and overrides Read/Write/Close,
but not the deadline methods, so those reached the connection every
stream shares. Any deadline set by one stream then outlived it and
decided when the others could read.
net/http makes this reachable: tearing down a connection parks a
pending read with SetReadDeadline(time.Unix(1, 0)) and clears it only
when a read was actually in flight. Without a read in flight the shared
conn is left with a deadline in 1970, and the next Accept on the
session fails immediately with an i/o timeout — reported as a listener
error even though the session was healthy.
Delegate the three deadline methods to the stream, which has always had
its own. The test drives one request end to end over a stream, applies
that teardown deadline, and asserts the session still hands out the
next stream.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A remote listener (rtcp/runix/rudp) wraps an accept failure in core's
AcceptError — which always answers Temporary() — to say "rebuild the transport
and try again", and a closed pipe under it is one of the failures that recovers
that way, so the retry stays exactly as it was. What was wrong is what it said: a
stop arrives down the same path, and a warning that promises a retry reads like a
hiccup in the log of a service that is simply being stopped. The retry is logged
at debug when the error means a closed listener, pipe or socket; every other
failure still warns, and the tail still logs real failures at error.
No behaviour change: the retry, its backoff and the state transitions are
untouched — the classification decides only the log level.
Stopping a service closes its listener, and a closed listener answers Accept
with core's listener.ErrClosed — which the guard never matched (it knew only
net.ErrClosed), so every stop reached the log at error level and read like a
crash. The tun handler logged the same shape a second time: its client wrapped a
closed device with %s, throwing the cause away, so the caller could not tell a
device closed under the reader from a real read failure. %w keeps the chain and
the handler logs the closed case at debug.
A configured output path without a rotation block is a plain file that nothing
rotates: it grows until the disk does something about it. The default logger —
the one an unconfigured program gets — always carries a rotation, so this only
happens to a hand-written config, and the parser is the only place that sees
both the path and the rotation that is missing. Reported on stderr, once, at
parse time; nothing about the behaviour changes.
ParseLogger reported the failure through logger.Default(), which is nil until a
caller installs one — and this call is usually what installs it. An unwritable
path therefore dereferenced a nil logger and panicked; inside a c-shared library
that aborts the process, which on Android took an app down a second or two after
start with nothing readable left behind (an app's stderr goes nowhere, and the
tombstone carries no abort text).
Measured with Output "file" — a path, not a mode — in an app whose working
directory is "/" and not writable.
The failure goes to stderr instead, which is always there, and the default
output stays in place. The regression test forks a child process, because a
fresh one is the only place the default logger is still unset: TestMain skips
installing its logger for that child.
An unprivileged Android app cannot open /dev/net/tun, so the device can only
come from a VpnService: the Java side creates and configures it (addresses,
routes, MTU, DNS via VpnService.Builder) and hands its fd over as the "fd"
metadata, either as a fixed descriptor or — for an embedder, which is what the
platform's own VPN is — as a provider resolved when each device is built, so a
device rebuilt after its session ended never copies a closed or recycled
descriptor.
The device is created with tun.CreateUnmonitoredTUNFromFD (no netlink listener,
no setMTU — the builder already set it), which makes the Linux variant, with
its netlink and resolvectl calls, inapplicable: it is now linux && !android.
The interface lookup in the accept loop was fatal on any error; it is
informational, and on Android the device may not be listed in this process.
Bump github.com/go-gost/plugin to v0.8.1, which reports transport stats
over the p2p Status RPC (direct/derp peer gauges plus punch/stream
counters), and pick up the matching go.sum entries.
SupportedDialer: also accept wss, grpc, http2, h2, ssh, sshd and mwss as
inner protocols on top of a p2p tunnel. The list stays fail-closed; each
entry is a dialer verified to dial one plain conn and use it as its base.
Node auth on the HTTP forwarder path read the origin credential
(Authorization) and challenged with 401/WWW-Authenticate. When the
forwarded backend also used Basic auth, the node consumed the credential
meant for the backend, so the two could not coexist.
Read the proxy credential (Proxy-Authorization) instead and challenge
with 407/Proxy-Authenticate. Authorization is left untouched so it
reaches the backend, and Proxy-Authorization is stripped before the
upstream write since it is hop-by-hop (RFC 9110 11.7.2).
Handlers write one recorder entry when a session ends, so long-lived
SOCKS5 tunnels, streaming HTTP responses and UDP associations stay
invisible while they run and report their traffic as one late record.
With recorder.period set, a session now emits interim records on the
interval while it is active and one final record when it ends. Each
record carries the cumulative counters alongside the increments since the
previous one, so a consumer sums records instead of replacing them, plus
sessionID, recordIndex and phase. A session that ends inside one period
still writes a single record, as it does today.
One goroutine per handler samples every live session on the tick and
writes its record straight to the recorder. There is no queue and no
retry loop, and none is needed: a session moves its cursor past a record
only once the recorder has taken it, so a refused write costs no bytes.
They stay in the delta and go out with the next record. Holding on to
what the sink cannot take right now is the sink's job.
Interim and Finish take a context and return an error, so a handler
learns what became of every record it asked for. Start writes nothing, so
it takes neither. Close stops the sampling and reports the sessions its
owner had not finished; nothing is buffered, so there is nothing to
drain.
With recorder.period unset or non-positive the previous behaviour is
kept: one record per session, written inline, in the old shape. An object
its handler never stamped, and one whose record mode is off, is dropped
in both modes.
Three fixes found while wiring the p2p hub topology:
- The client keepalive was gated on network == "udp", but on a p2p link
raddr is the peer's base64 key (no port), so SplitHostPort fails and
network is "ip": the keepalive never ran, a tun server never learned the
client's route, and the read deadline was never armed. Gate on the udp
network OR a configured keepalive period; the udp branch keeps the one-shot
registration handshake it always sent.
- Liveness was refreshed only by the keepalive echo; a transient channel
hiccup can drop that single packet and redial a healthy tunnel after
3xperiod. Refresh the read deadline after any successful read instead.
- Routes were never removed: a departed client's IP -> addr stayed forever
and the device kept blackholing to a dead address. Expire a route once its
last keepalive is older than 3x the configured period, checked lazily on
the write path (no sweeper). The TTL is opt-in: without a server-side
keepalive/ttl, routes keep their current never-expiring behavior.
The read deadline set for ClientHello sniffing was reset via defer, so it
remained on the client connection until Handle returned, i.e. for the whole
session. xnet.Pipe is called without a read timeout for TLS/HTTP, so nothing
refreshed it and any connection was torn down after readTimeout (15s default)
regardless of traffic. Clear the deadline immediately after sniffing, matching
the relay/router/tunnel/socks4 handlers.
Regression test asserts the deadline is zero when the upstream dial fires.
Fixes#904
The test fed all 715 chunks before the first Read; with the cap-1 pump
handoff and a 64-slot fake stream the feeder blocks long before the
reader starts — a deterministic deadlock the earlier "passing" runs
masked (a `go test | tail` pipeline hid the exit code). Feed from a
goroutine.
readDatagram kept appending into one backing array; an idle conn held
onto the largest frame's array indefinitely (bounded, but anchored until
a later append crossed capacity). Drain it to nil once a frame boundary
empties it, and assert the release plus the received-bytes-only bound in
the tests.
TunnelProvider becomes OpenTunnelStream: OpenTunnel authorizes the tunnel
and issues a single-use id, the Tunnel stream carries the data. The new
x/p2p/streamconn conn wraps the stream as a net.Conn with real
Set*Deadline support (bounded Recv pump, idempotent Close, framed mode
for udp). The tunnel base dialer collapses to the stream; its failure
path closes the conn nil-safely instead of leaking.
udp.Relay.Run ignored ctx cancellation, so an idle MASQUE association (or a
disconnecting client) leaked its target UDP socket and both copy goroutines,
and service shutdown blocked on the stuck handlers. quic-go's
ReceiveDatagram only unblocks on ctx cancellation, never on stream close, so
closing the datagram conn could not release the read either.
Make Relay.Run cancellation-aware and close both conns on exit, add
WithReadTimeout for an idle read deadline, give DatagramConn a cancelable
context that Close cancels, and apply the parsed idleTimeout to the UDP
relay.
Ref: https://github.com/go-gost/x/issues/128
ginuerzh