118 lines
6.7 KiB
Markdown
118 lines
6.7 KiB
Markdown
# High availability deployment
|
|
|
|
LinkPort can run as two independent gateway nodes behind an external TCP
|
|
load balancer. The example in `deploy/ha/haproxy.cfg` uses node A as primary and
|
|
node B as standby. It is a deployment recipe, not an embedded cluster or a
|
|
validated production cluster.
|
|
|
|
## Node layout
|
|
|
|
| Component | Node A | Node B |
|
|
| --- | --- | --- |
|
|
| Host | `10.20.0.11` | `10.20.0.12` |
|
|
| Gateway listener | `0.0.0.0:3307` | `0.0.0.0:3307` |
|
|
| Management listener | `10.20.0.11:8080` | `10.20.0.12:8080` |
|
|
| SQLite | `/var/lib/linkport-a/linkport.sqlite` | `/var/lib/linkport-b/linkport.sqlite` |
|
|
| Audit | Node A data directory | Node B data directory |
|
|
|
|
Use the same linkport release and gateway/target configuration on both
|
|
nodes. Adapt the environment examples to your service manager and load values
|
|
from your secret manager. The `.env.example` files are templates; linkport
|
|
does not load them automatically. Restrict network access to gateway ports and
|
|
use an authenticated HTTPS reverse proxy or an SSH tunnel for management.
|
|
Firewall management port 8080 to the load balancer and operator network only;
|
|
the listener itself is HTTP and must not be exposed to untrusted networks.
|
|
|
|
**Never let two instances open the same SQLite database.** Do not put SQLite,
|
|
its WAL files, or audit files on a shared network volume for this arrangement.
|
|
Management sessions, configuration revisions, metrics, and audit records are
|
|
independent on each node. There is no distributed leader election, replication,
|
|
or shared session state.
|
|
|
|
## Provision the standby
|
|
|
|
1. Configure and test the primary against isolated test accounts first. Confirm
|
|
both nodes can reach the configured target services and outbound transports.
|
|
2. Stop the owned primary service before taking an offline snapshot. Back up
|
|
the complete data directory, including the database, any WAL files, audit
|
|
directory, and TLS material. Keep the original backup immutable.
|
|
3. Restore a copy of the snapshot into the standby's own data directory.
|
|
The portable SQLite database contains its key, audit and automatic TLS
|
|
material; preserve private database permissions (0600 on macOS/Linux).
|
|
`LINKPORT_MASTER_KEY` is not used. Migrate legacy external-key databases
|
|
offline before deployment, retaining the complete original backup; see
|
|
[single-file storage](single-file-storage.md).
|
|
4. Configure each node's gateway bind address and port. Provision gateway TLS
|
|
certificates valid for the public load-balancer hostname. Pin target SSH
|
|
host keys independently and check their expected fingerprints.
|
|
5. Start both services and their bound gateways, check management `/readyz`
|
|
locally, test each gateway
|
|
directly with a disposable account, and only then enable load-balancer
|
|
routing. An offline data copy inherits configuration; it does not establish
|
|
ongoing synchronization.
|
|
|
|
Future configuration changes must be applied to both nodes explicitly. Keep
|
|
one authoritative change record, back up each node before an update, and verify
|
|
target/account mappings, account permissions, TLS identity and listener ports
|
|
on both nodes. A configuration import does not merge active sessions or audit
|
|
history. Avoid overwriting a running node's database to synchronize it.
|
|
Preview the import on the destination node, stop all its gateways, and submit
|
|
the returned revision with the import. A stale preview is rejected with HTTP
|
|
409. Imported gateways remain stopped until explicitly started. Templates and
|
|
redacted exports omit credentials; encrypted configuration backups retain them.
|
|
Health-check intervals persist locally and resume after restart, checking every
|
|
target account. Configure and verify monitoring on each node independently.
|
|
|
|
## Health and failover
|
|
|
|
The example HAProxy checks call each node's management `/readyz` endpoint.
|
|
It returns HTTP 200 with `{"status":"ready"}` only when at least one gateway
|
|
has a bound target and all bound gateways have running listeners. Unbound
|
|
drafts are ignored. Empty configuration, a stopped bound gateway, or a
|
|
configuration read failure returns HTTP 503 with `{"status":"not_ready"}`.
|
|
The unauthenticated response contains no configuration or credential details
|
|
and disables caching. `/healthz` remains a process-liveness endpoint.
|
|
|
|
Readiness is node-wide: stopping any bound gateway removes the node from all
|
|
backends using this probe, even when another gateway is still running. Drain
|
|
the node before planned maintenance. Readiness checks listener state, not
|
|
target connectivity, target authentication or matching configuration between
|
|
nodes. Use
|
|
`POST /api/v1/monitoring/check` with `targetId` and `accountId` for an authenticated
|
|
protocol health probe. Periodic checks are enabled explicitly through
|
|
`POST /api/v1/monitoring/settings` with `intervalSeconds` between 10 and 86400;
|
|
zero disables them. Settings and metrics reset after service restart. Periodic
|
|
checks select the first configured target account and do not automatically
|
|
change load-balancer membership.
|
|
|
|
Health probe retries are limited to three attempts and 25 seconds total.
|
|
Authentication and certificate errors are not retried. These probes never
|
|
replay traffic from a real gateway session. Do not replace the example health
|
|
check with recurring unauthenticated connections to protocol ports: those
|
|
probes inflate socket-attempt and unauthenticated-closure metrics. Verify
|
|
gateway listener availability with controlled authenticated client checks
|
|
when changing configuration. Interpret connection and authentication failure
|
|
rates separately.
|
|
|
|
The standby receives new connections after the primary fails its checks.
|
|
Existing database transactions, subscriptions, cursors and SSH sessions cannot
|
|
be transferred: clients must reconnect, and application owners must decide
|
|
whether an operation can be retried. LinkPort does not replay commands.
|
|
With the example's 5-second interval and three failures, detection takes
|
|
roughly 15 seconds plus probe timeout. Tune timeouts for the application's
|
|
long-lived sessions. The load balancer itself also needs redundancy.
|
|
|
|
TCP proxying changes the peer IP visible to linkport to the load balancer's
|
|
address. Source-IP rules therefore match that address; enforce original-client
|
|
IP restrictions at the load balancer/firewall. Do not enable HAProxy
|
|
`send-proxy`: linkport does not consume the PROXY protocol preamble.
|
|
|
|
## Validation before production
|
|
|
|
Use isolated databases, credentials and ports to verify direct authentication
|
|
on both nodes, rejection of disabled accounts, account permissions, TLS and SSH
|
|
identity checks, failover of new connections, client reconnect behavior, and
|
|
return of the primary. Validate the HAProxy configuration with
|
|
`haproxy -c -f deploy/ha/haproxy.cfg` in the deployment environment. This example
|
|
has not been exercised against a live two-node cluster.
|