LinkPort/docs/high-availability.md

High availability deployment

Serviceport can run as two independent gateway nodes behind an external TCP load balancer. The example in deploy/ha/haproxy.cfg uses node A as primary and node B as standby. It is a deployment recipe, not an embedded cluster or a validated production cluster.

Node layout

Component	Node A	Node B
Host	10.20.0.11	10.20.0.12
Gateway listener	0.0.0.0:3307	0.0.0.0:3307
Management listener	10.20.0.11:8080	10.20.0.12:8080
SQLite	/var/lib/serviceport-a/serviceport.sqlite	/var/lib/serviceport-b/serviceport.sqlite
Audit	Node A data directory	Node B data directory

Use the same serviceport release and gateway/target configuration on both nodes. Adapt the environment examples to your service manager and load values from your secret manager. The .env.example files are templates; serviceport does not load them automatically. Restrict network access to gateway ports and use an authenticated HTTPS reverse proxy or an SSH tunnel for management. Firewall management port 8080 to the load balancer and operator network only; the listener itself is HTTP and must not be exposed to untrusted networks.

Never let two instances open the same SQLite database. Do not put SQLite, its WAL files, or audit files on a shared network volume for this arrangement. Management sessions, configuration revisions, metrics, and audit records are independent on each node. There is no distributed leader election, replication, or shared session state.

Provision the standby

1. Configure and test the primary against isolated test accounts first. Confirm both nodes can reach the configured target services and outbound transports.
2. Stop the owned primary service before taking an offline snapshot. Back up the complete data directory, including the database, any WAL files, audit directory, and TLS material. Keep the original backup immutable.
3. Restore a copy of the snapshot into the standby's own data directory. Supply the same externally managed master key required to decrypt the snapshot. A Windows master-key.dpapi is tied to its original Windows user and cannot simply be copied to another machine as a portable key. For that case, provision credentials through the destination's management interface or use POST /api/v1/configuration/backup with a passphrase and POST /api/v1/configuration/import with the encrypted backup and passphrase.
4. Configure each node's gateway bind address and port. Provision gateway TLS certificates valid for the public load-balancer hostname. Pin target SSH host keys independently and check their expected fingerprints.
5. Start both services and their bound gateways, check management /readyz locally, test each gateway directly with a disposable account, and only then enable load-balancer routing. An offline data copy inherits configuration; it does not establish ongoing synchronization.

Future configuration changes must be applied to both nodes explicitly. Keep one authoritative change record, back up each node before an update, and verify target/account mappings, account permissions, TLS identity and listener ports on both nodes. A configuration import does not merge active sessions or audit history. Avoid overwriting a running node's database to synchronize it. Preview the import on the destination node, stop all its gateways, and submit the returned revision with the import. A stale preview is rejected with HTTP 409. Imported gateways remain stopped until explicitly started. Templates and redacted exports omit credentials; encrypted configuration backups retain them. Health-check intervals persist locally and resume after restart, checking every target account. Configure and verify monitoring on each node independently.

Health and failover

The example HAProxy checks call each node's management /readyz endpoint. It returns HTTP 200 with {"status":"ready"} only when at least one gateway has a bound target and all bound gateways have running listeners. Unbound drafts are ignored. Empty configuration, a stopped bound gateway, or a configuration read failure returns HTTP 503 with {"status":"not_ready"}. The unauthenticated response contains no configuration or credential details and disables caching. /healthz remains a process-liveness endpoint.

Readiness is node-wide: stopping any bound gateway removes the node from all backends using this probe, even when another gateway is still running. Drain the node before planned maintenance. Readiness checks listener state, not target connectivity, target authentication or matching configuration between nodes. Use POST /api/v1/monitoring/check with targetId and accountId for an authenticated protocol health probe. Periodic checks are enabled explicitly through POST /api/v1/monitoring/settings with intervalSeconds between 10 and 86400; zero disables them. Settings and metrics reset after service restart. Periodic checks select the first configured target account and do not automatically change load-balancer membership.

Health probe retries are limited to three attempts and 25 seconds total. Authentication and certificate errors are not retried. These probes never replay traffic from a real gateway session. Do not replace the example health check with recurring unauthenticated connections to protocol ports: those probes inflate socket-attempt and unauthenticated-closure metrics. Verify gateway listener availability with controlled authenticated client checks when changing configuration. Interpret connection and authentication failure rates separately.

The standby receives new connections after the primary fails its checks. Existing database transactions, subscriptions, cursors and SSH sessions cannot be transferred: clients must reconnect, and application owners must decide whether an operation can be retried. Serviceport does not replay commands. With the example's 5-second interval and three failures, detection takes roughly 15 seconds plus probe timeout. Tune timeouts for the application's long-lived sessions. The load balancer itself also needs redundancy.

TCP proxying changes the peer IP visible to serviceport to the load balancer's address. Source-IP rules therefore match that address; enforce original-client IP restrictions at the load balancer/firewall. Do not enable HAProxy send-proxy: serviceport does not consume the PROXY protocol preamble.

Validation before production

Use isolated databases, credentials and ports to verify direct authentication on both nodes, rejection of disabled accounts, account permissions, TLS and SSH identity checks, failover of new connections, client reconnect behavior, and return of the primary. Validate the HAProxy configuration with haproxy -c -f deploy/ha/haproxy.cfg in the deployment environment. This example has not been exercised against a live two-node cluster.