Clustering#

Many hosts, one actor system. The single-activation guarantee still holds across the fleet — that is what the directory is for.

import { defineActorApp } from '@sigx/actors/host';
import { cluster } from '@sigx/actors/cluster';
import { redisCluster } from '@sigx/actors-redis';

export const app = defineActorApp({ actors, storage }).use(
    cluster({
        providers: redisCluster({ url: process.env.REDIS_URL }), // membership + directory
        advertise: 'http://10.0.4.7:7311',   // this host's peer-reachable origin
        secret: process.env.HOST_SECRET,     // declared ONCE
    }),
);

The two providers#

Membership answers "which hosts are alive?" — a heartbeat plus a view of peers. Fencing falls out of it: a host that has lost its place in membership refuses every activation.

The directory answers "which host owns this actor?" — the claim that makes single activation true cluster-wide.

They are independent choices. Redis, Postgres and SurrealDB provide both; Kubernetes provides membership only, on Leases you are already running, and pairs with either directory:

cluster({
    providers: { membership: k8sMembership(), directory: redisDirectory(client) },
    advertise,
    secret: process.env.HOST_SECRET,
});

One handler, both mounts#

The plugin contributes the internal host-to-host mount as a route, so secret and internalBase are stated once instead of repeated at the endpoint, and any adapter that mounts app.routes picks up host-to-host traffic automatically.

const handler = createAppHandler(app);   // public endpoint AND internal route

let stopping = false;
const server = createServer((req, res) => {
    if (stopping) res.setHeader('connection', 'close');
    handler(req, res);
});

// Listen BEFORE starting: app.start() joins membership, and from that moment
// peers may place actors here and call them. Bind first and there is no
// window where this host is routable but nothing is listening.
await new Promise<void>((resolve) => server.listen(7311, resolve));
const host = await app.start();

attachSignalHandlers(host, { server, onStopBegin: () => (stopping = true) });

Shutdown: pass the server, not just the host#

Stopping the actors is only half a graceful shutdown. An orchestrator's preStop sleep and readiness-503 steer new connections away, but connections already established survive endpoint removal — conntrack — and ride into the exiting pod, where they are reset when the process exits.

On a real cluster that measured as 122 lost calls out of ~1.7M on a rolling restart. All connection-level, none of them visible from the actor layer, which reported a clean hand-off.

The sequence is deliberately not the obvious one:

1. onStopBegin() — start answering connection: close. This is what actually drains the pools, one response at a time, interrupting nothing.
2. host.stop() — the actor drain. Pooled connections keep flowing; peers whose dials are refused see unreachable, which is retryable by design.
3. server.close() + closeAllConnections() — last.

Closing the listener first looks more decisive and is worse. On Node ≥ 19, close() also destroys idle connections, and "idle" from the server's side includes a socket the client is at that instant writing its next request onto — producing exactly the reset the sequence exists to prevent.

Options#

fetch and endpoint remain as sugar for exactly those two httpTransport() options, and passing either alongside an explicit transport throws.

The secret is not optional#

cluster() throws at construction without a secret, in every build but development.

The internal host-to-host mount runs no policies — the public edge is expected to have decided already — and it is contributed as an ordinary route, so it lands on the same listener as the public actor endpoint. Without a secret, every registered actor type, key and method is reachable unauthenticated by anything that can address that listener.

cluster({ providers, advertise, secret: process.env.HOST_SECRET });   // normal
cluster({ providers, advertise, secret: null });                     // explicit opt-out

secret: null is for a mount genuinely unreachable by an untrusted caller — an mTLS mesh, a private network with no other tenant. It is a claim you are making, which is why it has to be written down. An empty or whitespace-only secret throws in every build, development included.

The HMAC authenticates the peer, not the payload.

Fencing#

A host fences itself when it can no longer prove it holds its place in membership. Fenced is terminal: activations are refused, liveness fails, and an orchestrator restart is the way back — a host that has been aged out by its peers cannot talk itself back in.

Three things trigger it:

A fenced host also withdraws from membership, so peers stop routing to it immediately rather than waiting out its TTL.

Writing a provider#

Drive heartbeatClock() from your heartbeat rather than hand-rolling a TTL comparison:

import { heartbeatClock } from '@sigx/actors/cluster';

const clock = heartbeatClock({ ttlMs, onSuspect: () => fenceThisHost() });

clock.arm();                                    // once, when beating starts
// each beat:
clock.beat();                                   // before the write
try { await writeHeartbeat(); clock.confirmed(); }
catch { clock.failed(); }

It watches the monotonic and the wall clock together, because CLOCK_MONOTONIC does not advance across a VM suspend — and setTimeout rides that same clock, so a suspended host would otherwise conclude every beat had been punctual. The window is stamped when the beat is armed, not at the join write, so a slow join cannot fence a host at startup.

clock.lost() is the fourth door, for a provider holding proof its record is gone rather than a suspicion inferred from elapsed time — a Kubernetes Lease deleted out from under its host, say. It fires immediately whatever the clocks say, because the next write would succeed promptly and look perfectly healthy. Like every other trigger it is latched, cleared by the next confirmed(), and silent before arm().

Testing it#

memoryClusterHub() can simulate both shapes of loss:

hub.kill(hostId);      // a crash the host notices — fires onSelfSuspect
hub.expire(hostId);    // a TTL lapse it is never told about

expire() is the one worth reaching for: no cleanup, no notification, exactly the case a provider has to detect for itself.

Membership refresh coalescing#

Membership providers that support push notifications coalesce their refreshes: leading-edge immediate, then single-flight with version-gated hint skipping, so a burst of N changes costs one re-read rather than N per subscriber.

coalesceMs (default 0) is the knob, on redisCluster/redisMembership, pgCluster/pgMembership and surrealCluster/surrealMembership:

redisCluster({ url, coalesceMs: 50 })

0 keeps the leading-edge behaviour with no trailing window — every notification that is not already in flight refreshes immediately. Raise it when a noisy membership channel is costing more than the staleness it saves.

Writing a custom ClusterMembership? refreshCoalescer() from @sigx/actors/cluster is the same primitive: demand() preserves ClusterMembership.refresh()'s started-at-or-after contract, and settled() is for teardown.

What clustering does not change#

Your actor code. Nothing in a definition knows whether it is running on one host or fifty — placement, the directory and the transport are all host concerns.

Two things are worth knowing anyway: a live subscription is never routed or redirected, so it keeps paying the cross-host hop; and stateless workers are always local and invisible to the directory entirely.

Next steps#

• Placement policies — where new activations go.
• Transports — how hosts talk, and which to pick.
• Locality routing — getting the edge to help.
• Rebalancing — correcting a lumpy fleet.