Clustering#

Many hosts, one actor system. The single-activation guarantee still holds across the fleet — that is what the directory is for.

import { defineActorApp } from '@sigx/actors/host';
import { cluster } from '@sigx/actors/cluster';
import { redisCluster } from '@sigx/actors-redis';

export const app = defineActorApp({ actors, storage }).use(
    cluster({
        providers: redisCluster({ url: process.env.REDIS_URL }), // membership + directory
        advertise: 'http://10.0.4.7:7311',   // this host's peer-reachable origin
        secret: process.env.HOST_SECRET,     // declared ONCE
    }),
);

The two providers#

Membership answers "which hosts are alive?" — a heartbeat plus a view of peers. Fencing falls out of it: a host that has lost its place in membership refuses every activation.

The directory answers "which host owns this actor?" — the claim that makes single activation true cluster-wide.

They are independent choices. Redis, Postgres and SurrealDB provide both; Kubernetes provides membership only, on Leases you are already running, and pairs with either directory:

cluster({
    providers: { membership: k8sMembership(), directory: redisDirectory(client) },
    advertise,
    secret: process.env.HOST_SECRET,
});

One handler, both mounts#

The plugin contributes the internal host-to-host mount as a route, so secret and internalBase are stated once instead of repeated at the endpoint, and any adapter that mounts app.routes picks up host-to-host traffic automatically.

const handler = createAppHandler(app);   // public endpoint AND internal route

let stopping = false;
const server = createServer((req, res) => {
    if (stopping) res.setHeader('connection', 'close');
    handler(req, res);
});

// Listen BEFORE starting: app.start() joins membership, and from that moment
// peers may place actors here and call them. Bind first and there is no
// window where this host is routable but nothing is listening.
await new Promise<void>((resolve) => server.listen(7311, resolve));
const host = await app.start();

attachSignalHandlers(host, { server, onStopBegin: () => (stopping = true) });

Shutdown: pass the server, not just the host#

Stopping the actors is only half a graceful shutdown. An orchestrator's preStop sleep and readiness-503 steer new connections away, but connections already established survive endpoint removal — conntrack — and ride into the exiting pod, where they are reset when the process exits.

On a real cluster that measured as 122 lost calls out of ~1.7M on a rolling restart. All connection-level, none of them visible from the actor layer, which reported a clean hand-off.

The sequence is deliberately not the obvious one:

1. onStopBegin() — start answering connection: close. This is what actually drains the pools, one response at a time, interrupting nothing.
2. host.stop() — the actor drain. Pooled connections keep flowing; peers whose dials are refused see unreachable, which is retryable by design.
3. server.close() + closeAllConnections() — last.

Closing the listener first looks more decisive and is worse. On Node ≥ 19, close() also destroys idle connections, and "idle" from the server's side includes a socket the client is at that instant writing its next request onto — producing exactly the reset the sequence exists to prevent.

Options#

fetch and endpoint remain as sugar for exactly those two httpTransport() options, and passing either alongside an explicit transport throws.

The secret is not optional#

cluster() throws at construction without a secret, in every build but development.

The internal host-to-host mount runs no policies — the public edge is expected to have decided already — and it is contributed as an ordinary route, so it lands on the same listener as the public actor endpoint. Without a secret, every registered actor type, key and method is reachable unauthenticated by anything that can address that listener.

cluster({ providers, advertise, secret: process.env.HOST_SECRET });   // normal
cluster({ providers, advertise, secret: null });                     // explicit opt-out

secret: null is for a mount genuinely unreachable by an untrusted caller — an mTLS mesh, a private network with no other tenant. It is a claim you are making, which is why it has to be written down. An empty or whitespace-only secret throws in every build, development included.

The HMAC authenticates the peer, not the payload.

Fencing#

A host fences itself when it can no longer prove it holds its place in membership. Fenced is terminal: activations are refused, liveness fails, and an orchestrator restart is the way back — a host that has been aged out by its peers cannot talk itself back in.

Three things trigger it:

A fenced host also withdraws from membership, so peers stop routing to it immediately rather than waiting out its TTL.

Writing a provider#

Drive heartbeatClock() from your heartbeat rather than hand-rolling a TTL comparison:

import { heartbeatClock } from '@sigx/actors/cluster';

const clock = heartbeatClock({ ttlMs, onSuspect: () => fenceThisHost() });

clock.arm();                                    // once, when beating starts
// each beat:
clock.beat();                                   // before the write
try { await writeHeartbeat(); clock.confirmed(); }
catch { clock.failed(); }

It watches the monotonic and the wall clock together, because CLOCK_MONOTONIC does not advance across a VM suspend — and setTimeout rides that same clock, so a suspended host would otherwise conclude every beat had been punctual. The window is stamped when the beat is armed, not at the join write, so a slow join cannot fence a host at startup.

clock.lost() is the fourth door, for a provider holding proof its record is gone rather than a suspicion inferred from elapsed time — a Kubernetes Lease deleted out from under its host, say. It fires immediately whatever the clocks say, because the next write would succeed promptly and look perfectly healthy. Like every other trigger it is latched, cleared by the next confirmed(), and silent before arm().

Testing it#

memoryClusterHub() can simulate both shapes of loss:

hub.kill(hostId);      // a crash the host notices — fires onSelfSuspect
hub.expire(hostId);    // a TTL lapse it is never told about

expire() is the one worth reaching for: no cleanup, no notification, exactly the case a provider has to detect for itself.

Membership refresh coalescing#

Membership providers that support push notifications coalesce their refreshes: leading-edge immediate, then single-flight with version-gated hint skipping, so a burst of N changes costs one re-read rather than N per subscriber.

coalesceMs (default 0) is the knob, on redisCluster/redisMembership, pgCluster/pgMembership and surrealCluster/surrealMembership:

redisCluster({ url, coalesceMs: 50 })

0 keeps the leading-edge behaviour with no trailing window — every notification that is not already in flight refreshes immediately. Raise it when a noisy membership channel is costing more than the staleness it saves.

Writing a custom ClusterMembership? refreshCoalescer() from @sigx/actors/cluster is the same primitive: demand() preserves ClusterMembership.refresh()'s started-at-or-after contract, and settled() is for teardown.

Heterogeneous clusters: registration-aware placement#

A cluster member is only ever chosen to host an actor type its app registers — defineActorApp({ actors }). Different roles may register different actor apps and join one cluster: web pods registering only a relay worker and a cache actor, engine pods registering everything. A type never lands on a host that does not register it, structurally.

Each host publishes its registered type names in its membership descriptor — HostDescriptor.types — and Host.registeredTypes() returns the same sorted list locally. Placement enforces eligibility at every decision point: a cached route naming a non-registering host is dropped, a directory entry pointing at one is evicted and the actor re-placed, and the policy is handed a view already narrowed to the hosts registering the type.

When no active host registers a type, placement throws ActorUnplaceableError (kind: 'unplaceable') rather than silently widening to the full view — silently widening is how a type lands on a host that never registered it. It is retried against a refreshed membership view, because the one pod registering a type being mid-join is a rolling deploy, and it surfaces as the cause of the final ActorActivationError once the retries are spent:

actor type "Report" is registered by no ACTIVE host (view: 3 hosts, 3 active).
Add "Report" to some host's defineActorApp({ actors }), or check that the pods registering it are up.

A host that is still joining or already leaving does not count.

The receiving side enforces the same rule. An inbound cluster call for a type this host does not register answers wrong-host with no owner hint, so the caller evicts its route and re-places — rather than a 404, which is terminal.

Mixed versions. A descriptor without types reads as "registers everything" — the only safe direction, since absent-means-ineligible would empty every view mid-rolling-deploy. Consequently, while a fleet is half-published, older pods stay eligible for a brand-new type; a newer receiving pod refuses it with wrong-host so the caller re-places instead of caching a poisoned route.

Reminder shard ownership stays rendezvous over the full active view — shards are host-level, not typed — and a host registering only workers has nothing to fence and keeps serving them.

Targeted worker calls: members() and dispatchOn()#

Placement routes a call to wherever the actor lives. A stateless worker executes on the host that receives the call — that is what a worker is — so for workers, choosing the host is meaningful. ClusterPlacement exposes the two primitives:

import { workerOn } from '@sigx/actors/cluster';

// every active member registering the worker — the pod registry you would otherwise hand-roll
const relays = placement.members({ registers: 'WebsocketRelay' });

await Promise.all(
    relays.map((member) => workerOn(placement, member, WebsocketRelay).broadcast(roomId, payload)),
);

members(filter?) enumerates the current view, self included — status: 'active' by default ('any' disables the filter), optionally narrowed with registers: '<Type>'. A synchronous read with no I/O. A descriptor without types matches any filter, the same rule placement applies.

dispatchOn(target, ref, method, args, options?) invokes a worker method on a chosen member — a HostDescriptor or a host id. workerOn(placement, target, def, key?, options?) is the typed sugar, returning a client whose methods are the worker's. options is { timeoutMs?: 30_000, signal? } for the single attempt.

Three rules keep it honest:

• Workers only. A stateful type is refused when its definition is locally resolvable — targeted delivery would fight placement over where the activation lives.
• One attempt. No retry, no route cache, no directory. The caller chose the host, so unreachable (departed) and wrong-host (does not register the type) are answers, propagated branded, never consumed as a re-route.
• Self-targets run in-process. A host often cannot reach its own advertised address, and a fenced host can still serve its own workers.

targetedDispatches counts these apart from remoteDispatches in placement.counters(). Both members are optional on the ClusterPlacement interface, so a hand-rolled placement predating them keeps compiling; workerOn throws on a placement without dispatchOn.

The use case this exists for: a per-pod relay or fan-out worker on the web-tier members — a WebSocket relay that must reach every pod holding connections — without a hand-rolled pod registry.

What clustering does not change#

Your actor code. Nothing in a definition knows whether it is running on one host or fifty — placement, the directory and the transport are all host concerns.

Two things are worth knowing anyway: a live subscription is never routed or redirected, so it keeps paying the cross-host hop; and stateless workers are always local and invisible to the directory entirely.

Next steps#

• Placement policies — where new activations go.
• Stateless workers — the targets of dispatchOn.
• Transports — how hosts talk, and which to pick.
• Locality routing — getting the edge to help.
• Rebalancing — correcting a lumpy fleet.