Clustering#

Many hosts, one actor system. The single-activation guarantee still holds across the fleet — that is what the directory is for.

import { defineActorApp } from '@sigx/actors/host';
import { cluster } from '@sigx/actors/cluster';
import { redisCluster } from '@sigx/actors-redis';

export const app = defineActorApp({ actors, storage }).use(
    cluster({
        providers: redisCluster({ url: process.env.REDIS_URL }), // membership + directory
        advertise: 'http://10.0.4.7:7311',   // this host's peer-reachable origin
        secret: process.env.HOST_SECRET,     // declared ONCE
    }),
);

The two providers#

Membership answers "which hosts are alive?" — a heartbeat plus a view of peers. Fencing falls out of it: a host that has lost its place in membership refuses every activation.

The directory answers "which host owns this actor?" — the claim that makes single activation true cluster-wide.

They are independent choices. Redis, Postgres and SurrealDB provide both; Kubernetes provides membership only, on Leases you are already running, and pairs with either directory:

cluster({
    providers: { membership: k8sMembership(), directory: redisDirectory(client) },
    advertise,
    secret: process.env.HOST_SECRET,
});

One handler, both mounts#

The plugin contributes the internal host-to-host mount as a route, so secret and internalBase are stated once instead of repeated at the endpoint, and any adapter that mounts app.routes picks up host-to-host traffic automatically.

const handler = createAppHandler(app);   // public endpoint AND internal route

let stopping = false;
const server = createServer((req, res) => {
    if (stopping) res.setHeader('connection', 'close');
    handler(req, res);
});

// Listen BEFORE starting: app.start() joins membership, and from that moment
// peers may place actors here and call them. Bind first and there is no
// window where this host is routable but nothing is listening.
await new Promise<void>((resolve) => server.listen(7311, resolve));
const host = await app.start();

attachSignalHandlers(host, { server, onStopBegin: () => (stopping = true) });

Shutdown: pass the server, not just the host#

Stopping the actors is only half a graceful shutdown. An orchestrator's preStop sleep and readiness-503 steer new connections away, but connections already established survive endpoint removal — conntrack — and ride into the exiting pod, where they are reset when the process exits.

On a real cluster that measured as 122 lost calls out of ~1.7M on a rolling restart. All connection-level, none of them visible from the actor layer, which reported a clean hand-off.

The sequence is deliberately not the obvious one:

1. onStopBegin() — start answering connection: close. This is what actually drains the pools, one response at a time, interrupting nothing.
2. host.stop() — the actor drain. Pooled connections keep flowing; peers whose dials are refused see unreachable, which is retryable by design.
3. server.close() + closeAllConnections() — last.

Closing the listener first looks more decisive and is worse. On Node ≥ 19, close() also destroys idle connections, and "idle" from the server's side includes a socket the client is at that instant writing its next request onto — producing exactly the reset the sequence exists to prevent.

Options#

fetch and endpoint remain as sugar for exactly those two httpTransport() options, and passing either alongside an explicit transport throws.

meta.node: the machine a host runs on#

meta is free-form, with one key that has an agreed meaning: node, the machine the host runs on. The monitor, the CLI and the dashboard count its distinct values into 3 host(s) / 1 node(s) and show a per-host node column — the difference between three replicas spread over three machines and three replicas packed onto one, which a replica count reads the same either way.

On Kubernetes, fill it from the downward API:

env:
  - name: NODE_NAME
    valueFrom: { fieldRef: { fieldPath: spec.nodeName } }

cluster({ providers, advertise, secret, meta: { node: process.env.NODE_NAME } });

A fleet that reports no node shows — in the node column and no nodes figure — "said nothing about where it runs", never a guess.

The secret is not optional#

cluster() throws at construction without a secret, in every build but development.

The internal host-to-host mount runs no policies — the public edge is expected to have decided already — and it is contributed as an ordinary route, so it lands on the same listener as the public actor endpoint. Without a secret, every registered actor type, key and method is reachable unauthenticated by anything that can address that listener.

cluster({ providers, advertise, secret: process.env.HOST_SECRET });   // normal
cluster({ providers, advertise, secret: null });                     // explicit opt-out

secret: null is for a mount genuinely unreachable by an untrusted caller — an mTLS mesh, a private network with no other tenant. It is a claim you are making, which is why it has to be written down. An empty or whitespace-only secret throws in every build, development included.

The HMAC authenticates the peer, not the payload.

A synchronous HMAC on Node#

A sign and a verify sit on every secured cross-host call. The default implementation, webCryptoHmac, uses WebCrypto and runs everywhere — but crypto.subtle.sign is asynchronous, and on Node it runs on the libuv threadpool at ~27µs per call. node:crypto computes the same HMAC-SHA-256 synchronously, on the calling thread, in ~2µs. Node hosts should pass it:

import { nodeHmac } from '@sigx/actors/node';

cluster({ providers, advertise, secret: process.env.HOST_SECRET, hmac: nodeHmac() });

The bytes on the wire are identical, so hosts on either implementation interoperate and a fleet can switch host by host during a rolling deploy.

A transport built outside this repo signs and verifies through the same seam: @sigx/actors/cluster exports the HostHmac type, webCryptoHmac, and signAuthWith / verifyAuthWith; signAuth / verifyAuth keep their promise signature and take an optional implementation.

Fencing#

A host fences itself when it can no longer prove it holds its place in membership. For a host that registers any stateful actor type, fenced is terminal: activations are refused, liveness fails, and an orchestrator restart is the way back — a host that has been aged out by its peers cannot talk itself back in, because a survivor may already be serving the actors it used to own.

Three things trigger it:

A fenced host also withdraws from membership, so peers stop routing to it immediately rather than waiting out its TTL.

What a fenced host answers#

A call dispatched on a fenced host that would resolve to a local activation throws ActorFencedError (kind: 'fenced'). It is terminal — the routing loop does not retry it, since re-resolving lands on the same refusal and each attempt would cost a directory lookup and a membership refresh against the store that most likely just failed. On the public actor mount and on the host-to-host wire it answers 503.

A peer calling into a fenced host gets unreachable instead — the answer it can act on: go somewhere else.

Worker-only hosts rejoin#

Fencing defends the single-activation invariant, so it applies to hosts that hold directory claims. A host that registers no stateful actor type — a defineWorker-only tier, such as a web tier that only calls actors other hosts own — holds no claim by construction: nothing anywhere names it, and nothing was re-placed when it vanished.

So on a membership lapse such a host rejoins under the same identity, retrying with capped backoff until the store takes it back, and keeps serving its worker pools throughout. It never throws ActorFencedError. One stateful registration brings the terminal fence back.

placement.counters() reports both paths: selfFences, and beside it rejoinAttempts (lapses answered by rejoining, counted at the decision) and rejoins (those that got back in). rejoinAttempts − rejoins is the number of rejoin loops still running — 0 or 1; a store that is down shows as attempts climbing while rejoins stays put.

A CPU-bound turn can fence its own host#

Actor turns, health probes and the membership heartbeat share one event loop. A turn that computes synchronously past the membership TTL — redisCluster defaults to a 5-second beat and a 15-second TTL — starves the host's own liveness signal. While the loop is held, peers age the host out and release its claims; when it resumes, the host sees its beat landed past the TTL and fences, by design.

So a method that blocks long enough does not merely go slow: it removes its host from the cluster and takes every other actor on that host with it. Treat a method whose CPU cost scales with client input as a denial-of-service surface — a guard keeps anonymous callers out, but not a signed-in caller from wedging the host — and cap the work a single call may ask for.

Mitigations, in the order to reach for them:

1. Chunk the work and yield between slices. Every few hundred iterations, await new Promise((resolve) => setImmediate(resolve)) on Node, or await new Promise((resolve) => setTimeout(resolve, 0)) where setImmediate does not exist. The yield is the load-bearing detail: it is what lets the heartbeat, the probes and every other actor's turns run.
2. Move long work off the request turn with ctx.tasks, so the caller is not held. A detached task still runs on the same loop, so it must chunk and yield too.
3. Move genuinely heavy compute onto worker_threads. A defineWorker pool does not do this: it multiplies activations — concurrency at await points — not threads, so a pool of synchronous hashers blocks exactly like one.

Development builds flag offenders: a turn that holds the loop longer than slowTurnMs (default 5s) logs a warning naming the actor and method.

Writing a provider#

Drive heartbeatClock() from your heartbeat rather than hand-rolling a TTL comparison:

import { heartbeatClock } from '@sigx/actors/cluster';

const clock = heartbeatClock({ ttlMs, onSuspect: () => fenceThisHost() });

clock.arm();                                    // once, when beating starts
// each beat:
clock.beat();                                   // before the write
try { await writeHeartbeat(); clock.confirmed(); }
catch { clock.failed(); }

It watches the monotonic and the wall clock together, because CLOCK_MONOTONIC does not advance across a VM suspend — and setTimeout rides that same clock, so a suspended host would otherwise conclude every beat had been punctual. The window is stamped when the beat is armed, not at the join write, so a slow join cannot fence a host at startup.

clock.lost() is the fourth door, for a provider holding proof its record is gone rather than a suspicion inferred from elapsed time — a Kubernetes Lease deleted out from under its host, say. It fires immediately whatever the clocks say, because the next write would succeed promptly and look perfectly healthy. Like every other trigger it is latched, cleared by the next confirmed(), and silent before arm().

Testing it#

memoryClusterHub() can simulate both shapes of loss:

hub.kill(hostId);      // a crash the host notices — fires onSelfSuspect
hub.expire(hostId);    // a TTL lapse it is never told about

expire() is the one worth reaching for: no cleanup, no notification, exactly the case a provider has to detect for itself.

Membership refresh coalescing#

Membership providers that support push notifications coalesce their refreshes: leading-edge immediate, then single-flight with version-gated hint skipping, so a burst of N changes costs one re-read rather than N per subscriber.

coalesceMs (default 0) is the knob, on redisCluster/redisMembership, pgCluster/pgMembership and surrealCluster/surrealMembership:

redisCluster({ url, coalesceMs: 50 })

0 keeps the leading-edge behaviour with no trailing window — every notification that is not already in flight refreshes immediately. Raise it when a noisy membership channel is costing more than the staleness it saves.

Writing a custom ClusterMembership? refreshCoalescer() from @sigx/actors/cluster is the same primitive: demand() preserves ClusterMembership.refresh()'s started-at-or-after contract, and settled() is for teardown.

Heterogeneous clusters: registration-aware placement#

A cluster member is only ever chosen to host an actor type its app registers — defineActorApp({ actors }). Different roles may register different actor apps and join one cluster: web pods registering only a relay worker and a cache actor, engine pods registering everything. A type never lands on a host that does not register it, structurally.

Each host publishes its registered type names in its membership descriptor — HostDescriptor.types — and Host.registeredTypes() returns the same sorted list locally. Placement enforces eligibility at every decision point: a cached route naming a non-registering host is dropped, a directory entry pointing at one is evicted and the actor re-placed, and the policy is handed a view already narrowed to the hosts registering the type.

When no active host registers a type, placement throws ActorUnplaceableError (kind: 'unplaceable') rather than silently widening to the full view — silently widening is how a type lands on a host that never registered it. It is retried against a refreshed membership view, because the one pod registering a type being mid-join is a rolling deploy, and it surfaces as the cause of the final ActorActivationError once the retries are spent:

actor type "Report" is registered by no ACTIVE host (view: 3 hosts, 3 active).
Add "Report" to some host's defineActorApp({ actors }), or check that the pods registering it are up.

A host that is still joining or already leaving does not count.

The receiving side enforces the same rule. An inbound cluster call for a type this host does not register answers wrong-host with no owner hint, so the caller evicts its route and re-places — rather than a 404, which is terminal.

Mixed versions. A descriptor without types reads as "registers everything" — the only safe direction, since absent-means-ineligible would empty every view mid-rolling-deploy. Consequently, while a fleet is half-published, older pods stay eligible for a brand-new type; a newer receiving pod refuses it with wrong-host so the caller re-places instead of caching a poisoned route.

Reminder shard ownership stays rendezvous over the full active view — shards are host-level, not typed — and a host registering only workers has nothing to fence and keeps serving them.

Targeted worker calls: members() and dispatchOn()#

Placement routes a call to wherever the actor lives. A stateless worker executes on the host that receives the call — that is what a worker is — so for workers, choosing the host is meaningful. ClusterPlacement exposes the two primitives:

import { workerOn } from '@sigx/actors/cluster';

// every active member registering the worker — the pod registry you would otherwise hand-roll
const relays = placement.members({ registers: 'WebsocketRelay' });

await Promise.all(
    relays.map((member) => workerOn(placement, member, WebsocketRelay).broadcast(roomId, payload)),
);

members(filter?) enumerates the current view, self included — status: 'active' by default ('any' disables the filter), optionally narrowed with registers: '<Type>'. A synchronous read with no I/O. A descriptor without types matches any filter, the same rule placement applies.

dispatchOn(target, ref, method, args, options?) invokes a worker method on a chosen member — a HostDescriptor or a host id. workerOn(placement, target, def, key?, options?) is the typed sugar, returning a client whose methods are the worker's. options is { timeoutMs?: 30_000, signal? } for the single attempt.

Three rules keep it honest:

• Workers only. A stateful type is refused when its definition is locally resolvable — targeted delivery would fight placement over where the activation lives.
• One attempt. No retry, no route cache, no directory. The caller chose the host, so unreachable (departed) and wrong-host (does not register the type) are answers, propagated branded, never consumed as a re-route.
• Self-targets run in-process. A host often cannot reach its own advertised address, and a fenced host can still serve its own workers.

targetedDispatches counts these apart from remoteDispatches in placement.counters(). Both members are optional on the ClusterPlacement interface, so a hand-rolled placement predating them keeps compiling; workerOn throws on a placement without dispatchOn.

The use case this exists for: a per-pod relay or fan-out worker on the web-tier members — a WebSocket relay that must reach every pod holding connections — without a hand-rolled pod registry.

Reading membership on a hot path#

members() is cheap, but a fan-out concurrency cap or a per-call routing decision reads it on every call. Cache what you derive from the view — and key the cache on the view object, never on view().version.

A provider builds a new MembershipView object for every change it observes — including a peer expiring on the store's TTL clock, which no host writes — and hands back the same object until the next one. Object identity is the invalidation key that is always right.

membersMemo(placement, filter?) is that pattern, exported from @sigx/actors/cluster. Create it once — at module or actor scope — and call the thunk on the hot path:

import { membersMemo } from '@sigx/actors/cluster';

const relays = membersMemo(placement, { registers: 'WebsocketRelay' });

// per call: recomputes only when placement.view() hands back a new object
const cap = relays().length * 8;

The filter is members()'s (status: 'active' | 'any', registers). It accepts anything with a view() method, so it works against any placement.

placement.onChange(cb) covers anything derived from the view beyond what the memo does. It passes the provider's membership change stream straight through, firing with the new view object on every observed change, and returns the unsubscribe:

const off = placement.onChange((view) => { rebuildRing(view.hosts); });

The callback runs synchronously inside the provider's refresh path and must not throw — a throw aborts the remaining subscribers and rejects that refresh. onChange is optional on the ClusterPlacement interface, as members is.

What version means#

MembershipView.version is a per-process change token. It strictly increases on every change this process's view observes — expiry included — and is unchanged when nothing changed.

It is monotonic per process view, not per cluster: on a change nobody wrote, a provider advances it locally past the store's counter, so two hosts can hold different values for the same converged membership. Compare it only with earlier values from the same handle. The same reason means membershipVersion in cluster stats can spread across hosts without their views having diverged.

When a call is retried#

The routing loop (retries, default 3) re-dispatches a call on exactly three answers, and each is a claim about execution, not about the network:

Everything else surfaces to the caller unchanged. So a call cannot run twice because of a retry: a retried call is one that never ran.

A call already on the wire when its connection drops is not unreachable. The owner may have run it and the reply died with the socket. Over a connection-oriented transport such as tcpTransport(), a unary call in that state rejects with an ordinary, un-branded Error — isActorError() is false for it:

[sigx actors] host connection closed with this call in flight (<reason>): outcome unknown, not retried

You know whether the method is idempotent; the runtime does not, so re-issuing it is your decision. Streams and watches are unchanged: they fail unreachable before the first chunk, so a watch re-subscribes.

When a host departs#

A host that leaves the view leaves its directory claims behind unless it released them itself. Two mechanisms clean up:

• Lazy eviction on lookup — always on, and the backstop. A caller that finds a claim whose host is gone evicts it and re-resolves.
• The departure sweep — proactive hygiene. One live host sweeps each departure: the one whose id sorts first among the hosts that saw it leave. The others leave it to that host.

A host seen announcing leaving is not swept at all: host.stop() released every claim as it drained. A sweeper that dies before sweeping, or a graceful leaver that died mid-drain, falls back to lazy eviction — no actor is lost, only a few stale entries wait for a lookup.

In placement.counters(), hostSweeps and sweptEntries count the sweeps this host ran and the entries they removed — across a fleet, hostSweeps sums to the departures. sweepsDelegated counts departures this host left to the elected sweeper, and sweepsSkippedGraceful the departures nobody swept because the host left gracefully.

What clustering does not change#

Your actor code. Nothing in a definition knows whether it is running on one host or fifty — placement, the directory and the transport are all host concerns.

Two things are worth knowing anyway: a live subscription is never routed or redirected, so it keeps paying the cross-host hop; and stateless workers are always local and invisible to the directory entirely.

Two habits for a fleet#

Never await another host inside a turn without a deadline. A turn awaiting a cross-host call holds a pooled connection until the peer answers, and a mesh of such awaits wedges the pool. Every call carries callTimeoutMs by default — timer ticks and task turns included — so keep it on, and give a known-long call its own .with({ deadlineMs }) rather than disabling it.

Pick the reminder provider for the population you sleep. The default shardedReminders() is right for hundreds of sleeping actors; thousands want the indexed provider your store already has, and adding hosts does not change that. See choosing a provider.

Next steps#

• Placement policies — where new activations go.
• Stateless workers — the targets of dispatchOn.
• Transports — how hosts talk, and which to pick.
• Locality routing — getting the edge to help.
• Rebalancing — correcting a lumpy fleet.