Kubernetes#

An actor host is an ordinary HTTP service with two unusual requirements: it needs a startup probe, and its shutdown order matters more than most.

Membership without an extra store#

@sigx/actors-k8s implements membership on coordination.k8s.io Leases — one Lease per host, renewTime as the heartbeat, peers discovered by a label-selected Lease watch, and self-fencing when the host loses its place in membership.

cluster({
    providers: { membership: k8sMembership(), directory: redisDirectory(client) },
    advertise: `http://${process.env.POD_IP}:7311`,
    secret: process.env.HOST_SECRET,
});

You still need a directory — Leases answer which hosts are alive, not which host owns this actor. Redis, Postgres and SurrealDB each provide one; pair the Lease membership with whichever you already run.

RBAC needs get/list/watch/create/update on leases in the namespace. No kubeconfig is parsed; in-cluster service-account credentials are used, and kubectl proxy covers local development.

A deleted Lease fences the host#

The Lease is the membership token, so a renewal answering 404 fences — it does not recreate it. By the time the Lease is gone, peers have aged the host out of their views, evictHost has released every directory claim it held, and a survivor may already be serving those actors. Recreating the Lease would re-advertise a host whose claims are forfeit, and no elapsed-time check could catch it: the recreate succeeds promptly and everything looks healthy.

So the host fences, liveness fails, and the restart mints a fresh identity that rejoins cleanly.

The one exception is the host's own leave() racing an in-flight renewal: a graceful exit, and it never fences.

Probes#

startupProbe:
  httpGet: { path: /_sigx/health, port: 7311 }
  failureThreshold: 30
  periodSeconds: 2
livenessProbe:
  httpGet: { path: /_sigx/health, port: 7311 }
readinessProbe:
  httpGet: { path: /_sigx/health/ready, port: 7311 }

The startupProbe is not optional. Before app.start() the whole mount answers 503, including liveness — so without it a slow boot gets the pod killed in a loop.

The two probes are allowed to disagree, and the states they distinguish map onto exactly what you want Kubernetes to do: a draining host stays live and goes not-ready; a fenced host answers 503 {status:'fatal'} on liveness so the pod restarts.

Shutdown#

terminationGracePeriodSeconds: 45
lifecycle:
  preStop:
    exec: { command: ["sleep", "10"] }

The preStop sleep gives endpoint removal time to propagate before the process starts draining. It is necessary but not sufficient — connections already established survive endpoint removal via conntrack and ride into the exiting pod.

That is why the entry must answer connection: close from the moment shutdown begins, and why the ordering is onStopBegin → host.stop() → server.close(). Skipping it measured as 122 lost calls out of ~1.7M on a rolling restart.

Set terminationGracePeriodSeconds above your preStop sleep plus the actor drain plus taskGraceMs.

One host per core#

A host is one Node process on one JS thread. Nothing in the runtime uses a second core — a defineWorker pool multiplies activations, not threads — so the way to use a multi-core node is several hosts on it. That is a supported shape: host identity is per process, so N processes on one machine are N cluster members that happen to share a kernel, each with its own listen port and advertise address.

It scales: eight hosts on one box measured 5.93× the throughput of one, with the shared components — directory, reminder table — flat across the ladder.

Size requests at ~1.4 cores per host, not one: a saturated host is the JS loop plus V8's GC helpers, libuv and its store client's I/O. The rule is

sum(requests) ≤ allocatable − 1 core,   limits = requests

which on an 8-vCPU node is five hosts at requests.cpu: 1300m:

resources:
  requests: { cpu: 1300m }
  limits:   { cpu: 1300m }

Why not worker_threads? They were measured and kept out of the runtime: a ~3.4× ceiling with a ~200µs crossover and a 1 MB payload cliff, where more hosts keep scaling.

The fence is the watchdog. Packing does not create a new hazard — each host has its own loop, so a CPU-bound turn starves only its own heartbeat. The hazard is oversubscription: limits summing past the cores, so the CFS throttles a host across whole periods and its beat lands past the TTL with no turn being slow. That host fences, reports fatal on liveness, and the kubelet restarts it — no supervisor is needed beyond the kubelet. The answer to a host fencing under load is the sizing rule, not a longer TTL.

Set meta.node from the downward API so the monitor, CLI and dashboard show how many hosts share each node:

env:
  - name: NODE_NAME
    valueFrom: { fieldRef: { fieldPath: spec.nodeName } }

cluster({ providers, advertise, secret, meta: { node: process.env.NODE_NAME } });

Scaling#

An HPA works normally. Two things to know:

• New pods start cold and attract work only as new activations are created — unless you run activationCountPolicy(), which reads a host with no known load as cold and steers to it immediately.
• Scaling down does not move actors back. After a scale event the fleet can stay lumpy indefinitely; rebalancing is the correction, and it is off unless configured.

What to expose#

The public actor endpoint goes behind your ingress. /_sigx/host, /_sigx/ops and /_sigx/health* are internal only — probes cannot be authenticated, and ops() is authenticated but still not something to publish.

If you configure locality routing, the ingress is where the routing token gets hashed. On ingress-nginx, give that hash an empty-key fallback: worker calls carry no token, and its consistent hash otherwise pins every one of them onto a single pod.

A worked rig — Helm chart with deployment, HPA, PDB, service, RBAC, Redis and a load generator, plus a runbook — is perf/aks in the actors repo.

Next steps#

• @sigx/actors-k8s — options and Lease anatomy.
• Health & readiness — the three states.
• Clustering — the shutdown sequence in full.