Cluster stats#

One call, every host: activations, queue depths, per-type counts, reminder shard ownership, merged metrics and a health tally — plus an honest account of who did not answer.

TypeScript 
import { clusterStats } from '@sigx/actors/cluster';

const report = await clusterStats(placement, { detail: true, activations: 20 });

It rides the internal mount as a reserved symbol, so it needs the cluster secret and inherits the HMAC.

Partial results, never an exception#

It never throws for a sick peer. Unreachable hosts come back in unreachable[] with a classified reason, and partial: true is set.

That is the correct behaviour for a diagnostic: the moment you most need cluster stats is the moment some host is not answering.

totals.metrics.hosts is the denominator. A partial-fleet total is a lower bound, and reading it as a whole-fleet number during an incident is how you talk yourself into the wrong diagnosis. It is null when nothing is instrumented.

Latency merges, it does not average#

Percentiles are re-derived from summed histogram buckets, not averaged across hosts.

Averaging per-host p99s is simply wrong — it is a mean of order statistics, which is not an order statistic of anything. Ten hosts at p99 = 10ms and one at p99 = 900ms do not make p99 = 91ms; they make a fleet whose p99 depends entirely on how the traffic was distributed. Summing the buckets first and then taking the percentile gives the number you meant.

HISTOGRAM_LAYOUT, mergeHistogramDigests(), digestSnapshot() and createMetricsAccumulator() are exported from @sigx/actors/host if you want to do the same folding yourself.

Reminder shard ownership#

The reminder table is split into 16 hash shards, each ticked by exactly one host via rendezvous hashing over the membership view. The report tells you who owns what, and two readings matter:

  • Two claimants for one shard — views have diverged. Safe, because the per-shard etag CAS keeps delivery at-most-once, but it means membership is unsettled.
  • An empty shard — nothing is ticking those reminders.

Counters worth watching#

placement.counters() gives routing, route cache, directory, membership and authFailures.

The one to put on a dashboard is locateRemote / locates — the miss rate your edge is producing. If you have configured locality routing and this is not near zero, the edge is not hashing the token, or placement is not preferLocalPolicy().

dispatchesLocal / (dispatchesLocal + dispatchesRemote) is the per-request locality fraction. Each call, stream or watch subscriber this host initiates counts once, in one of the two, decided at its first resolved target — on the warm local fast path and the routed path alike. A re-route after wrong-host or unreachable is a retries event, not a second dispatch. Stateless workers and dispatchOn() count in neither — nothing placed them — and a call that never resolves a target counts only in routingFailures. Reminder, task and topic deliveries go through the same path, so read the fraction rather than the sum against an inbound request rate.

routedLocal does not measure locality. It is a placement count: the warm fast path never reaches the routing loop, so routedLocal reads near zero on a perfectly local fleet. remoteDispatches is per attempt, retries included. Neither pair gives the fraction.

selfFences, rejoinAttempts and rejoins cover fencing — a worker-only host rejoins instead of fencing. hostSweeps, sweptEntries, sweepsDelegated and sweepsSkippedGraceful cover directory hygiene when a host departs.

membershipVersion is each host's own view version. A spread across hosts can mean views have not converged — or only that hosts observed a different number of changes nobody wrote, because the version is per process.

transportFallbacks tells you a transport chain is doing real work — useful mid-deploy, worrying afterwards. rebalanceRounds and rebalanceMigrations cover rebalancing. targetedDispatches counts dispatchOn() calls this host initiated, apart from remoteDispatches.

Per-host stats.watchLoops is the live fan-out gauge: a loop count tracking the subscriber count on a hot actor is the per-identity split building.

Counters are summed across hosts with a missing field read as 0, so a mixed-version fleet — a peer whose report predates a counter — still totals rather than failing the fold.

Detail is opt-in and clamped#

GET /_sigx/ops/cluster?detail=1&activations=20&host=<id>

Requested limits are clamped by the responder, not trusted. The activation list is off by default because actor keys can be personal data — see The ops endpoint.

Reading it#

The CLI dashboard renders all of this, and every panel states its scope — cluster-wide or one host — because that distinction is easy to lose and expensive to get wrong.

Next steps#