Observability#

@sigx/actors-otel turns the host's own metrics into something your monitoring stack already speaks — without making OpenTelemetry a dependency of anything that does not want it.

pnpm add @sigx/actors-otel

npm install @sigx/actors-otel

yarn add @sigx/actors-otel

bun add @sigx/actors-otel

Prometheus#

The /prometheus entry imports no OpenTelemetry at all:

import { prometheusOps } from '@sigx/actors-otel/prometheus';

app.use(metrics()).use(prometheusOps({ secret: process.env.SIGX_OPS_SECRET }));

Mounts GET /_sigx/metrics beside ops(), with the same bearer posture — the secret is mandatory outside dev.

scrape_configs:
  - job_name: sigx-actors
    metrics_path: /_sigx/metrics
    authorization: { credentials_file: /etc/prometheus/sigx-ops-secret }
    static_configs: [{ targets: ['actors-host:7311'] }]

Options: path (default /_sigx/metrics), secret, prefix (default sigx_actors_), bucketsSeconds — a per-octave grid from 1µs to ~134s, about 28 bounds — and cluster, for dispatch locality.

renderPrometheus(digest, stats, options) is the pure function underneath, if you would rather serve it yourself. Among what it emits is {prefix}one_way_failures_total — a one-way call that failed after acceptance has nowhere else to be reported, so this counter is the only place it surfaces in Prometheus.

The cardinality rule: labels are type and method — never actor keys. An actor key is unbounded by construction, and a key-labelled metric will take your Prometheus down. This is enforced, not advisory.

metrics().reset() breaks monotonicity. Prometheus counters are expected only ever to increase; a reset() looks like a counter restart. That is survivable — rate() handles restarts — but do not wire reset() to anything periodic.

OpenTelemetry traces#

import { otelTraces } from '@sigx/actors-otel';

app.use(otelTraces({ turnSpans: true }));

One CLIENT span per dispatch and one SERVER span per turn, joined across hosts by the W3C traceparent the runtime propagates: captured from the public endpoint header and carried on the cluster envelope.

Span attributes use a key hash, not the key, for the same reason the routing token does — and with the same caveat that a hash is log hygiene, not privacy.

@opentelemetry/api is an optional peer dependency. With no tracer provider registered the plugin is inert and costs nothing.

The metrics bridge#

import { otelMetricsBridge } from '@sigx/actors-otel';

app.use(otelMetricsBridge({ percentileGauges: true }));

Feeds the host's counters and histograms into whatever OTel meter provider you have configured, for stacks that collect metrics over OTLP rather than by scraping — including sigx.actors.calls.one_way_failures, the OTLP spelling of the counter above.

Socket sessions#

Socket connections are counted separately, because none of the things worth watching about them — how many are open, how long they live, how many were refused — are visible to the turn metrics.

socketStats() is one recorder per host or listener, handed to every session:

import { socketStats } from '@sigx/actors/server';

const stats = socketStats();
attachActorSocket(server, { host, stats });

Publish it as an ops section, and it appears under ops.sockets behind the existing bearer posture — no endpoint changes:

registry.reportOps('sockets', () => stats.snapshot());

snapshot() carries the totals — connections opened, closed and refused (origin or auth, turned away before serving a byte); calls started and failed; subscriptions opened and closed, and subscriptionsShed — those closed by the slow-consumer cap, also counted in subscriptionsClosed; protocolBreaches (1003/1009) and lifetimeCloses (1008 from revalidateMs or maxConnectionMs) — plus three live gauges summed across open sessions (open, inFlight, subscriptions) and a connection-lifetime histogram, null until there is data.

Four more fields describe live delivery:

deliveries is the number to watch. A delivery is ~77% socket write, one write syscall per subscriber, so it is what a fan-out host's cost is proportional to. Pair it with open and subscriptions: the same subscription count on fewer connections measured 6–10× cheaper per delivery, so subscriptions per connection is a ratio worth showing.

It is a gauge, and it is polled: a burst between two scrapes is invisible, so it under-reports peaks the way open does. The Node adapter supplies it automatically from the socket's bufferedAmount; a custom adapter passes bufferedBytes() to createActorSocketSession if its transport can report a queue depth, or accepts that the host cannot report its own backpressure.

stats.digest() is the mergeable form, in the same log-linear layout as every other histogram, so a fleet-wide view merges the way cluster stats does.

Wiring the exporters and the dashboards#

Two registrations, one per consumer — this is the one step the app has to do:

registry.reportOps('sockets', () => stats.snapshot());   // ops → the monitor, CLI and dashboard
registry.reportDigest('sockets', () => stats.digest());  // → prometheusOps and otelMetricsBridge

With the digest registered, prometheusOps() emits the {prefix}socket_* families:

socket_delivery_bytes_total is in UTF-16 code units, like deliveryBytes above.

No digest, no family. A host that registers none emits no socket_* family at all — "no socket mount" is not "zero connections", and a scraper that sees the family appear knows the transport is mounted. renderPrometheus() takes the digest as options.sockets (SocketStatsDigest, exported).

otelMetricsBridge() observes the same set as sigx.actors.socket.* instruments, plus sigx.actors.socket.connection_duration.p50 / .p90 / .p99 under percentileGauges.

On the ops side, the monitor, the CLI and the dashboard render the section for the polled host only — the cluster fan-out carries no socket digest, so every other host's socket figures read as "said nothing".

Cluster dispatch locality#

Hand the exporters this host's cluster counters and they publish the per-request locality pair:

const clustered = cluster({ providers, advertise, secret: process.env.HOST_SECRET });

app.use(clustered)
   .use(metrics())
   .use(prometheusOps({
       secret: process.env.SIGX_OPS_SECRET,
       cluster: () => clustered.placement.counters(),
   }));

cluster is a thunk read on every scrape or collection, so it may answer undefined until placement is up. It adds:

There is no precomputed ratio gauge. The fleet-wide local-hit fraction is a query — add by (instance) to both sums for the per-host one:

sum(rate(sigx_actors_cluster_dispatches_total{locality="local"}[5m]))
  / sum(rate(sigx_actors_cluster_dispatches_total[5m]))

Without the option, or while the thunk answers undefined, no cluster family is emitted — a single host is not a cluster with zero dispatches. renderPrometheus() takes the totals as options.cluster.

Choosing#

Core plumbing this rides on: ActorCallContext.traceparent, the trailing call parameter on ActorTurnObserver, and bucketUpperBoundsUs() / timingSafeEquals from @sigx/actors/host.

Next steps#

• @sigx/actors-otel — the package reference.
• Metrics — what is being exported.
• Cluster stats — the fleet-wide view.