Cluster and coordination

The moment more than one jaque process shares a log, something has to decide who owns what and how commands find the right owner. That’s what this page covers, in mechanism rather than in scenario — see Topologies for the shapes this mechanism enables.

-queue

Required for -target engine and worker. memory only makes sense when engine and worker share one binary in dev; nats://host:port is the real option for running them on separate machines.

-coordination and membership

-coordination (default memory, in-process) is where engines announce themselves; nats://host:port is a shared store the same JetStream can serve, or a separate one. -heartbeat-ttl (default 10s) is how long a missed heartbeat is tolerated before the rest of the cluster considers an engine dead; each engine refreshes at a third of that interval. With one engine (replicas: 1, the default), none of this is observable — it matters once more than one process shares the coordination store.

Each role that scales independently gets its own membership bucket: engines under membership/, -target sink under sink/membership/, -target notifier under notifier/membership/ — siblings, not nested. Ownership within each bucket is computed by rendezvous hashing (HRW) over that bucket’s live member list, so the assignment is stable and doesn’t require a coordinator to broadcast a plan.

Heartbeat TTL, in practice

• replicas: 1 — no change from a single-process deployment.
• replicas: 2 — each object is owned by exactly one of the two; if one dies, the other recomputes ownership and picks up its objects (failover).
• replicas: 3+ — the same hashing spreads objects across all of them; every engine is active, none idle.

Epoch fencing

An epoch fence protects correctness during a partition: every membership change bumps an epoch, every event carries the epoch it was written under, and an up-to-date engine discards any event with a stale epoch (jaque_engine_stale_epoch_total) rather than let a network-partitioned engine corrupt state another engine already claimed. Worst case, an isolated engine runs one extra check — it can never write a state transition that contradicts the current owner’s.

Command forwarding, one hop

A command that lands on an engine that doesn’t own the target object is forwarded once to whichever engine currently does, over jaque.cmd.<engine-id> (the same NATS -coordination uses), and applied there. jaque_engine_commands_forwarded_total{outcome} counts the attempt (sent/error). If the object still has no owner after that one hop — membership changed again in the meantime — the command is dropped and counted (jaque_engine_not_owned_total); there is never a second hop. With -coordination memory nothing is forwarded, because there’s only one engine.

UI submission via ingress

A -target ui process has no engine loop of its own, so a write it receives is submitted to the engine fleet through jaque.work.cmd.ingress — exactly one engine claims each submitted command, resolves the real owner, and forwards it by object the same way any other command is forwarded. This requires -coordination; without it, -target ui rejects writes with 503.