> Section: [7. Running it](https://jaque.sh/docs/deployment/choose.md) > Next: deployment/kubernetes > Index: https://jaque.sh/llms.txt The moment more than one jaque process shares a log, something has to decide which process owns which object and how a command finds the owner. This page is that mechanism, stated as mechanism rather than as scenario: membership buckets, rendezvous hashing over their live members, a heartbeat TTL that declares a member dead, an epoch that fences a partitioned engine, and exactly one forwarding hop for a command that lands on the wrong engine. [Topologies](https://jaque.sh/docs/deployment/topologies.md) is the set of shapes this enables. None of it is observable with a single engine, and none of it needs a coordinator process: the coordination store is the same log server the event log lives on. ## 1. `-queue` Required for `-target engine` and `worker`. `memory` only makes sense when engine and worker share one binary in development; `nats://host:port` is the option for running them on separate machines. ## 2. `-coordination` and membership `-coordination` (default `memory`, in-process) is where engines announce themselves. `nats://host:port` names a shared store, which the same log server the log uses can serve, or a separate one. `-heartbeat-ttl` (default `10s`) is how long a missed heartbeat is tolerated before the rest of the cluster considers an engine dead; each engine refreshes at a third of that interval. With one engine (`replicas: 1`, the default) none of this is observable; it matters once more than one process shares the coordination store. Each role that scales independently gets its own membership bucket: engines under `membership/`, `-target sink` under `sink/membership/`, `-target notifier` under `notifier/membership/`. The three are siblings; none is nested inside another. Ownership within each bucket is computed by rendezvous hashing (HRW) over that bucket's live member list. Every member computes the same answer from the same list, so the assignment is stable across members and no coordinator has to broadcast a plan. ## 3. Heartbeat TTL, in practice With `replicas: 1` nothing changes from a single-process deployment. With `replicas: 2` each object is owned by exactly one of the two. If one dies, the other recomputes ownership after `-heartbeat-ttl` and picks up its objects; that recomputation is the failover. With `replicas: 3+` the same hashing spreads objects across all of them. Every engine is active; none is a standby. ## 4. Epoch fencing An epoch fence protects correctness during a handover and during a partition. Every membership change bumps an epoch, and every event carries the epoch it was written under and the id of the engine that wrote it. Three rules follow from that stamp: - An engine discards any event with an epoch older than the last one it folded for that object, counting it in `jaque_engine_stale_epoch_total`, whether the event arrives on the live tail or during replay at start. - An engine skips only the events it wrote itself. An event for an object it owns, written by the previous owner, is the state it takes over from. - An engine that has seen a newer epoch on the log for an object it owns stops writing that object until its own membership view catches up, counting the wait in `jaque_engine_epoch_behind_total`. The worst case for an isolated engine is one extra check run; it can never write a state transition that contradicts the current owner's. This handover protocol is model-checked. It is written as a TLA+ specification in the repository's `docs/formal` directory, and the checker exhausts every interleaving of a two-engine cluster: writes, membership changes, partitions, restarts and replays, over seventeen million states. The invariant is that two live engines whose view is current and whose tail is at the end of the log never disagree on an object. The same specification, checked against the fences as they were before these three rules, finds a seven-step counterexample that needs no partition at all. `just tla` reruns every configuration against the verdict it declares. An engine that loses the coordination store does not stay isolated for long. Once its heartbeats have failed for a whole `-heartbeat-ttl` the process exits non-zero, because by then its peers have declared it dead and taken its objects; the orchestrator restarts it and it rejoins with a fresh view. The same rule covers every background part of a process: if the membership watch, a queue subscription or a log follower stops on its own, the process exits rather than keep answering health checks while doing part of its job. ## 5. Command forwarding, one hop A command that lands on an engine that does not own the target object is forwarded once, to whichever engine currently does, over `jaque.cmd.` on the same log server that `-coordination` uses, and applied there. `jaque_engine_commands_forwarded_total{outcome}` counts the attempt as `sent` or `error`. If the object still has no owner after that one hop, because membership changed again in the meantime, the command is dropped and counted in `jaque_engine_not_owned_total`. There is never a second hop; a command that cannot find its owner in one bounce is lost rather than circulated. With `-coordination memory` nothing is forwarded, because there is only one engine. ## 6. UI submission via ingress A `-target ui` process has no engine loop of its own, so a write it receives is submitted to the engine fleet through `jaque.work.cmd.ingress`. Exactly one engine claims each submitted command, resolves the real owner and forwards it by object the same way as in section 5. This requires `-coordination`; without it, `-target ui` rejects writes with 503. ## 7. Security considerations The coordination store is a write surface. Any process that can publish to `jaque.cmd.` or `jaque.work.cmd.ingress` can have a command applied by an engine, with no check beyond what the log server enforces; `-api-token` guards the API listener, and it does not guard these subjects. That same access can just as easily publish forged check results on the work subjects and write them into the event log, so coordination credentials amount to full control of the monitoring system; the risk is not limited to ingress. There is no per-subject check inside jaque and none is planned; the log server's own credentials and network are the gate. Treat the log server's network as part of the perimeter described in [Security](https://jaque.sh/docs/deployment/security.md).