# Scaling and high availability How many monitors one installation keeps on cadence, what happens when a worker dies, and how to add capacity. Every number on this page comes from an artefact in `docs/evidence/bench/`, and `npm run bench:check` fails if a cell here stops matching the run it quotes. `scripts/bench/scheduler.mjs` is how to produce a new one. --- ## 1. What was already highly available The first useful thing this exercise produced was the removal of an assumption. **The control plane does not depend on any one process, and did not before this release.** Three locks stand between two workers and duplicated work, and Postgres holds all three: | Question | Who arbitrates | How | | --------------------------- | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Whose clock fires the cron? | pg-boss | Conditional `UPDATE` on its own `version.cron_on`; whichever instance wins the update sends, and the send carries a one-minute singleton key | | Which worker runs the tick? | pg-boss | The tick queue's `singleton` policy is a unique index over `(name, singleton_key)` where `state = 'active'`, so any replica may be the scheduler and exactly one is | | Which worker runs a check? | pg-boss | Jobs are fetched `FOR UPDATE SKIP LOCKED`, and the check queue's `stately` policy is a unique index permitting one queued and one active job per monitor | A lease table beside those would have been a second answer to a question that already had one, and the second answer is the one that goes wrong. **There is no lease table in this feature.** Adding a worker adds capacity and needs no configuration, no leader election and no coordination beyond starting the process. What the measurements found instead were five capacity defects and one blind spot. None of them was an availability defect; all of them were about how much one installation can hold, and none was visible from inside a running system. ## 2. Invariants The properties the scheduler holds, and where each is proven. | Invariant | Held by | Proven in | | ------------------------------------------------------------ | ------------------------------------------------------------ | -------------------------------------------------- | | Two ticks running at once enqueue each monitor at most once | `for update skip locked` in `claimMonitorsForDispatch` | `tests/integration/monitor-tick.test.ts` | | A monitor whose check is still running is not enqueued again | The claim leases `dispatch_claimed_until`, its own column | `tests/integration/scheduler-claim.test.ts` | | A monitor whose job was lost returns to the due list | The claim's lease is finite (90s) | `tests/integration/monitor-tick.test.ts` | | A monitor probed between selection and claim is not delayed | The due predicate is re-checked under the lock | `tests/integration/scheduler-claim.test.ts` | | No tenant is starved by a larger one | `row_number() over (partition by organization_id …)` | `tests/integration/scheduler-claim.test.ts` | | A monitor never falls off the scheduler | Most-overdue-first: anything skipped sorts earlier next tick | `tests/integration/scheduler-claim.test.ts` | | The sub-minute path cannot enqueue around the claim | It takes the same lease, and stands down if it loses | `tests/integration/fast-follow-up.test.ts` | | A monitor another plane owns does not eat selection slots | Deferred one tick period while a LIVE lease covers it | `tests/integration/high-frequency.test.ts` | | A monitor under maintenance is still probed | Maintenance suppresses the page, never the check | `tests/integration/maintenance-scheduling.test.ts` | | A dead worker stops being believed | Liveness is a recent heartbeat AND no recorded exit | `tests/integration/cluster-registry.test.ts` | | A deploy is distinguishable from a crash | `stopped_reason` is `signal:*` or `stale` | `tests/integration/cluster-registry.test.ts` | An assertion that two concurrent claims do not overlap proves nothing on its own: it passes just as well if the two never overlapped. So the guards are also checked by MUTATION - the same tests run against a build with the guard deliberately broken, which has to go red. `tests/integration/scheduler-claim.test.ts` carries a committed mutation test. The rest were run at the terminal while the guards were written, against these specific mutations, each of which turned at least one test red: dropping the lease predicate from the follow-up reservation; measuring the interval from the clock instead of from `last_checked_at`; making the tick-period boundary exclusive, or a second too generous, or removing it; enqueueing after losing the reservation; not deferring a monitor the high-frequency plane owns; testing that plane's FLAG instead of its lease; and omitting `returnId` from the queue insert. Two of those are worth naming. Testing the flag instead of the lease passes every test about deferral and silently removes the failover, because a monitor whose high-frequency worker has died still has the flag set. And omitting `returnId` is the one that reached production in this very release: pg-boss returns `null` rather than ids unless asked, so the enqueued count read zero on every real tick while a test double that returned ids regardless kept the suite green. ## 3. Capacity 1000 monitors on a 60-second interval, which requires **16.67 checks per second**. Local HTTP targets on a spread of ports; workers, Postgres and targets share one machine, so these are lower bounds for a deployment where they do not compete. ### Before (v1.20.0) | Workers | Achieved | Required | Coverage | Missed | Duplicates | Lag p95 | Lag max | Queue p95 | Tick | Catch-up | | ------: | -------: | -------: | -------: | -----: | ---------: | ------: | ------: | --------: | -----: | -------: | | 1 | 10.10/s | 16.67/s | 96.00% | 793 | 0 | 185.4s | 190.5s | 1000 | 115 ms | never | | 2 | 16.57/s | 16.67/s | 100.00% | 11 | 0 | 106.0s | 111.0s | 1007 | 85 ms | never | | 3 | 16.10/s | 16.67/s | 100.00% | 68 | 0 | 2.0s | 2.1s | 1000 | 63 ms | 47s | | 4 | 17.04/s | 16.67/s | 100.00% | 0 | 0 | 2.0s | 2.1s | 1013 | 68 ms | 39s | ### After | Workers | Achieved | Required | Coverage | Missed | Duplicates | Lag p95 | Lag max | Queue p95 | Tick | Catch-up | | ------: | -------: | -------: | -------: | -----: | ---------: | ------: | ------: | --------: | ----: | -------: | | 1 | 10.04/s | 16.67/s | 100.00% | 800 | 0 | 97.8s | 102.9s | 1000 | 20 ms | never | | 2 | 16.42/s | 16.67/s | 100.00% | 29 | 0 | 2.1s | 2.2s | 1006 | 20 ms | 53s | | 3 | 16.29/s | 16.67/s | 100.00% | 45 | 0 | 2.1s | 2.1s | 1002 | 13 ms | 35s | | 4 | 16.21/s | 16.67/s | 100.00% | 55 | 0 | 2.0s | 2.1s | 1005 | 26 ms | 27s | ### Reading it **Throughput barely moved, and that is not the result.** Both builds are demand-bound at two workers and above: 1000 monitors on a 60-second interval ask for 16.67 checks per second and both deliver about that. The one worker cell is bound by the worker, not by the scheduler, in both. What moved is everything about being BEHIND. - **Backlog age at two workers: 106 seconds to 2.1 seconds.** The old build kept up with the cadence while never catching up on the burst. It looked healthy by throughput and was fifty times further behind. - **The burst clears.** At two workers the old build never drained the catch-up backlog inside the window; the new one clears it in 53 seconds. At three and four workers it clears in 35 and 27 seconds where it used to take 47 and 39. - **Coverage at one worker: 96.00% to 100%.** Four percent of the fleet went unprobed for the whole two-minute window under a build that was otherwise reporting a healthy rate. That is the shape of this defect throughout: uptime computed over less of the window than it should be, with nothing failing and nothing logged. - **The Tick column is NOT a like-for-like comparison, and should not be read as one.** It falls from 115 ms to 20 ms at one worker, and the obvious reading - that the batched insert replaced a round trip per monitor - is refuted inside this same evidence set. The 120-second cap run uses the identical one-insert fan-out and reports 115.8 ms, the very figure attributed to the old code; the 2000-monitor 60-second run reports 8.0 ms, cheaper than the 1000-monitor one. What the number tracks is how many monitors the tick actually SELECTS, and in the After sweep it selects almost nothing, because the fleet has scheduled itself and holds dispatch claims. Twenty milliseconds is an idle tick, not a cheaper one. The sample is also one or two ticks per run, since a 120-second window contains two minute-boundaries. An average over two samples is what it is; it is published because leaving the column out would hide the measurement rather than qualify it. **Duplicates were zero in every cell of both builds.** An earlier draft of this page claimed the claim removed dozens of them. It did not, because there were none to remove: the runs that showed 8 and 41 were measured on dirty trees, and at least one of them with workers leaked from a previous run still contending for the same monitors. The claim is still worth having - it makes the exclusion a stated invariant instead of an emergent property of a queue index, and it is what lets the sub-minute path participate in the same exclusion - but this measurement is not evidence that it fixed a duplicate problem, and it should not be quoted as if it were. ### What the batch cap does, isolated The before-and-after tables do not answer "does `MONITOR_SCHEDULER_BATCH` matter", and the honest answer turned out to be "only sometimes, and not where this release originally claimed". The cap sits on the TICK, and what uses the tick is decided by the EFFECTIVE interval the scheduling policy just chose, not by the one the operator configured. A monitor whose next evaluation is at or below the tick period away schedules its own next check when the current one lands, so no batch size can constrain it. That is a narrower statement than "60-second monitors never use the tick", and the difference matters: `CALM_MULTIPLIER` doubles the interval of a monitor with ten consecutive clean observations, so a healthy 60-second monitor settles at an effective 120 seconds and goes back to the tick. The runs below are 190 seconds long and never reach that run length, so they measure the busy state, not the settled one. What the cap does to a large fleet of long-settled healthy monitors is NOT measured here, and the 120-second rows are the closest thing to it. That is why the first isolating run found nothing: 2000 monitors on a 60-second interval delivered 32.67 checks per second at batch 500 and 32.45 at batch 5000, both against 33.33 demanded. Same build, same fleet, same four workers, batch the only variable, and no difference at all. For the cap to bind, two things have to be true at once: the interval must be ABOVE the tick period, so the tick is the only path, and a tick's worth of due monitors must exceed the batch. 3000 monitors on a 120-second interval puts 1500 in front of a 500 cap and satisfies both. Both runs are below, because the null result is the more useful one: it is the reason the claim "a fleet larger than 500 monitors ran under its configured cadence" was wrong for the fleet most people have. | Batch | Interval | Monitors | Workers | Achieved | Required | Coverage | Missed | Lag p95 | | ----: | -------: | -------: | ------: | -------: | -------: | -------: | -----: | ------: | | 500 | 60s | 2000 | 4 | 32.67/s | 33.33/s | 100.00% | 80 | 2.1s | | 5000 | 60s | 2000 | 4 | 32.45/s | 33.33/s | 100.00% | 106 | 2.0s | | 500 | 120s | 3000 | 4 | 10.50/s | 25.00/s | 42.33% | 1753 | 73.9s | | 5000 | 120s | 3000 | 4 | 14.89/s | 25.00/s | 58.10% | 1222 | 71.8s | **Neither 120-second cell met demand**, and that is the second thing this run says. The same four workers delivered 32.45 checks per second to a 2000-monitor 60-second fleet and only 14.89 to a 3000-monitor 120-second fleet that asks for less. Monitors above the tick period get materially worse service than monitors below it: the tick path arrives in bursts of a minute's work at once, and the self-scheduling path arrives as a smooth stream the workers never fall behind on. Raising the batch recovers part of it - 42% more throughput and 37% more of the fleet covered - but not the gap itself. That gap is the next thing to fix, and it is not fixed here. ### The metric that was wrong first The first version of this benchmark reported 23 to 59 "duplicate observations" per run, and the number is worth recording because it is exactly the kind of measurement that produces a fix for something that is not broken. Re-measured against the same baseline with the corrected threshold, on a clean tree and a machine with no leaked workers on it, the real figure is **zero, in every cell of both builds**. Two wrong numbers, in sequence, and the second survived longer than the first. The threshold was corrected from half an interval to interval/16, which took 59 down to 8 - and 8 was still wrong, because that run was measured on a dirty tree while workers leaked from an earlier run were probing the same monitors. Only the third measurement, from a pristine worktree on a quiet machine, gave the answer that was true all along. It counted pairs closer together than half an interval. The scheduler is adaptive: `nextEvaluationAt` tightens a monitor with any failed observation in its recent history to a sixteenth of its interval, so a 60-second monitor that failed once is legitimately probed every 3.75 seconds. Half an interval counted every one of those as duplicate work. The threshold is now that sixteenth, below which two observations cannot be the policy's doing, and the artefacts record the failure count beside it so a reader can see whether they are holding a run against healthy targets or a struggling one. ### How the claim interacts with the other two planes Neither plane needed changing, and it is worth saying why rather than leaving a reader to check. **Remote probes.** The tick claims a monitor, `runMonitorCheck` hands it to `dispatchToProbes`, and that function opens a round and writes its own `next_evaluation_at` reservation from the monitor's interval. A second round cannot be opened while one is in flight: a unique index over open rounds per monitor arbitrates that, and it predates this work. The claim outlives that write, which is the point of separating the two columns. `dispatchToProbes` reserves the monitor's next DUE time; the claim says a dispatch is in flight right now, and it is cleared by `recordCheckOutcome` when the round settles - the same call that lands the observation, for a probe monitor exactly as for a local one. A round that never settles releases the monitor when the lease expires instead. Under the old design, where the claim WAS `next_evaluation_at`, this write silently released a lease that was still meant to be held. **The high-frequency plane.** Monitors a live shard lease covers are filtered out BEFORE the claim, not after: claiming one and then dropping it would hold a lease nothing was going to use. They are also deferred one tick period, because nothing advances `next_evaluation_at` while that plane owns them - so without the deferral they are permanently the most overdue rows in their tenant and head the fair ranking on every tick, spending selection slots that can never become work. The deferral tests the LEASE and never the flag. A monitor whose high-frequency worker has died still has the flag set, and deferring on the flag would leave it deferred forever with nothing probing it: the difference between degrading to a slower cadence and degrading to silence. **The rule underneath all three**: `next_evaluation_at` says WHEN a monitor is next owed a probe and anything may write it, last writer wins. `dispatch_claimed_until` says WHO is dispatching it right now. Collapsing them into one column is the mistake this release made and then found: it made claiming a backlog hide the backlog, and it made every writer of the due time an accidental releaser of the lease. ### What the achieved rate is not **The achieved rate is bounded by demand, not by capacity.** 1000 monitors on a 60-second interval require 16.67 checks per second, so a run that reports 16.6 means the fleet was kept up with; it is not the largest rate the workers could have reached. The only cells that measure a ceiling are the ones that fall short of what the cadence required. ## 4. Where the ceiling actually is Throughput per worker is bounded by the **ledger actor chain**, not by the probe and not by the scheduler. Every observation takes a row lock on its actor to advance that actor's sequence, and one worker process is one actor. `modules/ledger/service.ts` says so in its own docstring: "one actor's facts serialise against each other. That is bounded by design, chains are per-actor, so more writers means more chains rather than more contention on one." So the product was designed for horizontal worker scaling, and the measurements are what that design predicts: one worker has a fixed per-process rate, and adding replicas adds rate rather than contention. The scheduler was the part that could not feed them. **Practical guidance.** Take the checks-per-second one worker delivers on your hardware from the table below, and size the fleet at `fleet_size / interval_seconds / that_rate`, rounded up, plus one for headroom during a rolling restart. ## 5. Configuration | Setting | Default | What it does | | ----------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `MONITOR_SCHEDULER_BATCH` | 5000 | How many due monitors one tick enqueues. Bounded on purpose: an unbounded tick turns a restart into one enormous burst of outbound requests | | `NOTIFICATION_BATCH_SIZE` | 250 | Unchanged; the notification drain's own bound | | `NOTIFICATION_DELIVERY_CONCURRENCY` | derived from the pool | Unchanged | | `VIGIL_NODE_ID` | `host:pid:boot` | Overrides how a replica names itself in the fleet view | | `VIGIL_ACTOR_NAME` | hostname | The ledger identity. **Give each replica a distinct one**, or they share a chain and contend on its sequence | **A single-node install needs none of this.** One worker, no settings, same behaviour as before except that the tick now enqueues up to 5000 monitors instead of 500 and the Workers page exists. ## 6. Operator surface **Settings, Workers.** Three readings, and the split between them is a tenancy boundary rather than a preference. - **Scheduler backlog**, and whether monitoring is running at all: shown to every member, always. Both are this workspace's own facts, and the stalled banner in particular has to reach the responder, whose job is to notice it. The backlog's AGE is the number that matters: a large backlog seconds old is a healthy tick doing its job, and a small one minutes old is a scheduler that has stopped. - **Workers** and **Queues**: hostnames, process ids, build versions and per-queue depth. These span every organization on the installation, so they are shown only when `VIGIL_FLEET_DETAIL=true` AND the member has `organization: update`. The switch exists because there is no installation-level role in this product and inventing one would have been a permissions change wearing a feature's clothes. `VIGIL_FLEET_DETAIL=true` says "every organization here is mine", which is true of a single-tenant deployment and false of an agency one. Off by default: the safe default for a leak is closed. **Lateness is measured independently of the scheduler's claim.** The claim leases a monitor in its own column and leaves `next_evaluation_at` alone, so a backlog does not vanish the moment a tick looks at it. The first version moved the due time instead, which flattened every reading of how far behind the fleet was to at most one lease. **The health endpoints disclose nothing.** All three are unauthenticated and deliberately carry no version and no topology. `/api/ready` answers "may this process be sent traffic" (Postgres reachable and every migration in this build applied), which is what a load balancer needs and all it should be told; which of the failures it was goes to the log. `/api/live` answers "is this process running", and must not be used to route traffic. **The readiness probe has its own two connections.** Not the application's, because the failure it exists to catch is a database that accepts a connection and then stops answering, and that never gives a connection back: a probe on the shared pool would exhaust it in ten ticks and the outage would be the probe. Its connections are bounded at 1.5 seconds to establish and a second per statement, which is a probe's arithmetic and would be far too short for anything the product does. Two per web replica, on top of the ten the application takes. **Topology history is `worker_nodes`, not the audit log.** A worker serves every organization, so writing "worker-3 left" into each tenant's audit trail would be the same event N times, and writing it into one arbitrary tenant's would be a lie. The rows persist for seven days with `stopped_reason`, which is how "we deployed at 14:03 and the fleet was down to one worker for ninety seconds" stays answerable on Tuesday. ## 7. What is still not highly available Stated rather than implied, because a page about HA that omits its own limits is marketing. - **Postgres is a single point of failure.** Everything coordinates through it. Use managed Postgres with a replica and automatic failover, or accept that its outage is the product's outage. - **The app's rate limiter is per-process.** `lib/rate-limit.ts` says so: with several app replicas each enforces its own window. Acceptable for a cost guard, wrong if you need a strict global limit. It is the only such thing, and that was checked rather than assumed. Every other piece of module-level state in the request path is either a cache of something immutable that each replica derives for itself - ledger actor ids, `Intl` formatters, the decrypted channel key, the Anthropic client - or a value read from the environment at boot, which is by definition the same on every replica. Sessions, rate-limit decisions that matter, and every write are in Postgres. So app replicas are horizontally safe: put them behind a load balancer with no session affinity and nothing needs to know how many there are. - **The high-frequency plane's cadence degrades on failover, by design.** A shard whose holder died is picked up after the lease expires; until then those monitors fall back to the ordinary tick. Degrading from 500 ms to 2 s is a service level; degrading to silence would not be. - **A rolling restart drops in-flight escalation rungs, not pages.** The notification outbox is durable and fenced; a pg-boss job that was active when a worker was killed is retried per its queue's policy, and `monitor-check` has `retryLimit: 0` because a re-run would double-probe. The next tick re-enqueues it. - **Catch-up after an outage is bounded, not instant.** A fleet that was entirely unmonitored comes back over several ticks. That is the intended behaviour: the alternative is one burst of thousands of simultaneous outbound requests, which is a self-inflicted incident. Incidents are not manufactured by the gap. The failure window is measured in time, not in consecutive failures. ## 8. Upgrading Nothing to do. Two migrations: a `worker_nodes` table Core does not have and nothing in either edition coordinates through, and a nullable `dispatch_claimed_until` column on `monitors` that both editions get. Neither has a backfill - an unset claim reads as unclaimed, which is what every existing row should say. The scheduler change is a larger default bound and that claim, both of which apply to an existing fleet without configuration. Two things are worth knowing on the way in: - **A fleet that was silently behind will catch up on the first tick after the upgrade**, because the bound that was holding it back is gone. If your installation has more than 500 monitors, expect one burst of checks and a backlog that drops to zero over a few minutes. That burst is bounded by `MONITOR_SCHEDULER_BATCH`, and the checks it produces are the ones that should have been happening all along. - **If you already run more than one worker**, give each a distinct `VIGIL_ACTOR_NAME`. Two replicas sharing one ledger identity share one chain and contend on its sequence, which is the one configuration that makes a second worker slower rather than faster. This was true before this release too; the fleet view is what makes it visible. ## 9. Reproducing the measurements ```bash # a local target on a spread of ports PORT=38080 PORTS=20 node scripts/bench/target.mjs & # a fleet, in a THROWAWAY database. The suite truncates observations, # deletes incidents and pauses monitors, so it refuses to run unless the # database name contains "bench" and BENCH_DESTRUCTIVE is set. # Re-run the migration after ANY schema change, including between the # two halves of a before-and-after sweep. A bench database one migration # behind the tree makes every tick throw, and the run that produced that # reported 0.00 checks/s, zero duplicates and a growing backlog - all # arithmetically correct, all about a system that was not running. The # harness now exits 3 on it rather than writing a plausible artefact. createdb vigil_bench DATABASE_URL=postgresql://…/vigil_bench npx drizzle-kit migrate DATABASE_URL=postgresql://…/vigil_bench npx tsx scripts/bench/seed-scheduled.ts 1000 60 # the sweep: N monitors x W workers, resetting between cells. # The label matters: `bench-check` reads docs/evidence/bench/sched-*.json, # so a run labelled anything else produces artefacts it cannot see. BENCH_DESTRUCTIVE=yes DATABASE_URL=postgresql://…/vigil_bench \ scripts/bench/scheduler-suite.sh sched-after "1000" "1 2 3 4" # publishing a run: copy the artefacts in and regenerate the table. # Nothing in the table is typed by hand. cp bench-results/sched-after-*.json docs/evidence/bench/ npm run bench:tables npm run bench:check ``` Each cell writes an artefact to `bench-results/` carrying the commit, the tree's cleanliness, the machine, the Postgres settings, the window, and the caveats that say what the number is not. A result with a dirty tree is recorded as such rather than quietly published: the SHA does not describe what ran. `scripts/bench/scheduler-suite.sh` pauses surplus monitors rather than deleting them, so every cell measures the same fleet. The ids decide nothing here, but re-seeding between cells would change what "1000 monitors" meant from one row to the next.