Everything here happened in the Vigil codebase and shipped in a
release you can read. A note gets written when a fix says something
about the design — not on a calendar. Each note carries the release
it came from and is dated to that release, newest first.
Since 1.6.0 the public status page has cached its query for 60
seconds. That page is the one screen guaranteed to get traffic
exactly when the system it describes is on fire, and going to
Postgres once per viewer during an outage is a bad trade. So:
unstable_cache around the query, sixty-second
revalidate, done.
It worked. Then it returned a 500 — not on the first view, on the
second one, and only on a page that had an incident to show.
startedAt.toISOString is not a function.
unstable_cache JSON-serialises what it stores. On a miss
you get back the value the function just returned, with real
Date objects on it. On a hit you get back whatever
survived a round trip through JSON, which is a string. So the page
rendered, populated the cache, and then broke for the next sixty
seconds against exactly the data it had just rendered fine.
The types agreed Serialisation happens past the compiler
The cached wrapper is still typed as returning the page object, so
startedAt is a Date on both paths as far
as the compiler is concerned. The serialisation
happens inside the framework, at runtime, on the far side of the
type. There was nothing there for it to catch.
The test only saw a miss One visit is never a cache hit
The end-to-end smoke test opened the status page, asserted on it
and moved on — from a cold cache, every run. It only ever
exercised the path that worked. A test that loads
a page once cannot see a bug that needs the cache to be
warm. This shipped in 1.6.0 and was fixed in 1.9.0 with CI
green the whole way.
The fix revives the incident timestamps at the boundary, before
anything downstream touches them: new Date(x) is a parse
on a string and a copy on a date, so both paths converge on the same
shape before the template sees them. The smoke test now reloads the
page and asserts again, which is the one assertion that would have
failed. The lesson is unglamorous and worth restating: a cache
boundary is a serialisation boundary. What comes back across it is
data, not objects, whatever the signature says — treat it the way you
would treat a response off the wire.
· shipped in 1.4.0
A bound per incident is not a bound per monitor.
Automatic recovery has been bounded since it shipped in 1.2.0:
one to five attempts per incident, a cooldown between them, and
when the attempts run out the incident simply stays open for a
human. That bound was written for the failure everyone pictures
— something is down and stays down, and you do not want a hook
hammering it.
It does nothing for the failure that actually loops. A flapping
target goes down, recovery fires, the target comes back, and the
next passing check resolves the incident. A few minutes later it
is down again — and that is a new incident, with its own
counter, starting at one. Every attempt is legal inside the
incident it belongs to. The counter was never wrong. It was
counting the wrong noun.
What bounds a recoveryScope
attempts1–5 per incident, with a cooldown — set per action
triggers10 per monitor per rolling 24 hours — fixed constant
countedonly attempts whose trigger was actually delivered
on capattempt recorded as skipped, with the count that stopped it
alertsanything held for recovery fires immediately
The guard counts executed triggers per monitor over a rolling
24 hours and stands down at ten. Standing down is loud, not quiet:
the attempt is written to the recovery record as skipped, with the
count that suppressed it; the incident timeline says that repeated
recoveries point to a deeper problem and it is waiting for a human;
and any alerts that were being held while recovery ran fire at once.
Ten restarts in a day is not a repair, it is a lid on something that
needs a person.
Only delivered triggers count. An attempt that stopped because the
pre-recovery probe found the target already healthy never reached
your endpoint, so it does not move the counter — the cap measures
what was done to your infrastructure, not how often the worker woke
up.
Ten is a constant in the source. So is the 24-hour window. Neither is
exposed in the UI, and that is the part worth arguing about: a safety
limit with a settings field is a limit that gets raised at 03:00
during the exact incident it exists for, by someone who will not be
around to lower it again. If ten a day is genuinely wrong for a
monitor, the honest answer is not a larger number — it is turning
recovery off for that monitor, which is one switch and does not
quietly change what the guard means for everything else.
· shipped in 1.3.0
Three writers, one page.
1.3.0 let a recovery action hold an incident's alerts. If recovery
verifies the target healthy, nobody is paged and the incident
resolves quietly — the timeline and the recovery record still say
everything that happened. It is off by default, because deferring a
page is a real risk and should be a decision somebody makes, not a
default they inherit.
Holding an alert means something eventually has to send it. Three
code paths can decide that moment has arrived, and they can be in
flight at the same time in different jobs:
01The open path — nothing is held, so the incident notifies as it always did
02The recovery loop — attempts exhausted, or the restart-loop guard standing it down
03The deadline failsafe — the worst case has passed and nothing else has spoken
Two of them firing is a duplicate page at three in the morning. None
of them firing is an outage nobody was told about. Both failures are
worse than not holding alerts in the first place, which is why the
feature is only as good as its arbitration.
The arbitration is one conditional update: set
notified_at where the incident id matches
andnotified_at is still null, returning the
row. Whoever gets a row back owns the notification and sends it;
everyone else gets nothing back and returns without a word. Postgres
decides, once, durably, in the same database the queue already lives
in. No advisory locks, no dedup key, no second system to be right
about.
The failsafe is the part that is easy to get subtly wrong. It is
scheduled when the incident opens, in the same place the first
recovery job is enqueued — not at the end of the recovery chain.
Scheduled by the chain, it would share the chain's fate: a worker
killed mid-attempt takes the page with it and the incident sits open
and silent. Scheduled up front, the incident carries its own deadline
from the first second, and no amount of crashing downstream can
remove it.
That deadline is deliberately generous — it is the worst case for the
whole chain: roughly five probe timeouts plus the verify delay per
attempt, the cooldowns between attempts, then one monitor interval
and a minute of slack for the auto-resolve to land. A late page beats
a duplicate one, and the claim makes duplicates impossible anyway, so
there is nothing to be won by cutting it fine.