Incident operations
Commercial edition

The record does not stop at detected.

Detection is the cheap half. Everything after it is people, and Vigil keeps one timeline through all of it.

4 rolesowner, admin, responder, viewer
append-onlyattempts, never edited
62 s to 6 hretry walk on a delivery
one timelinefrom page to postmortem

One incident, end to end

Five moves, one timeline.

  1. Route it

    A routing policy decides which channels an event class reaches, and the scope it applies to is stored rather than inferred, so a deleted rule cannot quietly widen an audience. Maintenance windows suppress by reading timestamps, not by flipping state, and they are DST-safe.

  2. Page a person

    On-call rotas and escalation ladders decide who is woken and when the next person is tried. Delivery is durable: attempts are append-only, retries walk from about a minute to six hours, and a lease is fenced so two workers cannot both own one message.

  3. Run the runbook

    Bounded steps against the incident, from a typed action registry. Two of those actions involve a person, and when one runs the whole run suspends rather than pretending to continue.

  4. Hand a step to a human

    The hand-over becomes a task with a deadline and an owner. The answer that comes back is stored beside the step that asked for it, and the run resumes when the task is completed or cancelled, or when the step it waits on reaches its own deadline.

  5. Publish and close

    A status update posts from the incident screen to the public page, an objective's error budget is computed from the same observations the incident came from, and the postmortem is drafted from the real timeline instead of a blank page.

One Vigil runbook run: why it ran, the plan version pinned to it, and each step with the payload it sent, the attempts it spent and the result it recorded, including the hand-over to a person.

runbook runevery step keeps what it sent, what came back, and what it verified afterwards

Who is woken

A page that was never delivered is not a page.

Escalation steps reach people by email, SMS or voice call. Email is always available; SMS and voice route through your own Twilio account, and steps on channels you have not configured are logged rather than silently skipped.

Every message has a ledger: queue pressure, attempts spent, when the next retry is due, and the reason it stopped. An attempt that never reported an outcome is visible as exactly that rather than counted as a success.

The delivery ledger

Vigil escalation settings on a phone: the channels a step can page through, an on-call schedule that rotates every seven days between three responders, and the escalation policies attached to monitors.

escalationrota, ladder, and what each step may use

Objectives

An error budget and an uptime page cannot disagree.

An objective measures the same observations the incidents came from. Compliance over the window, how much budget is left, and the burn windows that are firing, computed from one source of truth rather than from a second pipeline that drifts.

A Vigil service level objective: 99.832 percent compliance against a 99.95 target, no error budget left, burn rates of 643.7 times over one hour and 105.1 over six, and a day-by-day bar for the window.

objectivethe budget is spent, and the burn windows say how fast it went

Running it

Updates should have a way back.

The supported path is Bash and Docker Compose. vigilctl checks real health rather than container liveness, validates a backup archive before it moves anything, and refuses a state it cannot own safely rather than guessing at it.

Nothing rolls back on its own. A rollback undoes code and database together, and an operator asks for it.

install
exit 0 / 10
start or repair the Docker Compose install
doctor
read-only
diagnose it without changing anything
backup
validated
dump the database and check the archive restores
restore
then verify
replace the database from an archive
update --to
backup first
move to a version you name
rollback
on request
undo the last update, code and database together