Vigil Product tour Buy Vigil
Menu

Guide

Restart a container when it is unhealthy, or when its URL stops answering.

Docker will restart a container whose process has exited. It will not restart a container that is still running and marked unhealthy. The free setup is a healthcheck plus something that acts on it. Vigil can watch the URL, and the Commercial edition can call a restart hook you own and check that the service came back.

A restart policy does not watch health

Docker's restart policies decide what happens when a container exits, or when the daemon restarts. no, on-failure, always and unless-stopped are all about that exit. None of them mention the health status.

HEALTHCHECK is a separate instruction. Docker runs your command and, after enough consecutive failures, marks the container unhealthy. The process can still be running. A web server stuck and refusing connections is the case the Dockerfile reference gives. docker ps shows that container as Up, with an unhealthy note if you ask for it.

The same Up line is what you see in a crash loop, for a few seconds between exits. Uptime Kuma issue #7956 is a report of that: the Docker monitor read a crash-looping container as up, because the inspect happened to land while the container was running. A check that only asks "is it running right now?" will do that. Vigil compares the restart count across samples. The Docker section below is the exact rule.

Check the URL from inside the container

Put a command in the image that requests the URL the service is supposed to answer. The image has to contain whatever you call. curl is not in every base image. 

HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
  CMD curl -fsS http://127.0.0.1:8080/health || exit 1

Exit 0 is healthy. Exit 1 is a failed check. Docker waits for --retries failures in a row before it flips the status to unhealthy. --start-period is the grace time while the process is still booting. Those defaults and the status values are in the Dockerfile reference linked above.

The same check in Compose, if you would rather not rebuild:

services:
  api:
    image: your-api
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8080/health"]
      interval: 30s
      timeout: 5s
      retries: 3

restart: unless-stopped still only reacts when the container exits. The healthcheck on its own does not restart anything.

Restart it when the healthcheck fails

willfarrell/autoheal watches the Docker socket and restarts containers you have labeled, once their health status is unhealthy. The project describes itself as a stand-in for a Docker feature that was proposed and not shipped. Its README is the source for the labels below. 

services:
  api:
    image: your-api
    restart: unless-stopped
    labels:
      autoheal: true
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://127.0.0.1:8080/health"]
      interval: 30s
      timeout: 5s
      retries: 3

  autoheal:
    image: willfarrell/autoheal
    restart: unless-stopped
    environment:
      AUTOHEAL_CONTAINER_LABEL: autoheal
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

The default label name is autoheal, and the value has to be true. Setting AUTOHEAL_CONTAINER_LABEL=all restarts every unhealthy container that daemon can see. Leave that off unless you mean it.

The socket mount is root-equivalent on that Docker host. The autoheal container can start, stop and remove containers. Run it only on a machine where that is acceptable, and do not publish it.

There is a second free trick: make the healthcheck kill pid 1 when curl fails, so the process exits and the restart policy applies. It works, and it is blunt. A healthcheck that signals the wrong process, or that fires during startup, takes the container down on purpose. Autoheal leaves that decision in a separate container.

Neither of those checks that the URL answered again afterwards. A crash loop satisfies "I restarted it" forever.

Or restart from a monitor webhook, only on down

If something outside Docker already notices that the URL is unreachable, point its down notification at a small receiver. The receiver holds the secret and the map. The payload is not allowed to name a shell command.

The webhook provider in the current stable release (2.5.0) POSTs {"heartbeat", "monitor", "msg"} unless you replace the body. heartbeat.status is 0 for down, 1 for up, 2 for pending and 3 for maintenance (src/util.ts on that tag). Act only on 0. In Kuma's webhook settings, put the secret in additional headers: {"Authorization": "Bearer <secret>"}. 

import { createServer } from "node:http";
import { timingSafeEqual } from "node:crypto";
import { execFile } from "node:child_process";

const SECRET = process.env.RESTART_SECRET;
const PORT = Number(process.env.PORT || 8909);

// Keys are monitor names you chose. Values are container names.
const CONTAINERS = {
  "api health": "api",
  "vpn": "gluetun",
};

function authorized(header) {
  const token = header.startsWith("Bearer ") ? header.slice(7) : "";
  if (!SECRET || token.length !== SECRET.length) return false;
  return timingSafeEqual(Buffer.from(token), Buffer.from(SECRET));
}

createServer((req, res) => {
  if (req.method !== "POST") {
    res.writeHead(405).end();
    return;
  }
  const chunks = [];
  req.on("data", (chunk) => chunks.push(chunk));
  req.on("end", () => {
    const header = req.headers.authorization;
    const value = Array.isArray(header) ? header[0] : header || "";
    if (!authorized(value)) {
      res.writeHead(401).end();
      return;
    }
    let body;
    try {
      body = JSON.parse(Buffer.concat(chunks).toString("utf8"));
    } catch {
      res.writeHead(400).end();
      return;
    }
    if (!body.heartbeat || body.heartbeat.status !== 0) {
      res.writeHead(200).end("ignored");
      return;
    }
    const container = CONTAINERS[body.monitor && body.monitor.name];
    if (!container) {
      res.writeHead(200).end("no mapping");
      return;
    }
    res.writeHead(202).end("restarting");
    execFile("docker", ["restart", container], (error) => {
      if (error) console.error(container, error.message);
      else console.log("restarted", container);
    });
  });
}).listen(PORT);

Run it on the Docker host, with RESTART_SECRET set, and do not expose the port past that host. This answers the webhook and restarts one named container. It does not probe the URL afterwards, and it will restart again on the next down notification. That is the gap.

What the Docker check calls down

Vigil Core includes a Docker container check. It asks the engine API for one inspect, GET /containers/<name>/json, over the worker's socket or a tcp:// address (src/modules/monitors/types/probes/docker.ts). The worker has to be able to open that socket. A missing socket is reported as misconfigured, not as the container being down.

The inspect is stored as these facts:

  • running, from State.Running
  • state, from State.Status
  • health, from State.Health.Status, or null when the image has no HEALTHCHECK
  • restartCount, from RestartCount, shown on the check as Restarts
  • containerId, from Id, so a recreated container is not compared with the one it replaced

The verdict is in src/modules/monitors/types/specs/docker.ts. The comparison with the previous sample is in src/modules/monitors/types/docker-restart.ts.

  • Not running is down, including a 404 from the daemon. A deleted container is treated as down, not as "the daemon could not be reached".
  • A rising restart count is down. The default is 3 restarts inside 10 minutes. Both numbers are on the monitor, under Advanced, and detection can be turned off. The first sample only records the count, so a container that has already restarted hundreds of times is not an alarm. A new container id, a count that went backwards, or a gap longer than the window starts that record again. The check says restart loop: N restarts in M min. M is the window you set, not the age of the oldest restart. The failure window still has to elapse before an incident opens. Its column default is 120 seconds (src/db/schema/monitors.ts).
  • Health unhealthy is down by default. Health starting, and a container with no HEALTHCHECK, are not failures. One unhealthy sample does not open an incident. The same failure window has to elapse, which is what "sustained" means here. Set Unhealthy counts as down to no and unhealthy stays degraded. Degraded does not open an incident.

An incident opened by a restart loop, or by unhealthy counted as down, stays open if a later sample is only degraded. It closes when a check is up (src/modules/monitors/status-controller.ts). The incident timeline gets the restart-loop sentence, and "The container's healthcheck is failing" when that was a down cause. Those lines are on the operator timeline.

A sample that lands in the few seconds the container has exited is down because it is not running. The restart count is what catches the loop when the sample lands while it is Up.

A container that is running, not restarting, and has no HEALTHCHECK, while the process inside is dead, is still up. Watch the URL. An HTTP or TCP monitor fails when nothing answers, which is the case where Docker says the container is up and the service is not reachable. One refused connection inside the failure window is not yet an incident.

The comparison uses the samples stored for this monitor on the worker that probes it. A Docker monitor sent to remote probes is judged from the inspect those probes return, without that memory, so a crash loop can still read as up there.

Commercial recovery calls your hook, then checks again

Core detects the failure, opens the incident, writes the timeline and notifies the channels you configured. It does not restart a container. Automatic recovery is Commercial, and it is off until you enable a recovery action on that monitor.

The loop is in src/worker/jobs/recovery.ts. The limits are in src/modules/recovery/schemas.ts and src/modules/recovery/engine.ts.

  1. An incident has to be open. That means the monitor was down for its failure window, not merely degraded.
  2. The worker probes again before it does anything. If the probe cannot measure the target, it records a skip and does not call the hook. If the target is already healthy, it skips the same way.
  3. Ten executed recovery triggers for that monitor inside 24 hours is treated as a loop. The hook is not called, and a person is paged. The cap is fixed.
  4. Before the POST, it checks again that the incident is still the same outage, the monitor is not paused, and the action is still enabled with the same URL and secret.
  5. It POSTs a JSON body to the URL you configured. The signature is HMAC-SHA256 of the exact body, in X-Vigil-Signature: sha256=<hex>. X-Vigil-Event is recovery.execute. The body names the monitor, the incident and the attempt number. It does not contain a shell command. The command lives in your receiver.
  6. Delivery is up to three tries inside the webhook sender. A delivery that never gets a 2xx is a failed attempt.
  7. After the verify delay (default 30 seconds, allowed range 5 to 600) it probes again with the same check that opened the incident. A pass is written on the timeline as recovery verified. The ordinary check loop is what marks the monitor up and resolves the incident. Recovery does not close it by itself.
  8. If the verify probe still fails, the next attempt waits for the cooldown (default 300 seconds, allowed range 30 to 3600). Attempts per incident are 1 to 5, and the default is 1.
  9. When the attempts are used up, the timeline says recovery is waiting for a person, and the incident stays open.

Hold alerts is a second switch, and it defaults to off. Off means the incident notification goes out when the incident opens, and recovery runs as well. On means the page waits while recovery runs, and goes out if recovery is exhausted, if the 24-hour loop guard stands down, or if the chain misses its deadline. A maintenance window, which is also Commercial, stops the hook and the page for that period. Core has no maintenance windows.

A test trigger sends recovery.test. The receiver should check the signature and not run the command. The Commercial source ships that receiver as examples/recovery-receiver.mjs. The decision in it is the same one to copy: verify X-Vigil-Signature, run only for recovery.execute, and map the monitor id to a container name you already chose. 

import { createHmac, timingSafeEqual } from "node:crypto";

function signed(secret, rawBody, header) {
  const expected =
    "sha256=" + createHmac("sha256", secret).update(rawBody).digest("hex");
  if (!header || expected.length !== header.length) return false;
  return timingSafeEqual(Buffer.from(expected), Buffer.from(header));
}

// After a verified recovery.execute:
//   const id = payload.data.monitor.id;
//   const container = CONTAINERS[id];
//   if (container) execFile("docker", ["restart", container], ...);

The seeded tour has both outcomes: a recovery that verified and a recovery that exhausted its attempts. The configuration screen is on the monitor. Those are sample records, not a live result.

Which edition

The checks, the incident, the timeline and a signed notification webhook are in Vigil Core, Apache-2.0. You can point that webhook at a receiver like the one above. Core will not re-probe after the restart, and it will not apply attempt limits.

The recovery loop on this page is Commercial: $149 once per company, not per seat. Pricing and recovery and operations are the longer pages. Escalation is the page about who gets paged when the restart does not work.