Self-healing monitor stack

BOA runs no external monitoring agent and no resident daemon. Every box watches and repairs itself from a single root crontab that fires a handful of short-lived bash scripts every minute. Those scripts sample load, restart dead services, kill runaway processes, scan auth logs for abuse, and drain the Ægir task queue — then exit. There is nothing long-running to crash; the watchdog is the cron tick.

This topic covers the service / load / process side of that machinery:

• what cron launches
• the self-looping fan-out that gives it sub-minute reaction time
• the per-service watchdogs
• the load auto-pause ladder
• the process guards and auth scanners
• the box-class throttle that keeps the fan-out from pinning idle load on small hosts
• the loadreport profiler used to measure it

The security-facing members of the same /var/xdrago/monitor/ family — the nginx log scorer scan_nginx.sh and the CSF ban pipeline — live in the Abuse Guard topic, and the per-host firewall those bans drive is in Security & isolation. This topic points at those rather than repeating them.

The root crontab

The installed root crontab lives at /var/spool/cron/crontabs/root. Its master copy is aegir/tools/system/cron/crontabs/root, deployed to /var/xdrago/cron/crontabs/root and copied into place on every install/upgrade. Do not hand-edit the installed file — its own header says so, and the next upgrade overwrites it. Per-box additions go in /var/xdrago/cron/custom.txt, which BOA appends after the copy.

The per-minute monitors (second.sh, minute.sh, guest-fire.sh, runner.sh) are the live self-healing core. The 2-minute and longer jobs are slower-changing maintenance. The nightly owl.sh run has its own page — Nightly maintenance (owl.sh); the dump/backup jobs are out of scope here — see Backups.

Where the pieces live

The repo source under aegir/tools/system/ is deployed verbatim to /var/xdrago/ on the box (cp -af .../aegir/tools/system/* /var/xdrago/). So a file the source tree calls aegir/tools/system/minute.sh is /var/xdrago/minute.sh at runtime. The single-shot monitors live one level down in /var/xdrago/monitor/check/. loadreport is the exception — it is deployed to /opt/local/bin/loadreport with a /usr/local/bin/loadreport symlink, like fpmreport.

Every per-service watchdog launched by minute.sh re-sources /root/.barracuda.cnf on entry (so every _VAR override below is read fresh each pass) and exits immediately unless /var/log/boa/reset_no_new_password.pid exists — i.e. the auto-healing watchdogs stay dormant until the box is a fully installed BOA system. The second.sh process guards and the launchers themselves do not gate on that marker.

Cross-cutting

• The security half of this same /var/xdrago/monitor/ machinery — scan_nginx.sh, the request scorer, and the guest-fire → guest-water → nginx_deny ban pipeline — is the Abuse Guard. This topic deliberately does not repeat it.
• The firewall those guards drive — temp bans, the allow list, LFD, SYNPROXY — is Security & isolation. The flood guards in this topic also pkill/purge runaway csf processes.
• Every monitor re-sources its config from /root/.barracuda.cnf on each spawn; the _CPU_*_RATIO load thresholds, the _INCIDENT_REPORT alert level, and the _MONITOR_* throttle overrides are catalogued in the host control-file reference and the Reference appendix.