Service auto-healing watchdogs

BOA keeps the stack alive without operator intervention through a fleet of short-lived watchdog scripts that the minute.sh launcher fans out, on the box, every few seconds. Each watchdog checks one service and — if it finds it down, stuck, or misconfigured — repairs or restarts it in place.

There is no daemon: the watchdogs are spawned, run to completion, and exit. Cron re-launches minute.sh every minute; the launcher self-loops to drive a tighter cadence inside the minute.

This page documents the per-service watchdog fan-out (_launch_auto_healing in aegir/tools/system/minute.sh) and the single-shot monitors under monitor/check/. The related parts of the same machinery are documented elsewhere:

• Load-reactive side of the same machinery — the high-load brakes that write /run/max_load.pid and /run/critical_load.pid, which almost every watchdog here honours — is in Load control.
• Cadence (how many times per minute the fan-out runs, and how that is throttled on small or CI boxes) is in Cron cadence & idle-load throttle.
• Security-facing member of this same /var/xdrago/monitor/ family, scan_nginx.sh, is reached through this fan-out — it is looped by nginx_guard.sh (one of the watchdogs minute.sh spawns), not run directly — and is documented in the Abuse Guard topic.

Where it runs from

On a box the launchers are deployed under /var/xdrago/; the per-service watchdogs are under /var/xdrago/monitor/check/. In the repository the same files live under aegir/tools/system/ and aegir/tools/system/monitor/check/.

cron (every minute)
   │
   ├── /var/xdrago/second.sh   ── self-loops; load-control + process guards
   │        └── (its own fan-out — see Load control / Process guards)
   │
   └── /var/xdrago/minute.sh   ── self-loops, drives the SERVICE watchdogs
            └── _launch_auto_healing()  ── spawns, per pass:
                   system.sh   unbound.sh   valkey.sh|redis.sh
                   mysql.sh    php.sh       fpm_tune.sh   sqlprobe.sh
                   batch_guard.sh   task_guard.sh   nginx.sh   nginx_guard.sh   java.sh
                                  └── loops scan_nginx.sh (Abuse Guard)

Each watchdog re-sources /root/.barracuda.cnf on entry (so every _VAR override below is read fresh each pass) and exits immediately unless /var/log/boa/reset_no_new_password.pid exists — i.e. nothing runs until the box is a fully installed BOA system.

What minute.sh does before the fan-out: flood guards

minute.sh is not only a launcher. Before it spawns any watchdog it runs two flood guards of its own. These protect the box from a runaway monitor or a CSF reload storm spinning up faster than it drains — exactly the failure mode that would otherwise pin a small box at high load.

Both guards are skipped while a BOA run is in progress (/run/boa_run.pid) so they never fight an upgrade. _csf_flood_guard is additionally gated on the csf binary being present (/usr/sbin/csf) and no /run/water.pid (the guest-water drain) being active, and it re-asserts SYNPROXY (synproxy_reassert -p "443 80" --no-quic) when /etc/csf/csfpost.d/synproxy.sh is present.

The fan-out itself is then run _ITER times with _SLEEP seconds between passes; on a NORMAL box that is 9 passes 5 s apart, giving ~5 s monitoring granularity across the minute. See Cron cadence & idle-load throttle for the box-class logic and the _MONITOR_FANOUT_ITER / _MONITOR_FANOUT_SLEEP overrides.

Re-entrancy: the shared lock and the kill guard

Every watchdog (and the launchers) open with the same _manage_single_lock pattern. It sources lock.inc from /opt/local/bin/ or /opt/local/lib/ and, when that shared single-instance library is present, takes its lock. If the library is absent it falls back to a legacy pgrep -fc guard: if more than 2 copies of the same script are already running, it logs to /var/log/boa/too.many.log and exits 0. So a slow watchdog can never stack up more than briefly — the next launch detects the pile-up and backs off instead of restarting the service underneath a still-running instance.

The per-service watchdogs

_launch_auto_healing spawns each of the following with nohup … &, so they run concurrently and independently. The Valkey/Redis choice is made by an init-script probe: valkey.sh runs if /etc/init.d/valkey-server exists, else redis.sh if /etc/init.d/redis-server exists. fpm_tune.sh, sqlprobe.sh, batch_guard.sh and task_guard.sh are launched only if their files are present (newer nodes).

A few load-bearing details worth calling out:

• system.sh answers a memory emergency by freeing memory, not by taking the site down. Pressure is read from MemAvailable, which already counts reclaimable cache, and the ≤5% condition must hold across three samples five seconds apart — it stands down the moment one recovers, so a brief allocation burst the kernel absorbs on its own is not an emergency. The response is then a single kill -9 of the largest resident process that is safe to kill. The database, the SSH lifeline and the provision/backup chain (mysqld, sshd, provision, drush, mydumper, duplicity, aegir.sh, backboa, multiback) are never chosen: killing one of those means a broken task or a corrupt dump rather than freed memory. If everything large is exempt, it says so in the log, pages, and leaves the rest to the kernel OOM killer, which picks its victim with far better information than a shell script has. Between 5% and 10% free it only culls runaway wkhtmltopdf (more than 2 instances).

  Repeats are paced and then stopped, exactly as the database heals are: _OOM_COOLDOWN_SECS (60 s) between kills, a ledger at /var/log/boa/oom.kill.ledger, and a breaker that latches /run/boa_oom_latched.pid once _OOM_FLAP_MAX (3) kills have run inside _OOM_FLAP_WINDOW_SECS (3600 s), pages once, and refuses to act again until an operator deletes the latch or _OOM_FLAP_LATCH_MINS (60) passes. The window is floored against the pace the path enforces on itself, because a window too narrow to hold the allowed kills is a breaker that never opens.

  This replaced a cascade that, on one measured box, killed seven services within three seconds of a single free -mt sample — wkhtmltopdf, every PHP CLI, nginx, every PHP-FPM pool, Java, New Relic and the cache server with its data files deleted — then restarted Percona through the whole-stack path and never restarted the web tier it had killed. The kernel recorded zero OOM kills during that entire event: it was coping on its own.

• mysql.sh only restarts a database that will not answer. A needless MySQL restart is itself an outage, so the test is a functional one: a mysqladmin ping reply of any kind — including an authentication error or Too many connections — proves the server is alive, and a first failed probe is re-tried across a short grace before the server is called down. That grace matters on a host where mysqld_safe supervises Percona, because it respawns a dead mysqld within a second or two, and the socket and PID file are legitimately absent while it does. A busy-but-responsive server is therefore never restarted: high load with a high thread count (_LOAD_THRESHOLD 33.0 and _THREAD_THRESHOLD 99), and a Too many connections flood, both fall through to the real remedies — flush-hosts and the long-query killer — instead.

• fpm_tune.sh never tunes anything live. It is purely the data collector for FPM capacity sizing; the JSONL it writes is consumed by fpmreport. See PHP-FPM & performance for how that data drives worker and memory sizing.

• sqlprobe.sh never tunes anything live either. It is the SQL-side sibling: the data collector behind the memorytuner advisory, which only names _SQL_*_FORCE pin lines — applying them stays the operator's decision. See Percona setup and tuning for the advisory and the measurement stores it keeps fresh.

• batch_guard.sh can only kill what is provably dead — and load level is never consulted, in either direction. It arms on the storm's own two-floor signal (/bgp-start/-specific own-IP 499 volume plus many distinct looping bids), never on load: a pool-sized storm pins one FPM pool and degrades that site while a many-core box's average stays under any workable per-core floor — observed live at 0.69 per core, for over 40 minutes, invisible to the per-core arm gate this design replaced — and in the other direction it never waits for calm either, because its heal is cheap SQL, not a restart. Age is never a criterion: Drupal 7 writes {batch}.timestamp once, at creation, so a long-running live batch and an abandoned one look identical by age, and any age-based cleanup would eventually delete a live one. A bid heals only if it is looping through /bgp-start/ right now, is driven by a live background_process row — the very row the module's cron re-launches, which browser-driven and update.php batches never have, so those are structurally out of reach and a forged request wall cannot make a victim site's batches eligible — and its blob fingerprint plus queue count survived at least eight minutes unchanged, with the final DELETE conditional on all three of those same values. It also never gates on the /run/max_load.pid tier markers: second.sh sets those even on passes where it declines to pause (the backup/iowait exemption), so a marker gate would disarm the guard for whole backup nights. What it requires instead is service continuity: no candidate confirms unless nginx, every FPM master and mysqld have been up, uninterrupted, since before its first sighting — pidfile birth times and the server's own uptime are the witnesses — because any teardown (the load auto-pause, a watchdog heal, a database restart) freezes every batch blob and would make a live batch read as dead. A teardown that came and went entirely between two passes still invalidates the interval. The affected site's user sees "No active batch." and re-runs their action — acceptable for a loop that is provably dead. The operator dry run is the --detect-only flag, which beats any configuration line.

• task_guard.sh can only fail what is provably dead — and it never re-runs anything. The judgement is the process table: an orphaned PROCESSING row is touched only when it is the task's current revision, is past the grace window, and its instance user has no live task runner and no live provision backend. One live process holds the whole instance for that pass — with per-user evidence the reaper never guesses which row a process belongs to, so the failure direction is a delayed heal, never a killed live task. The per-row reset is conditional on the row still being PROCESSING, so the frontend's own dispatcher reaper (on newer hostmaster releases) and the manual queue button always win a race. Rows become failed with a truthful log entry; re-running a crashed task — a migrate or clone in particular — stays the operator's decision.

• nginx_guard.sh is the bridge to the security layer. scan_nginx.sh is detection-only and post-hoc, and is launched only here (10 spawns, 5 s apart) on this minute.sh fan-out — second.sh does not run it. See Abuse Guard.

• The self-service restart sentinels are gated by plan or by a box-wide marker. A tenant asks for a restart by touching a sentinel in their own control directory — ~/static/control/run-nginx-restart.pid for the web tier, run-valkey-restart.pid (or the older run-redis-restart.pid, either name works) for the cache — and the watchdog acts on it only if the box qualifies. Qualifying is an OR of two tests: a plan grep that finds POWER, PHANTOM, CLUSTER, ULTRA or MONSTER in any instance's .octopus.cnf under /root/, or the operator marker — /etc/boa/.allow.nginx.restart.cnf for nginx.sh, /etc/boa/.allow.valkey.restart.cnf for valkey.sh. Neither test is per-account: the plan grep reads every instance's control file, the sentinel scan walks every account's static/control directory, and the restart itself cycles the whole service — so one qualifying instance, or one marker, opens the gate box-wide. valkey.sh honours the redis-named marker /etc/boa/.allow.redis.restart.cnf identically, which is how a box that predates the Valkey rename keeps working; the retired redis.sh knows only that redis-named marker, and tests it twice in the same OR chain — a harmless duplicate arm with no effect either way. All three markers are read from /etc/boa only, with no /root fallback; they are operator-created, and nothing in BOA writes or erases them. See Host control files & INI for the one-time /root to /etc/boa relocation that moved them.

The daily graceful.sh cleanup

graceful.sh is not part of the minute.sh fan-out — cron runs it once a day, at 03:01 — but it is the same auto-healing family, and two operator markers switch parts of it off, so it belongs here.

A pass stands down entirely while heavy maintenance is paused (/root/.pause_heavy_tasks_maint.cnf) or while a PHP-idle stack swap holds its quiesce marker (/run/boa_php_idle_quiesce.pid — honoured only while the owner PID recorded in it is alive; a dead-owner marker is cleaned up and the pass continues). Proxy nodes are no longer skipped: since the 2026-08-09 marker narrowing the daily hygiene — including the rsyslog watchdog the intrusion scanners depend on — runs on a finalized proxy like any other box. Past them, the whole action body is skipped whenever a BOA install or upgrade pass is in flight — any of /run/boa_run.pid, /run/boa_wait.pid (the queued window) or /run/octopus_install_run.pid (the only one the chained install's octopus leg holds), a running BARRACUDA.sh.txt or OCTOPUS.sh.txt from either the /var/backups or the /var/opt/boa-dist staging path, a running barracuda or octopus wrapper under /opt/local/bin or /usr/local/bin, a boa in-* install wrapper from the same two paths, or a live AegirSetupA/B/C child — or when /root/.skip_cleanup.cnf exists. Those seven signals are the same _boa_pass_active predicate owl.sh applies, mirrored deliberately rather than shared. The in-flight test and the marker are one test, so the marker is simply the operator's way of telling the box to behave, for that night, as though a BOA run were under way. Unlike the markers converted to /root/.barracuda.cnf variables in recent releases, this one has no variable form and deliberately keeps none — the file is the control, because the hosted wipe described below is a published guarantee that a sticky cnf line would quietly undo.

What a full pass does — and therefore what .skip_cleanup.cnf suppresses in one go: refresh the Devuan mirror list (ffdevuan), purge the postfix queue (postsuper -d ALL), restart whichever syslog daemon is installed (rsyslog, sysklogd or inetutils-syslogd), reset swap and, above 90% RAM usage, drop the disk caches, fetch the GeoIP city database if it is missing, remove the leftover session files from /opt/tmp, and run the Speed-Booster purge. A completed pass stamps /var/log/boa/graceful.done.pid.

Two details in the middle of that list are worth knowing. The swap and cache step runs only where /dev/disk exists, and the swap reset is skipped outright on /root/.no.swap.clear.cnf. Where it does run, swapoff -a happens unconditionally but swapon -a is held back while duplicity is running — so on a box mid-backup the pass can legitimately leave swap off until the next night.

The Speed-Booster purge has a marker of its own. /root/.giant_traffic.cnf suppresses that one step and leaves the rest of the pass running. The step lowers the nginx cache directory depth in /var/aegir/config/server_master/nginx.conf (levels=2:2:2 becomes levels=2:2, a change it leaves in place), reloads nginx, sweeps everything older than a day out of /var/lib/nginx/speed/, then reloads nginx a second time — a long walk over a very large cache plus two reloads, which is precisely what a high-traffic box may want to skip. It is already gated on no duplicity running and on no /run/speed_cleanup.pid left by a purge still in progress.

/root/.giant_traffic.cnf has a second, unrelated effect on the watchdog side, and it shares that one with /root/.high_traffic.cnf: either marker suppresses the segfault_alert.pl spawn in php.sh, so no PHP-FPM or nginx segfault report is ever mailed from that box. Both tests are negated — presence is the off switch, absence the working default. (segfault_alert.pl also exits at once on a box with no /data/u, and today it only reports; the vhost-quarantine step its own report text describes is commented out in the script.)

/root/.high_traffic.cnf has no other live effect. The daily cache-server restart, the locked/fpmcheck skip, buffered nginx logging and the hourly FPM reload that older changelog entries attach to this name all have no reader left in the tree, so do not expect them from it.

Hosted boxes overwrite these three markers

BOA's own hosted-fleet normalisation runs inside _update_agents on every five-minute SKYNET self-update tick (see The serial & fetch pipeline (SKYNET)), on boxes whose VM family is HOSTED and that carry the hosting-master Drush alias, /data/u and /var/xdrago. That block deletes /root/.skip_cleanup.cnf and /root/.giant_traffic.cnf on every pass, so a touch of either survives only until the next tick.

/root/.high_traffic.cnf goes the other way. The same block creates it once, when it is absent, together with /root/.no.swap.clear.cnf, writing an instance count into both that nothing ever reads back — every test on these files is a presence test, never a content test. It never re-fires and never removes either file, so on a hosted box segfault alerting is off permanently and the nightly swap reset never runs. Self-hosted boxes see none of this.

Cooldowns, double-checks, and load gating

Six patterns recur across the watchdogs and are worth understanding once:

1. Double-check before acting. Almost every restart path re-reads the symptom after a short sleep (2–5 s) and acts only if it persists. This filters out the transient flap — a socket mid-reload, a log line from a restart that already happened — that would otherwise trigger a needless restart.

2. Per-service cooldown stamp. A restart writes date +%s to a /run/<service>-monitor.cooldown (or per-version /run/phpNN-fpm.cooldown) file. A subsequent failure within the cooldown window (default 30 s) logs a skip instead of restarting again, breaking restart storms when a service is crash-looping for a reason a restart cannot fix. Overridable per service: _UNBOUND_COOLDOWN_SECS, _VALKEY_COOLDOWN_SECS, _REDIS_COOLDOWN_SECS, _FPM_COOLDOWN_SECS, _NGINX_COOLDOWN_SECS, and _CRON_/_POSTFIX_/_LFD_COOLDOWN_SECS (in system.sh). The database is the exception to the 30 s figure — _SQL_COOLDOWN_SECS defaults to 120 s, because a database restart takes far longer to settle than a web-tier one.

3. A flap circuit breaker on repeated heals. The cooldown spaces two heals; it cannot see a run of them. mysql.sh therefore also keeps a restart ledger at /var/log/boa/mysql.restart.ledger — one timestamp per heal, pruned to _SQL_FLAP_WINDOW_SECS (3600 s) and never counted further back than the last boot. Once _SQL_FLAP_MAX (3) heals have run inside that window the circuit opens: the watchdog writes /run/boa_mysql_restart_latched.pid, logs CIRCUIT OPEN to /var/log/boa/mysql.incident.log, sends exactly one incident e-mail, and stops restarting the database altogether. A database that has just been restarted three times and is down again is not one a fourth restart will fix, and continuing to try makes the host worse each time.

   Recovering a latched host is deliberately a human decision: fix the cause, then delete the latch file, and auto-healing resumes immediately with the count back at zero. If nobody does, it re-arms itself anyway after _SQL_FLAP_LATCH_MINS (60). The file says as much in plain words, so whoever finds it needs no other reference. Note that a server that recovers on its own — a mysqld_safe respawn, say — is never counted against the budget, because the watchdog stands down before it acts.

   The cache server is guarded by the same shape: valkey.sh keeps its ledger at /var/log/boa/valkey.restart.ledger and latches /run/boa_valkey_restart_latched.pid once _VALKEY_FLAP_MAX (3) restarts have run inside _VALKEY_FLAP_WINDOW_SECS (3600 s), pages once, and re-arms after _VALKEY_FLAP_LATCH_MINS (60) — the same operator contract, written into the latch file. Its window is floored generously against the pace heals can actually land at: a pass that has to confirm a wedged server and then walk the whole stop/start sequence overruns the dispatch tick, so consecutive heals quantise to a multiple of it, and a window too narrow to hold the allowed heals would be a breaker that never opens. The machinery is kept inline in valkey.sh, exactly as mysql.sh keeps its own copy and for the same reason: each watchdog is fetched under its own serial, and a box that has taken a new watchdog but not yet a new shared library must not lose its breaker to that skew. (system.sh's memory-pressure path, delivered in lockstep with the library that hosts the shared primitives, consumes those instead — see above.)

   The batch guard consumes the shared primitives too, but defensively: every call is guarded, so a box holding a new batch_guard.sh against an old shared library degrades to cooldown-only pacing — never to a half-armed breaker. Its ledger lives at /var/log/boa/batch_guard.heal.ledger, its latch at /run/boa_batch_guard_latched.pid, on _BATCH_GUARD_FLAP_MAX (3) / _BATCH_GUARD_FLAP_WINDOW_SECS (7200 s) / _BATCH_GUARD_FLAP_LATCH_MINS (60), the window floored against its own five-minute tick.

4. Load and run gating. Most watchdogs stand down while /run/max_load.pid or /run/critical_load.pid exists. Under a load emergency the box is already shedding work via Load control; piling service restarts on top would make recovery slower, not faster. They also stand down during a BOA run (/run/boa_run.pid) and while another auto-heal of the same service is mid-flight (/run/boa_<service>_auto_healing.pid).

5. Standing down for a database restart. move_sql.sh stops Nginx and every PHP-FPM pool in order to restart the database, and it does not start them again — these watchdogs are the intended recovery (inside an install or upgrade pass the flow itself brings the tier back with webserver up once the hostmaster vhost is ready). So nginx.sh, php.sh, valkey.sh, redis.sh, batch_guard.sh and task_guard.sh (whose probes and heals are all SQL) each stand their whole pass down while a database restart is in flight, which they read from the two markers runner.sh and system.sh already honour: /run/mysql_restart_running.pid (written by move_sql.sh and by mycnfup) and /run/boa_mysql_auto_healing.pid (held by the database watchdog for the length of a heal). Without that, each pass saw its service down and started it again mid-teardown, standing the web tier in front of a database that was still coming up: workers pile onto failed connections, and the whole herd then arrives at a cold cache the moment the database returns.

   Recovery is deferred, not cancelled — the next pass, seconds after the restart finishes, brings the service back. And the markers are honoured only while recent (_SQL_MUTATION_MAX_MINS, default 15 minutes): a healer killed mid-flight leaves one behind, clear.sh only reaps that one after an hour, and an hour with no web auto-healing would be a worse outcome than the cascade being avoided. Each stood-down pass logs one line to the service's own .incident.log, so the deferral is visible rather than silent.

6. Keeping table repair out of the way of database work. sqlcheck.sh watches the server log for crashed-table markers and hands off to checksql.sh, which runs a full-server mysqlcheck and then a generated repair script. Both take locks across every database, so both now decline while any of the SQL work markers is present: a backup (boa_sql_backup.pid, boa_sql_cluster_backup.pid), a maintenance operation (boa_sql_maintenance.pid), an auto-heal (boa_mysql_auto_healing.pid) or a restart (mysql_restart_running.pid). A crashed table is not urgent, and sqlcheck.sh runs every minute, so the repair simply happens once the window closes.

Email reporting

When a watchdog takes a corrective action it appends to its own /var/log/boa/<service>.incident.log and may e-mail ${_MY_EMAIL}, gated by _INCIDENT_REPORT (sourced from /root/.barracuda.cnf). The normalisation is uniform — NO maps to OFF, anything unrecognised maps to the script's default — but the send threshold differs by watchdog:

The important caveat: in most per-service watchdogs (unbound.sh, php.sh, nginx.sh, java.sh) the e-mail is sent only when _INCIDENT_REPORT is ALL — their MINI/CRIT settings still log the incident but mail nothing. mysql.sh sends on anything other than OFF (its own default is MINI). system.sh implements the OFF / CRIT (= ALERT-only) / ALL ladder. valkey.sh implements the full four-step ladder: a genuine restart of the cache server mails at the default MINI and up — a cache restart empties the cache, and while that mail was ALL-only an operator learnt about a flapping cache server from the load graphs rather than from a mail — the circuit-breaker page is an ALERT that CRIT still hears, and informational notices stay ALL-only. (redis.sh, retired from delivery, keeps its old ALL-only gate on the boxes that still run their local copy.) In every case the corrective action and the .incident.log entry happen regardless of the e-mail setting.

Reporting is additionally suppressed during the post-boot uptime grace period (_check_uptime_grace_period, from the shared lock.inc), so a reboot does not spray "service was down, restarted" mail while the stack is still coming up. See Load control for the full grace-gate conditions.

One alert per incident, not one per pass

A service that keeps failing used to mail every time a watchdog acted on it: one false database outage once produced nine near-identical alerts in fourteen minutes, which tells an operator nothing except that something is wrong somewhere. Each watchdog therefore holds an alert back when it has already sent one for that class of incident inside a cooldown (_INCIDENT_EMAIL_COOLDOWN_SECS, default 1800 s), and the alert that does go out says in its subject how many it stood in for: (+8 more since the last alert).

Nothing is lost by holding one back. The incident is still appended to the service's own .incident.log either way, and the mail body is the tail of that log, so the next alert carries the ones that did not go out. A held-back alert also logs its own line, so the suppression is visible rather than silent.

Classes that are not the same incident keep separate cooldowns, because otherwise a cheap alert spends the budget an expensive one needs. Each cooldown is a stamp under /run/incident.<key>.stamp:

The throttle lives in the shared lock.inc, and every watchdog calls it defensively: a box still holding an older copy of that library mails exactly as it did before.

Both the mail body and the on-disk history are bounded. Every _incident_email_report sends only the last 200 lines of the service's incident log (tail -n 200), applied identically in second.sh and each of the eight mail-sending monitor/check/ watchdogs (system, nginx, php, mysql, java, unbound, valkey, redis) — an alert shows the latest entries, never the whole file history. The logs themselves are rotated by a shipped logrotate policy (aegir/conf/var/logrotate.d.boa.conf, deployed to /etc/logrotate.d/boa on both the upgrade and INIT paths): /var/log/boa/*.incident.log is rotated weekly, keeping 4 rotations, with compress + delaycompress, missingok, notifempty. No postrotate signal or copytruncate is needed — the short-lived monitors reopen the log on each append, so logrotate's default create-after-rotate is safe. Full history stays on disk in the rotated files; only the e-mail body is capped.

Verify

# recent auto-heal activity, per service
tail -n 20 /var/log/boa/*.incident.log 2>/dev/null
ls -1 /var/log/boa/*.kill.log 2>/dev/null

# is a watchdog mid-flight, or is the box load-paused?
ls -1 /run/boa_*_auto_healing.pid /run/*_load.pid 2>/dev/null

# cooldown stamps currently in force
ls -1 /run/*-monitor.cooldown /run/phpNN-fpm.cooldown 2>/dev/null

# incident-log rotation policy deployed on this box?
cat /etc/logrotate.d/boa 2>/dev/null

# did last night's graceful.sh pass complete, and is any of it switched off?
ls -l /var/log/boa/graceful.done.pid 2>/dev/null
ls -l /root/.skip_cleanup.cnf /root/.giant_traffic.cnf /root/.high_traffic.cnf 2>/dev/null

# box-wide self-service restart markers currently in force
ls -l /etc/boa/.allow.nginx.restart.cnf /etc/boa/.allow.valkey.restart.cnf \
      /etc/boa/.allow.redis.restart.cnf 2>/dev/null

Related

• Load control & auto-pause — the load-reactive brakes (max_load.pid / critical_load.pid) that gate these watchdogs, and the uptime grace gate shared with this page.
• Cron cadence & idle-load throttle — box-class fan-out cadence and the _MONITOR_FANOUT_* / _MONITOR_HEAVY_EVERY knobs.
• Process guards & auth scanners — the second.sh side: the _proc_control service guards and the auth scanners.
• Abuse Guard — the security scanner (scan_nginx.sh) driven by the same monitor home via nginx_guard.sh on this minute.sh fan-out.
• PHP-FPM & performance — how the fpm_tune.sh JSONL feeds fpmreport and capacity sizing.
• Host control files & INI — the full marker catalogue, and the one-time /root to /etc/boa relocation that moved the three service-control markers named above.
• Reference appendix — every _VAR override named above with its default and source.