Nightly maintenance (owl.sh)

owl.sh is BOA's nightly maintenance run — the successor to the daily.sh monolith. The rename came with a decomposition: /var/xdrago/owl.sh is now a thin orchestrator that freezes the run context, walks the Octopus accounts, and dispatches the real work to worker scripts under /var/xdrago/night/. Per-account work runs as a subprocess per account with its own log file, and can optionally fan out in parallel. The default remains serial and behaviour-equivalent to the old inline run.

This page covers the orchestrator: layout, gates, the run-context freeze, the per-account subprocess model, parallelism, and the daily.sh transition. The ghost/empty cleanup family that runs inside the same nightly window is the companion leaf Ghost & empty cleanup; Let's Encrypt renewal and the per-account failure reports the night workers assemble are covered in SSL operations.

Layout: orchestrator + night/ workers

owl.sh aborts with FATAL ERROR if night.inc.sh or 90-global-post.sh cannot be sourced — it never runs half-defined. All five files are serial-gated _fetch_versioned entries in BOA.sh.txt, so every box re-fetches the current copy automatically whenever one changes; you do not pull updates by hand.

Two legacy names are kept by design: the run lock is still /run/daily-fix.pid, and the log directory is still /var/log/boa/daily/.

Launch, gates and lock

The root crontab launches the run at 04:15, throttled below interactive work (aegir/tools/system/cron/crontabs/root):

15 4 * * *  /usr/bin/nice -n5 /usr/bin/ionice -c2 -n7 bash /var/xdrago/owl.sh  >/dev/null 2>&1

Before doing anything, owl.sh:

• exits 1 if the root filesystem is over 90% full (ERROR: Your disk space is almost full) — nightly maintenance does not run on a nearly-full disk;
• exits 0 immediately if /root/.proxy.cnf, /root/.pause_heavy_tasks_maint.cnf or /root/.standby.cnf exists — proxy-only nodes, paused boxes and replication standbys run no nightly maintenance at all; on a standby the hostmaster drush calls, the per-site LE renewals and the ghost cleanups would all write into state that arrives from the source by replication and rsync, so the whole run stands down (see Failover and recovery scenarios);
• takes the run lock /run/daily-fix.pid. If the lock already exists, it touches /var/log/boa/wait-for-daily and exits 1; the lock is removed at the very end of the run, after the global post-steps.

To disable the nightly run on a box, the toggle is /var/xdrago/.owl.sh.off — xoct and xcopy park the script there (mv owl.sh .owl.sh.off + pkill) during operations that must not collide with nightly maintenance, and restore it afterwards. xmass does not manage this toggle. A legacy .daily.sh.off left by the pre-rename tooling is carried over to .owl.sh.off during self-update (see the transition section below).

Run order

Inside the lock, one run proceeds as:

1. Global pre-steps — a once-per-upgrade permissions sweep over distro/*/*/sites/all/{libraries,modules,themes} (stamped permissions-fix-post-up-<serial>.info so it reruns only after a BOA upgrade), then hostmaster housekeeping as the aegir user: drush8 cc drush, disabling update/syslog/dblog on @hostmaster, hostmaster cron, two cache-clears, and utf8mb4-convert-databases.
2. _daily_action — logged to /var/log/boa/daily/daily-<NOW>.log: weblogx preparation (when goaccess is enabled), the run-freeze, the per-account loop (below), and the global cleanup family from 90-global-post.sh.
3. Post steps — incident detection, the Nginx forward-secrecy/DH-param refresh (single service nginx reload), global cleanup, log archiving, lock release, and a final INFO: Daily maintenance complete.

GoAccess reporting rides inside step 2, and it is off unless _ENABLE_GOACCESS=YES in /root/.barracuda.cnf: that variable alone gates the weblogx preparation and teardown around the account loop, and per site a report is built only for a vhost that also carries its own <account>/static/control/goaccess/<domain>.info marker. /etc/boa/.goaccess.all.cnf adds one further report per account — with both the variable and the marker present, the account worker also runs the pseudo-site ALL report at the end of each account's sequence. Two co-guards apply to every report either way: the account must be on a POWER, PHANTOM, ULTRA, MONSTER or CLUSTER plan (read from its /root/.<account>.octopus.cnf), and weblogx must be installed and executable. Be clear about what ALL means before enabling it — weblogx widens its filter to every request line in the merged /var/log/nginx/access* set, so the report copied into <account>/static/goaccess/ALL/ is a box-level aggregate, not that account's own traffic. Nothing removes the marker: it is a persistent toggle, not a one-shot trigger. A legacy /root/.goaccess.all.cnf from before the markers moved is copied to /etc/boa/ once during self-update, and the original is left in place.

Standalone vhosts — sites defined directly in /etc/nginx/sites-enabled rather than through Aegir — are covered by a separate arm: with _GOACCESS_VHOSTS=YES (and _ENABLE_GOACCESS=YES), the orchestrator enumerates every literal server_name in those vhost files after the account loop and builds one GoAccess report per name, plus its own box-wide ALL aggregate. Reports are built in the standard weblogx working area (/var/www/adminer/access/vhosts/<name>/) and the finished HTML is then published atomically to /var/log/boa/xdr9000/goaccess/<name>/index.html — the observability archive tree, so fleet pull tooling picks the reports up without extra wiring. No account plan co-guard applies here, and the tenant-facing static/goaccess/ copies are not involved.

Excluding your own visits — _GOACCESS_EXCLUDE_IPS in /root/.barracuda.cnf takes a space-separated list of visitor IPs or dash-ranges like 1.2.3.4-1.2.3.9 (IPv4/IPv6, GoAccess --exclude-ip syntax; CIDR is refused because GoAccess silently ignores slash notation); weblogx appends each as a CLI exclusion on every report it builds, merged with the built-in 127.0.0.1 exclusion, so operator workstation and monitoring traffic stops skewing the numbers. The raw nginx logs are untouched — only the reports filter.

The run-context freeze: /run/night/run.env

Because each account now runs in its own subprocess, the run context must be inherited, not re-derived. After the global pre-steps and before the account loop, the orchestrator calls night_emit_run_env (night.inc.sh), which writes the exact per-run state as export lines to /run/night/run.env, chmod 0600:

• _NOW — the run keystone, date +%y%m%d-%H%M%S. It names every *-${_NOW}.info idempotency guard and every per-account log file, so a worker must see the orchestrator's value, never compute its own;
• _DOW, _xSrl (the BOA serial), _FORCE_SITES_VERIFY, _CTRL_TPL_FORCE_UPDATE;
• the module lists: _O_CONTRIB* per core version and the _MODULES_ON_* / _MODULES_OFF_* / _MODULES_FORCE sets;
• the cnf flags a worker acts on: _APT_UPDATE, _PERMISSIONS_FIX, _MODULES_FIX, _CLEAR_BOOST, _ENABLE_GOACCESS, _hostedSys;
• the reporting identity: _ADMIN_EMAIL (from _MY_EMAIL), _INCIDENT_REPORT, _LE_CLIENT_NOTIFY, _GHOST_CLIENT_NOTIFY.

A worker loads context with night_load_run_env: it sources /root/.barracuda.cnf first (for cnf flags and anything not in the freeze), then /run/night/run.env — the freeze is authoritative for the frozen per-run set.

_FORCE_SITES_VERIFY is the one frozen value an operator sets by hand, and the marker behind it does two unrelated things. owl.sh tests /root/.force.sites.verify.cnf at the top of the run, before the freeze, and sets _FORCE_SITES_VERIFY to YES (otherwise NO). The per-site worker acts on it inside _fix_permissions, so it only reaches sites whose permissions fix actually runs — _PERMISSIONS_FIX=YES, and no per-platform or box-wide permissions opt-out — and for each of those it runs hosting-task @<site> verify --force through the account's hostmaster. Without the marker that forced verify happens only for a site whose drushrc.php was missing the db_host line and had to be patched.

The second effect has nothing to do with verification: the nightly _load_control defaults _CPU_TASK_RATIO to 3.1 and then hard-overrides it to 4.1 whenever the marker exists, raising the per-core admission ceiling for the account loop from 310% to 410%. That override wins over any _CPU_TASK_RATIO you set in /root/.barracuda.cnf, so if what you want is a different nightly load ceiling, set _CPU_TASK_RATIO and leave this marker alone. Only the night copy of _load_control reads the marker; the other load-gated tools use _CPU_TASK_RATIO as configured.

On a hosted box the marker is transient. The five-minute clear.sh tick runs autoupboa whenever no BOA run is in progress, and its hosted normalisation block deletes /root/.force.sites.verify.cnf on every pass, so unless it is re-created shortly before 04:15 the nightly run never sees it. Nothing in BOA ever creates the marker.

Per-account subprocess model

The orchestrator walks /data/disk/ and processes each eligible account — one that has config/server_master/nginx/vhost.d and neither log/proxied.pid nor log/CANCELLED — by running:

bash /var/xdrago/night/10-account.sh <account>  >> /var/log/boa/daily/acct-<user>-<NOW>.log 2>&1

The log path comes from _acct_night_log (night.inc.sh) — derived identically on both sides from the frozen _NOW plus the account basename, so the orchestrator that redirects and the worker that later reads its own log back (to build the Let's Encrypt incident report and the ghost-site client notice) always agree on the file name. The orchestrator's own log for the run is /var/log/boa/daily/daily-<NOW>.log.

Before each account, the orchestrator re-samples CPU count and load (_count_cpu + _load_control); an account is only launched below the load ceiling (_O_LOAD_MAX, derived from _CPU_TASK_RATIO — default 3.1, forced to 4.1 while /root/.force.sites.verify.cnf exists; see Load control).

/var/log/boa/daily/ stays bounded without deleting anything: at the end of each run, _archive_old_daily_logs (90-global-post.sh) moves top-level *.log files whose mtime month differs from the current month into MM-YYYY/ subdirectories. The current run's daily-*.log and acct-*.log stay in place, already-archived files are never re-scanned, and nothing is ever deleted — the logs remain available for incident analysis.

Optional parallel fan-out

Two /root/.barracuda.cnf variables control per-account parallelism (lib/settings/barracuda.sh.cnf):

The admission gate differs between the modes in one operator-relevant way:

• Serial (default): an account whose turn arrives while load is over the ceiling is logged (...we have to wait...) and passed over for this run — the historical daily.sh behaviour.
• Parallel: each launch waits in a 5-second retry loop until there is both a free slot (fewer than _NIGHT_MAX_PARALLEL running workers) and load headroom, then fans out. Parallelism therefore does not raise the silent-skip rate — an overloaded moment delays an account instead of dropping it.

After the loop, a parallel run waits for all workers to join before the global post-steps run — 90-global-post.sh touches shared resources (the master /var/aegir tree, /data/all, /etc/ssl/private, one Nginx reload) and must run exactly once, after all per-account work has finished.

Task-queue interplay

The Ægir task-queue runner defers to the nightly run: runner.sh refuses to drain the queue while owl.sh is running (pgrep -fc owl.sh) or while a SQL dump is in flight (mysql_backup.sh, mysql_cluster_backup.sh, or any mydumper process). It touches /var/log/boa/wait-runner.pid, logs Another BOA task is running, we will try again later..., and exits — the queue resumes on a later tick. See Tasks & queue for the runner's full back-off list.

Transition from daily.sh

The rename is designed so that no box loses its nightly run mid-transition (BOA.sh.txt):

• The self-update path installs owl.sh and the four night/ workers via _fetch_versioned.
• A leftover /var/xdrago/.daily.sh.off (a box an operator or xoct/xcopy had disabled under the old name) is converted: owl.sh is parked as .owl.sh.off and the legacy marker removed — a disabled box stays disabled.
• Legacy /var/xdrago/daily.sh and its ctrl stamps are deleted only once the live /var/spool/cron/crontabs/root no longer invokes daily.sh — i.e. after a full system upgrade has re-rendered the crontab to call owl.sh. Until then the deployed crontab still calls daily.sh, so removing it early would silently stop nightly maintenance; gating on the live crontab keeps the retirement self-correcting and idempotent. Whether a given box has already crossed that gate is per-box state — check its live root crontab.

Related

• Ghost & empty cleanup — the opt-in, dry-run-by-default cleanup family (_GHOST_*_CLEANUP, _SHARED_CODEBASES_CLEANUP) that runs inside this nightly window.
• SSL operations — Let's Encrypt renewal and the per-account failure reports assembled from the acct-*.log night logs.
• Tasks & queue — runner.sh and its back-off while owl.sh or a SQL dump runs.
• Load control — _load_control and the _CPU_TASK_RATIO ceiling that gates each account launch.
• Host control files & INI — /root/.proxy.cnf and /root/.pause_heavy_tasks_maint.cnf, the box-level skip gates.
• Reference appendix — _NIGHT_PARALLEL / _NIGHT_MAX_PARALLEL and the other /root/.barracuda.cnf variables.