Nightly maintenance (owl.sh)

owl.sh is BOA's nightly maintenance run — the successor to the daily.sh monolith. The rename came with a decomposition: /var/xdrago/owl.sh is now a thin orchestrator that freezes the run context, walks the Octopus accounts, and dispatches the real work to worker scripts under /var/xdrago/night/. Per-account work runs as a subprocess per account with its own log file, and can optionally fan out in parallel. The default remains serial and behaviour-equivalent to the old inline run.

This page covers the orchestrator: layout, gates, the run-context freeze, the per-account subprocess model, parallelism, and the daily.sh transition. The ghost/empty cleanup family that runs inside the same nightly window is the companion leaf Ghost & empty cleanup; Let's Encrypt renewal and the per-account failure reports the night workers assemble are covered in SSL operations.

Layout: orchestrator + night/ workers

owl.sh aborts with FATAL ERROR if night.inc.sh or 90-global-post.sh cannot be sourced — it never runs half-defined. All five files are serial-gated _fetch_versioned entries in BOA.sh.txt, so every box re-fetches the current copy automatically whenever one changes; you do not pull updates by hand.

Two legacy names are kept by design: the run lock is still /run/daily-fix.pid, and the log directory is still /var/log/boa/daily/.

Launch, gates and lock

The root crontab launches the run at 04:15, throttled below interactive work (aegir/tools/system/cron/crontabs/root):

15 4 * * *  /usr/bin/nice -n5 /usr/bin/ionice -c2 -n7 bash /var/xdrago/owl.sh  >/dev/null 2>&1

Before doing anything, owl.sh:

• exits 1 if the root filesystem is over 90% full (ERROR: Your disk space is almost full) — nightly maintenance does not run on a nearly-full disk;
• exits 0 immediately if /root/.proxy.cnf or /root/.pause_heavy_tasks_maint.cnf exists — proxy-only nodes and paused boxes run no nightly maintenance at all;
• takes the run lock /run/daily-fix.pid. If the lock already exists, it touches /var/log/boa/wait-for-daily and exits 1; the lock is removed at the very end of the run, after the global post-steps.

To disable the nightly run on a box, the toggle is /var/xdrago/.owl.sh.off — xoct and xcopy park the script there (mv owl.sh .owl.sh.off + pkill) during operations that must not collide with nightly maintenance, and restore it afterwards. xmass does not manage this toggle. A legacy .daily.sh.off left by the pre-rename tooling is carried over to .owl.sh.off during self-update (see the transition section below).

Run order

Inside the lock, one run proceeds as:

1. Global pre-steps — a once-per-upgrade permissions sweep over distro/*/*/sites/all/{libraries,modules,themes} (stamped permissions-fix-post-up-<serial>.info so it reruns only after a BOA upgrade), then hostmaster housekeeping as the aegir user: drush8 cc drush, disabling update/syslog/dblog on @hostmaster, hostmaster cron, two cache-clears, and utf8mb4-convert-databases.
2. _daily_action — logged to /var/log/boa/daily/daily-<NOW>.log: weblogx preparation (when goaccess is enabled), the run-freeze, the per-account loop (below), and the global cleanup family from 90-global-post.sh.
3. Post steps — incident detection, the Nginx forward-secrecy/DH-param refresh (single service nginx reload), global cleanup, log archiving, lock release, and a final INFO: Daily maintenance complete.

The run-context freeze: /run/night/run.env

Because each account now runs in its own subprocess, the run context must be inherited, not re-derived. After the global pre-steps and before the account loop, the orchestrator calls night_emit_run_env (night.inc.sh), which writes the exact per-run state as export lines to /run/night/run.env, chmod 0600:

• _NOW — the run keystone, date +%y%m%d-%H%M%S. It names every *-${_NOW}.info idempotency guard and every per-account log file, so a worker must see the orchestrator's value, never compute its own;
• _DOW, _xSrl (the BOA serial), _FORCE_SITES_VERIFY, _CTRL_TPL_FORCE_UPDATE;
• the module lists: _O_CONTRIB* per core version and the _MODULES_ON_* / _MODULES_OFF_* / _MODULES_FORCE sets;
• the cnf flags a worker acts on: _APT_UPDATE, _PERMISSIONS_FIX, _MODULES_FIX, _CLEAR_BOOST, _ENABLE_GOACCESS, _hostedSys;
• the reporting identity: _ADMIN_EMAIL (from _MY_EMAIL), _INCIDENT_REPORT, _LE_CLIENT_NOTIFY, _GHOST_CLIENT_NOTIFY.

A worker loads context with night_load_run_env: it sources /root/.barracuda.cnf first (for cnf flags and anything not in the freeze), then /run/night/run.env — the freeze is authoritative for the frozen per-run set.

Per-account subprocess model

The orchestrator walks /data/disk/ and processes each eligible account — one that has config/server_master/nginx/vhost.d and neither log/proxied.pid nor log/CANCELLED — by running:

bash /var/xdrago/night/10-account.sh <account>  >> /var/log/boa/daily/acct-<user>-<NOW>.log 2>&1

The log path comes from _acct_night_log (night.inc.sh) — derived identically on both sides from the frozen _NOW plus the account basename, so the orchestrator that redirects and the worker that later reads its own log back (to build the Let's Encrypt incident report and the ghost-site client notice) always agree on the file name. The orchestrator's own log for the run is /var/log/boa/daily/daily-<NOW>.log.

Before each account, the orchestrator re-samples CPU count and load (_count_cpu + _load_control); an account is only launched below the load ceiling (_O_LOAD_MAX, derived from _CPU_TASK_RATIO — see Load control).

/var/log/boa/daily/ stays bounded without deleting anything: at the end of each run, _archive_old_daily_logs (90-global-post.sh) moves top-level *.log files whose mtime month differs from the current month into MM-YYYY/ subdirectories. The current run's daily-*.log and acct-*.log stay in place, already-archived files are never re-scanned, and nothing is ever deleted — the logs remain available for incident analysis.

Optional parallel fan-out

Two /root/.barracuda.cnf variables control per-account parallelism (lib/settings/barracuda.sh.cnf):

The admission gate differs between the modes in one operator-relevant way:

• Serial (default): an account whose turn arrives while load is over the ceiling is logged (...we have to wait...) and passed over for this run — the historical daily.sh behaviour.
• Parallel: each launch waits in a 5-second retry loop until there is both a free slot (fewer than _NIGHT_MAX_PARALLEL running workers) and load headroom, then fans out. Parallelism therefore does not raise the silent-skip rate — an overloaded moment delays an account instead of dropping it.

After the loop, a parallel run waits for all workers to join before the global post-steps run — 90-global-post.sh touches shared resources (the master /var/aegir tree, /data/all, /etc/ssl/private, one Nginx reload) and must run exactly once, after all per-account work has finished.

Task-queue interplay

The Ægir task-queue runner defers to the nightly run: runner.sh refuses to drain the queue while owl.sh is running (pgrep -fc owl.sh) or while a SQL dump is in flight (mysql_backup.sh, mysql_cluster_backup.sh, or any mydumper process). It touches /var/log/boa/wait-runner.pid, logs Another BOA task is running, we will try again later..., and exits — the queue resumes on a later tick. See Tasks & queue for the runner's full back-off list.

Transition from daily.sh

The rename is designed so that no box loses its nightly run mid-transition (BOA.sh.txt):

• The self-update path installs owl.sh and the four night/ workers via _fetch_versioned.
• A leftover /var/xdrago/.daily.sh.off (a box an operator or xoct/xcopy had disabled under the old name) is converted: owl.sh is parked as .owl.sh.off and the legacy marker removed — a disabled box stays disabled.
• Legacy /var/xdrago/daily.sh and its ctrl stamps are deleted only once the live /var/spool/cron/crontabs/root no longer invokes daily.sh — i.e. after a full system upgrade has re-rendered the crontab to call owl.sh. Until then the deployed crontab still calls daily.sh, so removing it early would silently stop nightly maintenance; gating on the live crontab keeps the retirement self-correcting and idempotent. Whether a given box has already crossed that gate is per-box state — check its live root crontab.

Related

• Ghost & empty cleanup — the opt-in, dry-run-by-default cleanup family (_GHOST_*_CLEANUP, _SHARED_CODEBASES_CLEANUP) that runs inside this nightly window.
• SSL operations — Let's Encrypt renewal and the per-account failure reports assembled from the acct-*.log night logs.
• Tasks & queue — runner.sh and its back-off while owl.sh or a SQL dump runs.
• Load control — _load_control and the _CPU_TASK_RATIO ceiling that gates each account launch.
• Host control files & INI — /root/.proxy.cnf and /root/.pause_heavy_tasks_maint.cnf, the box-level skip gates.
• Reference appendix — _NIGHT_PARALLEL / _NIGHT_MAX_PARALLEL and the other /root/.barracuda.cnf variables.