Nightly worker architecture (owl.sh + night/)

The nightly maintenance monolith that used to be /var/xdrago/daily.sh was renamed to owl.sh and split into an orchestrator plus a worker family under /var/xdrago/night/. The orchestrator keeps the run-level control flow; the workers carry the per-account, per-site and once-per-run bodies.

Every body was moved verbatim during the split — behaviour is identical to the pre-split monolith on the serial path — so the split is a maintainability boundary, not a behaviour change.

Deployed layout:

/var/xdrago/owl.sh              orchestrator (was daily.sh)
/var/xdrago/night/night.inc.sh  shared helper library + run-freeze
/var/xdrago/night/10-account.sh per-Octopus-account worker (subprocess)
/var/xdrago/night/20-sites.sh   per-site family + the _daily_process loop
/var/xdrago/night/90-global-post.sh once-per-run global post steps

Source in the fork: aegir/tools/system/owl.sh and aegir/tools/system/night/*. 10-account.sh and 20-sites.sh also define functions only when sourced, so they are unit-testable in isolation; only the orchestrator and 90-global-post.sh carry top-level flow.

owl.sh — what stayed in the orchestrator

owl.sh runs from the root crontab at 04:15 and owns everything that must happen once per run, in order, before and after the account loop:

• Prologue and gates. _check_root (ionice/renice self, disk-full abort at

  90%), the proxy/pause/standby skips (/root/.proxy.cnf, /root/.pause_heavy_tasks_maint.cnf, /root/.standby.cnf → exit 0 — a replication standby runs no nightly maintenance sweep, because the hostmaster drush calls, the per-site LE renewals and the ghost cleanups all write into state that arrives from the source by replication and rsync), the PHP-idle quiesce gate (/run/boa_php_idle_quiesce.pid — exits only while the recorded owner PID is alive, deletes a dead-owner marker and proceeds; this replaced barracuda's old trick of synthesising the proxy marker during PHP-version surgery), and the bounded queue wait (_boa_pass_active — boa_run.pid, boa_wait.pid, octopus_install_run.pid or a live installer process — spun on for at most ~2 h, after which the nightly is skipped with a log line) all live here.

• Run keystone. _NOW=$(date +%y%m%d-%H%M%S) and _DOW=$(date +%u) are computed once. _NOW names every *-${_NOW}.info idempotency guard and every per-account log, so it must be frozen and inherited, never re-derived in a worker (see The run-freeze).
• Module lists and force flags. The Tuesday-gated (_DOW=2) _MODULES_ON_*/_MODULES_OFF_* lists, the day-independent _MODULES_FORCE set, _FORCE_SITES_VERIFY, _CTRL_TPL_FORCE_UPDATE, and the _O_CONTRIB* shared-tree roots are all resolved in the orchestrator.
• Version check + notice. BARRACUDA release fetch and the "upgrade available" mail (gated on _SKYNET_MODE).
• Hostmaster pre-work. The @hostmaster drush cache/cron/utf8mb4 block and the permissions-fix-post-up-${_xSrl}.info-gated /data/.../sites/all chmod sweep run once, under the /run/daily-fix.pid re-entrancy lock the orchestrator owns.
• The account loop (_daily_action), described below.
• Global post (90-global-post.sh bodies), incident detection, forward-secrecy refresh, _global_cleanup, and _archive_old_daily_logs.

The helper library and 90-global-post.sh are sourced with a hard abort if a command -v probe for a known function from each does not resolve — a half-installed night/ set aborts cleanly rather than running with missing bodies:

[ -r "/var/xdrago/night/night.inc.sh" ] && . /var/xdrago/night/night.inc.sh
if ! command -v _run_drush8_hmr_cmd > /dev/null 2>&1; then
  echo "FATAL ERROR: /var/xdrago/night/night.inc.sh not loaded; aborting"
  exit 1
fi

10-account.sh and 20-sites.sh are deliberately not sourced by the orchestrator: the account worker is invoked as a subprocess (it sources its own deps, including 20-sites.sh), and only 90-global-post.sh is sourced-and-called in place because its bodies touch shared/global resources that must never run inside the per-account fan-out.

night.inc.sh — the shared helper library

night.inc.sh is functions and constants only, with no top-level side effects; each caller owns its own PATH/HOME prologue and _check_root. It is sourced by the orchestrator and by every worker, so a single copy is shared instead of duplicated. What it defines:

• Tier-0 constants. _WEBG, the _crlGet/_wgetGet fetch flags (iCab UA), the apt flags, and the drush config/variable verb shorthands _cGet/_cSet/_vGet/ _vSet used throughout the per-account and per-site work.
• Pure helpers. _validate_safe_dir (realpath must resolve under /data/disk/*, /var/aegir/* or /home/* — the guard that stops a compromised aegir-context alias from redirecting a chown onto system paths), _sanitize_number, _detect_real_docroot (version-agnostic docroot detection: index.php at the root or under docroot/, html/, web/), _cnf_flag_yes (grep-not-source a control flag so load order never clobbers caller scope), _provision_running (the master cleanup interlock), and the consecutive-ghost counter pair _ghost_seen_enough/_ghost_seen_reset.
• The interlock matches genuine task executions, not any mention of the word. It was a bare pgrep -f provision, which reported "running" for any command line that merely mentioned a provision path — a checksum, an editor, an operator's ssh probe — and silently skipped that night's cleanups. It now matches a provision-<task> drush command, an anchored hosting-task dispatch (the trailing boundary keeps it off the per-minute hosting-tasks/hosting-dispatch pollers, which would otherwise pin the gate on), or a php/su/env/drush process carrying provision on its command line. Because the workers and the library that defines the helper are fetched separately, each worker also defines a conservative fallback when the library predates it — an undefined function returns 127, which reads as "nothing running" and would let a cleanup walk through a live task.
• Load helpers. _count_cpu, _get_load (loadavg-1 over CPU count, ×100), and _load_control (derives _O_LOAD_MAX from _CPU_TASK_RATIO, default 3.1, bumped to 4.1 while .force.sites.verify.cnf is present).
• chattr helpers. _enable_chattr/_disable_chattr, which read _HM_U from the caller's scope at call time.
• drush8 wrappers. _run_drush8_cmd, _run_drush8_hmr_cmd, _run_drush8_hmr_master_cmd, _run_drush8_nosilent_cmd — all read _HM_U/_Dom from caller scope and prefer the pinned /opt/php74/bin/php where present, and all four share one verbose gate on /root/.debug_daily.info (below). Plus _hmr_context_exists, the one wrapper that captures output: it counts a site name's hosting_context row on @hostmaster and echoes YES/NO/UNKNOWN (see Per-account ghost-site notice).
• Night log + LE parsing. _acct_night_log, _le_extract_failures, _le_reason (see Per-account LE reporting).
• The run-freeze. night_emit_run_env / night_load_run_env.

All four drush8 wrappers open with the same gate on /root/.debug_daily.info, and echoing is the marker's whole effect — the presence test adds one timestamped line naming the account user and the exact command about to run, then the wrapper proceeds unchanged:

if [ -e "/root/.debug_daily.info" ]; then
  _nOw=$(date +%y%m%d-%H%M%S)
  echo "${_nOw} ${_HM_U} running drush8 @${_Dom} $1"
fi

The three silent wrappers still discard the command's own output and _run_drush8_nosilent_cmd still does not; _run_drush8_hmr_master_cmd names aegir rather than the account user, matching the user it runs as. Two of the four also name a different alias: _run_drush8_hmr_cmd and _run_drush8_hmr_master_cmd log @hostmaster rather than the site alias, so the block above is verbatim for _run_drush8_cmd and _run_drush8_nosilent_cmd only. The line lands wherever the calling process already writes — /var/log/boa/daily/daily-<NOW>.log for wrappers called from owl.sh or 90-global-post.sh, the account's own acct-<user>-<NOW>.log for everything the 10-account.sh subprocess drives. The marker is operator-created only: nothing in the tree writes or removes it, no hosted normalisation pass touches it, and it is not consumed on read, so it stays until it is deleted. One divergence to know about — xoct and xcopy carry their own same-named _run_drush8_hmr_cmd / _run_drush8_nosilent_cmd copies with no debug gate at all, so this marker has no effect on what those tools log.

The run-freeze

Because each account is now a separate subprocess, the per-run state the orchestrator computed must cross the process boundary intact. night_emit_run_env writes it once, after the hostmaster pre-work and before the account loop, into /run/night/run.env (mode 0600) as a block of export lines:

night_emit_run_env    # owl.sh:189, defined night.inc.sh

The frozen set is the keystone _NOW, plus _DOW, _xSrl, _FORCE_SITES_VERIFY, _CTRL_TPL_FORCE_UPDATE, the _O_CONTRIB* roots, all _MODULES_* lists, _hostedSys, _APT_UPDATE, the _PERMISSIONS_FIX/_MODULES_FIX/_CLEAR_BOOST toggles, _ENABLE_GOACCESS, the resolved admin address (_ADMIN_EMAIL, from _MY_EMAIL), _INCIDENT_REPORT, and _LE_CLIENT_NOTIFY.

A worker loads the context with night_load_run_env, which sources /root/.barracuda.cnf first (for cnf flags and any variable not in the freeze), then the freeze second so the frozen per-run set is authoritative for anything it sets:

night_load_run_env() {
  [ -e "/root/.barracuda.cnf" ] && . /root/.barracuda.cnf
  [ -r "/run/night/run.env" ] && . /run/night/run.env
}

The ordering matters: _NOW in particular must come from the freeze, never be re-derived, or a worker's idempotency stamps (*-${_NOW}.info) would key off a different timestamp than the orchestrator expects. .barracuda.cnf-only variables (e.g. the cleanup opt-in flags read directly by _cnf_flag_yes) are not in the freeze and are re-read live in the worker.

The worker contract

An owl.sh night worker is a script under /var/xdrago/night/ with a fixed shape. 10-account.sh and 90-global-post.sh are the two concrete templates:

1. Prologue. Same export HOME/SHELL/PATH/_tRee/_xSrl block as the orchestrator.
2. Source deps. night.inc.sh unconditionally (guarded [ -r ... ] && .); a per-account worker additionally sources 20-sites.sh.
3. Load run context (per-account worker only): night_load_run_env before doing any work, so _NOW and the module lists match the orchestrator's.
4. Define bodies, then either run them (subprocess/sourced-in-place worker) or leave them defined (a pure library like 20-sites.sh).

10-account.sh — the per-account worker

Invoked by the orchestrator as bash /var/xdrago/night/10-account.sh <account-path>, once per Octopus account — this subprocess is the unit of per-account parallelism.

_account_process is the whole per-account sequence:

• derive _HM_U from the account path;
• drush prep;
• sync _MY_OCTO_EMAIL/_CLIENT_EMAIL out of the account .octopus.cnf;
• the @hostmaster _vSet block (cron intervals, queue frequencies, hosting_delete_force);
• relocate backups (see Backups relocation);
• the per-site loop _daily_process (in 20-sites.sh);
• platform GC (_check_old_empty_platforms, the per-account undo/ reaper);
• cruft purge (_purge_cruft_machine);
• hostmaster LE cert copy (_le_hm_ssl_check_update);
• goaccess;
• the LE account report (_le_account_report);
• the ghost-site client notice (_ghost_account_report);
• and the final _enable_chattr relock.

The account worker also carries the account-scoped ghost-platform reaper, which moves a ghost alias into <account>/undo/ behind _GHOST_PLATFORMS_CLEANUP (per-account .octopus.cnf override wins over the .barracuda.cnf default).

20-sites.sh — the per-site family

Sourced by 10-account.sh, never run directly. It defines the per-site (per-vhost) procedures and the loop driver _daily_process, which:

• iterates the account's config/server_master/nginx/vhost.d entries;
• resolves _Dom/_Dir/_Plr from the drush alias;
• runs _validate_safe_dir on both derived paths (skip the site on a path outside the allowed roots);
• and calls the per-site fixups (control-file conversion, symlink and permission fixes, module force on/off, per-site LE, the _cleanup_ghost_vhosts/_cleanup_ghost_drushrc reapers — the site reaper classifies each post-grace candidate first and arms itself on the first enabled run per account; see ghost cleanup).

It reads the frozen _NOW/_DOW/_O_CONTRIB*/_MODULES_* context and the _usEr/_HM_U loop vars set by _account_process.

90-global-post.sh — once-per-run global post

Sourced by owl.sh and called in place after the account loop joins. Its bodies touch shared/global resources — the master /var/aegir tree, /data/all, /etc/ssl/private, a single service nginx reload — so they must never run inside the per-account fan-out; they run exactly once, after every account worker has finished.

It carries _shared_codebases_cleanup, _ghost_codebases_cleanup, _check_old_empty_hostmaster_platforms, _incident_detection/_incident_email_report, _fix_nginx_forward_secrecy, _global_cleanup, _purge_shared_aegir_backups, _archive_old_daily_logs, _goaccess_vhosts (only when _ENABLE_GOACCESS=YES and _GOACCESS_VHOSTS=YES, behind its own command -v skew guard: one GoAccess report per literal server_name found in /etc/nginx/sites-enabled, plus a box-wide ALL aggregate, built under /var/www/adminer/access/vhosts/ and published atomically to /var/log/boa/xdr9000/goaccess/), _cleanup_weblogx (called only when _ENABLE_GOACCESS=YES) and _migrate_source_sweep_all — the nightly migration-source grant sweep, called from _daily_action behind a command -v guard because the two files carry independent fNN serials, so a mixed vintage skips the call instead of dying on an undefined function.

The /run/daily-fix.pid lock stays owned by the orchestrator and is not released here.

Per-account subprocess execution

_daily_action in owl.sh drives the account loop. Each account directory under /data/disk/ is a candidate; an account is processed only when it has a config/server_master/nginx/vhost.d tree and neither log/proxied.pid nor log/CANCELLED is present. For each eligible account the orchestrator re-samples CPU and load, then hands the account to the worker subprocess:

bash /var/xdrago/night/10-account.sh "${_usEr}" \
  >> "$(_acct_night_log "${_usEr}")" 2>&1

Every account's stdout/stderr is redirected to its own log, /var/log/boa/daily/acct-<user>-<NOW>.log, derived identically by orchestrator and worker from the frozen _NOW, so both sides agree on the filename and the worker can read its own log back to build the LE report.

The orchestrator's own top-level output goes to /var/log/boa/daily/daily-<NOW>.log; 90-global-post.sh's incident scan reads that global log, while the per-account LE parsing reads the per-account log.

The run model: serial and parallel

The account loop has two modes, admission-gated identically on the per-core load ceiling (_O_LOAD < _O_LOAD_MAX, from _CPU_TASK_RATIO):

• Serial (default, _NIGHT_PARALLEL=NO). One account subprocess at a time, called synchronously; behaviour-equivalent to the former inline call. If load is over the ceiling the loop logs and waits (re-sampling next iteration).
• Parallel (_NIGHT_PARALLEL=YES). Accounts fan out concurrently up to _NIGHT_MAX_PARALLEL slots (default = CPU core count, sanitised via _sanitize_number; floor 1). Before each fan-out the loop spins on a combined gate — a free job slot and load headroom:

while :; do
  _count_cpu
  _load_control
  if [ "$(jobs -rp | wc -l)" -lt "${_NIGHT_MAX_PARALLEL}" ] \
    && (( $(echo "${_O_LOAD} < ${_O_LOAD_MAX}" | bc -l) )); then
    break
  fi
  sleep 5
done

The parallel path waits for headroom rather than skipping an overloaded account, so raising parallelism does not raise the silent-skip rate. Each worker writes to its own account log, so concurrent output never interleaves.

After the loop, the orchestrator waits for all backgrounded workers before running the global-post step — this join is what guarantees 90-global-post.sh's shared-resource bodies never race an in-flight account worker.

_NIGHT_PARALLEL and _NIGHT_MAX_PARALLEL are the only two variables introduced by the split; both live in /root/.barracuda.cnf (defaults seeded idempotently on install/upgrade). Leaving _NIGHT_MAX_PARALLEL empty defaults it to the core count.

Backups relocation and the queue-hold interlock

_relocate_backups_to_static_fs (in 10-account.sh) moves an account's backups/ and backup-exports/ onto the static/files filesystem behind symlinks, so large dereferenced backups cannot fill the root partition.

It is a deliberate no-op unless static/files resolves to a different device than the account root (the case for large accounts moved to attached storage), and it self-skips on either kill-switch (/data/conf/disable_backups_on_static_fs.cnf box-wide, or the per-account static/control/no_backups_on_static_fs.info).

The interesting path is the one-time migration of an existing real directory. It runs inside the parallel fan-out, so it must serialise against sibling account workers and must not let a new Ægir task start mid-move. Two interlocks stack:

1. A real flock on /run/.boa_backups_relocate.flock (flock -n, non-blocking; the kernel releases it on process death, so there is no stale-lock guessing). Only one account worker performs a migration at a time.
2. The task-queue hold, /run/boa_queue_stop.pid. The worker writes its own PID into the stop file only if it is absent, records that it created it (_madeStop), drains any in-flight provision task (bounded, ~60 s via _provision_running), does the rsync --remove-source-files move, then removes the stop file only if it created it and still owns the recorded PID:

[ -e "${_stop}" ] || { echo "$$" > "${_stop}" && _madeStop=YES; }
# ... drain, relocate ...
[ "${_madeStop}" = "YES" ] && [ "$(cat "${_stop}")" = "$$" ] && rm -f "${_stop}"

runner.sh (the frequent task-queue agent) honours /run/boa_queue_stop.pid two ways — a parent-level exit 0 when it is present, and a per-child skip inside its loop — so no new queue task starts while the move is in flight; the _provision_running drain handles any task already running.

The stop file is self-healing: clear.sh purges a leaked one after checking its PID is dead, and /run is cleared on reboot, so a crashed relocation can never permanently freeze the queue.

This same /run/boa_queue_stop.pid is the shared hold the filesystem migration tools (migratefs, updatesymlinks) use, so the night relocation is a cooperative participant in an existing interlock, not a new mechanism.

Per-account LE reporting

_le_account_report (in 10-account.sh) runs at the end of each account's sequence and builds the Let's Encrypt renewal report from that account's own log only — which is what guarantees a client notice can never be cross-attributed to another account.

_le_extract_failures (in night.inc.sh) parses the dehydrated failure blocks and emits one record per failure with fields joined by the 0x1f control byte (not a tab — the alt-names field is empty for single-domain certs and a whitespace read IFS would collapse the empty middle field, shifting every later field left). _le_reason maps an ACME error type/detail to a plain-language cause.

Two recipients get mail:

• The operator (_ADMIN_EMAIL) gets the full per-account report every night, gated by _INCIDENT_REPORT (normalised to OFF/ALL/MINI/CRIT in owl.sh; legacy NO→OFF, YES→MINI), plus a catch-all for non-validation backend/ACME errors so the move to per-account logs does not lose the old combined-log coverage.
• The affected client (_CLIENT_EMAIL) gets an actionable notice, throttled to once per 7 days per failing site via a marker under <account>/log/ctrl/, ON unless disabled with _LE_CLIENT_NOTIFY=NO (per-account .octopus.cnf override wins; only the literal NO disables). The client notice carries a Reply-To of the account owner (fallback: server admin) so replies do not bounce off the root@<host> envelope sender.

Per-account ghost-site notice

_ghost_account_report (also 10-account.sh, right after the LE report) reuses the same read-your-own-log pattern for confirmed ghost sites. The contract lives in the log-line formats _cleanup_ghost_drushrc (20-sites.sh) emits for a post-grace candidate:

• GHOST drushrc for <site> … — a true ghost whose front-end record still exists. Before choosing this line the sweep asks the account's own hostmaster via _hmr_context_exists (night.inc.sh, next to _run_drush8_hmr_cmd but capturing output): drush8 @hostmaster sqlq counts the site's hosting_context row, which is registered with the domain on node insert and deleted with the node.
• GHOST backend leftover for <site> (no front-end record|front-end check failed) … — the record is gone (node already deleted) or the front-end could not be asked; cleaned without any client mail.
• GHOST candidate <site> SKIPPED (<class>: …) — the classified operator-review cases (front-end / platform-gone / stranded), never moved, never notified.

The notice's awk matches only the first family's post-grace variants (detected and moved to / detected (dry-run), so the grace line and everything else stay operator-only. Recipient, isolation, and Reply-To follow the LE model; the throttle is 30 days per site (ghost-notify.<site>.info markers under <account>/log/ctrl/), gated by _GHOST_CLIENT_NOTIFY (default YES, only the literal NO disables, per-account .octopus.cnf wins).

Deploy and fetch surfaces for the night/ family

The self-update path in BOA.sh.txt provisions the whole family. It creates the directory (mkdir -p /var/xdrago/night), then fetches each file with the serial-gated fetcher:

_fetch_versioned /var/xdrago/night/night.inc.sh  .../tools/system/night/night.inc.sh  fNN
_fetch_versioned /var/xdrago/night/20-sites.sh   .../tools/system/night/20-sites.sh    fNN
_fetch_versioned /var/xdrago/night/10-account.sh .../tools/system/night/10-account.sh  fNN
_fetch_versioned /var/xdrago/night/90-global-post.sh .../tools/system/night/90-global-post.sh fNN
_fetch_versioned /var/xdrago/owl.sh              .../tools/system/owl.sh               fNN

Each file carries its own fNN fetch serial, so a maintainer bumps only the serial(s) of the file(s) actually changed and only those boxes re-fetch. Serials count down (bump = decrement) and each is single-homed in BOA.sh.txt; do not cite a specific serial in a living doc — state the gating rule (each night file is serial-gated independently) instead.

The self-update path also drops stale *.ctrl.*.pid stamps for any night file whose deployed copy is missing, so a partial install re-fetches cleanly.

Two migration guards make the daily.sh → owl.sh rename safe on a live box:

• Disable state carries over. A box that had the nightly run disabled has the script renamed aside to /var/xdrago/.daily.sh.off (the same rename-aside disable the migration tools use, not a marker file). When .daily.sh.off is present, the self-update path renames the freshly fetched owl.sh aside to /var/xdrago/.owl.sh.off and removes .daily.sh.off, so the box stays disabled under the new name. The xoct/xcopy/xmass enable/disable of the nightly run now toggles between /var/xdrago/owl.sh and /var/xdrago/.owl.sh.off.
• Legacy daily.sh retired only when the crontab has moved. daily.sh and its ctrl stamps are removed only once the live root crontab no longer invokes daily.sh (i.e. a full system upgrade has re-rendered the crontab to call owl.sh). Until then the deployed crontab still calls daily.sh, so deleting it on the self-update path would silently stop the nightly run mid-transition. Gating on the live crontab keeps this self-correcting and idempotent.

The root crontab entry is 15 4 * * * ... bash /var/xdrago/owl.sh. Note that weblogx self-skips while owl.sh is running (a pgrep -fc /var/xdrago/owl.sh guard), preserving the old mutual-exclusion the monolith had.

Adding a night worker

Where a new body goes depends on its scope:

• A new once-per-run global step: put the body in 90-global-post.sh and call it from owl.sh after the account loop — nothing that touches the shared /var/aegir or /data/all trees, /etc/ssl/private, or issues a service reload may run inside the per-account fan-out.
• Per-account work: add the function to 10-account.sh and call it from _account_process.
• Per-site work: add it to 20-sites.sh and call it from _daily_process.

Rules the split enforces:

• No top-level side effects in a sourced library. night.inc.sh and 20-sites.sh are sourced by multiple callers; a top-level statement would run in every one. Define functions only; let the caller invoke them.
• Read the run context from the freeze, not the environment. Any per-run value a worker needs must be in night_emit_run_env's block, or it will be empty in the subprocess. If a new step needs a new per-run value, add its export line to the freeze (orchestrator side) and it appears automatically via night_load_run_env. Values that live only in .barracuda.cnf and are stable across the run can be read live with _cnf_flag_yes instead of freezing them.
• Gate anything shared/global behind the join. A body that mutates a shared path belongs in 90-global-post.sh, which runs after wait. A body that races sibling account workers on a shared resource (like the backups relocation) must carry its own flock and, if it must quiesce the task queue, the /run/boa_queue_stop.pid hold with the create-and-own discipline shown above.
• Bump the file's own fetch serial in BOA.sh.txt in the same change, so boxes actually re-fetch the modified worker.
• Keep bodies verbatim when moving them. The split's invariant is behavioural identity with the monolith; a move that also changes logic breaks the ability to bisect a regression to "the split" vs "a later edit".

Related

• Monitor deploy surfaces — how the /var/xdrago monitor and worker binaries are fetched and serial-gated during self-update.
• Abuse Guard internals — the real-time monitor family that shares the same /run pidfile and _fetch_versioned deploy conventions.
• Monitor and abuse internals — the chapter overview for this topic.
• Discontinued features — the retired Drupalgeddon daily hacked-site detector and its now-defunct control files, removed from this run.