Ghost & empty-artefact cleanup

Long-lived BOA boxes accumulate ghosts:

• orphaned Composer trees with no working docroot;
• legacy shared codebases no platform links to any more;
• platform aliases whose platform is gone;
• nginx vhosts with no matching site, and .restore vhosts left by a failed Clone or Migrate;
• stale alias copies for sites deleted long ago.

The nightly owl run and the every-3-minutes manage_ltd_users.sh cron detect all of these — and, only when explicitly enabled, move them aside.

Three rules govern every mover on this page:

• Opt-in, dry-run by default. Every flag defaults to NO: out of the box each check only logs what it would move and changes nothing. You flip one _VAR=YES per check, per box or per account, after reviewing the dry-run output. (One exception: on omega8.cc-hosted servers — hostname ending in .aegir.cc — BOA seeds _GHOST_SITES_CLEANUP=YES and _GHOST_VHOSTS_CLEANUP=YES into .barracuda.cnf as ordinary lines, only when absent, so the operator toggle always wins.)
• Moves, never deletes. An enabled check relocates the item to a backup or undo/ path; recovery is moving it back.
• Safeguards sit above the flags. The Provision interlock and the fail-closed liveness tests below apply on every run, enabled or not; the nightly per-site reapers additionally demand ghost state across consecutive nights before acting, and the site reaper classifies each confirmed candidate and only ever acts on a true per-site ghost.

The reaper family

The six nightly checks belong to the owl run (04:15 by default): the per-account workers handle platforms, vhosts, and sites for their account; after all accounts finish, the two codebase checks and the Hostmaster platform sweep run once globally.

The ltd-alias reaper lives in manage_ltd_users.sh (see lshell + manage_ltd_users) because that script already owns the tenant alias copies it inspects.

The .restore vhost test cleans up after a failed Clone or Migrate — but only half of it. The same abort also strands a sites/<domain>.restore directory on the platform, which no reaper in this family touches: that half comes off through the platform's Import leftover → Purge leftover flow, or by hand. See Ægir task failures for where the artefact comes from and both removal paths.

Version-agnostic detection

Whether a tree is "real" is judged by _detect_real_docroot (night/night.inc.sh): an index.php at the platform root or under docroot/, html/, or web/. That is the front controller present in every Drupal docroot from D6 to D11, regardless of build recipe, so Composer-based D8+ platforms — whose web root differs from the application root and which no longer ship sites/all — are never misread as ghosts.

Ghost detection never keys on sites/all. The -f test follows symlinks and is false for a dangling one, so a decoy like web -> /etc cannot register as a docroot.

For the marker-gated platform reapers a vendor/ tree alone also counts as live: a codebase mid-composer-build has no docroot yet, but it is not a ghost.

Always-on safeguards

These apply before any move, regardless of flags:

• Provision interlock. While any Ægir/Provision task runs (install, clone, migrate, verify, backup, restore) the site and platform trees are transiently inconsistent — files mid-repoint, aliases mid-rewrite, vhosts staged under a leading dot. Every cleanup calls _provision_running first and skips entirely, logging a notice, when a task is found. It retries on the next pass once the box is quiet. The interlock matches genuine task executions — a provision-<task> drush command, an anchored hosting-task dispatch, or a php/su/env/drush process carrying provision — and no longer the bare substring it used, which reported a task for any command line merely mentioning a provision path and so skipped that night's cleanups for a checksum or an ssh probe.
• Consecutive-night markers. The nightly site, vhost, and platform reapers never act on a single snapshot: _ghost_seen_enough requires the item to look ghost on 2 consecutive nightly runs, tracked in counter files under <account>/log/ctrl/ (ghost-*.seen). The moment an item looks valid the counter is reset (_ghost_seen_reset), so a recovered item never carries a stale count into a later reap. A transient state — a momentarily-unmounted store, a symlink mid-repoint — gets at least one grace run to recover.
• Symlink-aware liveness. A site is judged live on drushrc.php alone, present as a file or a symlink. files/ and private/ are deliberately not required: under native files-symlinking those point into a static store whose target can be transiently absent, and that must never count as gone.
• Fail-closed parsing. A degraded or mid-rewrite alias that parses to an empty or non-/data/disk site_path keeps the site and resets its marker — a bad parse never reaps.
• Candidate classification (site reaper). A confirmed ghost-site candidate is classified before anything moves, and three classes are always left in place for operator review, logged as SKIPPED: front-end (a site_path under aegir/distro — the account's own Ægir control panel or its hm/oN.<host> alias companions, e.g. a stale alias left by a hostname rename); platform-gone (the alias's platform root is missing entirely — a platform moved or removed outside Ægir would otherwise take every one of its sites' registrations down at once); and stranded (the site's directory turns up on another platform of the same account — an interrupted migrate; the data may be intact). Only a true per-site ghost — platform directory present, site directory nowhere — is ever reaped.
• Arming run on every enable flip. The consecutive-night counters keep counting while the site reaper is in dry-run, so flipping _GHOST_SITES_CLEANUP to YES would otherwise act on every long-accumulated ghost at once. Instead the first enabled run per account only arms the reaper (logged, nothing moved); moves start on the next night. Disabling the flag re-arms it for the next enable.
• Freshness and staging skips. Vhosts younger than 24 h (mid-import/activation, or a restore still in flight) reset their marker, and that age is read once per vhost, at the top of the run, before the same run's own vhost fixes can rewrite the file — a vhost BOA itself just edited must not read as freshly staged. The two vhost tests keep separate counters (ghost-vhost-<domain>.seen for the .restore test, ghost-vhost-noalias-<domain>.seen for the no-alias test) so neither can reset or double-count the other. Leading-dot companion vhosts (.example.com — staged or rollback originals from xoct/xcopy) are always skipped; the vhost sweep skips the whole account while log/exported.pid marks a migrate/export in flight; the ltd reaper skips aliases written under 60 minutes ago and re-checks the Provision interlock inline on its 3-minute cadence.
• No instant re-bake. On any manage_ltd_users.sh pass that reaped, the tenant Drush yml site-cache rebuild is skipped, so a deregistration is never immediately baked into the tenant's Drush state.

One platform check predates the markers: the legacy empty-platform sweep in the per-account worker (and its Hostmaster twin) carries no consecutive-night marker — it is age-gated instead, examining only platform_* alias files older than _DEL_OLD_EMPTY_PLATFORMS days (see the octopus.cnf reference).

Only its ghost-alias branch belongs to the flag family above; the sweep's original job — deleting an old legacy (sites/all) platform no site uses, through a Hostmaster task — is that variable's own feature, not gated by any _GHOST_* flag. The marker-gated platform reapers run independently of the age gate.

Control flags

All default NO. The two codebase checks walk the whole /data tree, so their flags are system-wide only.

Semantics for the dual-scope flags: YES in either file enables the move for that account — the per-account file lets you opt in one account at a time, but a per-account NO cannot veto a system-wide YES (only YES enables, anything else is ignored). The Hostmaster platform sweep reads /root/.barracuda.cnf only.

Flags are read at act time by grepping the control file (case-insensitive YES; a missing file or unset variable reads as disabled) — no reload or restart is involved, and the next nightly pass simply picks up the current value.

BOA is designed to seed the NO defaults into the live control files on install and to append any still-absent flag on the next barracuda up-* / octopus up-* upgrade, never overwriting a value you have set; a box whose control files predate the flags behaves identically, since unset reads as disabled.

Reading the dry-run output

Where each check logs:

Typical dry-run lines, verbatim:

GHOST platform /data/disk/o1/static/foo detected (dry-run; set _GHOST_PLATFORMS_CLEANUP=YES in /root/.o1.octopus.cnf or /root/.barracuda.cnf to move)
Ghost /data/disk/o1/distro/002/foo detected (dry-run; set _GHOST_CODEBASES_CLEANUP=YES in /root/.barracuda.cnf to move)
Unused /data/all/001/foo detected (dry-run; set _SHARED_CODEBASES_CLEANUP=YES in /root/.barracuda.cnf to move)
GHOST vhost for foo.example.com.restore detected (dry-run; set _GHOST_VHOSTS_CLEANUP=YES to move)
GHOST vhost for foo.example.com.restore detected (grace; moves on the next consecutive run)
GHOST vhost for foo.example.com with no drushrc detected (dry-run; set _GHOST_VHOSTS_CLEANUP=YES to move)
GHOST vhost for foo.example.com with no drushrc detected (grace; moves on the next consecutive run)
GHOST drushrc for foo.example.com detected (grace run, not moved)
GHOST drushrc for foo.example.com detected (dry-run; set _GHOST_SITES_CLEANUP=YES to move)
GHOST backend leftover for foo.example.com (no front-end record) detected (dry-run; set _GHOST_SITES_CLEANUP=YES to move)
GHOST candidate foo.example.com SKIPPED (platform-gone: operator review needed, nothing moved)
GHOST sites cleanup enabled -- arming run for o1, nothing moved tonight
GHOST foo.example.com.alias detected (dry-run; set _GHOST_ALIASES_CLEANUP=YES to move)
GHOST foo.example.com.alias sighted, held for 48h before any move
GHOST foo.example.com.alias sighted, still inside the 48h hold

(grace run, not moved) and (grace; moves on the next consecutive run) both mean the consecutive-night marker is still counting — the item looked ghost tonight but has not yet been ghost long enough to act on even with the flag enabled. (dry-run; set …=YES to move) means the opposite: the marker is satisfied, the flag is off. Read the two apart. A dry-run line every night on a box where the flag is YES is not normal and means something is resetting the counter — the vhost reaper's own 24 h freshness test did exactly that on every box until BOA stopped rewriting the whole vhost tree nightly.

The two post-grace site-ghost lines differ by what the account's own Ægir front-end says: GHOST drushrc means a front-end record for the name still exists (the site is still visible in the control panel — this is the line the client notice below is built from), while GHOST backend leftover means the front-end record is already gone (the node was deleted; only backend artefacts remained) or the front-end could not be asked — those are cleaned without any notice. SKIPPED (<class>: …) marks the three classified operator-review cases, which are never moved and never notified.

Client notification for confirmed ghost sites

A ghost site's registration lives in the account's own Ægir front-end, and only there can it be removed for good — the reapers can move the backend alias and vhost aside, but any task run on the surviving front-end record re-creates them. So when the nightly sweep confirms a true ghost site (post-grace, whether or not _GHOST_SITES_CLEANUP is enabled), it first asks that account's own front-end whether a record for the name still exists (its hosting_context row, registered with the domain when the site node is created and deleted with it):

• Record present — the account owner gets an email notice naming the site(s) with the exact control-panel steps: Disable, then run the Delete task; if Delete stalls, remove the stuck record directly via its node/NNN/delete address; or re-run the failed Install/Clone if the site is still wanted.
• Record absent (the customer already deleted the node — only backend leftovers remain, which they cannot see or touch) or front-end unreachable — no mail; the leftovers are cleaned silently, and with no front-end record left nothing can ever re-create them.

The notice follows the Let's Encrypt client-notice model: it is built from the account's own night log only and goes to that account's _CLIENT_EMAIL only (never cross-attributed), is throttled to once per 30 days per site via a marker under <account>/log/ctrl/, carries a Reply-To of the account owner, and is gated by _GHOST_CLIENT_NOTIFY — default YES; only the literal NO disables. Unlike the mover flags, the per-account value wins outright: _GHOST_CLIENT_NOTIFY=NO in /root/.<account>.octopus.cnf silences that account even when the system default is YES, because the account's octopus.cnf is sourced after the frozen run context.

Recovering a moved item

Nothing is deleted, so recovery is moving the item back from its destination:

• ghost codebases — mirrored under /var/backups/ghost-codebases-cleanup/ at their original path, in a <timestamp>/ subdirectory per move;
• shared codebases — /var/backups/codebases-cleanup/… (or /data/disk/codebases-cleanup/… on symlinked /data/all);
• platform aliases, platform dirs, vhosts, site aliases — the owning account's undo/ dir (/var/aegir/undo/ for Hostmaster); a reaped site's vhost lands as undo/ghost-vhost-<domain> and its files dir as undo/ghost-site-<domain>. The ~/.ftp/.drush mirror copy the ltd reaper deletes is derived data: restore the master alias and a live site's copy is re-created on the next 3-minute pass.

One retention caveat: the nightly run prunes the shared-codebase backup dirs after 7 days. Review what landed there within that window; the ghost-codebase mirror and the undo/ dirs are not aged out by the night family.

Enabling on production

The codebase and platform checks are the older, long-exercised pair. The per-site reapers (_GHOST_VHOSTS_CLEANUP, _GHOST_SITES_CLEANUP, _GHOST_SITE_FILES_CLEANUP, _GHOST_ALIASES_CLEANUP) act on live serving artefacts and are newer — exercise them in dry-run, then on a disposable VM, before enabling on production. A sane rollout:

1. Watch the dry-run output across at least two nights (the markers need two consecutive runs anyway) and confirm every listed path really is a ghost.
2. Enable per account in that account's octopus.cnf first; go system-wide in barracuda.cnf only once per-account runs are clean. Expect one extra night: the first enabled run per account only arms the site reaper.
3. Leave _GHOST_SITE_FILES_CLEANUP for last — it is split out precisely because it moves site data, not just registrations.

Note on _GHOST_ALIASES_CLEANUP: it runs on the 3-minute manage_ltd_users.sh cadence, and current builds protect it with a 48-hour first-sighting hold: the first pass that finds a ghost candidate only records a marker (log/ctrl/ghost-ltd-<domain>.seen) and logs the sighted, held for 48h line; the alias is moved only if it still looks ghost once the marker is 48h old, and any valid sighting in between clears the marker. Markers are recorded only while the flag is YES, so enabling it never mass-reaps a backlog on its first pass. Aliases whose site_path lives under aegir/distro (front-end control-panel companions) are never touched here — they stay with the nightly classifier's operator-review path. The hold exists because rsync -a preserves old alias mtimes, so during an xoct/xcopy account transfer the under-60-minutes freshness guard does not protect freshly-arrived aliases whose site directories have not landed yet — and /etc/boa/.pause_tasks_maint.cnf is NOT a usable interlock for this on a hosted system: the box automation removes a hand-set pause file within minutes whenever no barracuda run is active. On builds that predate the hold, the only reliable guard during an inbound transfer is to set _GHOST_ALIASES_CLEANUP=NO in both control files on the target until the import completes. Where the nightly site reaper is also enabled, it usually acts first (with classification and the client notice); this reaper is the between-nights backstop that also cleans the mirrored ltd-shell copy.

Related

• Nightly owl maintenance — the 04:15 run that hosts the six nightly checks: per-account fan-out, load gating, and the global post-account pass.
• lshell + manage_ltd_users — the orchestrator that owns the tenant alias copies and carries the _GHOST_ALIASES_CLEANUP reaper.
• barracuda.cnf and octopus.cnf — the control files the flags live in, and _DEL_OLD_EMPTY_PLATFORMS.
• Reference appendix — the catalogued _GHOST_* / _SHARED_CODEBASES_CLEANUP variables.