Skip to content
BOA Documentation

Powered by Grav

xmass — whole-server move

xmass — whole-server move

xmass is the whole-server tool: it moves a complete BOA host — with every account, database and Solr index — to a freshly installed target in a single operation, keeping the databases current with live replication so the actual cutover window stays short. This page is the full runbook. For where xmass sits against the per-account xoct/xcopy tools, and for the machinery all three share — the storage-aware DRY/--live gate, the http-off write freeze, and the migration proxy the source becomes — see the Cross-host migration overview.

Running the migration

xmass moves an entire BOA host — every Octopus account, every Solr index, every config — from source to target. The DB transport is xtrabackup snapshot + MySQL GTID replication, not per-account mydumper cycles, so it scales to fleets of large databases where per-account dump/restore would be impractical.

A common reason to reach for xmass is an OS upgrade by fresh install rather than in place: instead of walking an ageing host through the in-place OS upgrade paths — a multi-reboot codename chain whose only rollback is a VM-snapshot restore — install BOA fresh at the same release on a new host running the newer OS, xmass the whole estate across, and cut over, leaving the old box as a migration proxy until DNS is repointed.

The working steps (init, sync, status, cutover) all run on the source and take the target IP as their argument; replication and rsync flow source → target. post-mig is run on the target after DNS is updated, and the initial pre-mig is run on both source and target — pre-mig parks the BOA background runners (runner.sh, owl.sh, usage.sh, graceful.sh, manage_ltd_users.sh) so the task queue is frozen during the move, and exchanges root SSH keys between the hosts (the source also learns the target's SSH host key automatically). On both hosts pre-mig first forces the migration tool set current — the per-tool control markers are dropped, the housekeeping fetcher runs synchronously, and each tool's version is logged — so a migration is never run on stale tooling; the tool executing the command refreshes on its next verb, not mid-run. It then checks every Ægir root's control panel coherence on the box it runs on, fail-closed: the hostmaster alias must name a platform that exists with index.php and a site dir with settings.php, the panel database must resolve a hostmaster site context, and that site's platform row must name the alias's path — a panel the tools cannot resolve stops pre-mig here, where it is cheap to repair, instead of failing the rename on the target. A legacy <panel-fqdn>.alias.drushrc.php symlink to the hostmaster alias is reported; the ltd-users pass purges it on every box, and an account that still depended on it is repaired before it migrates. Run in target mode it additionally opens the firewall for the source (which the source can never arrange for itself) and removes this box's OWN published root public key from the nginx web root, not just the fetched copy — a box that was ever a migration source otherwise serves a root key at the undefined-host URL indefinitely, from which a later migration could fetch the wrong box's key.

SH
# Phase 0 — pre-mig, run on BOTH source then target
xmass pre-mig <source-fqdn>

# On SOURCE — prepare the target: CSF both ways, the Solr set gated
# against the source's measured use, PHP coverage, one installed+seeded
# Octopus account per eligible source account; a site-less account the
# target carries but the source does not (a clone's own satellite) is
# purged, one that carries sites is refused
xmass prep-target <target-ip> [--fix-php] [--fix-solr]

# On SOURCE — install xtrabackup, enable GTID, snapshot,
# restore on target, start the replica; declares the box-default
# proxy mode and prints each account's resolved mode
xmass init <target-ip> [--proxy-mode=temporary|permanent|ha-switch] [--proxy-deadline=YYYY-MM-DD|+Nd]

# On SOURCE — rsync platforms/files/configs/Solr (repeat freely);
# DRY plan by default, re-run with --live to transfer
xmass sync <target-ip>                                 # DRY plan
xmass sync <target-ip> --live

# On SOURCE — standing-mirror file cadence: arm AFTER the first
# 'sync --live'; minute.sh then repeats the live sync every N minutes
# (default 15, floor 5) until cutover or --off
xmass autosync <target-ip> [--every=N]
xmass autosync --status
xmass autosync --off

# On SOURCE — replication lag + last-sync timestamp (an unreadable
# replica is named, never printed as an empty lag)
xmass status <target-ip>

# On SOURCE — final cutover (block web, drain lag, promote target,
# renameaegirhost on target, convert source to proxy, every panel
# probed by its new name); the DRY run plans storage for all accounts
# + shared and stops before any destructive step
xmass cutover <target-ip> [--proxy-mode=...] [--proxy-deadline=...]          # DRY plan
xmass cutover <target-ip> [--proxy-mode=...] [--proxy-deadline=...] --live

# On SOURCE — read-only post-cutover sweep: sites, symlinks, the
# certificate mirror, every control panel by its new name
xmass verify [target-ip]

# On TARGET — finalise after DNS update
xmass post-mig

# On SOURCE — re-enable Solr on a rolled-back source after an
# aborted or reversed cutover (see below)
xmass restore-solr

Dispatch verbs: pre-mig, prep-target, init, sync, autosync, status, cutover, verify, post-mig, restore-solr. Every state-mutating verb takes a box-wide owner-PID lock: a concurrent run is refused non-zero naming the owning process, while status and verify stay unlocked so a migration can always be inspected mid-run (an automated autosync pass takes the same lock in defer mode — it quietly waits for the next cadence tick instead of erroring). The guard checks liveness, not the file, so a killed run leaves nothing to clean up. The sync and cutover verbs are gated by the DRY/--live model (see Storage-aware transfers and the DRY/--live gate on the overview); one DRY plans all accounts plus shared.

Pre-sync is incremental — once init runs, repeat sync over days or weeks until ready; the GTID replica keeps the target's databases current and sync carries the file-system deltas. For a standing mirror (a migration deliberately parked at syncing as a hot standby), xmass autosync <target-ip> automates exactly that: after your first manual DRY + --live cycle it repeats the live sync legs every N minutes (default 15) from the active side, so the mirror's files stay minutes — not days — behind its continuously replicating database. Every automated pass re-verifies the target still holds its standby marker (at pass start and at every leg boundary) and stands down loudly, syncing nothing, the moment it is gone — a promoted mirror is never written to, and unlike a manual live sync an automated pass never re-creates the marker. Passes defer quietly behind any other xmass verb, a running barracuda/octopus upgrade, and the cutover window; failures and every stand-down that needs a human are mailed through the standard incident throttle, and a daily alarm fires if no pass has completed for six cadences. Cutover completion disarms the cadence by itself; xmass autosync --off disarms it manually. Source-side deletions propagate on the data trees during any sync (manual or automated) — platform codebases, site files, the dump archives, the Solr data — because a mirror that only ever accretes grows nightly and can never reclaim: its own nightly cleanup is parked while it holds the standby marker. Control, credential and target-role legs stay additive, and so do the cutover legs, since that is the one window where a wrong deletion is unrecoverable. Four guards make the pruning safe: nothing is removed before a successful transfer, an oversized deletion is refused rather than executed (rsync stops at _XMASS_MAX_DELETE, default 5000, and the pass fails loudly for a human to read), the delete flags are never paired with the error-tolerance flag whose meaning is "delete through read errors", and an empty source tree never licenses deletion — an unmounted volume reads exactly like one. Expect the guard to trip on the first pruning pass against a mirror that has been accreting for months; that is it working. A fresh target's own install residue — the pre-pass backup sets under each account's backups/system/ — is dropped by init before the first sync, so it never trips the guard on a first live pass.

The cadence you arm is a ceiling on frequency, not a promise: a pass walks a lot of metadata, so the driver also enforces a duty cycle — the box rests at least three times the last measured pass before the next one, and xmass autosync --status prints the effective interval beside the armed one.

See the failover scenarios page for the standing-mirror runbook this belongs to.

Each sync run carries a fixed set of filesystem data to the target: shared BOA data (/data/all, /data/disk/all, /data/disk/arch, /data/disk/legacy), the static web root (/var/www/static), DNS zones (/etc/bind), usage logs (/var/log/boa/usage), the Solr indices, and per-account platforms, source trees (src/), site files, drush site-aliases, nginx vhosts, SSL/LE material, the sub-account registry (clients/), backups/ and undo/, the FTP account's SSH keys (/home/oN.ftp/.ssh) and the client's ruby-gem/npm trees under /opt/user/. MySQL data is the one thing not rsynced — GTID replication keeps it current continuously. Optional per-account config directories (pre.d, post.d, subdir.d, platform.d, the ssl.d trees, tools/le) are skipped with a logged "nothing to send" when absent rather than failing the run; a genuine transfer failure on a directory that IS present still aborts before anything destructive.

Each account's identity travels alongside those files: the portable values of /root/.<oN>.octopus.cnf merged into the target's own copy, the PHP pin files (static/control/{fpm,cli,multi-fpm}.info and the matching log/*.txt stamps), the client's shell credentials — the <oN>.ftp password hash paired with log/pass.txt, the sub-account password store and each sub-user's hash and keys — and the billing suspension flag, which lives outside the account tree (/data/conf/suspended/<oN>.pid) precisely so the account owner cannot delete it.

The pin files are force-copied without rsync's skip-newer flag, because the target's fresh install wrote its own defaults minutes earlier and would otherwise win. That direction matters: BOA derives the account cnf from fpm.info/cli.info, never the reverse, so those files are what actually make a pin stick — and an account left on the target's install-time PHP is a site silently served by the wrong interpreter.

Three files are deliberately excluded from the account log/ sync: proxied.pid and migproxy.cnf are target-role state that must never be overwritten from the source, and pass.txt rides the credential carry instead so it can never advertise a password the target's /etc/shadow does not hold.

Whole-server cutover downtime is typically 1–3 hours.

Matching Octopus accounts must exist on the target before init. Replication brings the databases, but the files rsync into /data/disk/<oN>/, the cutover renames each Ægir root and rewires each panel — all of which need a real install (system user, FPM pool, vhost include), not a directory rsync creates. xmass prep-target installs and seeds them; init, sync and cutover each re-verify and refuse without them.

The target's Solr set mirrors the source's measured use — gated, not assumed. A hosted-named box's first full barracuda pass installs all three Solr versions, but a migration target gets only what its source actually uses. prep-target classifies each version on the source by index-write recency — the one-year bar: queries leave no filesystem trace, so the newest data/index write across a version's real cores (dotted names only) is the measurable use signal, and a populated estate whose newest write is older than a year is a stale leftover, not a used version. Two carve-outs: live boa_site_control.ini bindings (or a running service with its SR* opt-in token) count as used regardless of write age — a bound service whose sites have not indexed yet — and registered-but-empty cores with zero bindings are reported for an operator ruling rather than classified either way. Each version's newest-write date is printed so every verdict is auditable. The gate then reads the target's install and deny state, reports both sets and refuses on any mismatch. --fix-solr reconciles: per-version denies (_DENY_JETTY9/_DENY_SOLR7/_DENY_SOLR9 plus the /etc/boa/.deny.*.cnf markers) are written for unused versions before any pass runs on the target, the blanket java pair is written only when the source uses no Solr at all, a denied or parked used version is un-parked by driving the target's own restore-solr, and a missing used version installs on one preflighted barracuda system pass — shared with --fix-php's pass when both are needed. The ordering exists because the timing is unforgiving both ways: the per-version denies only park a service after it installs (the 5-minutely autoupboa), and the file sync lands every existing source Solr tree on the target, where a landed data tree makes BOA's installer treat that version as already installed forever — data with no service, discovered as broken search only after promotion.

init, sync and cutover re-gate on the same comparison, refusing while a used version is not serviceable on the target (missing install, denied, parked, or an init script left non-executable by a proxy finalize). The classification measured while the source was healthy is recorded in /data/conf/xmass_solr_used.cnf, and the re-gates prefer that record once a cutover has denied Solr on the source — a post-cutover source reads as using nothing and would otherwise misclassify every version. A version that looks unused but runs (registered cores, all empty, zero site bindings) is reported for a per-box operator ruling and treated as used — it is never dropped automatically. For taking unused Solr versions off a box outside a migration, see Removing unused Solr versions.

xmass init auto-installs the matching percona-xtrabackup-* package and auto-enables GTID on both servers if not already active.

init also chooses its xtrabackup transfer method automatically from free space on the source /. With more than 1.5× the data-directory size free it stages — the backup is written to /var/backups/xmass_stage, prepared, then rsynced to the target. With less it streams — piped via xbstream over SSH straight to /var/backups/xmass_restore on the target and prepared there, needing no staging disk on the source. Either way the snapshot bounds its backup-lock wait to 15 minutes and kills a query that blocks it after 60 s, so an init cannot sit forever behind one long report.

Both init and prep-target first read the target's /root/.standby.cnf back over ssh and hard-refuse a role clash: a target already serving as a DIFFERENT source's standby is never overlaid (the marker records its source IP; a same-source marker is the normal re-init/repair path and passes), prep-target refuses ANY standby target (it builds fresh targets — its account installs would be refused by the standby's own read-only DB), and a box that itself carries the marker refuses to SOURCE a migration at all. The probe fails CLOSED — an unreachable target is refused with the ssh rc rather than assumed clean.

Three more things init does deserve to be known before they surprise you:

  • It leaves the target's cron ARMED — a standby is a working BOA box (IDS and the watchdogs all live), and its passivity comes from per-job gates on /root/.standby.cnf in every local writer: the task runner and the Ægir dispatch it parks, the nightly and hourly housekeeping, the Solr maintenance, the whole duplicity backup chain and the rest stand down while the marker exists. On top of the gates, init locks the replica's DB outright: it writes the xmass-standby-hold block (read_only + super_read_only) into the target's xmass_gtid.cnf — so the lock survives every restart — and sets it live once the replica threads verify. The appliers are exempt; everything else, root included, is refused at the server. The web tier is held DOWN the whole window (init-script gate, per-minute enforcer, and the BOA_STANDBY_WEB firewall chain), FTPS is kept dead, and tenant logins flip to nologin — the full roster and the .standby.serve.cnf preview escape hatch are on Failover and recovery scenarios. The gates exist for the reason cron used to be stopped outright: a writable replica running its own hourly housekeeping would TRUNCATE cache tables locally, which under ROW binlog stops the SQL thread or writes errant transactions into what becomes production. So a target whose cron is running during the sync window is healthy, not broken — do not stop it. (A target quiesced by pre-gate BOA bytes still had its cron stopped; the cutover restarts it regardless, as harmless vintage tolerance, and post-mig is the confirmation point.)
  • It purges unfinished delete tasks from every eligible account's panel queue before the databases travel: the whole hostmaster database replicates, the cutover runs the task queue with force on the target, and a stranded delete (xmass's own pre-mig kills the dispatcher, which is exactly how one strands) would execute against brand-new production — and init now refuses outright when a pending or interrupted delete cannot be cleared on any panel, naming the accounts, rather than carrying it across. Other task types are only reported — at cutover, per account, by type — so anything you do not want carried across can be cancelled first.
  • It holds Solr down on the target for the entire replication window — as a durable disarm, not a stop: each Solr/Jetty service is stopped, then its init-script exec bit and its rc links are dropped, and the set actually taken down is recorded in /var/log/boa/.xmass_solr_disarmed.list. The hold marker is /var/log/boa/.xmass_solr_hold.pid, and the per-minute java.sh watchdog re-asserts the disarm — and reaps any stray JVM — for as long as that marker exists, so the hold survives a reboot and a barracuda pass, and a Solr started by hand is taken back down within a minute. Cutover step 0 re-holds it as well, cutover step 14 and post-mig re-arm (exec bits and rc links restored, then the services started), and the first target-side Solr start happens only after the cores' tlog/ transaction logs are cleared, so writes buffered when the source Solr stopped are not double-indexed. Do not "fix" a target whose Solr looks down mid-sync — a non-executable /etc/init.d/solr9 there is the hold working, not damage.

xmass cutover invokes xoct proxy per account during the cutover phase to convert the source vhosts to proxies, plus renameaegirhost on the target for the master and each Octopus account. Before the rename, the cutover clears any demotion artefacts the target may still carry from having been demoted itself earlier (a failback: the pair switched once, now switches back): its own outbound proxy policy record, the proxied marker, its parked account dispatchers, the source latches the earlier demotion stamped, and — for every site whose name the rename does not touch — the proxy vhosts, restored from the dot-prefixed pre-proxy backups the conversion keeps. Without this a promoted box keeps relaying to its demoted sibling, which relays straight back, and the loop surfaces as 400 Request Header Or Cookie Too Large on every hop. On a first-time target none of these exist and the step does nothing.

Failing back — running the pair in reverse

The ha-switch pair is symmetric and the whole reverse pass uses the same verbs: on the promoted box run prep-target toward the old active, then init, then cutover, and finally retarget any long-lived migration proxy back with xoct proxy <oN> <ip> --repair --retarget. Two things are normal and expected on the reverse pass:

  • init refuses once on the box's own state file from its earlier source role, and the refusal names its remedy: remove the state file (back it up first) or use xmass reset-phase. Every failback hits this; it is the tool asking you to confirm you really are starting a new migration from this box.
  • The GTID history grows by one server identity per promotion, because the restore gives the promoted box a fresh server_uuid. The replication machinery carries the whole multi-identity set — do not be surprised by three, four or more UUIDs in gtid_executed after a few switches, and never health-check by expecting an exact count: check that the replica's executed set contains every identity the master's backup carried.

init also re-proves the source is master-ready — GTID and the binary log — and restarts MySQL itself if either is missing, so a box that spent time as a replica comes back as a valid master without manual intervention.

Per-account proxy modes

How long the source keeps forwarding, and what each customer is told about it, is decided per Octopus account, not per box. Each account carries a root-owned policy record (/data/disk/oN/log/migproxy.cnf, parsed never sourced) with one of four modes. Only a source-role record participates in resolution: a record whose role is target — the inbound half of the peer pair, stamped when this box was itself brought in by a migration — is never resolved as this box's own proxy mode, so on a chained move (a box migrated in yesterday, migrating out today) a travelled inbound record can never outrank the operator's explicit --proxy-mode. The modes:

  • temporary — forwarding is withdrawn once the migration settles; with a deadline the client mail names the exact date, without one it promises notice before withdrawal.
  • permanent — the old address stays in service; the client is told to move DNS at a normal pace, with no countdown.
  • ha-switch — the old address is presented as the switch point of a live main/mirror pair. For trust and teardown it behaves exactly as permanent; only the client notification differs.
  • retired — set by xoct proxy-retire; forwarding is (being) withdrawn and the target drops the trust for this account at the next reconcile.

--proxy-mode on init/cutover writes only the box default (/data/conf/migproxy_mode.txt); an account pinned with xoct proxy-mode oN <mode> always wins over it, and a box sweep never overwrites a pin. --proxy-deadline sets the box-default deadline quoted in the client mail (absolute date or +Nd; past dates are refused, and a stored deadline that has expired by send time is dropped with an alert rather than mailed). --permanent-proxy remains a deprecated alias for --proxy-mode=permanent. Flags on cutover override whatever was set at init.

The resolved mode drives three things per account:

  • the migration-complete notification xoct proxy sends — each mode has its own honestly-worded arrangement block, plus a shared block stating that only the sites which existed at migration time are forwarded (anything created later needs DNS pointed straight at the new address);
  • the migration-proxy realip/CSF trust on the target: any account resolving permanent or ha-switch keeps it past post-mig, and the target's migration_proxy_trust.sh reconcile recomputes the kept peer set from its per-account records whenever policy changes;
  • xmass cutover --live, which refuses to run while any account would fall back to the built-in default — init and the DRY cutover print every account's resolved mode precisely so this is discovered while it is still free to fix.

A per-account conversion failure at cutover keeps that account's 503 gate up and does not stamp it proxied.pid (its vhosts would otherwise serve the old local copy against a database that now lives on the target); the completion banner names the accounts to fix and re-run.

Inspect or change policy at any time on the source:

SH
xoct proxy-mode --all                          # the table: mode, deadline, scope, peer, last told
xoct proxy-mode o3 temporary --deadline=+30d   # pin one account
xoct proxy-retire o3                           # retire + send the withdrawal notice

Every policy change is pushed to the target's record and re-reconciled there, and the client is mailed a short follow-up whenever the promised arrangement changes (--no-notify suppresses and logs, --renotify forces).

Requirements

  • Identical Percona versions on source and target — series and patch level (for example both 8.4.13, never 8.4.10 against 8.4.13). GTID replication plus an xtrabackup snapshot of the live datadir is version-strict, which is the key difference from xoct/xcopy, cross-version safe via mydumper. Both halves are enforced gates at init, not advice. The series gate compares what mysql -V reports on each box — the client binary — and refuses a mismatch outright, pointing you at xoct instead. The patch gate then asks each server for SELECT VERSION(); an unreadable answer on either side is fatal, because a version nobody read cannot be vouched for. A deliberate forward skew — a target newer than the source — is accepted with _XMASS_ALLOW_PATCH_SKEW=YES. A target older than the source is refused and no flag overrides it: the restore puts the source's datadir on the target, and a newer datadir on an older server is an unsupported downgrade. Align both ends from the same repo snapshot instead.
  • BOA installed on the target at the same release as the source. This is an enforced gate, not advice: prep-target reads the release stamp from boa info on both ends at first target contact and refuses with no override when they differ — a target missing a central-map nginx variable a newer release introduced fails the box-wide config test and takes down every migrated site, not only the one that needed it. The tree may legitimately differ (lts vs pro); an unreadable stamp on either side is fatal too. The fix is a full run — barracuda AND octopus — on the older box.
  • Root SSH key access source → target, set up by xmass pre-mig.
  • An installed Octopus account on the target per eligible source account, with every PHP version those accounts pin — both established by xmass prep-target. Each account is installed by xoct create, which waits for a quiet target (no barracuda or octopus pass, no install lock, no armed run-upgrade.pid, two consecutive polls) before it starts, proves the install afterwards, and arms the enforced post-install upgrade only when the seeded control files actually changed — a re-entered prep-target never triggers a second upgrade for nothing. prep-target itself runs the full target-silence gate (below) before every account create, because each create leaves the target mid-motion and the next launcher refuses on any lock; an account whose target never settles is reported failed, never launched into a refusal.
  • No account on the target that the source does not have. A golden-master clone carries its own satellite, which the init datadir swap would leave without a panel database — a broken leftover. prep-target purges a site-less one itself with BOA's own boa cleanup (where a static/ tree exists the account's backups/, distro/, src/, static/ and undo/ trees are removed outright, whatever else the home holds is parked under /var/backups/zombie/purged/<oN>/, nginx configtest proven) and refuses one that carries sites; _XMASS_ALLOW_TARGET_ONLY=YES keeps it knowingly, broken after init until purged. "Site-less" is read from three independent sources on the target, all of which must be empty: the account's registered site aliases (its own control panel set aside), the sites/ directory of every registered platform (which also reaches a platform behind a symlink), and a sweep for site directories under distro/, static/, platforms/ and aegir/ to the depth BOA itself builds, with or without a web/ or docroot/ level; an inventory that cannot be read is never treated as empty.
  • CSF allowing the source on the target and the target on the source: the target dials back to source:3306 for replication, and getting it wrong fails init after the target's datadir has already been replaced. prep-target appends each peer to both csf.allow and csf.ignore (an allow alone still leaves the peer exposed to a guard temp-deny mid-migration), reloads CSF, and proves the reverse path before anything depends on it. The reverse-path proof is fatal by design; if your pair is deliberately firewalled differently, the documented escape hatch is _XMASS_SKIP_REVERSE_CHECK=YES.

init re-checks all of the above, and adds two of its own: the replication server_ids derived on each side must differ (they come from the last two IP octets only, so two boxes in one /16 collide and replication simply refuses to start), and the target must have room for the datadir restore (_XMASS_SKIP_DISK_GATE=YES overrides the disk-room gate when you know better — e.g. a mount init cannot see).

One more pre-flight worth doing on the source before the seed: this path replicates every schema on the box, orphans included. A source-side sqlclean pass first both shrinks the transfer and removes stale panel schemas the tooling would otherwise trip over (it refuses to guess between multiple hostmaster candidates).

The cutover sequence

Before any destructive step, cutover runs an automatic pre-flight: it confirms the phase is syncing, refuses to run outside screen/tmux (a dropped session mid-cutover strands the source on 503 — _XMASS_NO_SCREEN=YES overrides the refusal if your session management genuinely makes it safe), and re-verifies the target's accounts and PHP coverage. Only once every refusal is past does it stop cron and park the BOA background runners itself — the box's own cron restores a park done back at pre-mig time within minutes, so the park has to happen here, immediately before the destructive sequence.

Three verbs also wait for the target to be silent before they touch it, and the wait lives in the tool, not in an instruction: prep-target (before every account create and again before it completes) and init require no BOA pass, no install lock, no armed run-upgrade.pid and no queued or running panel task on any Ægir root, while cutover --live requires no pass and no lock only — a standby's runner never consumes an armed task and its panel rows are the source's, so queued rows are no signal there. Silence is two consecutive clean probes; a root whose queue cannot be read fails the gate rather than passing it. The ceiling is _XMASS_TARGET_SILENT_MAX_WAIT seconds (default 2400), and _XMASS_SKIP_TARGET_SILENCE=YES bypasses the wait knowingly.

Before the freeze, cutover --live also names the panel tasks still queued or running on each source account — current task revisions only, the dispatcher's own view (every finished task leaves its creation revision frozen at "queued" in hosting_task, so a naive count reports years of history as travelling work). Whatever it names will run on the target once its queue starts; cancel what you do not want. Interrupted delete tasks were already purged at init.

The cutover itself has built-in safety timings. It drains replica lag to zero, polling every 15 s up to a ceiling of _XMASS_SYNC_MAX_WAIT seconds (default 7200); on timeout it reports the first, previous and last lag samples and whether the lag is closing or growing in the most recent interval, because a growing lag will not be fixed by waiting.

It then takes a final static/files-only rsync to catch last-second uploads. That pass runs before the write freeze, not under it: the web freeze applied at the start of the cutover is what stops file writes, and a database lock never gated them, so holding one across a walk of every store gained nothing.

Only then does it stop writes — durably, not with a database read lock. A read-only flag is appended to /data/conf/global/global-extra.inc (the previous file kept beside it as .bak), and the cutover refuses to proceed if the freeze does not take: a session-scoped FLUSH TABLES WITH READ LOCK cannot hold anything once its client disconnects, which is why the old procedure's lock is gone — the surviving FLUSH TABLES only pushes buffers before the replication position is read. With the freeze in place it triple-checks lag = 0 at 10 s intervals; a failed check aborts before the promotion holds, so the source is still the only production box: the abort unlocks source MySQL and thaws the write freeze itself, handing the source back writable. A failed promotion command is handled by evidence, not by its exit code (transport can fail after mysql ran): the tool reads the target's replica state back — a target provably still a replica gets the same unlock-and-thaw abort; a promotion that actually committed parks resumably; an unreadable target keeps the source frozen (a thaw could silently lose writes) and the message spells out how to determine the state and which recovery to run. After a successful promotion the freeze flag is removed on the target (the new production box must never come up read-only); it deliberately stays on the source, which serves through the proxy from here.

Transfers are gated as well. A store that cannot be placed on the target, or an rsync that fails for any reason other than source files vanishing mid-copy, aborts the cutover before anything destructive happens. Every abort after the web freeze prints the exact commands to restore service on the source; when the write freeze is still in place as the recipe prints (a post-promotion park), the recipe includes the thaw line — restore /data/conf/global/global-extra.inc from the .bak kept beside it (or delete the # xmass migration freeze block) — and says when it is safe to use it: thaw only to abandon the cutover and keep the source as production, because after the promotion, writes accepted on the source can never reach the target. Clearing the 503 alone leaves every source site serving read-only.

Before converting the source into a proxy, the cutover proves the target's web layer: it runs nginx -t on the target and refuses the conversion if the config is invalid (printing the tail of the test output), then requires a real HTTP answer on the target's port 80. The proof sits at the head of the cutover tail, so every entry re-runs it — the first pass and each resume of a parked cutover; nothing later in the tail gates on the web layer (the rename pass's serve-wait measures and reports, and nothing else can start a stopped nginx). Either refusal parks resumably at phase=rename-failed and prints the source-restore recipe: the target stays promoted, the source stays 503-gated and frozen, and the SQL watchdogs stay paused. Fix nginx on the target, then re-run xmass cutover <target-ip> --live — the resume re-runs the proof and starts nginx itself.

Once replication is decoupled and nginx is serving proxied traffic on the target, cutover rewires panel database access for every Ægir root before the renames run. The init-time datadir swap replaced the target's MySQL wholesale, so the fresh-install panel databases the new host's control-panel dirs pointed at no longer exist; for each root xmass rediscovers the live (replicated) hostmaster database, resets that database user's password across its host variants, and rewrites the panel dir's stored credentials to match. The same step also reconciles the panel's platform: an aged source's control panel sits on a higher platform number than a fresh target's (the panel platform is rebuilt over the years as PHP pins change), and the replicated database keeps naming the source's path. The rewire always compares the platform the database names for the panel site with the platform the target's own hostmaster alias serves from, repoints the platform row at the on-disk one, and renames its platform_NNN context to the on-disk number when that name is free or held only by a deleted platform's leftover row, so alias file names keep matching paths. Left alone, the rename queue verifies a platform that does not exist on the target, hosting tries to rebuild it, and the panel vhost is rendered with that root — a 404 by the new name, while a master whose verify never got that far keeps a dead db_name. This must succeed before renameaegirhost can bootstrap, and it is idempotent — a root whose panel already names a live database and the on-disk platform is skipped.

The rename pass itself carries more than aliases and vhosts. A tenant site whose URI embeds the box hostname follows the box: its site directory is moved onto the new hostname together with every URI-keyed surface — the per-site Drush alias file, the static files store and the site's files/private symlinks into it, the client symlink, the URI values inside settings.php (including the backslash-escaped trusted_host_patterns spelling that produces an HTTP 400 when left stale), and the site's per-site PHP pin row. A fail-closed gate then refuses to run the Ægir task queue while any site directory still carries the old hostname — the queue would import those as brand-new sites with duplicate panel nodes; inside a cutover that refusal parks at phase=rename-failed. After the renames the tool waits for each renamed site to actually serve (up to 180 s per site, _RENAME_SERVE_WAIT; the box's catch-all page is discriminated so an unknown-host 200 never passes) — minutes spent here are the wait, not a hang, and a site reported NOT SERVING with a 400 usually means its trusted-host settings. The wait exists because settings.php is rewritten through its existing inode and PHP resolves it through a per-worker realpath cache, so a single early check proves nothing either way. The migrated-in http-off gate is cleared on the target before the renames and again after them, so the serve-wait measures the sites and never the gate. The rename also parks the old-name panel SSL-proxy include from the master root's pre.d (the account pass regenerates it for the new name once that name's certificate exists): left behind, it keeps naming a certificate under the old-name Let's Encrypt directory and fails nginx's configtest box-wide the day that tree goes.

Entering the cutover tail, and again on EVERY resume from a parked rename-failed, step 11.5 unlocks the promoted DB: SET GLOBAL super_read_only=OFF plus read_only=OFF, with the runtime readback verified (both variables) BEFORE the xmass-standby-hold block is stripped from xmass_gtid.cnf — a failed unlock parks the cutover rather than letting the renames march into a read-only DB, and with mysql unreachable the block deliberately survives as the watchdog's retry key. Step 12 then removes the BOA_STANDBY_WEB firewall chain on the target and proves the web tier both on loopback and EXTERNALLY — from the source, with a browser UA (curl's default lands in BOA's own crawler map), over HTTP and, when the target has a public 443 listener, HTTPS — because the loopback curl cannot see an INPUT-chain drop and a surviving firewall hold would otherwise surface only after the DNS flip, as total outage.

After promotion cutover removes the standby role marker on the target — the act that hands the promoted box back its task queue and every local writer, since the per-job gates keyed on that marker keep holding both while it exists. Only then does it start cron, which is a belt rather than the mechanism: a target quiesced by pre-gate BOA bytes still has its cron stopped, while a current one ran armed throughout the replication window. It then restores the parked runner scripts. The removal is re-checked over a second connection, and if the marker is still present or cron is not running the run warns and prints the one-line manual fix to apply on the target (rm -f /root/.standby.cnf; service cron start) rather than failing the cutover. It also writes a proxied.pid marker for every source account.

Before the final sync, cutover stops every Solr and Jetty service on the source and disables the stack there permanently, touching /root/.deny.java.cnf and setting _DENY_JAVA=YES in /root/.barracuda.cnf (on a box carrying no cnf it says so and writes the marker only). To reverse it on a rolled-back source, run xmass restore-solr there — it clears both the marker and the cnf value and brings the stack back up. Clearing the deny by hand is not enough: the file wins while it exists, and the verb also undoes what the deny switched off.

The order inside the verb is what makes it work. It clears the xmass Solr hold marker (/var/log/boa/.xmass_solr_hold.pid) first — while that file exists the per-minute java.sh watchdog disarms and kills whatever has just been started — then clears /root/.deny.java.cnf and sets _DENY_JAVA=NO, and only then re-arms each service: exec bit restored, boot links re-enabled (an enable as well as defaults, because the K links a disable left behind do not flip back on their own), any stale pid file removed, and the service started and waited on for up to 60 s on its localhost port. Where a service's init script is gone and no parked copy survives in /var/backups, restore-solr regenerates one for Solr 9 and Solr 7 from the distribution template shipped in /opt/<service>/bin/init.d/solr, substituting the install directory, environment file, run-as user and Provides: name BOA installs with (a service with neither a parked copy nor a usable template is reported and skipped). It honours each service's own deny (_DENY_SOLR9/_DENY_SOLR7/_DENY_JETTY9 and their /etc/boa/.deny.*.cnf markers), so clearing the blanket java deny never re-enables a version the box deliberately runs without, and it is idempotent — a service already in place and already listening is reported and left alone.

Two refusals. On a finalized PX0 proxy (/root/.proxy.cnf) it refuses outright: Solr is down there by design after xtrim finalize. On an active replication standby (/root/.standby.cnf) it also refuses — Solr is held down for the whole migration window, and starting it mid-window would open an index still arriving by rsync; finish or abort the migration first.

BOA does not self-heal any of this. _if_solr_nine (and its Solr 7 counterpart) re-runs the Solr installer only when the version's data directory or its version stamp is missing, and a rolled-back source still has both — so no barracuda up-* pass ever brings the init script back.

Once the target's Solr is up, cutover compares the source's core set against what the target actually registered and names any core that arrived as data but was never registered — a core a fresh build cannot regenerate would otherwise leave its search indexes silently returning wrong results on a cutover the tool called successful. The comparison asks Solr on the ports BOA actually installs it on, and counts a core as registered only when Solr reports it as a loaded core: a core whose configuration fails to resolve on the new box is still echoed by name in Solr's start-up failures, and is called out separately as arrived-but-unloadable rather than passed as healthy. Registration itself stays with the operator, because it is core-shape-specific.

It then restores cron and the five runners on the source too, which matters more than it sounds: a source kept as a long-lived proxy serves the migrated sites' TLS, renewals stop there by design, and the only thing keeping those certificates fresh is migration_proxy_certs.sh running from cron. cutover therefore also checks, before converting anything, that the mirror is both present and actually scheduled.

What the conversion covers is the hosted sites, never the control panels. Each account's panel is identified by its hostmaster alias, not by which alias files exist; it keeps its local vhost and is put into Drupal's own maintenance mode once the conversion is in — an admin can still log in, nobody else can queue tasks against a database that now lives on the target — and the master panel gets the same treatment in a step of its own. Each hold flushes the panel's page cache as well, because a variable change alone leaves the cached anonymous front page in place and the hold would stay invisible until the next cache clear. The old panels stay online on the proxy box as its monitoring canaries; the target never serves a source-named panel, so a proxied panel would only ever answer the new box's catch-all page. The cutover's last step then opens every panel on the target by its new name and names any that does not answer — a cutover that ends with a dead panel says so in its own output rather than reporting success.

Afterwards, xmass verify re-checks the result end to end without changing anything. Per account it samples two real sites: one under a client domain (its name does not change at cutover), fetched twice at least 5 s apart against the target (BOA answers fetch bursts with transient 200 "Page not found" bodies, so spacing is what makes the check meaningful) and once relayed through the source proxy; and one under the old box's own subdomain, which cutover renamed to the target's name — fetched direct by its new name and never relayed (the old name is gone by design, and only the target's catch-all would answer it). The account's control panel is never a sample; it is probed separately. An HTTPS probe runs where the site has certificates. A 401 or 403 that the direct and the relayed request agree on is reported as a protected site serving as configured, not as a breakage. Then the leftover http-off.pid files, the source's cron and certificate-mirror state, and every Ægir root's control panel on the target by its new name.

The replication transport is TLS

The replication stream carries every production row between two boxes that, in a cross-region estate, sit on different providers' networks, so init runs it over TLS: the xmass_repl user is created REQUIRE SSL (the source refuses a plain connection outright), the replica is configured with SOURCE_SSL=1 (MASTER_SSL=1 on 5.7/8.0) against the source's own server certificate (Percona generates one in the datadir; no CA is pinned — the stream is encrypted against passive capture, which is what the first cross-provider mirror required), and once the replica runs, init proves the session from both sides (Source_SSL_Allowed: Yes on the target, connection_type = SSL/TLS in the source's performance_schema.threads) and refuses a stream that is not TLS. A source whose MySQL has no TLS material stops before the replication user is created. xmass status prints the transport it can prove (TLS, PLAIN or unknown). The rsync legs already travel over ssh. _XMASS_PLAIN_REPLICATION=YES runs the stream in the clear deliberately and loudly — for a source that cannot serve TLS, never the default.

MySQL credentials and the datadir swap

Because the xtrabackup restore replaces the target's entire /var/lib/mysql — including the mysql system tables — the target's MySQL root password becomes the source's. xmass therefore ships /root/.my.pass.txt and /root/.my.cnf from source to target twice: right after the restore, so client tools work during replica setup, and again after promotion at cutover. This wholesale swap is also why the panel-database rewire above is required.

Finishing on the target — post-mig

xmass post-mig, run on the target once DNS is updated and traffic flows directly, is not optional housekeeping. It ensures Solr is running cleanly, reloads nginx, restores any remaining BOA runner scripts, and reconciles the migration-proxy trust from the per-account policy records: peers whose accounts resolved temporary are dropped, permanent/ha-switch peers stay trusted (restricted to the live peer set), and undeclared accounts leave everything as found and are reported.

It also rebuilds the pinned PHP pools. A migrated account arrives carrying the source's per-release FPM markers; both boxes run the same release, so without clearing them the target reads the pool set as already built, never creates pools for versions that exist only here, and never regenerates the per-site socket includes — every pinned site is then served by the account's DEFAULT pool, indefinitely, because nothing re-triggers until the release serial moves. post-mig clears the markers, lets the normal sweep rebuild, then prints per account either that every pinned pool is live or an ALRT naming each pin still missing. Treat any such ALRT as a stop: those sites are running on the wrong interpreter right now, and a 200 with correct content proves nothing — a Drupal 7 core tolerant of a newer PHP looks healthy while mis-pinned.

Finally, post-mig is what confirms cron is running: the current tool never stops a standby's cron (its passivity comes from the per-job gates), but the cutover restarts it regardless — harmless vintage tolerance for a target quiesced by pre-gate bytes — and a box left with cron down silently stops receiving fleet updates entirely. Verify with pgrep -x cron before walking away.

State machine and recovery

xmass tracks its progress in /data/conf/xmass_state.cnf (mode 600 — it holds the replication password), moving through init → syncing → cutover → complete. Each subcommand checks the current phase and refuses to run out of sequence.

phase=cutover is written only immediately before the write freeze, so anything that aborts earlier in the cutover leaves the phase at syncing and is recovered by re-running the dry run and --live. If a phase does end up wrong, correct it with xmass reset-phase <phase> rather than editing the state file by hand; it warns about the unsafe transitions and is exempt from the 90% disk precondition, since a stalled migration is a likely reason the disk filled. To abandon a migration and start over, remove the state file — but only after replication has been torn down on the target, or the next init collides with a live replica.

Recovery paths:

  • init fails before replication starts — remove the state file and retry.
  • init fails after replication starts — on the target run STOP SLAVE; RESET SLAVE ALL, drop the xmass_repl user on the source, then remove the state file.
  • cutover aborts mid-flight — the source stays on its 503 gate, and the tool prints the full restore recipe for the source; follow that rather than working from memory. An abort before the write freeze leaves the phase at syncing, so retrying is a fresh dry run plus --live with nothing else to undo. An abort after the freeze but before the promotion holds (the post-lock lag checks, or a failed promotion whose read-back proves the target is still a replica) unlocks source MySQL and thaws the write freeze itself; the phase is cutover, so retrying is xmass reset-phase syncing, a fresh dry run, then --live. A promotion failure whose target cannot be read back keeps the freeze — the message spells out how to determine the promotion state and which recovery to run.
  • The target's web-layer proof, a panel rewire, or a renameaegirhost pass fails — cutover parks resumably at phase=rename-failed and names the cause; the target is already promoted, so the source stays 503-gated and deliberately frozen. The printed recipe leads with the resume instruction; the restore lines below it, including the thaw, are only for abandoning the cutover. Fix the cause and re-run xmass cutover <target-ip> --live to resume — the resume first re-asserts the source's 503 gate and write freeze (an operator may have followed the abandon recipe before changing their mind), then the tail drops the replication user and re-runs the web-layer proof (starting the target's nginx itself); already-rewired and already-renamed roots no-op.

© 2026 BOA Documentation. All rights reserved.