Failover & recovery scenarios

The sibling pages in this chapter document the tools. This page is organised the other way round: by situation. Each scenario says what has actually happened, which commands to run, on which machine, and what to expect while they run. Nothing here replaces the tool pages — every sequence links back to the page that explains it in full.

The topology these scenarios assume is the one every cross-host migration leaves behind, extended by an optional passive mirror:

DNS / CDN edge (optionally a proxying CDN such as Cloudflare)
   |
[proxy]    the migrated-away old box: every site vhost relays to the active
   |       server; possibly shrunk (xtrim) or finalized (/root/.proxy.cnf)
[active]   the current master: serves every site, sole certificate issuer,
   |       replication master when a mirror exists
[mirror]   passive GTID replica, built with --proxy-mode=ha-switch;
           serves NOTHING while passive

Not every deployment has every layer, and the front decides how a switch is executed. Every scenario below refers back to this table:

First moves — read the state before acting

Whatever went wrong, start by reading what the boxes believe, not what the monitoring says:

# On the ACTIVE server (the replication master)
xmass status                  # phase, peer IPs, last sync, replication lag
xoct proxy-mode --all         # per-account promise: mode, deadline, peer
grep ^_XMASS_PHASE /data/conf/xmass_state.cnf
                              # never dump the whole file to a shared screen --
                              # it holds the replication credentials

# On the MIRROR
mysql --defaults-file=/root/.my.cnf -e 'SHOW REPLICA STATUS\G' \
  | grep -E 'Running:|Seconds|Error'
                              # Percona 5.7 and 8.0: SHOW SLAVE STATUS\G

A healthy pair shows phase syncing, lag 0s, both Running: fields Yes and empty Error fields. xmass status reports lag only — a stopped SQL thread shows as lag NULLs, which is why the thread check on the mirror is a separate command.

The phase field is the map of any half-finished switch:

Probe rules that apply to every check on this page: pin probes to an explicit IP with curl --resolve <name>:80:<ip> (live DNS lies during any switch), space repeated fetches at least 5 s apart and re-verify once before believing a failure (BOA answers fetch bursts with transient soft-404 bodies), and judge by page content, never by status code alone — an unknown Host name lands on the box-wide catch-all page, which answers a healthy-looking 200.

The active server is down

Impact. Every site is down. The proxy (if any) is up but relays to a dead origin, so visitors see gateway errors served from the proxy rather than timeouts.

Without a mirror this is a restore situation, not a failover. Do not try to serve from the proxy — a shrunk proxy holds no site data at all, and an unshrunk one holds a copy that has been stale since the day of the cutover. Revive the box, or rebuild a server at the same BOA release and restore from the remote backup history. Budget hours to days, not minutes.

With a mirror, decide first: revive or promote. If the active can plausibly return quickly (crashed process, reboot, transient outage at the provider), reviving it is the safe route. After it returns, verify that replication actually resumed — the thread check on the mirror, above. GTID auto-positioning normally reconnects by itself, but an unclean crash can lose a master's final transactions, leaving the mirror ahead of it; the IO thread then fails instead of resuming. In that state the pair has diverged: pick the box that will serve, and rebuild the other as the mirror.

While the active is down, note the clock that matters beyond the outage itself: the active is the sole certificate issuer for every proxied name, so nothing renews until it is back. The relaying box serves its frozen certificates meanwhile — check the horizon with xtrim status on it. An outage that approaches the renewal window forces the promote decision.

Emergency promotion of the mirror. The planned switch in the next section is the proven, supported path, and it needs a live active. This checklist reproduces by hand what that path automates — with one sharp edge it cannot reproduce, stated below.

# On the MIRROR (the box being promoted)
rm -f /var/log/boa/{xmass,xoct,xcopy,xtrim,renameaegirhost,ffmirror}.ctrl.*
bash /var/xdrago/clear.sh     # a mirror's cron runs and keeps tools current;
                              # dropping the markers forces a refresh anyway,
                              # covering a box that was down when a serial shipped
mysql --defaults-file=/root/.my.cnf -e 'SHOW REPLICA STATUS\G' | grep Gtid
                              # let the SQL thread finish applying what was
                              # already received (Retrieved_Gtid_Set contained
                              # in Executed_Gtid_Set) -- cutting it loose early
                              # discards received-but-unapplied transactions
mysql --defaults-file=/root/.my.cnf -e 'STOP REPLICA; RESET REPLICA ALL;'
                              # Percona 5.7 and 8.0: 'STOP SLAVE; RESET SLAVE ALL;'
rm -f /root/.standby.cnf      # the standby role marker: post-mig below also
                              # clears it, and the watchdog self-removes it once
                              # the box probes as no replica -- but explicit is
                              # faster than either, and the task queue is held
                              # until it is gone
rm -f /data/disk/*/static/control/http-off.pid
                              # a 503 gate can ride in on the file sync, and
                              # nothing on this box removes it by itself
renameaegirhost --aegir-root /var/aegir --force-old <active-fqdn>
                              # then once per /data/disk/<oN> root; each run is
                              # convergent and safe to repeat
xmass post-mig                # Solr up, cold cache flush, runners un-parked,
                              # proxy trust reconciled, carried FPM pins healed
service cron start            # normally already running (a mirror keeps cron
                              # armed); harmless belt for a box quiesced by
                              # pre-gate BOA bytes — then: pgrep -x cron
nginx -t && service nginx start
curl -s -o /dev/null -w '%{http_code}' --max-time 15 http://127.0.0.1/

Then switch the front per the front table above. If Solr cores fail to load afterwards, clear their transaction logs and restart Solr — post-mig starts it without the stale-tlog clear the planned cutover performs.

What this path cannot reproduce. The planned cutover rewires the promoted box's panel (Hostmaster) database access before any rename — the mirror's own fresh-install panel databases died in the mirror-build datadir swap, and the replicated panel users' passwords exist only as hashes. That rewire has no standalone command. After an emergency promotion, client sites serve (their vhosts and databases are local), but the Ægir control panels on the promoted box answer 500 and its task queues run against a Hostmaster that cannot bootstrap — expect renameaegirhost's queue passes to report failures even though its vhost and alias rewrites converge. Panel service returns with the first planned chain you run (the deliberate failback performs the rewire on the box it promotes); treat panel operations as unavailable until then.

Fence the failed box before it returns. From the moment of promotion the old active's data is stale, and it must never serve again as-is. When it comes back:

# On the RETURNING box, before anything else
/etc/init.d/cron stop; pkill -x cron       # its own jobs would write and self-heal
for _d in /data/disk/*/static/control; do echo 7200 > "${_d}/http-off.pid"; done
service nginx reload                       # every account answers 503 here

Prove no front still reaches it (curl --resolve per name at this box's IP), then choose: fail back deliberately, or rebuild it as the new mirror — both sequences are below.

Expect: recovery point = whatever the mirror had applied at the moment of failure (a healthy pair idles at lag 0s, but replication is asynchronous — nothing guarantees a crashing master's final writes arrived). No published recovery-time figure exists for this path; budget at least the planned-switch window plus diagnosis time.

The proxy box is down

Impact. Only names still resolving to the proxy's address go dark. Names whose DNS already points at the active server are unaffected, and the mirror is not involved at all. There is no data risk — the live data is on the active; the proxy only relays.

Move 1 — bring it back. A proxy is deliberately kept a live, self-recovering server: nginx and cron are registered to start at boot even on a finalized (/root/.proxy.cnf) box, so a rebooted proxy comes back serving with its watchdogs armed and its tool updates flowing. There is nothing to run after the boot — verify instead:

# On the revived proxy
xmass verify <active-ip>     # relayed + direct fetch per account, cron check
pgrep -x cron                # cron powers the daily certificate mirror

Move 2 — the proxy cannot be revived promptly. Take it out of the path by moving the front directly to the active server, per the front table: a CDN origin change (immediate), or repointing the A records (bounded by the old record TTL). This work happens in a DNS or CDN dashboard, not on any box; prove it from a workstation with curl --resolve <name>:443:<active-ip>. One check first: the active is already the sole certificate issuer, so no re-issuance is needed — but confirm it actually holds a live certificate store for every HTTPS name (the proxy's daily certificate mirror warns when a name has none; such a name loses HTTPS the moment the proxy is out of the path).

Afterwards. If the proxy is gone for good, the per-account policy table (xoct proxy-mode --all, readable on the active — the mode, deadline and peer carry to both ends) tells you which clients were promised that the old address would keep working; those clients need the DNS move now, not at their own pace. The withdrawal notice exists for exactly this: xoct proxy-retire.

Expect: minutes of impact for CDN-fronted names, up to one old TTL for direct-DNS names, and no data risk.

Replication is broken while both boxes are up

Impact: none that visitors can see — but you have no failover target right now. Treat it with the urgency of running without backups.

Read both ends (the First-moves block). Two shapes:

• A replica thread has stopped (Running: No, an Error field set): the mirror is dead weight until rebuilt. If the error says the replica is ahead of the master (it has transactions the master does not), the pair has diverged — rebuild the mirror from scratch (sequence below). For any other error, the safe generalist route is the same rebuild; init re-snapshots the active and re-attaches the replica cleanly.
• Lag keeps growing: the pair is intact but the mirror cannot keep up. A planned switch in this state would wait out the full lag drain (up to _XMASS_SYNC_MAX_WAIT, default 7200 s) and then refuse. Find the write source first — the cutover's own lag report distinguishes a closing trend from a growing one.

A mirror is a working BOA box — cron runs, the gates carry the passivity. Since the per-job standby design, a mirror keeps its cron armed for the whole window: the IDS battery, log rotation, the sensors and every service watchdog run exactly as on any BOA box, and system-layer upgrades and kernel reboots proceed as normal. What stands down is precisely the passivity-breaking set, each behind the durable role marker (/root/.standby.cnf):

• The database refuses every local write. The replica runs super_read_only — persisted as the xmass-standby-hold block in xmass_gtid.cnf (so it survives every mysqld restart and reboot) and set live at init. The replication appliers are exempt by definition; everything else, root included, gets refused at the server. The mysql watchdog re-asserts it every minute, converts a standing mirror built by older bytes the same way (it appends the block and locks the runtime — fleet delivery alone retrofits the estate), and releases it once the marker is gone — verifying the runtime unlock landed BEFORE it strips the block, so a failed unlock keeps its own retry key.

• The web tier is held DOWN, three layers deep. The start() gate in the shipped nginx init script refuses service nginx start/restart — including the boot rc links, which no cron enforcer can cover; the per-minute enforcer takes down anything that slipped through and keeps the BOA_STANDBY_WEB firewall chain asserted (IPv4+IPv6, DROP 80/443, loopback exempt); csfpost.sh re-adds the chain the moment any csf -r flushes it. A mirror answering HTTP would write sessions, caches and watchdog rows into its own panel DB — the measured cost was 13 errant own-UUID GTIDs from a single GET.

• FTPS is dead and its healer stands down. An authenticated upload would land in the synced trees and permanently shadow the active's copy under the -u sync legs.

• Tenant logins are denied. Every lshell/mysecureshell account flips to nologin with live sessions killed; the flip is recorded in /var/log/boa/standby-held-shells.txt, and promotion restores exactly the recorded users — each restore verified, failures kept for retry, so an admin-suspended account is never resurrected by mistake. Accounts born on a held mirror (the user store syncs from the active) are created nologin and recorded the same way.

• The whole backup chain is gated. mybackup, multiback, backboa and duobackboa exit quietly on the marker — the active owns the lineage, and a mirror chain's database dumps would in any case be as old as the last sync while their timestamps said today.

• The task runner holds the queue and parks the panel's own per-minute dispatch (restoring both the moment the role ends); the night worker, the Solr core manager, the batch guard, the binlog purge, the mysqlcheck repairs and the symlink fixer all stand down on the same marker.

• The SQL backup and cleanup tools refuse on the marker AND on any configured replica — the probe is exit-code-aware and fails closed, so broken or under-privileged credentials refuse rather than proceed; the replica-state half protects even hand-built replicas.

• A full barracuda up-* on a mirror is steered to its system flavor (the Ægir-master half writes into a replicated database and resumes at promotion); the enable/disable flag verbs are refused loudly rather than steered (the flag change would be silently dropped otherwise), the system flavor skips the 5.7 mysql-schema heal (binlogged DDL that would mint errant transactions on the replica), and the master-upgrade arm carries its own role gate as a backstop; up-* aegir and all octopus work refuse until promotion. Solr is held by a durable disarm the per-minute watchdog re-asserts — it survives reboots and upgrade passes.

• The usage accounting job, the tenant self-service restart triggers (run-nginx-restart.pid, run-php-fpm-reload.pid, run-valkey-restart.pid — those files arrive from the active with the synced static/control tree and belong to it), the PRO remote-backup command queue, the wiped-config self-heal's upgrade arming and the forced Let's Encrypt re-registration arm all stand down on the same marker — each is either replicated state the active owns or a writer into the synced trees.

After a reboot the mirror simply comes back as that same held box: cron up, replication reconnecting, the DB read-only straight from the cnf, nginx refused at the boot rc links by the init-script gate, the firewall hold re-added as csf starts, tenant shells still nologin. Nothing re-quiesces and nothing needs enforcing by hand:

# On the MIRROR after a reboot — verify
pgrep -x cron                            # expect cron RUNNING
mysql -e 'SHOW REPLICA STATUS\G' | grep -E 'IO_Running|SQL_Running'
mysql -e "SHOW VARIABLES LIKE 'super_read_only'"   # expect ON
pgrep -f 'nginx: master' || echo web-held           # expect web-held
iptables -w 5 -nL BOA_STANDBY_WEB | head -3         # expect the DROP chain
test -e /root/.standby.cnf && echo role-held
tail /var/log/boa/standby.quiesce.log    # the marker watchdog's record

Deliberately serving from a mirror (content preview, a read-only DR exercise): touch /root/.standby.serve.cnf and the web tier opens within a minute — nginx heals up and the firewall hold lifts — while the DB, tenant, FTPS and backup holds all STAY. It is a preview, not a promotion; the enforcer logs one line an hour while the marker exists so it cannot be forgotten silently. Remove the file to re-hold the web tier.

xmass init refuses a target whose BOA tools pre-date the per-job gates (they carry the protection now); every live xmass sync restores a missing marker.

The marker is removed automatically when the mirror is promoted (cleared before the cutover's own cron start — a belt for a box quiesced by pre-gate BOA bytes — and the cutover verifies the removal landed). Every hold releases with it: the cutover itself unlocks the DB (step 11.5, re-entrant on every resume, runtime verified before the cnf block is stripped) and opens the firewall before proving the web tier EXTERNALLY (a browser UA, from the source — the path client traffic takes after the DNS flip); the watchdog layers release the rest within about a minute — which is also the complete story for a promotion done by nothing more than removing the marker on a box whose source is gone. On a box that probes as definitively no replica — a clean probe with empty output — the watchdog removes a leftover marker itself and logs the removal; that is what revives the task queue within about a minute on a box promoted by hand after losing the source (the one promotion the cutover-side removal can never reach). A probe that errors (broken /root/.my.cnf credentials are the usual cause — a source-side root-password rotation replicates into the standby while its client config stays the init-time copy) proves nothing about the role: the watchdog then takes no action either way and logs role probe FAILED (credentials?) — treat that line as an alarm and fix the credentials, because none of the credential-dependent guards can see the box's role until you do.

Two deliberate scope notes: the nightly mysql_backup.sh carries the same markerless replica guard as the hourly cleanup (it performs the same class of local writes), while mysql_cluster_backup.sh does not — it targets the cluster's designated write node, so its writes replicate correctly by design. And a proxy-shaped replica target (an ha-switch failback leg: the demoted former source, finalized as a PX0 proxy, still relaying production traffic) keeps its cron armed throughout — its DB writers all exit on /root/.proxy.cnf already, and a live relay needs its certificate mirror and watchdogs.

Expect: no visitor impact; protection is restored when both threads read Yes and lag returns to 0s.

Planned switch to the mirror

This is the proven path, and the one the ha-switch proxy mode exists for. The whole machinery is the ordinary xmass cutover — a failover is a whole-server migration whose target happens to be the standing mirror.

Preconditions — check, do not assume:

• The pair is healthy: phase syncing, lag 0s, both replica threads Yes (First-moves block).
• boa info reports the same BOA release on both boxes. A mirror keeps its cron armed, but a release upgrade is still per box: a run made only on the active leaves the pair apart. The release gate is enforced when the pair is built, but nothing re-checks it at cutover — this check is yours. The remedy is a full barracuda AND octopus run on the older box; on a mirror that pass is steered to its system flavor — the half the release stamp comes from — while the Ægir and octopus halves resume at promotion.
• Both boxes run the same Percona patch level, not just the same series. init checks the series from the client binary and then asks each server for its own version, so a box whose packages moved on since the pair was built is caught here rather than mid-restore. Unequal levels are refused: where the target is the newer side the refusal names _XMASS_ALLOW_PATCH_SKEW=YES as a deliberate waiver, and where the target is the older side it is refused outright — restoring a newer datadir onto an older server is an unsupported downgrade and no flag accepts it. An unreadable server version on either end also aborts: fix the credentials or connectivity and re-run. The remedy for a real difference is to align the packages on both ends from one repo snapshot.
• Every account's proxy mode is declared. The DRY cutover prints exactly which accounts --live would refuse; declare with xoct proxy-mode <oN> <mode>.
• Proxy trust is pre-staged between the proxy box and the mirror: the proxy's root SSH key in the mirror's authorized_keys, and CSF allow AND ignore entries in both directions. Stage this at mirror-build time, while everything is healthy — the retarget refuses without it, and until it is fixed, traffic keeps chaining silently through the box you are switching away from.

The sequence:

# On the ACTIVE server, inside screen/tmux (the tool refuses outside one)
xmass cutover <mirror-ip>            # DRY — must end CLEAN; it arms --live
xmass cutover <mirror-ip> --live

# On the PROMOTED mirror, once the cutover completes
xmass post-mig

# On the proxy box, one line per account — point the relay at the new master
xoct proxy <oN> <mirror-ip> --repair --retarget

# On the promoted mirror
bash /var/xdrago/migration_proxy_trust.sh reconcile

After the per-account loop, sweep for stragglers: the peer column of xoct proxy-mode --all must name the new master for every account, and xmass verify <mirror-ip> on the proxy must pass. A half-retargeted account is also caught by the proxy's daily certificate mirror, which prints the exact --repair --retarget line to run. Note that a successful retarget mails each account's arrangement notice by default — --no-notify suppresses it (and logs the suppression), --renotify forces a re-send.

On a CDN-only front there is no proxy box: replace the retarget step with the CDN origin change for each zone, and treat it as a numbered step of the switch. If the proxy box happens to be down at switch time, the same applies — switch the front by DNS/CDN straight at the promoted box, and owe the retarget before the proxy is ever allowed back into the path.

What the cutover does, in order: 503-gates every account on the active, drains replica lag to zero, freezes writes durably, promotes the mirror, renames the Ægir host on it, and finally converts the old active into one more proxy hop relaying to the new master — the per-account mode records carry over unchanged (at the conversion step the tooling prints a reminder that it cannot itself verify a live mirror exists behind an ha-switch promise). Aborts before the promotion unlock and thaw the source themselves; aborts after it park resumably with the source deliberately kept frozen, and the printed recipe leads with the resume line. One edge case does neither: a promotion whose outcome cannot be read back leaves the source frozen on purpose — downtime is recoverable, lost writes are not — and hands you the decision. Follow the printout rather than reconstructing it. Details: state machine and recovery.

Expect: the downtime-table row below — a 503 window, not an outage.

Failing back to the original server

Failback is not a special mode — it is the same chain run in the opposite direction, from the current active (the promoted mirror) toward the returning original. Documented in full in the xmass failback section; the shape:

# On BOTH boxes, the same argument — the CURRENT active's own hostname
xmass pre-mig <current-active-fqdn>

# On the CURRENT ACTIVE (the box being demoted)
xmass prep-target <original-ip> --fix-php
xmass init <original-ip> --proxy-mode=ha-switch
xmass sync <original-ip>             # DRY, then:
xmass sync <original-ip> --live
xmass cutover <original-ip>          # DRY, then --live, inside screen/tmux

# On the RE-PROMOTED original
xmass post-mig

# On the proxy box, per account — point the relay home again
xoct proxy <oN> <original-ip> --repair --retarget

Clear this before you start:

• The failback reverses which box is the target, and with it which side the patch-level gate will refuse. The returning original is now the target, so a mirror that has moved to a newer Percona patch makes the returning box the older side — the arm init refuses with no override. Align the packages from one repo snapshot before the failback leg; _XMASS_ALLOW_PATCH_SKEW=YES waives only the opposite case, a target newer than the source.

Expect these one-time speed bumps — they are normal, not faults:

• init refuses once on the box's own state file left over from its previous role. Back it up and remove it, then re-run — this is the documented state lever, needed on every direction change: cp /data/conf/xmass_state.cnf /root/xmass_state.bak && rm -f /data/conf/xmass_state.cnf
• init may restart MySQL once on the current active: a box that spent time as a replica often has GTID on but its binlog off, and the master-readiness gate proves both and self-heals by restarting.
• The replication identity set grows by one on every promotion. Health-check that the replica's executed GTID set contains every identity from the master's backup — never expect an exact count.
• The re-promoted box wears its demotion: its own stale policy record, proxied.pid, parked dispatcher and leftover proxy vhosts are cleared automatically by the cutover before the rename.

Then re-arm steady state: rebuild the mirror from the re-promoted active using the sequence below — with the same one-time state lever at its init.

The mirror is down — and rebuilding one

A mirror outage has no visitor impact; you are simply running unprotected until it is back. If it merely rebooted, replication resumes by itself (GTID auto-positioning) — confirm both threads and falling lag, and confirm the box came back as the working standby it is: pgrep -x cron expects cron running, and test -e /root/.standby.cnf expects the role marker that holds the per-job gates. Nothing re-quiesces it — see the replication scenario above. If the mirror is gone, or replication has diverged, rebuild it:

# On the ACTIVE server — and the SAME command with the SAME argument on the
# mirror box (the argument is the ACTIVE's own /etc/hostname value; the tool
# derives each box's role from whether the argument matches its own hostname)
xmass pre-mig <active-fqdn>

# On the ACTIVE server
xmass prep-target <mirror-ip> --fix-php
xmass init <mirror-ip> --proxy-mode=ha-switch
xmass sync <mirror-ip>               # DRY, then:
xmass sync <mirror-ip> --live
xmass status <mirror-ip>             # lag 0s; then the thread check on the mirror

The ha-switch mode shapes the client notification and keeps the proxy trust permanent — nothing in it creates or checks the mirror itself. The mirror is real because you built and monitor it, not because the mode says so.

What downtime to expect

Two caveats behind the table. The 503 maintenance page the cutover raises is cacheable (its max-age can be as long as the gate's TTL, up to 7200 s) — behind a CDN, confirm the edge is not caching it beyond the switch window. And the recovery point of an emergency promotion is the un-replicated tail at the moment of failure; only the planned switch, which drains lag to zero and triple-checks it under a write freeze, moves with zero loss.

Verifying after any switch

# On the box now relaying (the old active, or the long-standing proxy)
xmass verify <current-active-ip>

Pass the new master's IP explicitly — without it the tool falls back to the target recorded in that box's own state file, which after an HA switch is the box you just switched away from, and the check silently validates the wrong origin.

Then the standing rules: probe direct and through every hop with curl --resolve, twice and at least 5 s apart; judge by content and data — the verification checklist proves a switch by markers and row counts, not status codes; and run pgrep -x cron on every box that should be live — the relaying box needs cron for the daily certificate mirror that keeps its HTTPS fresh, and a promoted box without cron silently stops self-updating.

Related

• Cross-host migration — the shared machinery — the write freeze, storage gate and migration proxy every scenario relies on.
• xmass — whole-server move — the cutover sequence, state machine, proxy modes and failback in full detail.
• xoct — account-level move — per-account moves, --repair --retarget, and the policy verbs.
• PX0 proxy & shrinking — what a long-lived proxy box is, what keeps running on it, and what xtrim removes.
• Percona versions & verifying — the before/after proof checklist used by every scenario here.
• Troubleshooting — symptom-first pages for failures that are not topology events.
• See the Reference appendix for the consolidated variable, command, and control-file tables.