Backup subsystem overview & architecture

A BOA host carries two generations of off-site backup tooling, all built on Duplicity and all installed together by BOA.sh.txt. They differ in config surface, bucket scheme, provider coverage, and whether they are licence-gated.

The orchestrators

dcysetup is the installer/wiring tool for the multiback suite; it is also licence-gated. backboa and duobackboa carry no _verify_boa_keys call, so they are physically present and runnable on every host regardless of tier — "LTS-only" is a usage recommendation, not an enforced restriction. The practical split is: multiback/dcysetup are the modern, licence-gated, multi-provider path; backboa/duobackboa are the simpler single-bucket AWS-S3 legacy path.

Supported providers

multiback and mybackup target nine storage providers across the eleven backend keys enumerated in multiback operations — Amazon S3 is one provider but carries three storage-class keys (aws, aws_one_zone, aws_standard_ia), which is how nine providers become eleven keys. The key whose credential file you fill in under /root/.remote_backups/credentials/ picks the destination; Regions & buckets covers the per-key target-URL and bucket-name mechanics. The two tables below compare the providers on capability and on cost to inform that choice.

The figures below are a point-in-time snapshot (as of 2026-07) and drift — storage and egress pricing especially. Verify the current numbers against each provider's own pricing and product pages before committing to a backend.

Capability comparison

Cost comparison

Why Duplicity

All four tools shell out to /usr/local/bin/duplicity (pinned 3.2.0.2, Python 3.14.6 for all four tools — dcysetup install and backboa install pin the identical versions; built/installed by dcysetup or backboa install). Duplicity provides:

• Encryption before upload. multiback/mybackup use a symmetric passphrase read from .secret.txt into PASSPHRASE; backboa/duobackboa use _AWS_PWD into the same env var. Some backend paths run --no-encryption (the custom path-set on hosted systems).
• Full + incremental chains. _set_mode picks full vs incremental per run from cache state and whether today's full-log already exists — not by reading FULL_BACKUP_FREQUENCY — never by a CLI verb. The full cadence is enforced separately by Duplicity's --full-if-older-than ${FULL_BACKUP_FREQUENCY}, embedded in the default backup command (_DCY_BUP_CMD). On hosted systems the global/data path-sets instead use _FBF_BUP_CMD, which omits --full-if-older-than and forces a full only on the first of the month (_DOM=1).
• Point-in-time restore via --time / --path-to-restore against a volsize-300 archive.

The Duplicity invocation runs at --concurrency scaled by core count (_useCpu = 1/2/4 for up-to-4 / up-to-8 / over-8 CPUs) and --volsize 300.

What multiback actually captures

multiback does not back up a hard-coded path list. It backs up whatever the selected path-set (<user> argument) resolves to, sourced from a paths.txt file via _load_paths. The root-side path-sets are generated by dcysetup setup (create_global_paths_config.sh):

• global — _SOURCE="/etc /opt/solr4 /var/aegir /var/solr7 /var/solr9 /var/www /var/xdrago" plus generated include globs for /data/disk/arch, /var/backups/csf, /var/backups/dragon, /var/backups/reports, and an include-regexp for /root/.*.cnf.
• data — empty _SOURCE; an auto-generated per-/data/disk/<user> include set, which also appends --include /data/all, --include /data/conf and --include /home. It excludes each tenant's .tmp, backup-exports, backups, clients, src, u, undo, static/{.tmp,restores,tmp,trash}, each /home/<user>.ftp account's own .tmp, backups, clients, platforms and static, plus /var/www. The excludes are written as literal paths (/data/disk/<user>/backups, /data/disk/<user>/backup-exports); there is no exclude for static/files/.backups or static/files/.backup-exports, so on a relocated account the excluded names are only symlinks while the real tarball stores sit inside the included tree — one more reason to read the generated paths.txt.
• custom — operator-supplied include/exclude lists only.

A per-tenant mybackup run uses /data/disk/<user>/remote_backups/paths/paths.txt.

Because _SOURCE and the include/exclude sets are config-driven, the exact captured set is host-specific; read the generated paths.txt to know what a given host backs up, rather than assuming a fixed list.

Per-account backup directories may be symlinks

Ægir's per-site backup tarballs land in /data/disk/<account>/backups (Provision derives backup_path as <aegir_root>/backups) and backup-download exports in /data/disk/<account>/backup-exports; the nightly per-account run re-asserts aegir_backup_export_path to <account>/backup-exports right after running the relocation check below.

Post-5.10.3, when an account's static/files resolves to a different device than the account root (stat -c '%d' on the account vs stat -L -c '%d' on static/files), the nightly per-account run moves both directories onto the static filesystem and replaces them with symlinks:

/data/disk/<account>/backups         -> /data/disk/<account>/static/files/.backups
/data/disk/<account>/backup-exports  -> /data/disk/<account>/static/files/.backup-exports

Backups of native-symlinked sites dereference the files/private symlinks, so the tarballs are large and would otherwise fill the root partition; on a default single-filesystem box the device check makes this a deliberate no-op. The Ægir paths keep working through the symlinks, and both stores are deliberately kept on one filesystem so the backup-download hardlinks between backups and backup-exports keep working — hardlinks cannot cross filesystems. The leading-dot store names are skipped by the site/orphan scan.

To check whether an account is relocated: readlink /data/disk/<account>/backups — empty output means a plain directory, not relocated. Kill-switches: box-wide /data/conf/disable_backups_on_static_fs.cnf, per-account /data/disk/<account>/static/control/no_backups_on_static_fs.info.

The migration mechanics — the incremental rsync --remove-source-files move, the /run/.boa_backups_relocate.flock serialisation, and the /run/boa_queue_stop.pid task-queue pause — are owned by Backups on the static filesystem.

The local-dump layer underneath

Off-site backup archives a host that already has local MySQL dumps in place. mysql_backup.sh (cron 15 1 daily) writes per-DB dumps to /data/disk/arch/sql/<host>-<date>/<db>/ — mydumper by default (--sync-thread-lock-mode=AUTO, forced FTWRL on Percona 5.7), with the mysql system schema always taken by mysqldump --routines --events. Those dumps live under /data/disk/arch, which the global path-set includes — so the off-site copy carries the local dump tree. Local-dump retention (_DB_BACKUPS_TTL, default 14 days) is independent of off-site retention. See Database.

Known failure — backup binary permissions

/usr/bin/mysqldump and /usr/bin/rsync must be mode 755 so the unprivileged aegir/Octopus user can execute them. A transient 700 (or 750) on these binaries leaves them unexecutable for that user and breaks all Octopus/Ægir per-site backups with mysqldump: Permission denied. autoupboa restores 755 in normal operation and deliberately sets 700 only on a restricted VM — when /root/.restrict_this_vm.cnf is present (killall -9 rsync; chmod 700 /usr/bin/rsync /usr/bin/mysqldump). If site backups fail with a permission error, check these two modes before anything else. See Troubleshooting.

Concurrency and gating

No tool runs two Duplicity chains in parallel, but they differ in what they do when the slot is busy. multiback, mybackup and backboa abort immediately on a positive pgrep -fc duplicity (multiback logs to /var/log/mybackup_waiting_queue.log and, when _MY_EMAIL is set and _INCIDENT_REPORT=ALL, emails a waiting report). duobackboa backup instead waits: it counts as busy either a live Duplicity or a live backboa process, identified by the real PID backboa writes into /run/<host>_backboa.pid, so even the gap between backboa's sequential Duplicity invocations holds the gate. It logs that it is waiting to the same queue log, then polls every five minutes for up to two hours — proceeding as soon as the slot frees, or giving up until the next scheduled run — which is what keeps the 03:15 backboa run, when it overruns, from costing the 05:15 duobackboa run its day. Every other duobackboa verb keeps the immediate refusal, so an operator restore never blocks for hours.

The three root-side tools — multiback, backboa and duobackboa — and mysql_backup.sh additionally honour /root/.pause_heavy_tasks_maint.cnf (exit 0 immediately) and abort when root disk usage is over 90%. mybackup carries neither gate: it keeps the Duplicity-concurrency check above, but neither the tenant-facing queue front nor the root-side consumer that clear.sh launches to drain those queues is held back by the pause marker or by a full root filesystem. On the same clear.sh pass the dcysetup call beside it does honour the marker, so during a pause the wiring step is skipped while the restore consumer still starts.

The pause marker has no /root/.barracuda.cnf variable form and deliberately keeps none — it is a transient latch, and removing the file is what resumes the paused work. It is not only a polite skip, either: while it exists, each autoupboa pass — one roughly every five minutes — also runs killall -9 mysqldump and killall -9 rsync, which is precisely why a sticky configuration line would be the wrong shape for it.

mysql_backup.sh additionally short-circuits on /root/.proxy.cnf and on the replication-standby role marker /root/.standby.cnf, and refuses on any box that is a configured replica — its TRUNCATE/DROP/OPTIMIZE housekeeping would stop the SQL thread under ROW binlog. That last check is markerless, so it protects hand-built replicas too, and it fails closed: a probe that errors — broken or under-privileged credentials — refuses rather than proceeds, and only a probe that runs cleanly and returns nothing counts as "not a replica". While mysqld is still unreachable the role cannot be judged at all, so the check defers there and is asked again once the daemon is up. mysql_cluster_backup.sh takes the role marker but deliberately no replica gate: it targets the cluster's designated write node, so its writes replicate correctly by design.

The duplicity chain is gated the same way: mybackup, multiback, backboa and duobackboa all exit quietly (one log line, rc 0) on a box carrying /root/.standby.cnf. The active owns the backup lineage — a mirror's own chain would duplicate the offsite cost, and under the standby's super_read_only its site-bootstrap steps would only log refusals. The gates release the moment the marker goes, so a promoted box resumes its backups on the next cron pass with no operator step.

Related

• multiback operations — config tree + install.
• Retention — KEEP_WITHIN / FULL_BACKUP_FREQUENCY.
• CLI reference — verbs and argument order.
• Reference appendix — consolidated variable table.