multiback operations — dcysetup, config tree, cron

The operator surface for the modern backup suite is dcysetup (install + config generation) plus the /root/.remote_backups/ tree it materialises. There is no /etc/boa/multiback.conf, no /etc/boa/backup-config.env, and no BACKUP_* variables — those do not exist. Everything lives under /root/.remote_backups/ and /root/.barracuda.cnf.

dcysetup — install and setup

dcysetup (/opt/local/bin/dcysetup) takes exactly one of three actions (update is an alias for setup):

dcysetup install     # install/upgrade Duplicity + Python + deps (idempotent)
dcysetup setup       # generate config tree, credential templates, secret, cron
dcysetup update      # alias for setup

dcysetup is licence-gated (_verify_boa_keys, tier pro or dev); on a host that fails the check it exits before doing anything. install stops cron, runs install_dependencies.sh, restarts cron. setup runs seven numbered steps from /root/.remote_backups/run/:

create_global_paths_config.sh — write global_paths.txt, data_paths.txt, custom_paths.txt; generate the root secret.
create_user_paths_config.sh — per-tenant paths.txt.
create_credentials_templates.sh — one <service>.txt template per backend.
create_cron_entries.sh — write the wrapper, the schedule, the cron entry.
create_readme.sh — global README files.
create_config_readme.sh — per-user config README files.
install_dependencies.sh --patch-only — apply the Duplicity b2 backend patch when the pinned venv lacks it (a silent millisecond no-op once applied; non-fatal by design, so a deferred patch never aborts the config regeneration — see Retention).

The config tree

Root-side (global / data / custom path-sets):

TXT

/root/.remote_backups/
  .secret.txt                       # Duplicity PASSPHRASE (chattr +i, mode 600)
  credentials/<service>.txt         # per-backend secrets + retention vars
  paths/global_paths.txt            # _SOURCE / _INCLUDE_PATHS / _EXCLUDE_PATHS / _INCLUDE_LIST / _EXCLUDE_LIST
  paths/data_paths.txt
  paths/custom_paths.txt
  paths/.backboa.*                  # generated include/exclude fragments + merge state
  run/                              # the six setup scripts + sequential_backups.sh wrapper
  schedule/backup_schedule.txt      # "<service> <user>" lines the wrapper iterates
  logs/                             # OK-<bucket>.log / ERR-<bucket>.log per-run copy

Per-tenant:

TXT

/data/disk/<user>/remote_backups/
  .secret.txt
  paths/paths.txt
/data/disk/<user>/static/control/remote_backups/
  credentials/<service>.txt
  credentials/README.txt            # credential-format self-help
  config/                           # tenant include/exclude directive files
  config/README.txt                 # directive-syntax self-help
  logs/

<service> is one of the eleven backend keys: aws, aws_one_zone, aws_standard_ia, azure, b2, cloudflare, do_spaces, gcs, ibm, linode, wasabi.

The tenant config/ directory (created by create_config_readme.sh, mode 700) is the one surface where a hosted user shapes their own backup without root. It holds up to four optional directive files — include.txt and exclude.txt (plain --include / --exclude lines) plus include_regexp.txt and exclude_regexp.txt (--include-regexp / --exclude-regexp patterns). On the next dcysetup setup, create_user_paths_config.sh validates each line and merges the valid ones on top of that tenant's default set: a line is exactly one directive, one space and one path or pattern in a restricted character set, absolute and inside the tenant's own static/ tree or their ~USER.ftp home; an invalid line is skipped and logged to /var/log/backup_validation_issues.log while the rest of the file and the built-in defaults still apply. Both README.txt files are auto-generated self-help the user reads for the exact credential and directive formats.

Credential files

create_credentials_templates.sh writes a per-backend template named exactly <service>.txt — so Cloudflare R2 is cloudflare.txt and DigitalOcean Spaces is do_spaces.txt (not r2.txt / do-spaces.txt). Each template is mode 600 and carries the backend's own variable names plus the two retention vars. Example, cloudflare.txt:

export R2_ACCOUNT_ID="your_account_id"
export R2_ACCESS_KEY_ID="your_access_key_id"
export R2_SECRET_ACCESS_KEY="your_secret_access_key"
export KEEP_WITHIN="3M"
export FULL_BACKUP_FREQUENCY="28D"

A credential file is only scheduled once its your_ placeholders are replaced — create_cron_entries.sh skips any file still matching your_. Each backend has its own variable set: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION; B2_ACCOUNT_ID / B2_APPLICATION_KEY; AZURE_STORAGE_ACCOUNT / AZURE_STORAGE_KEY; DO_SPACES_KEY / DO_SPACES_SECRET / DO_SPACES_REGION; GCS_ACCESS_KEY_ID / GCS_SECRET_ACCESS_KEY (HMAC interoperability keys); IBM_ACCESS_KEY_ID / IBM_SECRET_ACCESS_KEY / IBM_REGION (HMAC service credentials); LINODE_ACCESS_KEY / LINODE_SECRET_KEY / LINODE_REGION; WASABI_ACCESS_KEY / WASABI_SECRET_KEY / WASABI_REGION.

_validate_credentials reads each file line-by-line, strips export, rejects any value containing shell metacharacters, and (for b2 only) URL-encodes the value — b2 is the one backend whose key pair rides inside the Duplicity target URL, where a reserved character would corrupt the URL; every other backend authenticates through environment variables, where an encoded value would break auth, so those values stay verbatim. A malformed line is logged to /var/log/backup_validation_issues.log and skipped — one bad entry does not abort the run.

The paths.txt variables

A paths.txt is a plain shell file sourced by multiback before each run. The variables carry a leading underscore (a name without it is read by nothing):

Variable	Role
`_SOURCE`	space-separated absolute paths emitted as `--include` directives
`_INCLUDE_PATHS`	merged `--include` directive string (pre-built)
`_EXCLUDE_PATHS`	merged `--exclude` directive string
`_INCLUDE_LIST`	path to a `--include-filelist` file
`_EXCLUDE_LIST`	path to an `--exclude-filelist` file

global_paths.txt ships, for example, _SOURCE="/etc /opt/solr4 /var/aegir /var/solr7 /var/solr9 /var/www /var/xdrago". The data set ships an empty _SOURCE and a generated per-tenant include set. To change what is captured, edit the relevant paths.txt; the next run picks it up.

Two rules govern the paths you write into these directives (and into the --include / --exclude filelists). Every path must be a full absolute path beginning with / — a ~/ home shortcut or a relative form is never expanded, so it silently matches nothing and is ignored; write /root/projects, not ~/projects. (This is the opposite of the restore RESTORE_PATH argument, which is absolute without a leading slash — see CLI reference.)

Exclude wins over include. multiback assembles the Duplicity command with every --exclude ahead of every --include (… ${_BATCH_EXCLUDE} ${_BATCH_INCLUDE} --exclude '**' /), and Duplicity honours the first rule that matches a path. A path named in both an include and an exclude is therefore dropped before the include can re-add it — an exclude cannot be rescued by a later include, so order your own edits with that in mind.

How secrets reach Duplicity

The encryption passphrase is never in a credential file or the target URL. _load_credentials reads it from the secret file into the PASSPHRASE env var:

root-side: /root/.remote_backups/.secret.txt
tenant: /data/disk/<user>/remote_backups/.secret.txt

create_global_paths_config.sh generates the root secret with openssl rand -base64 32, sets mode 600, and chattr +i (immutable). A missing or empty .secret.txt aborts the run (exit 1). Provider secrets reach Duplicity through environment variables for every backend except b2 — the S3 family as the standard AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY pair, Azure as AZURE_STORAGE_KEY — with only the bucket (and, for Azure, the account name) in the target URL; b2 alone carries its percent-encoded key pair inside the b2:// target URL (see Regions & buckets). Lose the passphrase and the archives are unrecoverable — back up .secret.txt out of band.

Upgrade note (5.10.1). An earlier security-audit "hide credentials" change moved every backend's secrets out of the target URL at once; that broke Duplicity (the B2 backend has no backend-specific key env var, and that change's S3 endpoint form was built wrong), so the in-URL form was restored at the time. The current tools complete the move properly for the S3 family alone — the env-var pair plus --s3-endpoint-url / --s3-region-name, as above — while b2 keeps its working in-URL form. If a host misbehaves after a partial update, re-fetch multiback (barracuda up-<tier> system) and re-test.

Handing a tenant their passphrase

A tenant who wants server-independent access to their own off-site backups (the customer-facing walkthrough is Disaster-proof access) needs a copy of their .secret.txt. Copy, never move — the original is immutable (chattr +i) and the scheduled backups depend on it — and land the copy in the tenant's already-hardened credentials directory with the tenant as owner:

cp /data/disk/<user>/remote_backups/.secret.txt \
  /data/disk/<user>/static/control/remote_backups/credentials/.secret.txt
chown <user>.ftp:users \
  /data/disk/<user>/static/control/remote_backups/credentials/.secret.txt
chmod 600 \
  /data/disk/<user>/static/control/remote_backups/credentials/.secret.txt

The tenant downloads it over SFTP and deletes the copy; if it lingers, remove it for them — the copy is a convenience hand-off, not a permanent resident.

Email notifications

Per-run summaries are sent by s-nail to _MY_EMAIL (from /root/.barracuda.cnf) when _MY_EMAIL is set and _INCIDENT_REPORT is not OFF. _INCIDENT_REPORT (normalised to OFF/ALL/MINI/CRIT, default MINI; legacy NO becomes OFF, YES becomes MINI) controls verbosity — ALL also emits waiting-queue and validation-alert mails. There is no BACKUP_EMAIL variable.

The freshness alert is the one that says backups have stopped. Per-run failure mail is easy to miss in a busy inbox, so the engine additionally raises one distinct, daily-repeated Backup FRESHNESS Alert per scheduled service once no successful run has landed past an interval-scaled floor, judged from the per-service OK-/ERR- sentinel files in the logs/ directories. A fresh ERR with a stale OK means runs happen and fail (dead credentials after a key rotation, an unreachable bucket) and alerts on the early floor (5× _BACKUP_INTERVAL, minimum 30 h); nothing fresh at all means runs stopped entirely (dead cron, wedged wrapper) and alerts on a longer floor (8× interval, minimum 48 h) that clears a monthly full spanning several cycles. It never false-alarms by design: a box with no schedule, a standby mirror, a service that has never produced any sentinel, and — because the wrapper walks the schedule strictly one entry at a time — any service on a box where a backup stream is currently live all stay silent. That last gate is box-wide on purpose: an entry whose turn has not come yet in a running pass is pending, not stalled, and judging it by its own idle PID file alone produced a round of alerts for tail-of-schedule entries that then completed within the hour. The suppression is bounded, though: a stream still holding its lock after four cycles is stuck rather than busy, and earns its own daily Backup STALLED Alert naming it, while the entries queued behind it stay quiet so the cause is reported once instead of as a list of symptoms. Alerts also append to /var/log/boa/backup-freshness.log regardless of mail settings, and the notice repeats daily until a run completes.

The cron path — interval-based, not nightly

create_cron_entries.sh writes an interval cron, not a fixed nightly slot. It emits /etc/cron.d/duplicity_backup:

TXT

0 */$((_BACKUP_INTERVAL/60)) * * * root bash /root/.remote_backups/run/sequential_backups.sh

With the default _BACKUP_INTERVAL=360 (minutes) this is 0 */6 * * * — every six hours. _BACKUP_INTERVAL is overridable in /root/.barracuda.cnf; the script also auto-widens the interval if the last total runtime (/var/log/backup_runtime.log) would not fit. The bash prefix and the absolute path are deliberate: the entry must not depend on the wrapper's executable bit or on PATH. create_cron_entries.sh does chmod +x the wrapper when it writes it, but a mode lost to a later hand copy would otherwise fail silently. The wrapper sequential_backups.sh iterates schedule/backup_schedule.txt (one <service> <user> line per configured backup) and calls multiback backup <service> <user> for each, one at a time, with per-run PID files in /run/duplicity_<service>_<user>_sequential.pid (plus multiback's own /run/duplicity_<service>_<user>.pid one level down).

The locks self-heal. A leftover pid file whose recorded process is gone — or was recycled by an unrelated process (the liveness check matches the pid's command line, not just its existence), or is empty — is removed and re-claimed atomically; a live run's lock is never touched; and an interrupted run cleans its own lock on the way out (INT/TERM and normal exit alike), so a killed run no longer blocks every later cycle until an operator removes the file by hand.

A slow service never starves the schedule. A service still running from an earlier cycle — a monthly full can span many 6-hour cycles — is skipped for that cycle while the rest of the schedule proceeds, bounded by a two-stream cap: when two backup streams are already live box-wide, the remaining entries defer to a later cycle instead of stacking a third concurrent full. (Previously a single held lock aborted the whole wrapper, so one long full silently suspended every other configured backup for its entire runtime.)

There is no 3:01 AM nightly entry and no Sunday-weekly full entry — those belong to the separate legacy crontab path (see Legacy backboa).

To inspect the live schedule:

cat /etc/cron.d/duplicity_backup
cat /root/.remote_backups/schedule/backup_schedule.txt

Manual invocation

Every multiback action is <action> <SERVICE> <USER> (plus restore positionals). Full verb table in CLI reference:

multiback test wasabi data       # verify connectivity for the data path-set
multiback backup b2 data         # run a backup pass now
multiback cleanup aws john       # apply KEEP_WITHIN retention now
multiback list b2 jane           # list current files in the set

Retention — the two retention variables in detail.
Regions & buckets — bucket names + target URLs.
CLI reference — full verb + argument reference.
Reference appendix — _MY_EMAIL and _INCIDENT_REPORT in the variable index; the remote_backups/ path, credential and schedule files in the control-file index.