Backup retention policy

multiback/mybackup retention is time-based: after each backup run, anything older than the configured window is removed, while every chain inside the window stays intact. Two variables drive it, both validated and defaulted in _validate_or_default_duration.

export KEEP_WITHIN="3M"             # how long to keep backups
export FULL_BACKUP_FREQUENCY="28D"  # how often to take a new full backup

These live in the relevant paths.txt (or are inherited from a credential file's template). They are not in any /etc/boa/* file — that path does not exist.

KEEP_WITHIN

How long all backup files (full + incremental) are retained.

• Accepted units: M (months) or Y (years).
• Default: 3M.
• A value not matching the months/years form falls back to the default. The validator additionally enforces the M/Y-only rule specifically for KEEP_WITHIN (a days/weeks value is rejected and reset to default).

FULL_BACKUP_FREQUENCY

How often a new full backup is taken; between fulls, runs are incremental.

• Accepted unit: D (days).
• Range: 7D to 60D (enforced).
• Default: 28D.
• Out-of-range or wrong-unit values fall back to 28D.

Both variables default from script-level constants when unset: _DEFAULT_KEEP_WITHIN="3M", _DEFAULT_FULL_BACKUP_FREQUENCY="28D". On the legacy AWS-S3 tools the equivalents are _AWS_TTL and _AWS_FLC — see Legacy backboa.

Time-based cleanup

Age-based cleanup runs remove-older-than:

duplicity remove-older-than "${KEEP_WITHIN}" --force "${_BACKUP_TARGET}"

This deletes every full and incremental older than KEEP_WITHIN, then runs collection-status. It is invoked by the cleanup verb (_cleanup) and by the _weekly_cleanup path inside a backup run.

The weekly cleanup fires only when:

• an archive log exists;
• today is _DOW=7;
• today's cleanup has not already run;
• the cache is warm.

Within the retention window every chain is kept whole, so a restore to any point in the window resolves without reconstruction.

A damaged newest set is pruned first (Duplicity 3.2.0.2)

Under the fleet-pinned Duplicity 3.2.0.2, remove-older-than --force begins by deleting the newest backup set of the newest chain when that set is partial or incomplete — the leftovers of an interrupted upload, never a restorable set; the active chain itself is never deleted. Duplicity announces this only at notice verbosity. Every tool that prunes — backboa, duobackboa, and multiback (its weekly retention pass and the cleanup and repair verbs alike) — runs its deleting legs at a notice floor, so the deletion reaches the per-bucket logs. The purge wipe bypasses the Duplicity CLI and logs its per-file deletions unconditionally (see below). mybackup is restore-only and never deletes remote data.

Chain deletion probes name variants; misses are capped, not slow

Deleting a chain under Duplicity 3.2.0.2 probes several possible remote names per backup set — the per-set signature file is tried in every compression/encryption/partial spelling — so a retention pass always generates missing-file responses on the backend; that is routine, not damage. Stock Duplicity retried every such miss five times with spaced sleeps (its b2 backend ships without the _error_code hook that would classify a missing file as not-found), which stretched one observed weekly pass across multiple backup cycles while it held the backup locks. Two layers keep this fast: the deleting legs run with a capped retry budget (--num-retries 2 --backend-retry-delay 5), and on B2 the installers patch the pinned venv's backend with the missing hook so a not-found response never sleeps between attempts. The patch applies itself: dcysetup update runs it as its final step, and the update path runs from nightly maintenance on every configured box, so no operator action is needed — dcysetup install applies the same patch through the full dependencies run. The one-time Patched duplicity b2backend line in the step output confirms the application; every later pass is a silent no-op.

The capped budget scopes to the whole pruning invocation, including its listing legs, so a backend outage lasting longer than a few seconds fails that pass — the next weekly pass simply catches up, and the failure is loud in the per-bucket log.

To preview a pass without deleting anything, run the same age command by hand with --dry-run appended:

duplicity remove-older-than "${KEEP_WITHIN}" --force "${_BACKUP_TARGET}" --dry-run

It reports every set that falls outside the KEEP_WITHIN window while leaving the bucket untouched — handy for confirming a retention setting before it runs live. This is distinct from the test verb, which runs cleanup --dry-run: a connectivity and broken-chain check, not a retention preview.

purge is a full wipe, not a retention cleanup

The cleanup verb applies KEEP_WITHIN age-based retention. The purge verb is different and destructive: it empties the bucket completely. Duplicity itself has no verb for that — every removal verb keeps the newest chain by construction, and remove-all-but-n-full 0 errors out — so _wipe drives Duplicity's backend layer directly, under the same interpreter, target URL, and credentials the backup runs use: it lists the bucket, deletes every file Duplicity's own naming parser recognizes (foreign files are left untouched), and repeats the deletion until a re-list comes back clean — on a versioning bucket removing the newest version of a name can reveal the one below it (B2 deletes exactly one version per call), so a single pass is not always enough. It then removes the local /root/.cache/duplicity/<name> cache, and a collection-status truthfully reports an empty collection. A failed wipe is loud in the per-bucket log and on the terminal, and purge exits non-zero without printing any collection status.

The clean listing is the strongest proof this layer has. Bucket-level versioning or soft-delete, where enabled on the bucket, can retain unlisted older versions no Duplicity backend can see — a hidden B2 file, or every version behind an S3 delete marker. The wipe states this limit in its log note; such remnants follow the provider's own lifecycle settings and can only be removed from the provider's side.

Use purge only to empty a bucket deliberately; it is not an age-based trim. The only age-based path is remove-older-than, used by cleanup.

Per-tenant local DB-dump retention (separate concern)

Tenants also keep local-disk DB dumps under /data/disk/<user>/static/files/dbackup/, distinct from the off-site copies. Default retention is 14 days, overridable per tenant via a single integer (days, 1–9999 — the whole file must be that one number) in:

/data/disk/<user>/static/control/dBackupCycle.info

A value that isn't a plain non-zero number — 2 weeks, 0, extra lines, a symlink planted at that path — is ignored, not obeyed: the cleanup still runs at the 14-day default, and the ignored value is recorded in /var/log/backup_validation_issues.log. That log is the first place to look when a tenant reports their retention window "didn't take".

The host-level local SQL dump tree under /data/disk/arch/sql/ is pruned separately by mysql_backup.sh using _DB_BACKUPS_TTL (default 14 days; 3 days in basic mode) — see Database.

Local config-snapshot vault retention (separate concern)

/var/backups/dragon/ (root from ${_vBs:-/var/backups}) is the local pre-change safety vault. The nightly 90-global-post.sh prunes its top-level entries — each entry is removed wholesale once its own mtime exceeds 7 days — with a single exception for config:

find /var/backups/dragon/* -maxdepth 0 ! -name config -mtime +7 -exec rm -rf {} \;

/var/backups/dragon/config/ is a permanent archive, never auto-pruned; trimming it is an operator decision, by design.

What lands there:

• pre-rewrite snapshots of /root/.barracuda.cnf and every /root/.*.octopus.cnf (taken by BARRACUDA before its first on-the-fly cnf edit, and by OCTOPUS before _satellite_cnf);
• /root/.my.cnf and /root/.my.pass.txt.

All via the shared _backup_to_dragon helper: one chmod-0700 subdirectory per run named YYYY-MM-DD_HH-MM-SS, keeping the earliest pristine copy of each file (never overwritten later in the run).

On a run's first snapshot, any loose files still at the top of config/ (legacy flat *-pre-* snapshots) are swept into config/archive/.

The four auto* OS-upgrade tools (autobeowulf, autochimaera, autodaedalus, autoexcalibur) inline the same subdirectory scheme.

The snapshot mechanism itself is documented on the control-file system overview — this page only owns the retention rule.

Do not conflate the two dragon subtrees: /etc/mysql/my.cnf pre-upgrade copies still go to /var/backups/dragon/t/my.cnf-pre-<serial>-<version>-<timestamp> and, living outside config/, fall under the 7-day rule — the whole t/ subtree is removed once untouched for 7 days.

The off-site global path-set includes /var/backups/dragon (see Backups overview), so these snapshots also ride the off-site backup, where the normal KEEP_WITHIN window applies.

Cost vs. retention

Longer KEEP_WITHIN means a larger provider bill. Practical bands:

For multi-year archival, take an out-of-band multiback snapshot to a separate cold tier rather than extending KEEP_WITHIN indefinitely.

Related

• multiback operations — where the variables live.
• CLI reference — the cleanup vs. purge verbs.
• Legacy backboa — _AWS_TTL / _AWS_FLC equivalents and the monthly-cleanup toggle files.
• Control-file system overview — the _backup_to_dragon snapshot mechanism behind the dragon vault.
• Reference appendix — consolidated variable table.