The serial & fetch pipeline (SKYNET)

This page is the maintainer's contract for how a merged commit physically reaches every live box. There are exactly two transports, and every change you ship rides one of them:

1. Per-file, serial-gated fetches — _fetch_versioned in BOA.sh.txt, driven by the SKYNET self-update tick. Covers aegir/tools/bin/*, aegir/tools/system/* monitors, and the privileged fix-drupal-* helpers. Shipping a change here means decrementing an fNN serial — in both fetch sites if the tool has two.
2. Whole-release rides — lib/functions/*.sh.inc, lib/settings/*.sh.cnf, Nginx/PHP config templates, and everything else in the build tree. No per-file serial exists; the code lands when a box runs a full barracuda up-* pass against the release. Their "serial" is the release identity itself (_xSrl / _X_VERSION).

Pick the wrong transport model and the failure is silent: the commit merges, the mirror serves it, and no box ever installs it.

The heartbeat: clear.sh → BOA.sh.txt → autoupboa

The self-update tick is a root cron line, not a daemon (aegir/tools/system/cron/crontabs/root):

*/5 * * * * bash /var/xdrago/clear.sh >/dev/null 2>&1

Each pass, clear.sh prunes stale pidfiles, stands down only while a live PHP-idle stack swap holds /run/boa_php_idle_quiesce.pid (owner-PID keyed; a dead-owner marker is cleaned and the pass continues — proxy nodes run the full channel since the 2026-08-09 marker narrowing, so BOA.sh.txt is never stale on any box, proxies included), and — only when no install pass is in flight (/run/boa_run.pid, /run/boa_wait.pid and /run/octopus_install_run.pid all absent, and no live installer process) — re-executes the meta installer straight off the mirror, then the auto-upgrade scheduler:

wget -qO- https://${_USE_MIR}/versions/${_tRee}/boa/BOA.sh.txt | bash
bash /opt/local/bin/autoupboa

BOA.sh.txt is therefore never stale on a box — it is streamed fresh from the mirror every tick, and everything below hangs off whatever identity block that fresh copy carries.

Inside it, the dispatch at the bottom of the script:

• _if_boa_key_tools_update_allowed — parks the whole update path while a distro migration marker (/root/.run-to-<codename>.cnf) is present.
• _SKYNET_MODE=OFF → _if_update_boa_key_tools_only runs _update_boa_tools (the core tools still refresh, per-file serial-gated) and exit 0. This is the documented "critically important tools are still updated" floor — and since the once-per-release function latch was retired, an OFF box picks up a bumped core-tool serial on its next tick like any other box.
• SKYNET active → _boa_setup (mirror health check, then _update_boa_tools unconditionally — every line inside self-gates on its own serial stamp), then — only on an installed box (/var/log/barracuda_log.txt exists) — the _fix_* series, Drupal core patch checks (_if_drupal_patches_update, _fix_drupal_core_ten / _fix_drupal_core_eleven) and _update_agents.

Beyond fetching the current copy, two of those steps also verify that a core tool still works and rebuild it in place if it does not. _boa_setup — and the _SKYNET_MODE=OFF floor, which reaches it through _if_update_boa_key_tools_only — runs _if_reinstall_curl: if the curl binary's libraries have drifted out of sync (curl --version shows no OpenSSL, a stale libcurl.so.4, or empty output) it backs up the legacy libs and reinstalls curl, logging OOPS: cURL is broken! Re-installing... cURL is therefore repaired on every dispatch path, even with SKYNET off. On the SKYNET-active installed-box path the _fix_* series adds _if_fix_lshell, which reinstalls the lshell limited shell (with python3-pip) when its version check comes back wrong or its Python runtime fails with a Traceback, bad interpreter or ImportError. The serial fetches keep the tools current; these integrity checks keep them working, which is why an OS or library change rarely leaves BOA's own tooling stranded.

Separately, the nightly owl.sh (04:15 cron) fetches ${_urlHmr}/conf/version/barracuda-release.txt (shipped from aegir/conf/version/) and mails _MY_OCTO_EMAIL when /var/log/barracuda_log.txt does not yet contain that release string — the "new release available" notice. It runs only when _SKYNET_MODE is unset or ON — and only when owl.sh runs at all: not on a proxy node (/root/.proxy.cnf, a deliberately kept gate), not while heavy maintenance is paused, and not while a PHP-idle stack swap holds its quiesce marker.

The identity block

The first exported lines of BOA.sh.txt are the release identity every other mechanism keys off:

export _tRee=dev              # tree: dev | pro | lts (per published branch copy)
export _xSrl=<NNNN><tree>TNN  # whole-release serial — in EVERY ctrl stamp name
export _rLsn="BOA-X.Y.Z"      # release label

(Concrete values churn every release — read them from BOA.sh.txt at HEAD; NNNN squashes the version, e.g. X.Y.Z → XYZ-style digits.)

• _tRee selects the mirror tree: all serial-gated fetches resolve against _urlHmr="https://${_USE_MIR}/versions/${_tRee}/boa/aegir". The value is baked into each branch's published copy of the installer, so a box follows whichever tree's BOA.sh.txt its clear.sh streams.
• _xSrl appears in every control stamp (<name>.ctrl.<serial>.${_tRee}.${_xSrl}.pid). Bumping it at a release invalidates all stamps at once — the first tick after a release refetches the entire serial-gated tool set exactly once. One-shot migration blocks (e.g. cv-phar-symlink.ctrl.${_tRee}.${_xSrl}.pid) and the dragon backup names in lib/functions/ use the same key, which is why whole-release code needs no fNN of its own.
• A release bump is not the only thing that clears stamps. Every full upgrade pass that reaches the tools deploy drops the whole stamp set. _xdrago_install_upgrade (lib/functions/system.sh.inc) copies the release tarball's aegir/tools/system/* over /var/xdrago/, which reverts fetch-delivered tools to the release cut's vintage while their fNN stamps still claim the newer serial — a box upgrading between a mid-cycle tool wave and the next release cut would otherwise keep the reverted bytes with no stamp left to trigger a re-fetch. The function therefore runs rm -f ${_pthLog}/*.ctrl.*.pid immediately after the copy, and the next _update_boa_tools tick re-fetches every current serial within minutes. The glob is deliberately wide: the one-shot whole-release markers (cv-phar-symlink.ctrl.${_tRee}.${_xSrl}.pid and friends) match it too, so those idempotent blocks re-run once after an upgrade. Byte/marker coherence after a tar deploy therefore holds whether or not the release serial moved.
• _rLsn feeds _rlsE="${_rLsn}-${_tRee}"; the barracuda/octopus/ boa wrappers pin it into the fetched settings file as _X_VERSION=${_rlsE} (see the release-ride section below). The same values are duplicated at the top of the wrappers themselves — a release commit updates all of them together.
• _bTs is RETIRED (with the whole updateBOAtools function latch it fed). _update_boa_tools now runs unconditionally on every dispatch path and every tick; each tool line self-gates on its own fNN stamp, so there is no function-level serial to bump any more — a tool change ships with its own fNN bump alone, and stale updateBOAtools*.pid latches are swept on every run. (The up-counting <NNNN>vNN shape survives elsewhere: the /etc/crontab rewrite markers inside autoupboa still carry it as file-local literals, rewritten by the release bump — same rule there: bump means increment, never re-use.)

All stamps live in _pthLog=/var/log/boa (a one-time migration block at the top of BOA.sh.txt moved them from the legacy /var/xdrago/log).

_fetch_versioned — the per-file transport

One helper replaces every hand-rolled fetch block (defined in BOA.sh.txt, above _update_agents):

_fetch_versioned <dest> <url> <serial> \
  [--mode 700] [--owner root:root] [--symlink PATH] [--guard PATTERN] \
  [--require-token TOK]

Semantics, in order:

1. Up-to-date test. Return 0 when the stamp ${_pthLog}/<basename>.ctrl.<serial>.${_tRee}.${_xSrl}.pid exists and the destination is non-empty (-s) and, when --symlink was given, the symlink is intact and, when --require-token was given, the deployed copy still carries the token in its last 4 KiB and, on a box that carries the published tree itself and therefore holds this file locally, the deployed bytes still match that local copy exactly (cmp -s) and the stamp is younger than the revalidation window (_FETCH_REVALIDATE_MIN, default 60 minutes — an aged stamp re-checks the source, so a publish race or CDN skew self-heals within the hour instead of pinning stale bytes forever). The serial alone cannot spot a stamp written against content older than it implies: the local comparison catches that on the very next pass, and a box with no local copy has the window as its only backstop. An hour rather than a day is affordable because revalidation normally costs nothing — see item 4. Missing file, broken link, failed token, byte drift against the local copy or an aged stamp = refetch now.
2. Run guard. --guard PATTERN does a pgrep -fc; a running match skips the fetch entirely (no stamp written) — a live tool is never clobbered, and the fetch retries next tick. The match is anchored to genuine executions only — the tool as an interpreter's script argument, or its path as argv[0] — so an editor, a checksum or an operator's ssh command that merely names the path does not hold an update back. The guard is skipped entirely when the destination is missing or empty: such a copy cannot be running, and skipping the fetch would leave a dangling symlink behind.
3. A box that holds the published tree reads from disk, not the network. /var/www/static/getboa marks a box carrying the tree these URLs are served from. Where it exists the candidate source is resolved by stripping scheme and host from the URL and grafting the webroot on; if that file exists and is non-empty it is copied with cp -pf and no network request is made at all. Such a box must never fetch its own bytes back over HTTP, because an intermediary can answer with pre-publish content after the local tree is already current, and cache purges are best-effort. A URL that is not ours, or a tree the box does not hold, has no local file and falls through to curl unchanged.
4. Revalidation is a conditional request, and a 304 is success. When the deployed copy is intact (non-empty, and still carrying its --require-token sentinel where one is required), a marker for this serial already exists, and that marker is younger than _FETCH_HARD_REVALIDATE_MIN (default 1440 minutes), the fetch adds -z <dest>, sending If-Modified-Since from the deployed copy's mtime. An unchanged source then answers 304 with no body — treated as positive confirmation, not a failed fetch: the marker is re-armed and the optional symlink is still repaired, which is what a revalidation triggered by a missing link came here to fix. That is what makes a short revalidation window cheap. curl -R (and cp -p on the local path) are load-bearing here, not cosmetic: they stamp the deployed copy with the source's Last-Modified, so the comparison is "is the source newer than the version I hold" rather than "newer than the moment this box happened to write it". Each gate exists because a 304 would be the wrong answer without it — a changed serial means new content exists, so it is never revalidated conditionally, and once past the hard window the request goes out unconditionally. A deployed copy carrying a wrong mtime therefore degrades to the unconditional refetch this replaced: it can delay a heal, never prevent one.
5. Validated atomic replace. The payload lands in a private temp (<dest>.new.$$) — the live dest is never written directly. The temp is discarded when curl itself reports failure (a partial body would otherwise survive), when it is empty, when it starts with an HTML tag (the mirror answers a missing path with an HTTP 200 "Under Construction" stub that --fail cannot catch), or when a --require-token token is absent from its last 4 KiB (tail placement makes the sentinel a truncation check as well as an identity check — used for websh, which doubles as /bin/sh on hosted boxes). A surviving payload identical to the deployed copy just re-arms the stamp; a differing one gets chmod/chown (defaults 700 root:root) and an atomic mv onto the dest, then the optional symlink, all old <basename>.ctrl.* stamps removed and the new stamp touched. Any discard writes no stamp — mirror lag or a transient failure retries on the next 5-minute tick instead of being skipped forever.

Retry is two-layered: _crlGet itself carries --retry 9 --retry-delay 9 --fail (so a non-2xx is refused before the validation chain even starts), and the stamp-only-on-success rule makes the cron tick the outer retry loop.

This replaced two pre-5.10.3 mechanisms:

• The /opt/local/bin bulk curl loop (~30 tools) tested -e instead of -s, so a 0-byte --fail download could overwrite a good tool and delete its .prev backup, and it had no retry.
• The six privileged helpers (fix-drupal-*-permissions/ownership.sh, lock-local-drush-permissions.sh, fix-drupal-site-symlinks.sh) sat behind one updateFx31.ctrl one-shot marker that stamped even on a failed fetch, so a partial pull was never retried.

Both are now per-file _fetch_versioned calls (/opt/local/bin group and privileged helpers alike); the old updateFx*.ctrl markers are unused and harmless.

Two residual hand-rolled blocks remain in _update_agents (manage_ltd_users.sh, manage_solr_config.sh) — same stamp shape, inline logic; treat their serials identically.

fNN serials count DOWN — and every tool has exactly ONE fetch site

The in-code rule (comment above the /opt/local/bin group in _update_boa_tools): "bump a script's serial to force a refresh after editing it; move it to a value not currently deployed (stamps are wildcard-cleared, so direction does not matter; convention counts down)."

• Bump = decrement, by convention. f99 → f98 → … → f01. Concrete values churn every release — read them from BOA.sh.txt at HEAD, never from docs.
• Never re-use. The mechanism itself is direction-free (_fetch_versioned wipes old stamps on every successful fetch), but only for boxes that actually completed the previous fetch. A box that skipped an intermediate serial — offline, mirror lag, guard always busy — may still hold a stamp bearing an older value. "Bump" a serial back to a previously-used value and that box matches its stale stamp and silently skips the refetch. Monotonic decrement makes stamp collision impossible.
• One tool, one fetch line, one stamp. The historical second listing of the core tool group in _update_agents is gone — it existed only to work around the retired once-per-release _update_boa_tools latch, at the price of a paired-serial discipline (a divergent pair made the two sites wipe each other's stamp and refetch every 5-minute tick, fleet-wide). If you ever see two _fetch_versioned lines whose destinations share a basename, that is a defect, not a convention: the stamp namespace is keyed on the basename, so the pair will flip-flop. A tool's single line lives in _update_boa_tools if every box needs it, or in _update_agents if only installed Skynet-active boxes do.
• A serial mistake is corrected by decrementing again — never by putting an earlier value back.

_update_agents vs _update_boa_tools — two run contexts, disjoint lists

The split is by audience, not by duplication (the old "duplication is intentional, bump both together" contract is retired along with the function latch):

Whole-release rides: lib/functions and the build tree

lib/functions/*.sh.inc carry no fNN and are never fetched per-file. They reach a box only inside a full installer run:

1. The barracuda wrapper (itself a serial-gated tool) maps up-lts/up-pro/up-dev to _tRee and sets _rgUrl="https://files.boa.io/versions/${_tRee}/boa", _rlsE, _bRnh (_set_tree_vars). _proceed then fetches BARRACUDA.sh.txt and lib/settings/barracuda.sh.cnf fresh into /var/backups/ and calls _up_action, which seds the release pins into the fetched settings file: _X_VERSION=${_rlsE}, _BRANCH_PRN=5.x-${_tRee}, _AEGIR_VERSION=${_tRee}. boa and octopus do the equivalent for their chains.
2. BARRACUDA.sh.txt sources the fetched barracuda.sh.cnf, then _download_helpers_libs stages the whole build tree at _bldPth=/opt/tmp/boa: _DL_MODE=BATCH (default) pulls boa.tar.gz from https://<mirror>/dev/<tree>/ via _get_dev_ext (10 attempts, 9 s apart); _DL_MODE=GIT clones ${_BOA_REPO_GIT_URL}/boa.git at ${_BRANCH_BOA} instead.
3. The include list is fixed: _FL="helper dns system sql valkey redis nginx php solr master xtra firewall hotfix", each sourced from ${_bldPth}/lib/functions/ with _panic_exit on a missing file.
4. _xdrago_install_upgrade (lib/functions/system.sh.inc) then wholesale copies ${_bldPth}/aegir/tools/system/* over /var/xdrago/, after snapshotting the old tree and root crontab to ${_vBs}/dragon/…-pre-${_xSrl}-${_X_VERSION}-${_NOW}.

Consequence for shipping: a monitor or system script in the fNN set updates both ways (serial fetch within minutes, wholesale copy at the next full upgrade); a lib/functions change updates only at the next full barracuda up-* — there is no faster path, so never put an urgent fleet fix solely in lib/functions if a serial-gated tool can carry it.

The _*_VRN block at the top of BARRACUDA.sh.txt (_CURL_VRN, _NGINX_VRN, _CSF_VRN, _GIT_VRN, _MSS_VRN, …) is the software-pin layer of the same ride: the pin is defined there, but the build step that consumes it lives in the sourced lib/functions/*.sh.inc, and its build/rebuild stamp embeds the pin value together with _xSrl/_X_VERSION (e.g. the MySecureShell build in lib/functions/system.sh.inc gates on ${_pthLog}/mss-build-${_MSS_VRN}-${_xSrl}-${_X_VERSION}.log).

Changing a pin therefore re-triggers exactly that component's build on the next full run.

Mirrors and the _crlGet convention

Every fetch in the pipeline goes through one flag set, defined near the top of BOA.sh.txt and duplicated verbatim in the barracuda/boa/octopus wrappers and the BARRACUDA.sh.txt/OCTOPUS.sh.txt meta-installers (the lib/settings/*.sh.cnf files inherit it from the sourcing *.sh.txt, they do not redefine it):

_crlGet="-L --max-redirs 3 -s --fail --retry 9 --retry-delay 9 -A iCab"
_wgetGet="--max-redirect=3 -q --tries=9 --wait=9 --user-agent='iCab'"

--fail is load-bearing (non-2xx → empty destination → _fetch_versioned rollback instead of an HTML error page saved as a tool).

-A iCab is the fetch identity every BOA downloader sends; the mirror cluster is operated around that convention, so when hand-testing a mirror URL, test with curl -A iCab — a default UA is not a representative probe.

Mirror selection: _find_fast_mirror_early seeds /var/backups/boa-mirrors-2026-05.txt with the current pool — files.boa.io (default/fallback), files.o8.io, files.host8.biz — and runs ffmirror to pick the fastest; the winner becomes _USE_MIR. ffmirror probes each candidate with a hard two-second timeout and a single try, so a mirror that accepts a connection and then goes silent cannot stall the run probing it — that run may be holding a box-wide guard (the 5-minute channel, the nightly worker, the user-management sweep, a migration tool), so the bound matters fleet-wide, not just at install time. And when no candidate answers in time it returns the first listed mirror rather than an empty string, so mirror selection never yields an empty hostname or a malformed fetch URL.

The list filename is date-stamped, so shipping a new pool means a new filename (old boxes regenerate rather than merge). Two URL bases derive from it:

• _urlHmr="https://${_USE_MIR}/versions/${_tRee}/boa/aegir" — per-tree tool and config fetches (paths mirror the repo's aegir/ subtree).
• _urlDev="https://${_USE_MIR}/dev" — source tarballs and boa.tar.gz.

For the pro and dev trees, _verify_boa_keys additionally checks a per-hostname key (enc/2024/<md5(hostname)> on the mirror) before proceeding; lts has no licence gate.

How a tagged release propagates

Tags exist only on 5.x-pro and 5.x-lts; 5.x-dev (with its -base / -edge feeders) is consumed untagged by dev-tree boxes. The mirror cluster serves each tree's installers, tools and boa.tar.gz under versions/<tree>/ and dev/<tree>/; tagging a release is what promotes the new file set into the pro/lts trees (mirror-side publication — the repo carries the content, the tag marks it releasable).

From the box's point of view, everything after that is the mechanics above:

1. Within one 5-minute tick, clear.sh streams the new BOA.sh.txt. Its new identity block does the rest: a bumped _xSrl invalidates every ctrl stamp, so the full tool/monitor set refetches once; individually decremented fNN serials refetch just their files — between releases as promptly as at one.

2. The same tick runs autoupboa, which maintains the scheduled full-upgrade cron entries (barracuda up-${_AUTO_VER} … / octopus up-${_AUTO_VER} … in /etc/crontab) from the _AUTO_UP_* variables, and the weekly wrapper mode (autoupboa weekly-system up-<tier> system … noscreen). Two populations, one mechanism:

   • Hosted-fleet boxes (_if_hosted_sys — /root/.host8.cnf present or hostname ending .aegir.cc): autoupboa hard-sets the schedule in-code, with the full-upgrade date parked on _AUTO_UP_MONTH=2 / _AUTO_UP_DAY=29 (Feb 29 — a leap-day-only slot) while the weekly system upgrade stays live; release ops edit the shipped autoupboa to open a real fleet window and re-park it afterwards.
   • Operator boxes: the same variables are read from /root/.barracuda.cnf — no month+day set means _crontab_cleanup strips the full-upgrade lines.

   The crontab rewrite is itself a one-shot per ${_tRee}.${_xSrl} (an up-counting vNN ctrl stamp), reset when the /root/.turn.off.auto.update.cnf lock toggles; dev-server boxes (/root/.dev.server.cnf) never get auto-upgrade lines.

   That weekly wrapper line is also where a PHP 7.4 intl self-heal rides. On each weekly run autoupboa weekly-system checks for a 7.4 that an ICU bump left without intl, and only then idles the inactive PHP versions (barracuda php-idle disable) to shrink the rebuild surface before running the upgrade twice: the pin pass pins ICU to a 7.4-safe version and rebuilds the active PHPs onto it (7.4 regains intl), then the unpin pass drops the pin and rebuilds the 8.x versions back onto the OS-default ICU while 7.4 keeps its pinned-ICU intl. It is self-limiting — once 7.4 carries intl the plain upgrade runs — and tree-independent, and the orchestration lives only in autoupboa so barracuda stays a leaf call that cannot self-loop. Shipping it to existing boxes is a paired serial bump in one commit: decrement autoupboa's fNN (single fetch line in _update_boa_tools) so the new script lands per-file, and increment the up-counting crontab-update vNN marker so each box regenerates /etc/crontab once and picks up the routed weekly-system line.

3. Nightly, owl.sh compares barracuda-release.txt against /var/log/barracuda_log.txt and mails the operator until the box has been upgraded.
4. The full stack upgrade — new lib/functions, rebuilt services, /var/xdrago wholesale refresh — happens when barracuda up-lts|up-pro actually runs: manually, or at the scheduled autoupboa window.

So: tools and monitors converge fleet-wide in minutes; the deep stack converges at the next full run. Design any fix's rollout around that split.

Shipping checklist (serial mechanics only)

• Tool under aegir/tools/bin/ or aegir/tools/system/ changed → decrement its fNN in BOA.sh.txt at its single fetch line (grep the tool name; exactly one hit is the invariant — two hits sharing a basename is a defect). Same commit as the change.
• Never increment or re-use an fNN value; never copy a serial from another tool.
• _update_boa_tools body changed → nothing extra: the function runs every tick; the per-file serials are the only knobs. (The old _bTs function serial is retired.)
• lib/functions/*.sh.inc / lib/settings/*.sh.cnf / template change → no serial exists; it rides the next full barracuda up-*. If the fix is urgent fleet-wide, carry it (also) in a serial-gated tool.
• New tool → add a _fetch_versioned line starting at f99, plus the chmod group and symlink block in _update_boa_tools if it is operator-facing.
• Verify fetch URLs against the repo layout (_urlHmr maps to the aegir/ subtree) and test with curl -A iCab.

Related

• Code style & conventions — the lint gates and commit rules every serial-bumping commit must also pass.
• Contributing — how changes reach the public branches this pipeline publishes from.
• Reference appendix — consolidated _VAR tables, including the _AUTO_UP_* scheduler variables.