Building the test-codebase mirror

The other two leaves in this topic are about proving your fork change holds. This one is about the codebases you prove it against: the newer Drupal cores and distributions you install, clone and migrate to exercise BOA end to end. Building that set by hand — track upstream, resolve every version, work around each distro's Composer quirks, package, publish — is exactly the kind of manual procedure BOA wraps in a tool, and staticbuild (aegir/tools/bin/staticbuild) is it.

It is a first-class shipped tool, but a build-box one: _update_agents in BOA.sh.txt fetches it via _fetch_versioned and symlinks it to /usr/local/bin/staticbuild only on boxes carrying /root/.stackbuild.cnf — even an empty file opts a box in — alongside stackbuild, the prebuilt-package builder. That marker keeps the mirror-source VM always carrying the current builder without shipping it to the whole fleet, and the tool itself still refuses to act without root plus an existing build user. The built codebases are published to /var/www/static/{core,distro,dev/{dev,lts,pro}} — the tree the fleet's read mirrors replicate and installs fetch from.

Nothing is hardcoded: each distribution is built from its Composer spec (farmOS from its GitHub release), each vanilla core from its minor's latest patch, the core minors are auto-discovered, and the versions Composer actually resolves are read back out of the build and stamped into the tarball names. The only settings to review are the Composer specs and the core floor/exclude in the configuration block at the top of the script.

Run as root, build as the Octopus user

staticbuild is fail-closed about who it runs as. It refuses to start unless it has both:

• root (_require_root) — packaging, the per-build /usr/bin/php switch, and the writes into /var/www/static/ all need it;
• a real Octopus build user (_require_o8) — o8 by default, -u USER overrides. It must be an existing system account with a /data/disk/<user> home, and wget/curl must be present.

The builds themselves run as that user under /data/disk/<user>/static/<MM-DD>/; only the packaging and publishing steps use root. Crucially, it never scans /data/disk to guess an account — a bare o* glob there would run builds as an arbitrary customer instance, so a missing or wrong -u is a hard error, not a guess. A single-run lock (/run/staticbuild.lock) guards against re-entrancy, and an EXIT trap restores the original /usr/bin/php symlink and releases the lock even on a failed or interrupted run.

Subcommands

staticbuild check              # report the latest upstream versions a build would pull
staticbuild catalogue [tree ...]  # audit each PUBLISHED tree's catalogue against the mirror
staticbuild all                # build every target, package, publish
staticbuild build [name ...]   # build all, or only named targets
staticbuild package            # clean + tar (cores keep core/profiles, distros strip)
staticbuild distribute         # copy tarballs to /var/www/static/{core,distro,dev/{dev,lts,pro}}
staticbuild backdrop           # build + package + publish ONLY the Backdrop family

check is read-only and needs no root — run it each cycle to see what a build would pull before committing to one. Options precede the action: -d MM-DD picks the build-day directory (default today), -u USER the build user, -f forces a rebuild of an already-built platform, and -C skips the opening composer clearcache.

catalogue audits the published catalogue of each tree (default dev lts pro) against the distro mirror and fails if any entry's tarball is not there. The guard exists because a de-published tarball is not a build failure but a silent customer-facing one: the octopus run retries the fetch ten times, nine seconds apart, then carries on — costing every box a minute and a half per codebase generation and surfacing only as an OOPS line in one customer's console.

What it builds

Ten distributions to distro/, each from its own upstream, named by the authoritative distribution profile version (not the create-project template's, which is often a -dev alias) plus the resolved core version, e.g. localgov-4.0.2-11.4.5:

Raw vanilla cores to core/, one per supported minor at its latest patch, named drupal-<version>. Minors are auto-discovered: every released major.minor at or above the floor (_CORE_FLOOR, 10.2) minus the exclude list (_CORE_EXCLUDE, 11.0), so the newest minor is always built without editing the script — currently 10.2–10.6 and 11.1–11.4. Set an explicit _CORE_MINORS list only to pin a fixed matrix.

The Backdrop family (see below), built alongside everything else by staticbuild all.

Versions, and the older-core fallback

Every build resolves upstream live. staticbuild check reports what each distro and the core matrix would pull — drupal.org's release feed is the authoritative source for most distribution profile numbers, GitHub for farmOS, and Packagist for the core minors (and for commerce_kickstart, which has no drupal.org stable release). The distro version stamped into the tarball name is read from the built composer.lock, and the core version from the built docroot's Drupal.php VERSION constant, so the name always reflects what was really installed.

Because the newest core sometimes breaks a distribution — a distro's bundled core patch goes stale on a just-released core, for instance — a distro that fails to build on the latest core is retried pinned to progressively older core minors, and the newest one that builds is kept (_with_core_fallback). Varbase and OpenFed sit outside this mechanism — varbase installs from upstream's shipped lock, and an older-core pin would contradict the profile's own core requirement; OpenFed's profile pins its own core minor, and its merge-plugin build would not survive the fallback's vendor wipe (see each distro's quirk below). Each retry starts below the core the failed attempt had resolved, wipes the lock and vendor tree, and pins both drupal/core and drupal/core-recommended (a distro may require drupal/core directly, so pinning only the recommended metapackage would conflict rather than override).

Per-distribution quirks

Each of these is a real Composer-level workaround the manual procedure needed and the tool now applies automatically. Six of the ten distributions build under php85, and vanilla cores under php84; the remaining four build on an older interpreter, each noted in its quirk below — openfed, opigno and social on php83, the catalogue cap each will actually run under, and varbase on php84. staticbuild swaps the /usr/bin/php symlink per build and restores it on exit.

• allow-plugins — every distro builds with composer config allow-plugins true: nine get it via _composer_prepare, varbase sets it inline as its only config change. The distros pull varying plugin sets (composer/installers, cweagans/composer-patches, installers-extender, …), and current Composer blocks any unlisted plugin, failing the install; enumerating them per-distro is fragile.
• advisory blocking off — policy.advisories.block false is set for the nine _composer_prepare distros (varbase's pristine config never gets it) because commerce_kickstart enables Composer's security-advisory blocking, which refuses every advisory-affected core or dependency and also defeats the older-core fallback (older cores carry advisories). These are throwaway test platforms, so it is turned off. Commerce Kickstart also builds in two phases — install, add the centarro/certified-projects layer, then re-resolve and install.
• openculturas — its distribution ships no example.sites.php (one is copied in from the platform's own core scaffold, falling back to a sibling build). Its dependency patches are dropped via extra.composer-patches.ignore-dependency-patches — several are stale against the core the distribution itself requires, and composer-patches 2.x cannot apply patches to dist-installed packages (its git-based patcher skips any package without .git) — except one load-bearing patch applied with GNU patch after the build, fail-closed: the shipped views reference entity_reference_revisions relationship handlers that only exist with the ERR issue-2799479 patch, and without it every front-page request fails. The build also flattens the openculturas-distribution wrapper so its inner profile becomes the install profile.
• openfed — the template's own require is only three composer helpers; the whole distribution arrives through composer-merge-plugin (the template's composer.openfed.json), whose include list points inside packages the resolve itself installs (webform's and ckeditor_codemirror's composer.libraries.json). The plugin converges that itself during the bootstrap install — and then upstream's recursive post-update-cmd (composer install, their own include-file self-heal) exits non-zero against the mid-convergence lock, so the build tolerates the bootstrap's exit code and gates on the explicit resolve+install pair that follows, plus a fail-closed check that openfed/openfed really reached the lock (a bootstrap that died before the plugin hit disk would otherwise lock three helpers and no distribution). Trying to suppress the recursion with --no-scripts does not work — plugin event subscribers still run, and the nested update the plugin dispatches re-enables scripts on its own (measured, twice). One accepted trade rides the build: the profile pins entity_browser exactly and the XSS-fixed release sits outside the pin (SA-CONTRIB-2026-094), so the platform ships the advisory-affected version. Builds under PHP 8.3 (its catalogue cap); docroot docroot/, profile openfed, version read from the lock. Upstream keeps OpenFed 13.x on the Drupal 10 track (core-recommended ~10.6) — do not "fix" the core the build resolves.
• opigno (Opigno LMS) — upstream's documented create-project is broken as shipped, in three ways the build corrects, each an open issue in Opigno's own queues with no fixed release. The template replaces the h5p/h5p-core and h5p/h5p-editor PHP libraries without providing them, so the interface the H5P module implements never reaches disk and every install fatals (opigno_lms

  3574405): the build drops the replace block. With that corrected, h5p-core

  1.28.0 adds an interface method the bundled H5P module release does not implement (h5p #3578071): the build requires h5p/h5p-core:1.27.*, inside the module's own ^1.27 and the workaround endorsed in that issue. Finally, current Twig rejects opigno_learning_path's empty getOperators() on the first front-page render (opigno_learning_path #3561556, RTBC): the issue patch is applied with GNU patch, fail-closed. One accepted trade rides the build: Opigno pins dompdf ~2.0.0 and the fixed dompdf line (3.x) is outside it, so the platform ships an advisory-affected dompdf — which is also why upstream's own command fails outright before advisory blocking is turned off. Builds under PHP 8.3 (its catalogue cap); docroot web/, profile opigno_lms, version read from the lock.

• social (Open Social) — upstream ships no create-project template for its current major: goalgorilla/social_template has not moved since 2024 and its committed lock pins the 12.4 line on a long-superseded core, so following upstream's own documented command cannot produce a current build. The template is therefore used only as a chassis — it owns the html/ docroot, the scaffold locations and the installer-paths upstream itself defined — and the distribution alone is moved to the current major before resolving. Three consequences worth knowing: the docroot is html/, not web/; it builds under PHP 8.3 rather than the shared newer interpreter, because the distribution's floor and the platform's PHP cap are the same version; and the version is read from the lock, never from social.info.yml, whose 13.0.2 tag still declares 13.0.1.
• thunder — the template thunder/thunder-project versions independently from the actual distribution thunder/thunder-distribution (e.g. 5.0.0 vs 8.4.0); the platform is named by the distribution.
• varbase — installs from the shipped lock, uniquely among the distros here: upstream's committed lock is their tested resolve, including the composer-patches generation their ~76-patch set is written for, and a re-resolve discards exactly that (measured: the freshly resolved patches stack aborts on their own core patch, and every older-core fallback pin contradicts the profile's own core requirement). Composer config stays pristine except allow-plugins. The patches are fetched from ~76 URLs at install time and one flaked download aborts the whole install — but upstream's pinned composer-patches records the aborted set as applied, so a retried install in the same tree exits 0 with patches silently missing. Every retry therefore starts from a fresh create-project; the shared Composer cache still warms across attempts, which is what lets a later attempt complete — never "optimise" this to resume a failed tree. Those patch URLs are mutable branch refs with no hashes, so rebuilds are not reproducible: each published tarball is the frozen artefact. The build target is the stable 10 line (docroot/ layout, matching the catalogue), not the still-beta 11.0 line (web/); it builds under php84 rather than the shared newer interpreter, creates the default/content dir varbase expects under each recipes/, and is named by the vardot/varbase distribution package — drupal.org's number — not the project template's.
• cms — drupal_cms's post-update-cmd cleanup exits non-zero, so its drush require runs with --no-scripts; the whole build runs non-interactively so the "move this requirement?" prompt (drupal/cms dev-requires drush) never hangs it. That same require also pulls the migration pipeline contribs — migrate_plus, migrate_tools, migrate_upgrade — because upstream drupal/cms ships none of them (core's own migrate stack lives in drupal/core and needs no action); they are inert until a site enables them, and they are why a Drupal CMS platform can take on a Drupal 7 migration without per-site Composer work. The build then corrects pathauto's d7_pathauto_patterns migration definition, adding source_module: pathauto (upstream pathauto #3588684): Drupal 11 prefers the plugin's PHP attribute, which lost source_module in the annotation-to-attribute conversion, and without the correction migrate_drupal validation refuses every Drupal 7 → Drupal CMS upgrade at the credentials step. A grep guard makes it a no-op once a tagged pathauto release carries the fix, and it is fail-closed — a correction that does not apply fails the cms build.

The stages it automates

staticbuild all runs three stages in order; build, package and distribute run them individually against a build-day directory.

1. build — composer clearcache, then for each target: composer create-project into a hidden working dir (farmOS is a release tarball instead), allow-plugins and the advisory/patch config, composer update --no-install then composer install --no-dev, with the older-core fallback wrapping the install (varbase skips the update and the fallback, installing straight from its shipped lock; openfed skips the fallback and leads with its rc-tolerated bootstrap install — see its quirk). Vanilla cores install immediately and add drush; distributions stage without installing until the fix-ups are applied.
2. package — strip the artefacts that must never ship inside a platform tarball (sites/all/drush, stray o_contrib*, PID files, sites.php, …), then gzip the raw cores first, keeping their core/profiles, then strip the stock core/profiles/* from the distributions (they ship their own install profile) and gzip those. Re-running package is idempotent — cores always keep their profiles.
3. distribute — route each tarball by kind: drupal-*.tar.gz and backdrop-<ver>.tar.gz to core/ (the Backdrop publish also refreshes the version-less backdrop.tar.gz compat tarball, then the backdrop.txt stamp — tarball first, stamp last, so the stamp never points at a missing file), bee.tar.gz and backdrop-drush-extension.tar.gz to every per-tree dir — dev/dev/, dev/lts/, dev/pro/ — redis_backdrop-<tag>.tar.gz to the per-tree contrib shelf dev/{dev,lts,pro}/contrib/, and everything else to distro/.

Once published, register the built platforms in Ægir (paths like MONTH-DAY/drupal-11.4.5) and run site install, clone and migration against them — the three tasks the whole set exists to exercise. How BOA registers a pre-built mirror tarball as a platform is covered on The staged setup.

The Backdrop family

staticbuild is also the Backdrop-family builder. Backdrop is not Composer-based, so it is built apart from the Drupal distros and cores — no Composer, no /usr/bin/php switch, just git / wget / unzip — and always fetched at its newest upstream, so BOA never falls behind. staticbuild all builds it alongside everything else; staticbuild backdrop does a lightweight build + package + publish of only this family, scoped so it never touches unrelated build-day content. Four artefacts, always rebuilt at the latest upstream tag (pin any with the matching _*_TAG in the config block):

• backdrop — Backdrop CMS core (backdrop/backdrop), from its latest GitHub release backdrop.zip. Repackaged versioned as backdrop-<ver>.tar.gz (extracts to backdrop-<ver>/), classified as a core and managed on the mirror exactly like the Drupal cores: every published version is retained, the resolved version is written to backdrop.txt — BOA names the platform from that stamp and fetches the matching tarball — and a version-less backdrop.tar.gz compat tarball of the newest release (still extracting to backdrop/, the contract already-deployed BOA releases rely on) is refreshed alongside. No contrib is baked into the versioned core tarballs — the Valkey/Redis integration reaches platforms through the shared o_contrib_backdrop bundle instead (see the redis_backdrop artefact below). Only the version-less compat tarball still carries a baked modules/redis, injected at tar time, because the core/ shelf is shared by every release and the pre-bundle releases consuming that name probe only modules/redis. Published to core/.
• bee — the native Backdrop CLI (backdrop-contrib/bee), from its latest git tag, packaged version-less as bee.tar.gz (bee.php at the root). Published to dev/{dev,lts,pro}/ — each box fetches from its own tree dir.
• backdrop-drush-extension — the Backdrop Drush extension (backdrop-contrib/backdrop-drush-extension), from its latest git tag, packaged version-less and shipped pristine — BOA applies its own PHP 5.6 de-hint and __DIR__ include fix on deploy. Published to dev/{dev,lts,pro}/.
• redis_backdrop — the Backdrop redis contrib module (backdrop-contrib/redis; its cache implementation is Backdrop-native — the Drupal redis modules are not compatible), packaged versioned as redis_backdrop-<tag>.tar.gz wrapping a redis_backdrop/ directory (the tarball's top-level name is the deployed directory name under the shared contrib store). Published to the per-tree contrib shelf dev/{dev,lts,pro}/contrib/. Unlike the other family members it is consumed by a pinned version on the BOA side, so publishing a newer tag is inert until the pin is bumped in the same lockstep as the Drupal redis pins.

Downstream, satellites resolve _BACKDROP_V from that published backdrop.txt stamp at make time, name Backdrop platforms backdrop-<ver>-{dev,stage,prod} against the current release, and fetch the matching backdrop-<ver>.tar.gz; the version pinned in OCTOPUS.sh.txt is only a fallback for when the mirror is unreachable at make time.

Related

• Building & testing BOA changes — the four test gates for a BOA change, including edgetest, staticbuild's documented sibling tool, and the disposable-VM gate where the codebases built here get installed and migrated.
• Verifying the drush extension filter — a worked end-to-end verification of a security-sensitive backend change.
• The staged setup — the consumer side: how a box fetches a pre-built mirror tarball and registers it as a platform.
• The serial & fetch pipeline — how staticbuild itself ships: _fetch_versioned, the fNN serials, and how a bump propagates to the fleet.
• Commands and Variables — the consolidated tool and _VAR tables.