Task queue engine

Every change Ægir makes is a Task node, processed asynchronously. BOA drains that queue from cron, not from a daemon: a load-aware runner.sh fires a per-Octopus dispatch chain that ends in drush @hostmaster hosting-dispatch.

There is no long-running queue process — the optional upstream hosting_queued daemon ships in the fork but BOA never starts it.

This page is the operator reference for the queue engine:

the lifecycle
the cron chain
the fast/slow cadence
the registered queues
the controls

Task lifecycle

TXT

1. Front-end (Hostmaster UI) creates a Task node
       e.g. "Install site foo.example.com"  → status=Queued
2. Cron fires /var/xdrago/runner.sh (once a minute, load permitting)
3. runner.sh runs each Octopus's /var/xdrago/run-<USER>
4. run-<USER> runs /data/disk/<USER>/aegir.sh
       → drush @hostmaster hosting-dispatch
5. hosting-dispatch forks the due queue workers
       → hosting-tasks picks up Queued task nodes
6. The task forks drush @self hosting-task <NID>, which resolves to
       the matching provision-<type> backend command
7. Provision runs server-side, returns log + exit code
8. The Task node is updated:
       success → status=Successful, log attached
       failure → status=Failed, log + error attached
9. Front-end refresh shows the new status (UI polls the task node)

A failed task stops the queue for that entity (a failed Migrate on foo.example.com blocks further tasks on that site until resolved). The operator must Reset the task in the UI (or delete it) to unblock.

How BOA runs the queue

The engine is a chain of cron-fired shell scripts ending in a single Drush dispatcher. Nothing here is resident — every layer is a short-lived process started by cron.

1. Cron (aegir/tools/system/cron/crontabs/root) runs four scripts every minute:

TXT

* * * * *   bash /var/xdrago/second.sh
* * * * *   bash /var/xdrago/minute.sh
* * * * *   bash /var/xdrago/guest-fire.sh
* * * * *   /usr/bin/nice -n5 /usr/bin/ionice -c2 -n7 bash /var/xdrago/runner.sh

runner.sh is the only queue-relevant entry — the task-queue orchestrator, run niced + ioniced so it never competes with site traffic. (second.sh / minute.sh / guest-fire.sh are the other per-minute agents; none touches the task queue.)

2. runner.sh is load-aware. It:

Bails / waits when the box should not process tasks:
- a passive replication standby (/root/.standby.cnf) — checked first, above the proxy and pause gates on purpose, because a promoted failback target can still carry the proxy marker. On a standby runner.sh exits and also parks the Ægir master crontab (/var/spool/cron/crontabs/aegir moved aside to .aegir, cron reloaded), which carries the per-minute hosting-dispatch that runs outside runner.sh and so is reached by none of the gates below. The first pass with the marker gone moves the parked crontab back, again above the proxy and pause exits — except on a CI / Jenkins-style box, where the park belongs to the idle-queue control described below
- a proxy node (/root/.proxy.cnf — a rightful gate kept by the 2026-08-09 marker narrowing: a proxy has no local sites for tasks)
- an explicit maintenance pause (/etc/boa/.pause_tasks_maint.cnf)
- a PHP-idle stack swap in progress (/run/boa_php_idle_quiesce.pid, owner-PID keyed and self-clearing — before the narrowing this mute reached the queue via a synthetic proxy marker)
- the dedicated queue-stop file (/run/boa_queue_stop.pid — see Pause / resume)
- a high/critical load flag (/run/max_load.pid, /run/critical_load.pid)
- a running SQL backup (mysql_backup.sh / mysql_cluster_backup.sh / mydumper)
- the nightly owl.sh run (formerly daily.sh)
- a MySQL restart
- another BOA task already holding the queue (/run/boa_run.pid, /run/boa_cron_wait.pid)
On the SQL-backup / nightly back-off it touches /var/log/boa/wait-runner.pid, logs "Another BOA task is running, we will try again later...", and exits — queue draining defers until the dump or maintenance run finishes.
Throttles on load: it compares one-minute load to _CPU_TASK_RATIO (default 3.1, i.e. a ceiling of ratio × 100% per CPU) and skips a pass when over the ceiling.
Iterates the per-Octopus runners (_runner_action): for every /var/xdrago/run-<USER> it re-checks load, runs the script, then pauses a random 2–10 s before the next instance.
Accelerated mode overrides the load gates during a genuine Octopus install/upgrade. The CLI touches the marker /run/octopus_install_run.pid (set by octopus and by boa up-* runs, removed on completion) so the queue keeps running regardless of load and the freshly upgraded backend config and Drush aliases reload promptly: runner.sh skips the max_load.pid / critical_load.pid exit and runs each run-<USER> regardless of the _CPU_TASK_RATIO gate. The marker is honoured only when backed by a live /opt/local/bin/octopus process or when it is fresh (under 15 minutes); a stale marker is deleted, so a crashed run can never permanently bypass load protection. run-<USER> cooperates: while the marker is fresh it overrides its own build-in-progress skip so dispatch still fires. Normal-operation load protection is unchanged, and the SQL-backup / nightly back-off above is not bypassed by accelerated mode.

3. Per-Octopus run-<USER> (/var/xdrago/run-<USER>, generated from aegir/scripts/run-xdrago by satellite.sh.inc with EDIT_USER → the instance user, mode 0700) does cache/tmp cleanup, optional SFTP-password rotation, an optional Octopus self-upgrade, then runs drush8 cc drush and bash /data/disk/<USER>/aegir.sh as the instance user. It records /var/log/boa/last-run-<USER> and bails if /run/boa_wait.pid exists. Two guards shape when it actually dispatches:

Per-instance serialisation: it takes a non-blocking flock on fd 9 (/run/run-xdrago-<user>.lock) so overlapping cron runner.sh passes, the fast-schedule loop, and the accelerated install/upgrade loop cannot fire concurrent hosting-dispatch + drush8 cc drush storms for one instance. A concurrent run exits cleanly (exit 0); the lock is released automatically on process exit, so there is no stale lock to clear (the lockfile itself persists harmlessly), and it fails open — runs unlocked — if flock or a writable /run is unavailable. The front-end dispatcher's own serialisation (step 5 below) is the authoritative no-double-launch guard; this flock is best-effort de-duplication.
Build-in-progress skip: a _tmp_ Drush working dir under /data/disk/<USER>/.tmp marks a platform build in progress and makes run-<USER> skip that instance's dispatch (touching /var/log/boa/skip-run-<USER> instead of last-run-<USER>). It counts as an active build only while a live drush.php process for that instance user exists — an orphaned _tmp_ dir left by a crashed build does not freeze the instance queue until an unrelated cache clear. The check is re-evaluated on every run and deletes nothing.

4. Per-Octopus aegir.sh (/data/disk/<USER>/aegir.sh) is the actual dispatch call:

BASH

php .../tools/drush/drush.php @hostmaster hosting-dispatch
touch /data/disk/<USER>/<USER>-task.done

5. drush @hostmaster hosting-dispatch (hosting/dispatch.hosting.inc) is the real queue processor. It reads the registered queues (hook_hosting_queues) and, for each enabled queue whose frequency is due, forks the matching drush hosting-<queue> worker for up to N items, guarded by Drupal lock_acquire / lock_wait semaphores. One cron entry, frequency tunable from the front end, no resident process. The semaphore semantics:

The per-queue semaphore is named hosting_dispatch_<queue>_running and is acquired with a lifetime of HOSTING_QUEUE_LOCK_TIMEOUT = 900 s (15 min; reduced from the historical 3600 s). The lock is held only for the brief fork-and-dispatch, so the timeout is purely a recovery bound: a dispatcher that dies holding the lock blocks that queue for at most 15 minutes.
The dispatcher is serialised: while holding the dispatch semaphore it reloads the D7 variable table (variable_initialize(); D7 caches $conf per request) and re-reads hosting_queue_<queue>_last_run, then stamps last_run synchronously before forking the worker — so two overlapping dispatch runs are designed no longer to both pass the frequency gate and each fork a full batch (previously up to 2× the configured concurrency).
Concurrency is capped as calc_items minus running_items, where running_items counts PROCESSING tasks whose executed stamp is within the last 28800 s (8 h). This window is deliberately generous and dual-role — shortening it lets long installs/migrations age out of the count while still PROCESSING and re-opens the 2×-batch hole. Stuck-PROCESSING tasks are never re-dispatched (only QUEUED tasks are) — a reaped row is marked failed, never re-queued. Before the arithmetic, and only for the tasks queue, the dispatcher reaps orphans: any current-revision PROCESSING row more than two minutes old whose recorded runner PID ({hosting_task}.pid) is no longer a live hosting-task process in /proc — gone, a zombie, or recycled onto an unrelated command — is marked failed, with a task-log line and a watchdog warning naming the dead PID. It then re-reads running_items while still holding the dispatch semaphore, which also picks up a sibling dispatcher's just-forked batch that the pre-lock snapshot missed. So a row left behind by a runner killed without its shutdown handler — a signal, a code-tree swap during a hostmaster upgrade, a host reboot — stops capping the queue at the first dispatch pass after it is two minutes old, instead of holding a slot for the whole window. Rows carrying no recorded PID still age out only after 8 h: the pid column arrives with hosting_task update 7303, so revisions written before it, and any box whose schema is not yet updated, keep the old behaviour. The same reaper runs in hosting-pause's wait loop, so an upgrade no longer waits on corpses.

Dispatcher enablement. hosting-dispatch processes queues only while the hosting_dispatch_enabled variable is TRUE — otherwise it logs "Dispatching disabled." and exits. hosting-setup (run during install/upgrade) disables the dispatcher, self-tests one dispatch, and re-enables it on success; Hosting remembers the prior state and keeps the dispatcher enabled when the self-test fails on a system where dispatch was already enabled — logging "Dispatch self-test failed, but dispatch was already enabled; leaving it enabled." — while a fresh install whose self-test fails stays disabled as before. BOA adds belt-and-braces repair: _ensure_hosting_dispatch_enabled() checks and re-enables the variable via drush ev immediately before the hosting-dispatch calls on Barracuda master install/upgrade finalise (lib/functions/master.sh.inc) and, as a deliberately separate satellite copy, before hosting-dispatch on Octopus instance install/upgrade finalise (satellite.sh.inc; su mode when the caller is root pre-install, bare mode when already the instance user), logging INFO only when a correction was made. Manual one-line check/repair (also works on older versions), as the instance user:

drush8 @hostmaster vget hosting_dispatch_enabled
drush8 @hostmaster vset hosting_dispatch_enabled 1

Fast vs slow cadence

How aggressively runner.sh fires is set by BOA control files, not by a daemon poll interval:

Mode	Trigger	Behaviour
Slow	`/root/.slow.cron.cnf` present (and no `/root/.force.queue.runner.cnf`)	one `_runner_action` pass, padded 15 s before and after
Fast	`/root/.fast.cron.cnf` (or `/root/.force.queue.runner.cnf`)	loops `_runner_action` ×10 with 5 s sleeps
default	neither flag	a single `_runner_action` pass

The overlap ceiling is a separate test, and it keys on .slow.cron.cnf alone — not on the mode above. runner.sh tolerates 1 concurrent runner.sh process when /root/.slow.cron.cnf exists and 8 when it does not; a pass that finds more than that many copies already running touches /var/log/boa/wait-runner.pid and stands down. So a box carrying both .slow.cron.cnf and .force.queue.runner.cnf runs the Fast ×10 loop under a ceiling of 1.

Boxes with ≤ 4096 MB RAM are forced Slow: runner.sh writes /root/.slow.cron.cnf (made immutable with chattr +i) plus a .slow.cron.cnf.protected marker, so the slow cadence survives plan changes.

The cadence markers themselves are not written by the Octopus installer. .fast.cron.cnf's only writer in the tree is BOA's own normalisation pass — _update_agents in BOA.sh.txt, which clear.sh pulls and runs every five minutes whenever no barracuda / octopus / boa run holds /run/boa_run.pid — and it runs on omega8.cc-hosted boxes only (/root/.host8.cnf present, or a hostname ending .aegir.cc, plus the hosted layout: the Master Drush alias, /data/u and /var/xdrago). That pass counts the instances on the box (/data/disk/*/static/control/cli.info) and greps the per-Octopus control files for the plan tier, then:

creates .fast.cron.cnf if it is missing, before the ladder runs at all;
under 9 instances (and with no POWER / PHANTOM / CLUSTER / ULTRA / MONSTER plan), or between 9 and 50 instances, keeps Fast — it re-creates .fast.cron.cnf and removes .slow.cron.cnf and /root/.tg.cnf;
above 50 instances switches to Slow — it removes .fast.cron.cnf and writes .slow.cron.cnf and /root/.tg.cnf;
then, in separate arms that run afterwards, re-asserts Fast for a POWER, PHANTOM or CLUSTER plan (also writing /root/.tg.cnf), so a large-plan box above 50 instances ends up Fast again. ULTRA and MONSTER are grepped only to disqualify the under-9 arm; they have no arm of their own.

Every .slow.cron.cnf removal in that ladder is skipped when .slow.cron.cnf.protected exists — which is always the case on a box runner.sh forced Slow, and where the file is immutable anyway.

On a self-hosted box nothing writes .fast.cron.cnf, so it is a pure operator knob there. .slow.cron.cnf is different: runner.sh itself writes it — and makes it immutable — on any box with ≤ 4096 MB RAM, hosted or not.

On CI / Jenkins-style boxes (/etc/boa/.look.like.jenkins.cnf, or _FORCE_CI_BOX=YES in /root/.barracuda.cnf) the automatic queue is off by default. It runs only when a higher Octopus plan (POWER / PHANTOM / CLUSTER / ULTRA / MONSTER) is present, or /etc/boa/.allow.aegir.queue.cnf exists (or _ALLOW_AEGIR_QUEUE=YES is set in the same cnf), and at least one instance opts in with /data/disk/*/static/control/run-aegir-queue.info.

The registered queues

This hosting fork registers three queues by default via hook_hosting_queues: tasks, cron, and task_gc — and a fourth, migrate_source, arrives with the Migration source feature, which every Octopus install and upgrade enables. hosting-dispatch forks the matching hosting-<queue> worker when each is due — the same path for all of them:

tasks — the main task queue (hosting-tasks); processes Queued task nodes into provision-<type> calls.
cron — hosting_cron runs advanced Drupal cron on hosted sites. BOA additionally hardens this with its own reliability patch (aegir/patches/hosting_cron_queue-reliability.patch).
task_gc — hosting_task_gc garbage-collects old completed task nodes / task logs (keeps the last N completed per entity, deletes the rest; configurable per instance).
migrate_source (Migration source grants) — one whole-instance reconciliation pass of migration-source grants per run, serial, daily by default — and on BOA boxes the nightly maintenance window runs the sweep itself inside a box-wide task-queue pause and keeps this queue switched off per instance; registered while the Migration source feature is enabled — which every Octopus install and upgrade does — and tunable at Hosting → Queues like the rest.

There is no separate backup or statistics queue in this fork: a per-site backup is a backup task node processed by the tasks queue, not its own dispatch queue.

New task types are added by a hosting module implementing hook_hosting_tasks() (the object token — site, platform — is an array key, $tasks['site']['backup'], not part of the hook name).

A note on `hosting_queued`

Upstream Ægir offers two ways to drain the queue: the cron-driven dispatcher (above) and an optional long-running daemon (hosting_queued). The hosting/queued/ module ships in the omega8cc/hosting fork because it is inherited from upstream, but BOA does not run it — there is no init script, no service, no enablement anywhere in the BOA stack. So on a BOA host there is no hosting_queued daemon process, no /var/log/hosting_queued*.log, and no service hosting-queued-* … unit. Upstream docs describing a queue daemon do not apply.

Pause / resume

To pause all task processing on a host (maintenance window, manual DB fix), create the BOA maintenance-pause control file — runner.sh checks it on every run and exits immediately while it exists:

# Pause: runner.sh (and several other BOA jobs) stop touching the queue
touch /etc/boa/.pause_tasks_maint.cnf

# Resume
rm -f /etc/boa/.pause_tasks_maint.cnf

Six BOA jobs honour it: runner.sh, killer, weblogx, manage_ltd_users.sh and purge_binlogs.sh exit immediately while it exists, and autosymlink defers a narrow single-site files-store apply rather than interleaving it with the global sweep.

Do not pause by parking the aegir crontab instead. Moving /var/spool/cron/crontabs/aegir aside (the classic leading-dot park to crontabs/.aegir) does not stick: runner.sh calls _enable_master_cron on every pass, which moves a parked crontab straight back to crontabs/aegir and reloads cron — so a hand-park is silently undone within a minute. That holds off a standby only: on a box carrying /root/.standby.cnf the runner does the opposite and parks the crontab itself, and the master upgrade path writes the crontab in parked form while the marker exists. The marker file above is the supported pause precisely because it stops the runner's whole pass, un-parking included. (The reverse park exists in the code as _disable_master_cron, but it is the runner's own internal tool, not an operator interface.)

On an omega8.cc-hosted box a hand-set pause does not survive. There (/root/.host8.cnf present, or a hostname ending .aegir.cc) autoupboa owns the file, keyed on /run/boa_run.pid: it deletes the pause file on every pass whenever no BOA run holds that pid, and creates it when a pass does run while one is held. The five-minute driver is clear.sh, which starts autoupboa only when /run/boa_run.pid is absent — so on that path only the delete arm can fire, and an operator pause is cleared within minutes. It is not a usable interlock for anything longer (the ghost-cleanup 48-hour hold exists precisely because of this). On a self-hosted box nothing rewrites it and the create/remove pair above is the whole story.

Under high load BOA also pauses the queue automatically via /run/max_load.pid / /run/critical_load.pid until load recedes. There is no per-queue hostmaster-pause Drush command — the hosting-pause / hosting-resume commands are unrelated (they bracket a migration of the Hostmaster site itself to a new platform).

BOA's own nightly maintenance uses a separate, dedicated queue-stop file: /run/boa_queue_stop.pid holds the Ægir task queue while a maintenance operation moves data. Every holder follows the same discipline — take the marker only when it is free, skip the work rather than piggyback on someone else's window, and remove it only if the PID recorded inside is still its own. Four take it: the backups-to-static-fs relocation in night/10-account.sh, updatesymlinks --auto-fix, the nightly migration-source grant sweep in night/90-global-post.sh, and the operator-run migratefs storage relocation.

runner.sh honours it twice:

the parent exits immediately at startup;
each per-instance child dispatch is skipped if the file appears mid-pass ("Task queue paused: /run/boa_queue_stop.pid present -- skipping").

It is self-healing by design and can never freeze the queue for good: the file lives in /run (cleared on reboot) and clear.sh (cron */5) purges it as soon as the owner PID recorded inside it is gone, while a legitimately long move keeps its hold.

Treat it as a mechanism you may observe, not a control to set by hand — for planned maintenance use /etc/boa/.pause_tasks_maint.cnf as above.

When tasks fail

Two common operator-facing failures:

Task hangs ("spinning") — usually Drush could not reach Percona (a restart mid-task) or hit a network timeout. Confirm runner.sh is firing, then re-run.
Task fails immediately — read the task log in the UI; it carries the Drush + Provision output. Most common causes: bad file ownership, a network issue, or DB corruption.

To force the queue forward by hand:

# Run one Octopus instance's runner immediately
bash /var/xdrago/run-o1

# Or call the dispatcher directly for that instance
su -s /bin/bash - o1 -c "drush @hostmaster hosting-dispatch"

# Is the cron runner firing at all? (last-run marker + live process)
ls -l /var/log/boa/last-run-*
pgrep -fc 'runner.sh'

A crashed dispatch self-heals: the dispatch semaphore (hosting_dispatch_<queue>_running) is acquired with a 15-minute lifetime, so a dispatcher that died holding it blocks that queue for at most 15 minutes before the lock expires on its own. If you need the queue back sooner than the automatic bound, clear the semaphore for that instance by hand:

su -s /bin/bash - o1 -c \
  "drush @hostmaster sqlq \"DELETE FROM semaphore WHERE name LIKE 'hosting_dispatch_%'\""

The per-Octopus Hostmaster alias is @hostmaster (equivalently @hm), resolved from that instance's own ~/.drush/ — so run it as the instance user as above. There is no @hm-oN / @hm-o1 alias; that form does not exist and will fail with an unknown-alias error.

See Troubleshooting for the broader task-failure playbook.

Architecture overview — the layered model and runtime cadence table.
Entity & service model — the entities tasks operate on and their status constants.
Multi-Octopus model — why dispatch is per-instance.
Control files & INI — .fast.cron.cnf / .slow.cron.cnf / .pause_tasks_maint.cnf.
Reference appendix — the hosting-* and provision-* command index; the queue markers above are in the control-file index.