Skip to content
BOA Documentation

Powered by Grav

Recovery cycles & cache faults

Recovery cycles & cache faults

Two operationally distinct failure families share one symptom surface ("a task suddenly fails", "the site WSODs intermittently"):

  • platform/Drush state drift, fixed by the Verify/Unlock/Verify recovery cycle;
  • the APCu/Valkey/cron plugin-discovery cascade, fixed by sizing and cron discipline.

Diagnose which one you have before acting — the recovery cycle does nothing for a Valkey memory ceiling, and resizing Valkey does nothing for a platform whose local Drush is locked.

The platform recovery cycle

When a platform is "in a weird state" after a failed task — Drush version mismatch errors, lock state out of sync, Migrate/Clone/Verify aborting unexpectedly — run the three-step cycle:

TXT
1. Platform → Verify + Lock Drush
2. Platform → Unlock Local Drush
3. Platform → Verify + Lock Drush     (yes, again)

The middle Unlock Local Drush step un-patches Drupal core and removes the codebase permission locks; the final Verify + Lock re-establishes the canonical Verified+Locked state Ægir expects (dunlock/relock on Verify is implemented in the BOA hosting_platform/provision fork, not a generic toggle). This resolves:

  • LoggerChannel::log() declaration errors — from running a site task (Clone/Migrate/Verify) against a platform left Unlocked/unpatched: one never taken through Verify + Lock, or one not re-locked after site-local Drush or Composer work. Re-running Lock or Unlock Local Drush on a platform already in that state is itself a harmless no-op, not the trigger.
  • Site-local vdrush "doesn't work" — when the platform's local Drush is locked (the default post-Verify state), run Unlock Local Drush first; run Verify + Lock Drush again when finished. (The other cause is connecting as oN instead of oN.ftp — see lshell & limited users.)
  • Orphan site node with no working Drush alias after a failed Migrate/Clone — drush sa | grep -i foo.example.com empty means no on-disk alias; Verify the site to regenerate it from the node, and if that fails, run the full cycle at the platform level first.

Permission-drift Verify failures

Composer or manual file ops leave files with the wrong owner. Repair with the fix-drupal-* scripts (equals-form flags only) against the real platform path, then run the recovery cycle:

SH
# As oN.ftp
fix-drupal-platform-permissions.sh --root=/data/disk/<USER>/static/<platform>
fix-drupal-platform-ownership.sh   --root=/data/disk/<USER>/static/<platform>

Composer broke the platform ("code path X doesn't exist")

Composer run on a platform after sites were attached modifies vendor/ and composer.lock in ways Ægir does not expect. Do not try to repair in place. Instead:

  1. Roll the platform back to its pre-Composer state from git or backup.
  2. Build a new platform carrying the Composer changes.
  3. Migrate the affected sites onto it.

Anti-pattern — blanket re-Verify

Re-Verifying everything on a stuck box can make it worse: a Verify that hits a permission error mid-run can leave files in a worse state, and one that crashes mid-DB-update can corrupt schema state. Read the task log, fix the named root cause, then re-Verify.

APCu / Valkey / cron — the PluginNotFoundException cascade

The most-cited operational fault on busy hosts: intermittent PluginNotFoundException / PluginException / FieldException bursts that self-resolve after a few minutes and recur. BOA wires Drupal's bootstrap/discovery/config cache bins to cache.backend.chainedfast:

TXT
APCu (per-worker, fast)  →  Valkey/Redis (shared)  →  Database

Three distinct root causes, diagnosed in priority order. Do not override the chainedfast defaults to "fix" this — see the anti-pattern below.

Always check Valkey first

The cache password is in /root/.valkey.pass.txt (Redis hosts: /root/.redis.pass.txt); the PHP side reads /data/conf/valkey/pass.inc. There is no _VALKEY_PASSWORD control variable.

SH
_VPASS=$(tr -d '\r\n' < /root/.valkey.pass.txt)
valkey-cli -a "${_VPASS}" config get maxmemory
valkey-cli -a "${_VPASS}" info memory | grep -E 'used_memory_human|maxmemory_human|mem_fragmentation_ratio'
valkey-cli -a "${_VPASS}" info stats  | grep -E 'keyspace_hits|keyspace_misses|evicted_keys'
  • Healthy: used_memory_human well below maxmemory_human, evicted_keys low or zero, hit rate above 85–90 %.
  • Unhealthy: usage at the ceiling, evicted_keys climbing, or hit rate below 50 % on a busy host — all point to maxmemory too low.

Cause 1 — Valkey memory starvation (most common)

When Valkey is hard-capped too low, volatile-lfu eviction runs continuously, the discovery bin gets evicted, concurrent rebuilds write incomplete entries, and workers reading them throw PluginNotFoundException. Tell-tale: the same plugin named across incidents.

The shipped cap formula:

TXT
_MAX_MEM_VALKEY = _RAM / 3      # written to valkey.conf as ${_MAX_MEM_VALKEY}MB

_RAM is usable RAM in MiB (from free -mt) after the BOA reserve is subtracted: _RESERVED_RAM defaults to _RAM / 4 (overridable in /root/.barracuda.cnf). So on a 24 GB box the cap is ~RAM/3 of usable RAM after the reserve, not a clean 24/3.

BOA also sets maxmemory-policy volatile-lfu and disables RDB snapshots (save "") — this is a cache, not a store. Very old installs used _RAM / 6; if you are still carrying that, raise it.

Fix:

SH
valkey-cli -a "${_VPASS}" config set maxmemory 8gb
valkey-cli -a "${_VPASS}" config rewrite
watch -n 10 "valkey-cli -a ${_VPASS} info stats | grep -E 'keyspace_hits|keyspace_misses|evicted_keys'"

evicted_keys should drop to zero within minutes; hit rate re-warms over 15–30 min. If the box is genuinely under-RAMmed for its site count, this buys time but the real fix is more RAM.

Cause 2 — Valkey cache poisoning via CLI drush

Investigate only after Valkey memory and hit rate check out.

Signature:

  • bursts at regular intervals (e.g. hourly),
  • different plugins failing across incidents,
  • a full cache clear reliably ending the burst,
  • eviction stats fine.

Hypothesis: a CLI drush cache rebuild writes partial discovery entries into Valkey mid-rebuild; FPM workers read the poisoned entries until the next clean rebuild.

Mitigation:

  1. Disable Drupal core's Automated Cron on all Ægir-managed sites.
  2. Use Ægir's wget-based cron instead — it writes from FPM context with a fully rebuilt cache.

  3. If wget-cron times out (e.g. Scheduler module misses its window), raise the PHP execution-time band in the FPM common pool files (these overwrite on barracuda upgrade — re-apply afterward):

    TXT
    # /opt/etc/fpm/fpm-pool-common.conf (and -legacy.conf / -modern.conf)
php_admin_value[max_execution_time]     = 300
php_admin_value[max_input_time]         = 300
php_admin_value[default_socket_timeout] = 300

Cause 3 — class-not-found / unreadable on HDD-backed hosts

PluginException: Plugin instance class "…" does not exist while the .php file is on disk, self-resolving after minutes or on Verify. This is HDD I/O saturation: the autoloader hits a filesystem latency spike and times out. Common triggers: cache rebuilds, drush ops, Composer runs.

No software fix for under-resourced hardware:

  • reduce drush-based cron (Cause 2),
  • keep the chainedfast defaults so reads stay in the memory tiers, and
  • migrate rotational storage to NVMe.

That is the actual fix.

Anti-pattern — disabling chainedfast

PHP
// /sites/<domain>/local.settings.php  — do NOT do this
$settings['cache']['bins']['discovery'] = 'cache.backend.database';

This bypasses the cache stack entirely: a discovery DB hit on every request. Fix the root cause instead and revert any such override.

APCu sizing & the self-service graceful FPM reload

APCu is a per-worker cache inside the FPM worker processes; it cannot be flushed remotely.

A stale APCu entry (an old field definition, plugin registry, or Solr-core mapping) clears only when the worker is recycled — so after a config change, platform update, or Solr core rename, stale APCu is a common source of FieldException / PluginNotFoundException even when Valkey is perfectly healthy.

Sizing. Default apc.shm_size=256M, set in each version's per-version php.ini (/opt/php<ver>/etc/php<ver>.ini, with /opt/php<ver>/lib/php.ini as the alternate build layout — there is no separate apcu.ini); for 100+ sites raise the apc.shm_size line to 512M and reload that version's FPM. APCu sizing is secondary to Valkey sizing — raise Valkey first.

Self-service flush. BOA exposes a sentinel that triggers a graceful reload of every installed PHP-FPM version (clearing APCu pool-wide) without dropping active connections:

SH
# As the site owner, from the Octopus account
touch ~/static/control/run-php-fpm-reload.pid

The monitor (monitor/check/php.sh) detects the sentinel within seconds, performs the reload, and removes the file. A cooldown (_FPM_COOLDOWN_SECS, default 30 s, tracked via /run/php84-fpm.cooldown) prevents reload storms.

The sentinel is honoured only for:

  • accounts on POWER, PHANTOM, CLUSTER, ULTRA, or MONSTER plans, or
  • hosts carrying /etc/boa/.allow.php.fpm.reload.cnf.

On any other plan the plan gate returns before the file is even inspected, so the sentinel is simply ignored and left in place (not removed) — if you create it and it never clears, the account's plan does not qualify.

This is the correct fix for stale per-worker APCu — but only after ruling out Valkey starvation (Cause 1) and poisoning (Cause 2). Flushing APCu does nothing for a maxmemory set too low.

Correct Drush usage on BOA

Two mistakes produce errors easily misread as cache faults:

  • Run drush as oN.ftp under lshell, not as oN under bash.
  • For D8+, use site-local Drush (vdrush), not system drush8 — system Drush 8 is for D7 only.

See lshell & limited users for the limited-shell and Drush-version selection details.

© 2026 BOA Documentation. All rights reserved.