Cache & cron tuning

The most-cited operational failure mode on busy BOA hosts: intermittent PluginNotFoundException errors, self-resolving after a few minutes, recurring in bursts. Three distinct root causes, diagnosed here in priority order, plus the APCu-flush sentinel for clearing stale per-worker cache.

Applies to BOA 5.x, Drupal 8/9/10, the PHP-FPM + APCu + Valkey/Redis stack on Ægir-managed sites.

The cache stack

The default cache backend is cache.backend.redis (Valkey/Redis). On a standalone (non-cluster) host BOA then overrides the bootstrap / discovery / config bins to cache.backend.chainedfast:

APCu (per-worker, fast)  →  Valkey/Redis (shared)  →  Database

The chainedfast override is gated on not being a cluster node (if (!is_readable('/data/conf/clstr.cnf'))): on a cluster node those three bins stay on cache.backend.redis (the shared tier only), so the per-worker APCu tier does not apply and the APCu-flush guidance below is moot there. The redis_exclude_bins site/platform INI can additionally push named bins onto cache.backend.database (also only off a cluster node).

When Valkey is starved for memory the discovery cache is evicted; rebuilds under concurrent traffic can produce incomplete entries; plugin-not-found errors cascade. Do not override the chainedfast defaults without understanding the trade-offs (see the anti-pattern below).

Diagnose Valkey first

Memory starvation is the most common root cause on busy hosts. Get the Valkey password from /root/.valkey.pass.txt (the Redis variant uses /root/.redis.pass.txt; the PHP side reads /data/conf/valkey/pass.inc). There is no _VALKEY_PASSWORD variable in /root/.barracuda.cnf.

_PASS=$(cat /root/.valkey.pass.txt)

# Memory ceiling vs. current usage
valkey-cli -a "${_PASS}" config get maxmemory
valkey-cli -a "${_PASS}" info memory | grep -E 'used_memory_human|maxmemory_human|mem_fragmentation_ratio'

# Hit rate + eviction count
valkey-cli -a "${_PASS}" info stats | grep -E 'keyspace_hits|keyspace_misses|evicted_keys'

Healthy: used_memory_human well below maxmemory_human; evicted_keys low or zero; hit rate above 85–90% (keyspace_hits / (hits + misses)).

Unhealthy: used_memory_human at or near maxmemory_human (ceiling too low); evicted_keys non-zero and climbing (active eviction — the cause of discovery-cache loss); hit rate below 50% on a busy host (almost always maxmemory too low).

Issue 1 — Valkey memory starvation (most common)

Symptoms. Intermittent WSOD bursts that self-resolve after a few minutes; watchdog Drupal\Component\Plugin\Exception\PluginNotFoundException entries; the same plugin named across multiple incidents (vs. cache poisoning, where it varies); a full cache clear resolves temporarily but errors return.

Root cause. Valkey hard-capped at maxmemory. With the ceiling too low, eviction runs continuously (maxmemory-policy volatile-lfu), the discovery bin gets evicted, worker requests trigger rebuild from DB, concurrent rebuilds produce incomplete entries, and a worker reading an incomplete entry throws PluginNotFoundException.

The shipped default formula:

_MAX_MEM_VALKEY=$(( _RAM / 3 ))

_RAM is usable RAM in MiB (from free -mt) after the BOA reserve is subtracted: _RESERVED_RAM, when left at its default 0, is auto-set to a quarter of total RAM, then _RAM = total − _RESERVED_RAM. The cap is written to valkey.conf as ${_MAX_MEM_VALKEY}MB. So on a 24 GB host the cap is roughly a third of usable RAM after the reserve — not a clean 24/3 = 8 GB. The instance is also configured maxmemory-policy volatile-lfu with RDB snapshots disabled (save ""). The current formula is hard-coded to _RAM / 3 and rewritten on every tune run, so there is no per-install variable to carry forward.

Fix:

_PASS=$(cat /root/.valkey.pass.txt)
valkey-cli -a "${_PASS}" config set maxmemory 8gb
valkey-cli -a "${_PASS}" config rewrite

watch -n 10 "valkey-cli -a ${_PASS} info stats | grep -E 'keyspace_hits|keyspace_misses|evicted_keys'"

evicted_keys should drop to zero within minutes; hit rate re-warms over 15–30 min.

RAM sizing for D8/9/10 hosts:

If the host is under-RAMmed for its site count, increasing Valkey within existing RAM helps temporarily; the real fix is more RAM.

Issue 2 — Valkey cache poisoning via CLI drush

Investigate only after confirming Valkey memory + hit rate healthy.

Symptoms. Burst pattern at regular intervals (e.g. every hour); different plugins fail across incidents (vs. Issue 1's same-plugin pattern); a full cache clear reliably ends the burst; Valkey eviction stats look fine.

Hypothesis. CLI drush running a cache rebuild can write partial discovery entries into Valkey mid-rebuild; FPM workers read poisoned entries and fail until the next clean rebuild. This is also why Ægir's own scheduled cache clears don't trigger the fault: the nightly maintenance runs a full drush cache-clear all flush, so every discovery entry is dropped and workers rebuild cleanly on the next request — the opposite of the single poisoned entry a partial mid-rebuild CLI-drush write leaves behind.

Actions:

1. Disable Drupal core's Automated Cron on all Ægir-managed sites.
2. Switch Ægir to web-based cron instead of the drush backend. In the Hostmaster front-end this is the hosting_cron_use_backend setting: it defaults to TRUE (the drush/CLI backend — the poison source). For a regular managed site in backend mode hosting_cron_queue (hosting/cron/hosting_cron.module) runs two drush invokes, provision_backend_invoke … "elysia-cron", then sleep(3), then provision_backend_invoke … "cron"; the Hostmaster site itself is instead a single "cron" invoke, and it always uses the drush backend regardless of this toggle. Setting the toggle to FALSE makes the module hit each site over HTTP instead (https://<site>/cron/<cron_key>, or cron.php?cron_key= on pre-D8). That runs cron in FPM/web context against the live, fully rebuilt cache, which is what avoids the mid-rebuild poisoning.
3. If web-cron times out (e.g. Scheduler module misses its schedule), raise PHP execution time in /opt/etc/fpm/fpm-pool-common*.conf:
   INICopy

   php_admin_value[max_execution_time]     = 300
   php_admin_value[max_input_time]         = 300
   php_admin_value[default_socket_timeout] = 300
   

The fpm-pool-common*.conf files are overwritten on barracuda upgrade — re-apply after upgrades.

Issue 3 — class-not-found / file-unreadable on HDD-backed hosts

Symptoms. Drupal\Component\Plugin\Exception\PluginException: Plugin instance class "…" does not exist, even though the .php file IS on disk; self-resolves after a few minutes or on Verify.

Cause. HDD I/O saturation under disk pressure → the class autoloader hits a filesystem latency spike → PHP autoloader times out → "class not found". Common triggers: cache rebuilds, drush ops, Composer runs.

Mitigation. No software fix for under-resourced hardware. Reduce drush-based cron (Issue 2). Keep the chainedfast defaults intact so frequent reads stay in memory tiers, not disk. If the host has rotational storage, migrate to NVMe — that is the actual fix.

Stale cache namespaces after a site is recreated

Symptoms. A freshly installed site answers on a route belonging to a site of the same name deleted earlier: a phantom path, the old site's name in page titles, the front page returning 404. Neither a cache flush on the new site nor an FPM restart cures it.

Cause. Every hosted site shares one Valkey/Redis instance and each site's keys are scoped by prefix alone, so a site recreated under a name a deleted site used to hold inherits that namespace — the deleted site's compiled service container and routing included. A fresh install can then rebuild its own router table from the stale container, which turns cache poison into durable site state.

The purge that closes it. Provision clears the namespace itself. For Drupal sites the Delete task runs it as a post hook (site contexts only, and after the vhost is gone, so no web worker can repopulate the namespace mid-purge), and the Install and Deploy tasks run it as a pre hook — which is what covers clone, rename/migrate and restore, since those land a site without passing through provision-install. Backdrop sites have the equivalent hooks in the Backdrop half of the same tree.

What is swept. For Drupal: md5(<host>)_sn_* (D8+, compiled container and routing), md5(<host>)_n_* (D7) and <host>_z_* (front-end page cache), where <host> is the URI's host part, lowercased and www.-stripped. For Backdrop: md5(<db_host><db_name>). Every populated database is swept, enumerated from INFO keyspace with the standard 0–15 range as a fallback, because containers have been observed outside the configured base.

One shared-namespace consequence. Subdirectory sites share their parent host's namespace, so purging one flushes its siblings' cached entries. That is cache only, and unavoidable — the namespace does not distinguish them.

Non-fatal by design. A cache purge never fails the task that triggered it. A box with no cache-store wiring, or with no store installed at all, logs at info level and counts as a clean pass, and a URI whose host part is not a plain hostname is skipped with a warning.

What to grep in a task log. Every line carries the VALKEY/PURGE prefix. Success reads removed <n> stale cache keys in db <n> (prefix <prefix>), and a clean pass reads no stale cache keys found. A warning means the purge did not complete and stale keys may remain: could not connect to <remote>, authentication failed, connection lost …, database <n> refused, scan failed in db <n> for prefix <prefix>, and sweep bound reached for prefix <prefix> (the sweep re-runs up to eight times to catch keys written by still-in-flight workers, and reports when it hits that bound).

Anti-pattern — don't disable chainedfast

The tempting "fix":

// /sites/<domain>/local.settings.php
$settings['cache']['bins']['discovery'] = 'cache.backend.database';

This bypasses the cache stack entirely — a performance regression on every request, since discovery now hits the DB. Fix the root cause instead.

Correct Drush usage

Two mistakes produce errors easily misread as cache issues:

1. Run drush as oN.ftp under lshell, not as oN under bash. Running drush as the bare oN system user under bash bypasses the limited-shell wrapper and produces errors easily mistaken for cache faults.
2. For D8+, use the site-local Drush (vdrush), not system drush8 — system Drush 8 is for D7 only.

APCu memory sizing

apc.shm_size=256M    # template seed in phpNN.ini, NOT the live value

The 256M in phpNN.ini is only the shipped template seed. _tune_memory_limits rewrites it to the box-wide _USE_APC budget on every tune run and on barracuda upgrade (sed "s/256M/${_USE_APC}M/g"), so the live apc.shm_size already scales with RAM — a hand-edit to 512M is reverted on the next upgrade. _USE_APC = _USE_FPM is derived purely from usable RAM (the budget ladder in FPM capacity sizing) with no operator override, so the durable lever for more APCu is more RAM (or a smaller _RESERVED_RAM), not a phpNN.ini edit. For a quick one-version test you can still set it and service phpNN-fpm reload, but treat it as transient. APCu sizing is secondary to Valkey sizing — increase Valkey first.

Nightly Boost cache clearing (legacy D6/D7 platforms)

A different cache entirely: the Boost module's static page cache on legacy Drupal 6/7 platforms — unrelated to the D8+ chainedfast (APCu/Valkey) stack the rest of this page covers. On platforms carrying the o_contrib (Drupal 6) or o_contrib_seven (Drupal 7) bundles, the nightly per-site maintenance (aegir/tools/system/night/20-sites.sh) clears the platform-level Boost cache: _fix_boost_cache wipes <platform>/cache/* plus the cache/.boost and cache/.htaccess markers, creates the cache dir if missing (only when sites/all/drush/drushrc.php exists), and ensures it is owned oN:www-data mode 02775.

The call is gated by _CLEAR_BOOST=YES inside the o_contrib/o_contrib_seven platform branch. Default is YES (the shipped barracuda.sh.cnf default; the nightly run also falls back to YES when the variable is unset). Set _CLEAR_BOOST=NO in /root/.barracuda.cnf to keep Boost static caches across nights — the shipped cnf comment warns that with NO, boost cron and/or the Expire module must handle cache removal. _CLEAR_BOOST is a first-class persistent barracuda.cnf variable (written on fresh install, back-filled on upgrade) — see its entry in the barracuda.cnf reference.

Self-service APCu flush (graceful FPM reload)

APCu is a per-worker cache inside the FPM worker processes. Unlike Valkey it cannot be flushed remotely — a stale entry (old field definition, plugin registry, Solr core mapping) only clears when the worker is recycled. After a config change, platform update, or Solr core rename, stale APCu is a common source of FieldException / PluginNotFoundException even when Valkey is healthy.

BOA can graceful-reload every installed PHP-FPM version without dropping active connections, driven by a sentinel file the monitor watches. The sentinel lives in an Octopus account's control dir, so it is touched from that account (~ = /data/disk/<oN>):

# From the Octopus account (~ resolves to /data/disk/<oN>):
touch ~/static/control/run-php-fpm-reload.pid

The monitor (aegir/tools/system/monitor/check/php.sh) detects the sentinel, performs the graceful reload (clearing APCu pool-wide), and removes the file automatically. A cooldown (_FPM_COOLDOWN_SECS, default 30 s, shared across versions) prevents reload storms if the sentinel is recreated repeatedly. Know the cooldown's sharp edge: a sentinel dropped inside the cooldown window is discarded, not deferred — the file is consumed, no reload happens, and the incident log records the skip. Re-touch the sentinel after the window if the reload still matters. The window is also shared with the php84 health-restart path, so an unrelated auto-heal can start the clock.

Availability gate. Enabled only for accounts on POWER, PHANTOM, CLUSTER, ULTRA, or MONSTER plans (matched in /root/.<oN>.octopus.cnf), or box-wide via _ALLOW_PHP_FPM_RELOAD=YES in /root/.barracuda.cnf (the former /etc/boa/.allow.php.fpm.reload.cnf flag stays honoured during the conversion window; until a box has taken a full Release-N pass, a hand-set variable needs the flag as well). System cnf only — a same-named line in an octopus cnf is unsupported. On any other host the sentinel is ignored. This is the correct fix for stale per-worker APCu — after you have ruled out Valkey starvation (Issue 1) and cache poisoning (Issue 2). Flushing APCu does nothing for a maxmemory ceiling that is too low.

Diagnostic log paths

PHP-FPM error logs are a single directory, one file per version:

/var/log/php/phpNN-fpm-error.log     # e.g. /var/log/php/php84-fpm-error.log

There is no per-version /var/log/phpNN/ directory and no php-fpm.log file — every version logs to /var/log/php/phpNN-fpm-error.log (aegir/conf/php/phpNN-fpm.conf error_log = …; the directory is created once by mkdir -p /var/log/php).

Checklist

• Check Valkey hit rate + eviction count.
• If unhealthy: raise maxmemory to ~RAM/3.
• If RAM under-allocated for site count: escalate a RAM bump.
• Disable Drupal core's Automated Cron site-wide.
• If drush-based cron triggers bursts: set hosting_cron_use_backend = FALSE (web-based Ægir cron).
• If web cron times out: raise PHP execution time in fpm-pool-common*.conf.
• Revert any local.settings.php discovery-bin override.
• Confirm drush ops run as oN.ftp, not oN.
• Confirm site-local vdrush for D8+ sites.
• If APCu utilisation >75%: add RAM (lifts _USE_APC); a manual apc.shm_size bump is reverted on the next tune/upgrade.
• If stale APCu suspected (post config/platform/Solr-core change): touch ~/static/control/run-php-fpm-reload.pid for a graceful all-versions FPM reload (qualifying plans or _ALLOW_PHP_FPM_RELOAD=YES).

Related

• FPM capacity sizing — the worker/memory_limit sizing that prevents pool starvation.
• opcache & APCu — the cache shares, phpNN.ini settings, and the read-only fpm_tune/fpmreport samplers.
• Database (MySQL/Percona) — Percona side of the same hosts.
• Operator troubleshooting & recovery — the WSOD/502 forms of these errors and task-recovery.
• Self-healing monitor stack — the monitor that services the reload sentinel.