Nginx debugging

Three recurring Nginx-level failure modes on a BOA host:

• 502 Bad Gateway / 504 Gateway Timeout on specific URLs or a whole site — almost always PHP-FPM or downstream, not Nginx.
• High-load 503 — the reduced-feature mode second.sh switches on under CPU pressure.
• Redirect loop to /install.php after a clone-then-migrate.

The diagnostic value here is using the correct runtime paths: BOA's FPM sockets and logs are not where a generic Nginx guide would put them.

502 vs 504

Nginx's FastCGI read timeout is a fixed 180 s (server.tpl.php:196-198). Separately, _PHP_FPM_TIMEOUT in octopus.cnf (default AUTO = 180) sets the FPM pool's request_terminate_timeout, clamped to 60–180 (lib/functions/system.sh.inc:1315-1327) — it can only be lowered, and lowering it makes FPM reap a stuck request before Nginx times out, surfacing as a 502 rather than a 504. Both errors point at PHP-FPM or downstream (Drupal bootstrap, DB, Redis/Valkey), not at Nginx itself.

Where to look, in order

1. Nginx error log

tail -f /var/log/nginx/error.log

The upstream-related lines around the failure time:

upstream prematurely closed connection while reading response header from upstream   (→ 502: PHP segfaulted or was killed)
upstream timed out (110: Connection timed out) while reading response header          (→ 504: PHP is slow)
connect() to unix:/run/<pool>.fpm.socket failed                                       (→ 502: FPM pool not running)

The socket path in the third line is the real BOA FPM socket form (below) — if you see a different path, the vhost is referencing a stale upstream.

2. PHP-FPM error log

All PHP/FPM logs go to a single /var/log/php/ directory with version-tagged filenames — there is no per-version /var/log/phpNN/ directory:

tail -f /var/log/php/php84-fpm-error.log         # error_log, per version
tail -f /var/log/php/fpm-<pool>-slow.log         # slowlog, per pool
tail -f /var/log/php/opcache-<pool>-error.log    # opcache, per pool

Substitute the PHP version (php74, php83, php84, …) and pool name as needed. Look for:

• child <pid> exited on signal 11 (SIGSEGV) — segfault, often an APCu / opcache issue.
• child <pid>, script '…' executing too slow — a slow request approaching the timeout.
• server reached max_children setting — the pool is saturated; raise the pool size (see PHP-FPM & performance).

3. Drupal watchdog

# Legacy Drupal 6/7 (system Drush 8):
drush @<site-alias> watchdog-show --count=20
# Modern Drupal 8+ (site-local vdrush, from the platform app root; multi-dot
# aliases hyphenate every dot but the last — a bare domain's alias is unchanged):
vdrush @<site-alias> watchdog:show --count=20

When PHP-FPM logged nothing useful, Drupal's own log often carries the DB or cache failure behind the 502.

4. Confirm services are up

for v in 5.6 7.0 7.1 7.2 7.3 7.4 8.0 8.1 8.2 8.3 8.4 8.5; do
  vv=${v//./}
  echo -n "PHP-FPM $v: "; service php${vv}-fpm status 2>/dev/null | head -1
done
service mysql status | head -1
service redis-server status 2>/dev/null | head -1
service valkey-server status 2>/dev/null | head -1

Restart anything stopped with service <name> start and watch whether the 502/504 clears.

5. Check the high-load 503 path

Under high CPU load, second.sh activates the rule by renaming /data/conf/nginx_high_load_off.conf to nginx_high_load.conf — the inactive _off copy is the deployed default, and dropping the _off suffix is what arms it (the mv in second.sh's _nginx_high_load_on(); the file is pulled in via the include /data/conf/nginx_high_load.c*; glob in vhost_include.tpl.php).

It reloads Nginx, switching on reduced-feature mode: the fragment's if ($deny_on_high_load) { return 503; } returns 503 only (not 502) to the matched crawlers/spiders. The $deny_on_high_load map in the master config (server.tpl.php:866-868) flags only crawler-class user-agents (crawl|bot|spider|tracker|click|parser|google|yahoo|yandex|baidu|bing — the bare bot token matters: without it every UA self-identifying only as …bot would dodge high-load shedding) and leaves the default empty, so ordinary human visitors pass and bots are shed.

The toggle pair _nginx_high_load_on() / _nginx_high_load_off() (the reverse rename) lives in second.sh; the SPIDER-tier load checks arm it on either average and the NORMAL branch disarms it once both readings are back at or below the spider threshold.

It logs to /var/log/boa/high.load.incident.log:

tail -f /var/log/boa/high.load.incident.log

Recent entries near the failure time mean the box was shedding load.

Distinguish the full pause from a plain 502

Distinct mechanism, same incident log: the load controller's MAX/CRIT pause (second.sh _hold_services) fully stops Nginx and every PHP-FPM via init.d — that produces connection-refused, not 502/503. A 502 is a third, unrelated case (FPM crashed). Don't conflate the three.

Common 502/504 causes by frequency

Redirect loop to /install.php

Symptom: after a clone-then-migrate, Nginx redirects every request to /install.php, which itself returns 404.

Cause: the site directory is missing the sites/<domain>/settings.php that Ægir writes on Install — the migrate did not re-create it on the new platform.

Fix

1. Confirm the site directory exists on the new platform (Octopus instance root is /data/disk/<USER>, not /home/<USER>):
   SHCopy

   ls -la /data/disk/oN/static/<platform>/sites/<domain>/
   

2. If settings.php is missing, re-emit it — run Verify on the site, which re-writes settings.php from the Ægir template:
   SHCopy

   drush @<site-alias> provision-verify
   

3. If Verify also fails, the site's DB connection may be misconfigured — read the failing Verify task's log in the Ægir front end, or re-run drush @<site-alias> provision-verify -d and read the backend output. (Instance-level install/upgrade failures land in /data/disk/oN/log/install.log and log/octopus_log.txt; there is no per-site log under log/.)
4. Last resort: restore the site from a known-good backup.

Diagnostic recipes

Is Nginx the problem at all?

Bypass Nginx and hit PHP-FPM directly. Use the real FPM socket and the real Hostmaster path:

SCRIPT_NAME=/index.php \
SCRIPT_FILENAME=/var/aegir/host_master/<N>/index.php \
REQUEST_METHOD=GET \
  cgi-fcgi -bind -connect /run/aegir.fpm.socket | head -20

On the Master the Hostmaster root is /var/aegir/host_master/<N> (AegirUpgrade.sh.txt:119,501; the /var/aegir/hostmaster-<version> form exists only between install and the first upgrade, AegirSetupM.sh.txt:179). On an Octopus instance use /data/disk/oN/aegir/distro/<N>/index.php with /run/oN.fpm.socket (AegirSetupC.sh.txt:145-146). If this succeeds, Nginx is the issue; if it fails the same way, PHP-FPM / Drupal is.

Is the upstream socket reachable?

FPM sockets live under /run/, named <pool>.fpm.socket (PHP 7/8) or /run/php5-fpm.sock (the single PHP 5 socket). /opt/etc/fpm/ holds pool config, not sockets.

ls -la /run/*.fpm.socket
nc -U /run/o1.fpm.socket < /dev/null

If ls shows the socket but nc hangs, the FPM daemon is not reading it — restart the pool.

Force a fresh vhost

drush @<site-alias> provision-verify

Verify regenerates the vhost (see Config templates) and reloads Nginx — clears a stale vhost.

Related

• Custom rewrites & location blocks — rule out a custom include causing the redirect/loop.
• Config templates — the high-load 503 fragment and the rendered vhost set.
• PHP-FPM & performance — pool sizing, max_children, timeout, and version selection.
• Security — the load-control / CSF load-shedding layer.
• Troubleshooting — the broader operator troubleshooting catalogue.