SSL operations — LE renewal, custom certs

BOA issues and renews Let's Encrypt certificates automatically through the bundled dehydrated client driven from the nightly owl.sh run. This page covers the on-disk cert tree, the actual renewal cadence, renewal-failure reporting, and the procedure for replacing an LE cert with a custom OV/EV/wildcard cert behind a don't-overwrite marker.

Cert tree on disk

Per-site LE material lives under the frontend's tools/le/certs/<site>/:

<aegir-root>/tools/le/certs/<site>/
  privkey.pem      ← symlink (LE-managed)
  fullchain.pem    ← symlink (LE-managed)
  chain.pem        ← symlink (LE-managed)

<aegir-root> is /var/aegir on the Master or /data/disk/<USER> on an Octopus Satellite. The don't-overwrite markers live in a sibling tools/le/.ctrl/ directory.

Automatic LE renewal

Renewal runs from the daily cron, not weekly. The crontab fires the owl.sh orchestrator once per day at 15 4 * * * — bash /var/xdrago/owl.sh under nice/ionice (cron/crontabs/root:24).

owl.sh replaced the retired daily.sh; a host not yet through that transition may still name daily.sh in its root crontab, so crontab -l tells you which agent a given box runs.

The nightly dehydrated --cron --ipv4 --preferred-chain 'ISRG Root X1' invocation lives in two night workers under /var/xdrago/night/:

• 10-account.sh — _le_hm_ssl_check_update() (:394) renews each account's Hostmaster/frontend cert, running dehydrated as the account user via su (:435-441);
• 20-sites.sh — _le_ssl_check_update() (:1742) renews every LE site cert.

dehydrated --cron re-issues any cert with less than 30 days to expiry; certs with more runway are left untouched, so the daily pass stays cheap.

Monthly force-rebuild

A separate, roughly-monthly path force-rebuilds certs: each worker computes _DOM=$(date +%e) and a random _RDM=$((RANDOM%25+6)), and adds --force when _DOM equals _RDM (or when the per-account static/control/force-ssl-certs-rebuild.info control file is present). This applies to both the account frontend cert (10-account.sh:420-433) and every per-site cert (20-sites.sh:1791-1803).

The --force is further gated per domain: it is added only when no log/ctrl/site.<domain>.cert-x1-rebuilt.info marker exists; the marker is created on force and purged after 14 days (_PURGE_CTRL=14, 10-account.sh:657-661), so a forced re-issue cannot recur within ~14 days.

_SSL_FORCE_REINSTALL is not a certificate operation

Do not confuse this with _SSL_FORCE_REINSTALL=YES in /root/.barracuda.cnf — that is not a certificate operation. It is a one-shot toggle that forces an OpenSSL source rebuild on the next barracuda run (sets _SSL_LEGACY_INSTALL_REQUIRED / _SSL_MODERN_INSTALL_REQUIRED=YES) and cascades _SSH_FORCE_REINSTALL=YES, _NGX_FORCE_REINSTALL=YES and _PHP_FORCE_REINSTALL=YES (lib/functions/system.sh.inc:5043-5049); it is scrubbed from the live cnf after use (system.sh.inc:1592-1596) — set it, run barracuda, and it is consumed.

To force certificate re-issue, use the force-ssl-certs-rebuild.info control file above. See Barracuda cnf for the full *_FORCE_REINSTALL family.

LE issuance behind HTTP Basic Auth

Issuance and renewal work on sites protected with HTTP Basic Auth. The hosting_le_vhost injector emits the ACME challenge location with an explicit exemption (hosting_le_vhost.drush.inc:85-90):

location ^~ /.well-known/acme-challenge {
  allow all;
  alias <aegir-root>/tools/le/.acme-challenges;
  try_files $uri 404;
  auth_basic off;
}

The auth_basic off; (:89) exempts HTTP-01 validation from the site-wide Basic Auth, so password-protected staging/demo sites can enable and renew LE certificates without lifting the password. Before this shipped (bundled with 5.10.3; hosting_le issue #1790, commit a679a7b, contributed by Naurisr), the 401 on the challenge path blocked issuance and renewal.

Renewal-failure reporting

Post-5.10.3, each per-account night worker parses its own account log (/var/log/boa/daily/acct-<user>-<NOW>.log) at the end of its run (_le_account_report, 10-account.sh:475, invoked at :306) and extracts every failed renewal — the failing site, the certificate names, and the ACME error mapped to a plain-language cause (DNS NXDOMAIN, domain pointing elsewhere, key-auth mismatch, connection timeout).

Two mails come out of that parse:

• Operator report — the server admin (_MY_EMAIL from /root/.barracuda.cnf, exported to the workers as _ADMIN_EMAIL) gets the full per-account detail every night unless _INCIDENT_REPORT is OFF (gate at 10-account.sh:504-505; values normalized in owl.sh:76-90 to OFF/ALL/MINI/CRIT, default MINI; legacy NO maps to OFF, YES to MINI). A catch-all also carries non-validation ACME/backend errors — urn:*acme:error tokens including rateLimited / serverInternal, Traceback, Drush command terminated abnormally (10-account.sh:478-495).
• Client notice — the affected account's _CLIENT_EMAIL (from its .octopus.cnf) gets an actionable notice — fix the DNS, disable Encryption, or drop the dead alias — throttled to once per 7 days per failing site (_throttle=7 at 10-account.sh:494; marker log/ctrl/le-notify.<site>.info, purged after 14 days at :663-664). The notice sets Reply-To to the account owner, falling back to the server admin, so replies do not bounce off root@<host> (10-account.sh:560-570).

Client notices are on by default: disable with _LE_CLIENT_NOTIFY=NO globally in /root/.barracuda.cnf (shipped default YES, barracuda.sh.cnf:1278) or per account in /root/.<user>.octopus.cnf — the per-account value wins, and only a literal NO disables (_le_client_notify_on, 10-account.sh:458-463).

Because each worker reads only its own account's log and its own _CLIENT_EMAIL, a notice cannot be cross-attributed to another account's client. The _LE_CLIENT_NOTIFY config-table entry lives on Barracuda cnf.

Reading the Verify task log

The nightly parse above covers automated renewal; when you are chasing a single stuck issue or renewal interactively, read the site's Verify task log instead. The hosting_le extension tags every step it logs with a [hosting_le] prefix, so grepping the task log (or its on-disk .log) for [hosting_le] isolates the Let's Encrypt progress and failure lines — the SAN alias list it built, the redirect and DNS check results, and the exact reason an issue or renewal was skipped or failed. These lines come from the hosting_le post-verify hook and ride the backend log channel into the frontend task log, so they show in the control panel as well as on disk.

LE-skip keywords

provision_hosting_le.drush.inc skips LE issuance (and self-signs instead) for hostnames whose label matches .temp./.tmp./.temporary., .test./.testing., .dev./.devel., or .nodns. (the .dev/.test skip is suppressed when the dev-ctrl flag is set). Note this roster does not include stage/staging — a myapp.staging.example.org gets a real LE cert. (The stale stage|staging entry survives only in the retiring hosting_le README, not in the code.)

The same hook also skips issuance for the hosting service's own reserved names, and does so unconditionally — ahead of the dev-ctrl override and unaffected by it. Any site whose domain ends in a BOA service suffix (*.host8.biz, *.boa.io, *.aegir.cc, *.o8.cc, or any other host8|boa|aegir|o8 second-level name under .biz, .io, or .cc) returns early with no certificate requested (provision_hosting_le.drush.inc:136-141). These are the provider's own internal names, so only real client domains ever issue — a site created under a service name (as on a test estate) shows the skip by design. Unlike the keyword skips above, the dev-ctrl flag does not lift it.

Replacing an LE cert with a custom cert

Use a custom cert only when LE cannot issue what you need:

• a paid wildcard or multi-org SAN cert,
• an OV/EV cert required for compliance, or
• an internal-only site that cannot answer LE HTTP challenges.

Otherwise stay on LE.

The procedure replaces the three LE-managed symlinks with regular files and adds a per-site marker so the daily renewal pass leaves them alone.

1. Create the don't-overwrite marker

touch <aegir-root>/tools/le/.ctrl/dont-overwrite-foo.example.com.pid

Provision reads this marker (dont-overwrite-<site.uri>.pid under tools/le/.ctrl) and treats the cert as immutable — without it, the next Verify or daily renewal re-issues an LE cert over your files.

2. Replace the three PEM files

cd <aegir-root>/tools/le/certs/foo.example.com/

rm privkey.pem
# place your private key as privkey.pem (a regular file, not a symlink)

rm fullchain.pem
# place leaf cert FIRST, then every intermediate to (but not including) the root

rm chain.pem
# place ONLY the intermediates (no leaf, no root)

fullchain.pem is the leaf plus the intermediate chain; chain.pem is the same intermediates without the leaf prepended. The filenames must be exactly privkey.pem / fullchain.pem / chain.pem.

3. Ownership and permissions

chown aegir:aegir privkey.pem fullchain.pem chain.pem
chmod 600 privkey.pem
chmod 644 fullchain.pem chain.pem

4. Re-emit the vhost

drush @<site-alias> provision-verify

Verify regenerates the vhost referencing the cert files and reloads Nginx.

Confirming the custom cert is live

openssl s_client -connect foo.example.com:443 -servername foo.example.com < /dev/null 2>/dev/null | \
  openssl x509 -noout -issuer -subject -dates

issuer= should name your CA (e.g. O=DigiCert Inc), not O=Let's Encrypt.

Renewing a custom cert

BOA never touches a cert behind a dont-overwrite-<site>.pid marker, so custom-cert renewal is entirely manual — track the expiry yourself. To renew, overwrite the three PEM files in place and run drush @<site-alias> provision-verify (or service nginx reload). Leave the marker in place.

Reverting to Let's Encrypt

1. Remove the marker:
   SHCopy

   rm <aegir-root>/tools/le/.ctrl/dont-overwrite-foo.example.com.pid
   

2. Optionally clear the cert files (BOA re-issues anyway):
   SHCopy

   rm <aegir-root>/tools/le/certs/foo.example.com/{privkey,fullchain,chain}.pem
   

3. Run drush @<site-alias> provision-verify. BOA re-requests an LE cert and restores the symlinks once issuance succeeds.

Disabling Encryption cleans up the LE cert dir

When a site's Encryption is disabled and no dont-overwrite-<site>.pid marker exists, the following Verify tidies up the LE material: it runs dehydrated --cleanup and then symlinks -dr over the site's cert directory to strip the dangling LE-managed symlinks.

That is the whole disable path — it does not remove the per-site .ctrl/ control files (forced-renewal-02-<site>.pid, demo-<site>.pid, live-<site>.pid, <site>.pid); those markers persist on disk after Encryption is disabled (provision_hosting_le.drush.inc:97-109).

The dont-overwrite-<site>.pid immutable marker gates the whole cleanup: when it is present the disable branch is skipped entirely, so a manually installed certificate stays protected.

Clone and migrate operations auto-disable Encryption on the target name; see Site cloning ("TLS / Encryption auto-disable").

Wildcard custom certs

A true wildcard (*.example.com) covers every subdomain, but BOA's per-site cert directory model does not natively express a wildcard — place the wildcard PEM files in each covered site's certs/<site>/ directory, each behind its own don't-overwrite marker. A host-level shared-cert arrangement via a custom include (Custom rewrites & location blocks) is possible but outside BOA's managed cert path.

Migrating off the legacy dedicated-IP SSL model

For sites still on the deprecated dedicated-IP xoct ssl-gen custom-cert model, the move to LE goes:

1. Disable HTTPS redirects on the affected sites.
2. Repoint DNS at the default instance/server IP (if you were on dedicated IPs).
3. Remove any leftover HTTP/HTTPS proxy vhosts from /var/aegir/config/server_master/nginx/pre.d/ and reload Nginx.
4. touch ~/static/control/ssl-live-mode.info and wait five minutes.
5. Enable SSL for those sites.

On hosted BOA the account owner can carry out only the control-panel and DNS steps — 1, 2, 4 and 5. Step 3 is server-side — removing the leftover proxy vhosts from pre.d/ and reloading Nginx — and is the operator's job, done on the customer's behalf.

Related

• Config templates — nginx_wild_ssl.conf (the wildcard SSL listener) and HTTP/3.
• Nginx debugging — diagnosing TLS/vhost failures after a cert change.
• Custom rewrites & location blocks — host-level shared-cert include pattern.
• Discontinued features — the dropped dedicated-IP SSL model.