SSL operations — LE renewal, custom certs

BOA issues and renews Let's Encrypt certificates automatically through the bundled dehydrated client driven from the nightly owl.sh run. This page covers the on-disk cert tree, the actual renewal cadence, renewal-failure reporting, and the procedure for replacing an LE cert with a custom OV/EV/wildcard cert behind a don't-overwrite marker.

Cert tree on disk

Per-site LE material lives under the frontend's tools/le/certs/<site>/:

TXT

<aegir-root>/tools/le/certs/<site>/
  privkey.pem      ← symlink (LE-managed)
  fullchain.pem    ← symlink (LE-managed)
  chain.pem        ← symlink (LE-managed)

<aegir-root> is /var/aegir on the Master or /data/disk/<USER> on an Octopus Satellite. The don't-overwrite markers live in a sibling tools/le/.ctrl/ directory.

Automatic LE renewal

Renewal runs from the daily cron, not weekly. The crontab fires the owl.sh orchestrator once per day at 15 4 * * * — bash /var/xdrago/owl.sh under nice/ionice (cron/crontabs/root:26).

owl.sh replaced the retired daily.sh; a host not yet through that transition may still name daily.sh in its root crontab, so crontab -l tells you which agent a given box runs.

The nightly dehydrated --cron --ipv4 --preferred-chain 'ISRG Root X1' invocation lives in two night workers under /var/xdrago/night/:

10-account.sh — _le_hm_ssl_check_update() (:434) renews each account's Hostmaster/frontend cert, running dehydrated as the account user via su (:480-488);
20-sites.sh — _le_ssl_check_update() (:2004) renews every LE site cert.

dehydrated --cron re-issues any cert with less than 69 days to expiry — BOA's bundled client ships RENEW_DAYS="69" as its built-in default (aegir/helpers/dehydrated:426; stock upstream dehydrated uses 30), applied per cert by the expiry check at :2010. Certs with more runway are left untouched, so the daily pass stays cheap — in practice a fresh 90-day LE cert is re-issued about three weeks after it was obtained.

Both legs honour the dont-overwrite-<domain>.pid immutable marker: a marked domain is skipped before dehydrated runs at all, with an LE renewal skipped … immutable dont-overwrite marker present line in the account's night log (20-sites.sh:2007-2015, 10-account.sh:457-462). The hostmaster leg still copies changed PEM files to /etc/ssl/private/ afterwards, so a manually replaced frontend cert keeps propagating. The octopus install/upgrade leg honours the marker the same way: its hostmaster cert refresh — which otherwise re-issues on every octopus up pass, with --force on the on-demand re-register path — short-circuits with the same skip line, and the marker also shields the cert directory from that leg's stale-material cleanup (lib/functions/satellite.sh.inc:6952-6958).

Monthly force-rebuild

A separate, roughly-monthly path force-rebuilds certs: each worker computes _DOM=$(date +%e) and a random _RDM=$((RANDOM%25+6)), and adds --force when _DOM equals _RDM (or when the per-account static/control/force-ssl-certs-rebuild.info control file is present). This applies to both the account frontend cert (10-account.sh:466-479) and every per-site cert (20-sites.sh:2061-2074) — except domains behind a dont-overwrite-<domain>.pid marker, which are skipped before the force computation is reached.

The --force is further gated per domain: it is added only when no log/ctrl/site.<domain>.cert-x1-rebuilt.info marker exists; the marker is created on force and purged after 14 days (_PURGE_CTRL=14, 10-account.sh:793-796), so a forced re-issue cannot recur within ~14 days.

`_SSL_FORCE_REINSTALL` is not a certificate operation

Do not confuse this with _SSL_FORCE_REINSTALL=YES in /root/.barracuda.cnf — that is not a certificate operation. It is a one-shot toggle that forces an OpenSSL reinstall on the next barracuda run — prebuilt package where one applies, source rebuild otherwise (sets _SSL_LEGACY_INSTALL_REQUIRED / _SSL_MODERN_INSTALL_REQUIRED=YES) — and cascades _SSH_FORCE_REINSTALL=YES, _NGX_FORCE_REINSTALL=YES and _PHP_FORCE_REINSTALL=YES (lib/functions/system.sh.inc:6153-6159); it is scrubbed from the live cnf after use (system.sh.inc:2094-2098) — set it, run barracuda, and it is consumed.

To force certificate re-issue, use the force-ssl-certs-rebuild.info control file above. See Barracuda cnf for the full *_FORCE_REINSTALL family.

LE issuance behind HTTP Basic Auth

Issuance and renewal work on sites protected with HTTP Basic Auth. The hosting_le_vhost injector emits the ACME challenge location with an explicit exemption (hosting_le_vhost.drush.inc:85-90):

NGINX

location ^~ /.well-known/acme-challenge {
  allow all;
  alias <aegir-root>/tools/le/.acme-challenges;
  try_files $uri 404;
  auth_basic off;
}

The auth_basic off; (:89) exempts HTTP-01 validation from the site-wide Basic Auth, so password-protected staging/demo sites can enable and renew LE certificates without lifting the password. Before this shipped (bundled with 5.7.11; hosting_le issue #1790, commit a679a7b, contributed by Naurisr), the 401 on the challenge path blocked issuance and renewal.

Renewal-failure reporting

Since BOA-5.88.8, each per-account night worker parses its own account log (/var/log/boa/daily/acct-<user>-<NOW>.log) at the end of its run (_le_account_report, 10-account.sh:521, invoked at :345) and extracts every failed renewal — the failing site, the certificate names, and the ACME error mapped to a plain-language cause (DNS NXDOMAIN, domain pointing elsewhere, key-auth mismatch, connection timeout).

Two mails come out of that parse:

Operator report — the server admin (_MY_EMAIL from /root/.barracuda.cnf, exported to the workers as _ADMIN_EMAIL) gets the full per-account detail every night unless _INCIDENT_REPORT is OFF (gate at 10-account.sh:549-551; values normalised in owl.sh:95-108 to OFF/ALL/MINI/CRIT, default MINI; legacy NO maps to OFF, YES to MINI). A catch-all also carries non-validation ACME/backend errors — urn:*acme:error tokens including rateLimited / serverInternal, Traceback, Drush command terminated abnormally (10-account.sh:536-538).
Client notice — the affected account's _CLIENT_EMAIL (from its .octopus.cnf) gets an actionable notice — fix the DNS, disable Encryption, or drop the dead alias — throttled to once per 7 days per failing site (_throttle=7 at 10-account.sh:540; marker log/ctrl/le-notify.<domain>.info, purged after 14 days at :798-799). The notice sets Reply-To to the account owner, falling back to the server admin, so replies do not bounce off root@<host> (10-account.sh:606-624).

Client notices are on by default: disable with _LE_CLIENT_NOTIFY=NO globally in /root/.barracuda.cnf (shipped default YES, barracuda.sh.cnf:1576) or per account in /root/.<user>.octopus.cnf — the per-account value wins, and only a literal NO disables (_le_client_notify_on, 10-account.sh:504-509).

Because each worker reads only its own account's log and its own _CLIENT_EMAIL, a notice cannot be cross-attributed to another account's client. The _LE_CLIENT_NOTIFY config-table entry lives on Barracuda cnf.

Reading the Verify task log

The nightly parse above covers automated renewal; when you are chasing a single stuck issue or renewal interactively, read the site's Verify task log instead. The hosting_le extension tags every step it logs with a [hosting_le] prefix, so grepping the task log (or its on-disk .log) for [hosting_le] isolates the Let's Encrypt progress and failure lines — the SAN alias list it built, the redirect and DNS check results, and the exact reason an issue or renewal was skipped or failed. These lines come from the hosting_le post-verify hook and ride the backend log channel into the frontend task log, so they show in the control panel as well as on disk.

Two Verify outcomes that refuse rather than fail

The hook logs dehydrated's exit status as [hosting_le] ACME Exit Code: N right after the raw ACME Output: line, so both are in the task log for any run. The exit code is informational — it is logged but never branched on: the lock outcome below is decided by matching dehydrated's output, and the cert-file outcome by testing the files on disk — but it is worth reading alongside the lines that follow.

Two of those outcomes stop the run deliberately, leaving the site's existing certificate and vhost exactly as they were:

Another LE task holds the lock. When dehydrated aborts because a concurrent LE run holds <aegir-root>/tools/le/lock, the hook logs "Another LE task holds the lock … so this attempt was aborted", "No certificate has been issued or renewed for <domain> in this attempt" and "Please run Verify again for this site once the other LE task has completed", then returns without touching anything. Nothing was issued, and the fix is simply to re-run Verify for the site once the other LE task finishes. Two Verify tasks in the same account running back-to-back are the usual way to hit this.
An expected cert file is missing or unreadable. Before wiring the openssl.* symlinks, the hook checks every file a vhost actually loads — cert.pem, privkey.pem and fullchain.pem, in that order — and requires each to be readable, not merely present, so a dangling symlink fails the check too (cert.csr is skipped on purpose: no webserver reads it, and gating on it would refuse cert dirs which legitimately lack it). On the first file that fails it logs "Expected cert file <file> is missing or unreadable in <cert_dir>" and "Skipping symlinks wiring and webserver restart to keep the vhost config valid", then returns. This gate sits on the path taken when the hook has concluded a certificate was [re]generated and the site carries no dont-overwrite-<site>.pid marker.

Both refusals exist for the same reason: an openssl.* symlink pointing at a file that does not exist or cannot be read makes the vhost referencing it fail nginx -t, and a single such vhost fails the check for the whole box — a running nginx keeps serving from its loaded config, so the breakage only surfaces at the next reload. Skipping the wiring and the webserver restart keeps the vhost valid and the site on the certificate it already had.

LE-skip keywords

provision_hosting_le.drush.inc skips LE issuance (and self-signs instead) for hostnames whose label matches .temp./.tmp./.temporary., .test./.testing., .dev./.devel., or .nodns. (the dev-ctrl flag suppresses the whole .temp/.test/.dev family, :153-161; the .nodns. skip sits in the unconditional reserved-name gate below and is never lifted). Note this roster does not include stage/staging — a myapp.staging.example.org gets a real LE cert. (The stale stage|staging entry survives only in the retiring hosting_le README, not in the code.)

The same hook also skips issuance for the hosting service's own reserved names, and does so unconditionally — ahead of the dev-ctrl override and unaffected by it. Any site whose domain ends in a BOA service suffix (*.host8.biz, *.boa.io, *.aegir.cc, *.o8.cc, or any other host8|boa|aegir|o8 second-level name under .biz, .io, or .cc) returns early with no certificate requested (provision_hosting_le.drush.inc:146-152). These are the provider's own internal names, so only real client domains ever issue — a site created under a service name (as on a test estate) shows the skip by design. Unlike the keyword skips above, the dev-ctrl flag does not lift it.

Replacing an LE cert with a custom cert

Use a custom cert only when LE cannot issue what you need:

a paid wildcard or multi-org SAN cert,
an OV/EV cert required for compliance, or
an internal-only site that cannot answer LE HTTP challenges.

Otherwise stay on LE.

The procedure replaces the three LE-managed symlinks with regular files and adds a per-site marker so the daily renewal pass leaves them alone.

1. Create the don't-overwrite marker

touch <aegir-root>/tools/le/.ctrl/dont-overwrite-foo.example.com.pid

Provision reads this marker (dont-overwrite-<site.uri>.pid under tools/le/.ctrl) and treats the cert as immutable — without it, the next Verify or daily renewal re-issues an LE cert over your files.

2. Replace the three PEM files

cd <aegir-root>/tools/le/certs/foo.example.com/

rm privkey.pem
# place your private key as privkey.pem (a regular file, not a symlink)

rm fullchain.pem
# place leaf cert FIRST, then every intermediate to (but not including) the root

rm chain.pem
# place ONLY the intermediates (no leaf, no root)

fullchain.pem is the leaf plus the intermediate chain; chain.pem is the same intermediates without the leaf prepended. The filenames must be exactly privkey.pem / fullchain.pem / chain.pem.

3. Ownership and permissions

Match the account user that owns <aegir-root> — aegir:aegir on the Master, <USER>:users on an Octopus Satellite (e.g. o1:users), the same owner as the surrounding files in the cert directory:

chown <user>:<group> privkey.pem fullchain.pem chain.pem
chmod 600 privkey.pem
chmod 644 fullchain.pem chain.pem

4. Re-emit the vhost

drush @<site-alias> provision-verify

Verify regenerates the vhost referencing the cert files and reloads Nginx.

Confirming the custom cert is live

openssl s_client -connect foo.example.com:443 -servername foo.example.com < /dev/null 2>/dev/null | \
  openssl x509 -noout -issuer -subject -dates

issuer= should name your CA (e.g. O=DigiCert Inc), not O=Let's Encrypt.

Custom cert for the Ægir frontend

The same procedure protects the account's Hostmaster/frontend cert: name the marker after the frontend domain (tools/le/.ctrl/dont-overwrite-<frontend>.pid) and replace the PEM files in tools/le/certs/<frontend>/ the same way — the frontend's HTTPS vhost references fullchain.pem/privkey.pem in that directory. The nightly account leg then skips the LE renewal but still copies the changed PEM files to /etc/ssl/private/<frontend>.crt|.key for the auxiliary vhosts that read those copies (10-account.sh:457-462, :490-497).

Renewing a custom cert

BOA never touches a cert behind a dont-overwrite-<site>.pid marker, so custom-cert renewal is entirely manual — track the expiry yourself. To renew, overwrite the three PEM files in place and run drush @<site-alias> provision-verify (or service nginx reload). Leave the marker in place.

Reverting to Let's Encrypt

Remove the marker:

rm <aegir-root>/tools/le/.ctrl/dont-overwrite-foo.example.com.pid

Optionally clear the cert files (BOA re-issues anyway):

rm <aegir-root>/tools/le/certs/foo.example.com/{privkey,fullchain,chain}.pem

Run drush @<site-alias> provision-verify. BOA re-requests an LE cert and restores the symlinks once issuance succeeds.

Disabling Encryption cleans up the LE cert dir

When a site's Encryption is disabled and no dont-overwrite-<site>.pid marker exists, the following Verify tidies up the LE material: it runs dehydrated --cleanup and then symlinks -dr over the site's cert directory to strip the dangling LE-managed symlinks (provision_hosting_le.drush.inc:100-108).

The same branch also removes the site's auto-managed per-site .ctrl/ control files (forced-renewal-02-<site>.pid, demo-<site>.pid, live-<site>.pid, <site>.pid), so a later re-enable — or a rename that reuses the name — starts clean (provision_hosting_le.drush.inc:110-119).

The dont-overwrite-<site>.pid immutable marker gates the whole cleanup: when it is present the disable branch is skipped entirely (:97-99), so a manually installed certificate stays protected — and the marker itself is never auto-removed.

Clone and migrate operations auto-disable Encryption on the target name; see Site cloning ("TLS / Encryption auto-disable").

Wildcard custom certs

A true wildcard (*.example.com) covers every subdomain, but BOA's per-site cert directory model does not natively express a wildcard — place the wildcard PEM files in each covered site's certs/<site>/ directory, each behind its own don't-overwrite marker. A host-level shared-cert arrangement via a custom include (Custom rewrites & location blocks) is possible but outside BOA's managed cert path.

Migrating off the legacy dedicated-IP SSL model

For sites still on the deprecated dedicated-IP xoct ssl-gen custom-cert model, the move to LE goes:

Disable HTTPS redirects on the affected sites.
Repoint DNS at the default instance/server IP (if you were on dedicated IPs).
Remove any leftover HTTP/HTTPS proxy vhosts from /var/aegir/config/server_master/nginx/pre.d/ and reload Nginx.
touch ~/static/control/ssl-live-mode.info and wait five minutes.
Enable SSL for those sites.

On hosted BOA the account owner can carry out only the control-panel and DNS steps — 1, 2, 4 and 5. Step 3 is server-side — removing the leftover proxy vhosts from pre.d/ and reloading Nginx — and is the operator's job, done on the customer's behalf.

Config templates — nginx_wild_ssl.conf (the wildcard SSL listener) in the static template set.
HTTP/3 and KTLS — the QUIC reuseport listener placement, KTLS, and the HTTP/3 trusted-host fix.
Nginx debugging — diagnosing TLS/vhost failures after a cert change.
Custom rewrites & location blocks — host-level shared-cert include pattern.
Discontinued features — the dropped dedicated-IP SSL model.