PX0 proxies and shrinking a migrated source

When a whole server moves with xmass, or an account moves with xoct, the old box does not have to be switched off. It can stay in front of the new one as a web proxy, so that every name still resolving to the old address keeps working while DNS moves at the client's own pace.

That proxy role needs almost nothing: nginx, the certificates, and the per-account proxy vhosts. Everything else the box is still carrying — databases, platform trees, file stores, backups — is dead weight, and on a metered plan it is the difference between an expensive server and a cheap one. xtrim removes exactly that dead weight and nothing else.

Before you shrink anything

A proxy is only safe to shrink once it has proven itself quiet. xtrim enforces that with a minimum age on the conversion marker (_XTRIM_MIN_DAYS, default 14 days, hard floor 7) and a battery of refusals that all have to pass first:

• the account really is a converted proxy, and is not a cancelled account (a cancelled account belongs to boa cleanup, never to xtrim);
• every live vhost is a proxy vhost, and they all agree on ONE target — with one shape admitted beside the proxies: the 301 redirect the cutover itself writes for a site that was named under the old box name (its new name lives on the target). It is recognised by the cutover's own banner, carries no proxy and no content, stays in the map and is listed as its own class; any other redirect-only vhost still fails the check. An account whose sites were all box-named has no proxy to name its target and takes it from its policy record;
• the account's own policy record agrees with what the vhosts say (for an all-box-named account there is no vhost target to compare, so the record is taken as given);
• no certificate behind a retained HTTPS vhost has expired — a retired box-named name is outside this gate: the target no longer issues for it and the certificate mirror has no proxy to follow, so nothing renews its certificate and its expiry is reported, not refused (the 301 keeps working over HTTP; an HTTPS visitor of the old URL meets the browser's warning first);
• the root filesystem is under 90% full (refused before any dump is staged), there is at most ONE attached /mnt mount (with more, a resolved store path could not be attributed unambiguously — the whole run refuses), and the SQL endpoint answers as this box's own server — the classification is derived from this box's aliases, and a cluster or remote credential file would judge them against the wrong server;
• every server_name answers both directly on the target and through the proxy relay. The probe is not fooled by a status code: once per run the tool records what the target answers for an impossible hostname, and a 200 whose body matches that catch-all answer FAILS the probe (shown as direct=CATCH-ALL / relay=CATCH-ALL) — a BOA box answers any unknown Host with its "Under Construction" page, so a bare 200 proves nothing; a redirect is still accepted on status alone. One deliberate exception: after a whole-server move that renamed the box, the old box's host-derived site names are retired by design, so a failing name that embeds the old box name is re-probed as the same label under the new box's name — directly only, since the relay runs through the old box, which never served that name. A customer's own domain is never remapped and must prove itself by name;
• the target proves, over ssh, that it holds a populated database of each client site about to be dropped and serves at least one real (non-proxy) vhost — this is what stops a proxy chain being mistaken for a target;
• the account's own panel and dedicated site are proved differently, and deliberately so: a migration builds those with names derived from the new host, never the old one, so demanding a matching name would refuse every migrated source. Instead the target must show a live, working account of the same number — its account tree, its cores map and its panel alias all present;
• nothing else is running that could be mid-flight: no install or upgrade, no other migration tool, and no sqlclean (they share a lock).

If any of those fail the tool stops and says which one. Nothing is deleted on a refusal.

The three stages

Quiesce — reversible. The account is parked: its dispatcher is moved out of the way, its PHP-FPM pools and nginx includes are moved to quarantine, its Solr cores are unloaded (without deleting a single index) and set aside, nginx is reloaded, and every site is re-probed through the proxy. Nothing is deleted. If a probe regresses, the tool puts everything back by itself and aborts.

Shrink — one way. Every database is dumped and the dump verified, the target is re-proved one last time by exactly the same test, and only then are the databases and their users dropped and the retired trees removed. If anything fails before the first drop the account is left quiesced and still restorable, dumps and all. Store symlinks are resolved before deletion, so the real content goes and no orphan is left behind on attached storage. The proxy keeps serving throughout, because nothing removed here is part of what the proxy needs.

Finalize — box-wide. Once every account on the box is shrunk (finalize is all-accounts-or-nothing: one account short of stage-b refuses the whole run and names it with its phase, rather than half-standing-down a box that still serves; the check runs before anything else, so a dry xtrim finalize is a usable pre-flight for the whole box; a panel-only account is exempt — its control panel is local and it has no other live vhost of its own, so there is nothing to shrink and finalize does not wait for it, and its panel keeps serving through the MySQL that finalize keeps), the shared codebases go, Solr is stopped and disabled, and the PHP-FPM masters are stood down except the control panel front's and any master a local panel still answers through. MySQL is stopped and disabled only when no account keeps a local control panel on the box. Wherever a panel stayed local (the normal state after an xmass cutover, which leaves the panels local and never proxied), MySQL keeps running for those panel databases, the run says so (mysql KEPT running for the local control panel database(s)), and --drop-datadir is refused outright, in a dry run as well. Then the box is finally marked as a proxy node — which stands down only what a proxy genuinely has no use for. A finalized proxy remains a live, internet-facing server, and everything such a server needs keeps running: the intrusion detection and abuse scanners, the service watchdogs and nginx self-healing, the host security hardening, load profiling, and the update channel — the proxy keeps fetching tool updates like any other box. What stays down is the site-serving machinery: MySQL, Solr, the per-site PHP stack, the night maintenance worker, and every database tool — none of which a proxy has, and none of which the monitors will ever resurrect. nginx and cron are registered to start at boot, so a rebooted proxy comes back serving with its watchdogs armed. The nginx watchdog deliberately keeps running: that is exactly what a proxy box wants.

The sequence in practice

xtrim status                     # what is proxied, cert horizon, reclaimable space
xtrim plan    o1                 # the full battery plus an itemised plan
xtrim quiesce o1                 # dry run
xtrim quiesce o1 --live          # parked and reversible; nothing deleted
xtrim restore o1                 # undo, if you change your mind
xtrim shrink  o1                 # dry run
xtrim shrink  o1 --live          # the one-way step
xtrim finalize --live            # only when every account is shrunk
xtrim finalize --live --drop-datadir   # ALSO reclaim the MySQL data directory

The database data directory is kept by default at finalize; dropping it is a separate, explicit --drop-datadir decision, available only on a box where no account keeps a local control panel: with a local panel present finalize --drop-datadir refuses and names the panels.

Each verb has its own dry-run token: a clean dry run of one verb never arms a live run of another, and each live run consumes its token before doing any work. The finalize token additionally records whether the data directory was to be kept, so a dry run without --drop-datadir will not arm a live run with it — issue the dry and live runs with the same flags. all in place of an account name operates on the proxied accounts and skips the others with a notice; it also skips by name any account already past the verb instead of aborting the sweep.

Every verb is gated on a per-account phase, which xtrim status prints as phase= — none, stage-a, stage-b-started, stage-b or restored. An account whose only live vhost is its own local control panel is listed as panel-only with nothing to shrink, rather than with a target. A shrink interrupted mid-deletion leaves the account at stage-b-started, where quiesce ("deletion may be partial"), shrink, restore and finalize all refuse and only status and plan remain useful — inspect such an account by hand from the tool's working directory under /var/backups/xtrim/<oN>/.

What is never touched

The account's whole config/ tree including its SSL material, and tools/le/ in its entirety. Deleting the certificate store would leave every HTTPS proxy vhost pointing at a file that is not there, and nginx would refuse to start. The system firewall, the BOA tooling directories and the box's own configuration are likewise out of scope.

A local control panel is kept whole. When an account's panel is local by design (never proxied), quiesce leaves fpm_include_default.inc and the account's default-version PHP-FPM pool in place, and shrink keeps the panel database, the database of the account's own dedicated site and the panel platform tree under aegir/ out of the dump-and-drop and out of the tree removal. The run prints one line per account saying so.

Certificates on a long-lived proxy

A converted source stops issuing certificates the moment it becomes a proxy — it forwards ACME challenges to the target, which becomes the sole issuer. The frozen copies on the proxy would otherwise simply expire, so a daily mirror pulls each renewed certificate back from the target, verifying expiry and key match before installing it and reloading nginx behind a configuration test. That mirror keeps running long after a shrink; it is not something xtrim replaces.

One class of name it cannot refresh, because nothing can. A site that was named under the old box's own hostname was renamed on the target by the move, and the old name is answered here with a 301 to the new one: there is no proxy behind it to mirror from, and the target no longer issues for a retired name, so its certificate simply runs out. The mirror names every such name on each run and mails the admin once when the certificate enters the warning window (_MIGRATION_PROXY_CERT_WARN_DAYS, default 21) and once when it has expired. From then on an HTTPS visitor of the old URL meets the browser's warning before the redirect, while plain HTTP visitors are redirected as before. Nothing on the proxy needs fixing: have the old HTTPS links moved to the site's new name, or retire the proxy when it is no longer meant to serve.

Undoing things

A quiesce is fully reversible with xtrim restore. One thing it deliberately does not put back: a proxied account's dispatcher stays parked, moved aside where you can see it — at /var/backups/off-run/run-<oN>. Handing it back would let it regenerate the very vhosts the proxy conversion installed, quietly un-converting the proxy. (Note the location matters: the quiesce stage itself parks the dispatcher out of /var/xdrago but not into off-run/ — off-run/ is where restore moves it, which is also the path BOA's own restore machinery reads from.)

restore is honest about partial failure rather than unconditionally clean. It refuses an account name it does not recognise, and refuses when there is no quarantine to restore from, instead of reporting success on a no-op. Two partial paths have printed remedies: a Solr core that moves back but does not re-register leaves its index intact on disk — restart that Solr instance as the tool instructs, and the quarantine map is deliberately kept until every entry is done; and if the nginx configuration test or reload fails, the files are back but the running configuration is still the quiesced one — the tool says so and returns non-zero, and nginx -t plus a manual reload are owed before the account counts as live. A shrink is not reversible on that box: bringing the account back means migrating it back from the live target, using the map and archived aliases that xtrim records in its manifest before the first deletion. If the target itself is gone, that is a restore from the remote backup history — which is why the backup configuration is on the keep list.

Retiring the proxy altogether is a separate operation and belongs to boa cleanup.