Migration source wiring

A site owner can give one of their sites read-only access to another of their own sites' database, so Drupal's migration tooling can read the old site from inside the new one. The client-facing side is a single task; this page is what happens underneath, what bounds it, and how it comes enabled.

What it grants

Exactly one thing: GRANT SELECT ON <source database>.* TO <target site's database user>@<host>, on the loopback hosts the target's own grants already exist on.

No new database user is created, no password is generated or moved, and nothing is written to the source. The target's settings.php gains a migrate connection composed from the site's own credentials, so the file never carries a credential belonging to the source:

PHP

# Migration source: read-only access to another of this client's sites.
if (isset($databases['default']['default'])) {
  $databases['migrate']['default'] = $databases['default']['default'];
  $databases['migrate']['default']['database'] = '<source database>';
  $databases['migrate']['default']['prefix'] = '';
}

The table prefix is deliberately blanked: a target whose own connection carries a prefix must not apply it to the source's tables, or every migrate query would hit a non-existent name. The block is emitted by a hook_provision_drupal_config() implementation, so every verify rebuilds it from the site's saved context — unlike a hand edit to settings.php, which the next verify overwrites.

In the issued GRANT, the database name is escaped: _ and % are LIKE wildcards in the ON clause, and BOA mints names like site_0 through its collision suffix, so an unescaped grant would cover siteX0 too. SHOW GRANTS therefore shows the grant as ON `site\_0`.*; when revoking one by hand, revoke the spelling MySQL actually stored.

Both halves — the emitted block and the grant — run inside the same provision-verify, so neither can be triggered independently. The grant half applies checks the emitted block does not, so a refused grant still leaves the block in settings.php, inert: Drupal opens the connection lazily, so a site whose grant is missing or refused still serves normally and simply fails when a migration command tries to read.

What a verify also takes away. Every verify of a site this feature has wired — once, ever — converges that site's own grants, not only the ones this feature created: a bare GRANT SELECT ON <other database>.* held by the site's own database user — anything other than its own database and its currently intended migration source — is revoked in the same run and logged at ok level. The latch is the site's own alias (migrate_source_wired, emitted the first time a source is pushed and retained through every later push, the clearing one included), so a site this feature has never touched keeps its hand-made read grants through its verifies; the nightly sweep, which works from the instance's own records and proof of possession, is what reclaims stale grants there. Grants of any other shape (GRANT SELECT, SHOW VIEW …, GRANT ALL …) do not match and are untouched, and a grant issued to a separate user is never examined by the verify path. If you hand-grant a wired site's own database user read access to another database, it does not survive that site's next verify; use a dedicated user for monitoring or backup reads.

What bounds which database may be named

Two facts set the terms here. A site node's ownership bounds which node a client may name; the database that node records is a separate matter, because platforms live in a tenant-writable tree and Ægir's site import builds nodes from whatever settings.php a platform contains — so a legitimately-owned site node can record any database on the box. And a database's own shape in MySQL identifies nobody: every provisioned database on the box carries a same-named user holding a non-global grant on it, other accounts' databases included.

Ownership is therefore established by proof of possession, and it runs before anything else. The source site must open the database it names, using the credentials that site records for itself. Reading those credentials from a tenant-writable file is safe precisely because they are tested rather than trusted: a forged password does not connect, and a client's own real credentials aimed at somebody else's database do not connect either. What comes back is a fact rather than a claim. The probe only ever connects to the box's own server: a db_host other than loopback (localhost, 127.0.0.1, ::1) or a non-numeric db_port in that file fails the proof outright, so a server the client controls cannot vouch for a name. The grantee is proven the same way, so naming another account's database user requires that account's password on this server.

Four further checks sit around it, each failing closed on its own:

The front end is never selectable. Its node id is resolved from the aegir_hostmaster_site_nid variable with a positive fallback through the context registry, and an unresolvable front end is a hard refusal — the task refuses to offer anything at all rather than treat "cannot identify the front end" as "no exclusion needed". Two further gates need no variable: any candidate whose platform publish_path contains /aegir/distro is rejected, and the backend independently reads the front end's own database name from its drushrc.php and refuses it as a source.

The caller must own the target. The shared task-access gate performs no node-access check for sites (its platform branch does), so the task checks node_access('update') on the target itself, in both the access callback and the form validation.

The source must be a genuine site database, established by its own user holding a grant on it. This rejects a name that was never provisioned as a site database. It is a sanity check on the shape of the thing, and says nothing about who may read it — that is what proof of possession settles.

That user must not hold global privileges, which stops a superuser being named as the source's user. This is also what independently refuses the front-end database.

Underneath all of it sits the privilege boundary: issuing a grant requires the master_db credentials, readable only by the account user that runs Ægir tasks and not by the tenant's shell or web identity. It is deliberately not the only defence.

Enabling it

Nothing to do: every Octopus install and upgrade enables it for the instance. On a fresh instance the module is a dependency of the Hostmaster install profile, so it is enabled natively at install time; on upgrades it is a declared dependency of hosting_custom_settings and is also enabled directly by the installer. The create migrate_source task permission rides the same hosting_custom_settings features export that carries every other task permission, and the installer regenerates the instance drushrc when the module's drush/ directory is missing from it — so the backend half is loaded by the same pass that enables the front end.

On an instance installed before BOA gained this and not upgraded since, the manual pair still works:

TXT

drush @hostmaster en hosting_migrate_source -y
drush @hostmaster provision-verify

The verify is not optional there, and skipping it fails silently. The backend half of the feature only loads once its drush/ directory appears in $options['include'] in the instance's .drush/drushrc.php, and that list is regenerated on hostmaster verify. Enable the module without verifying and the task appears in the panel, runs, and reports success while granting nothing and emitting nothing. The automatic path above closes exactly this gap on its own.

The permission is granted to aegir client and aegir platform manager (and to the operator roles). Removing it from aegir client by hand does not stick: the hosting_custom_settings features revert restores the exported grants, and that revert runs in the nightly per-account maintenance pass as well as on every Octopus install and upgrade — a hand-removal is undone within a day, not at the next upgrade.

The reconciliation sweep

Every verify converges the site it runs on, but four situations leave a grant behind that no verify will ever look at again, and the sweep exists to reclaim them:

a source is deleted, its database dropped, and the target's grant on that name survives — and because database names are derived from a 16-character truncation of the URI and are global to the server, the freed name can later be taken by a site in another account;
a crash between issuing a grant and recording it leaves a grant nothing intends;
a target site is deleted, and Provision keeps its MySQL user alive precisely because the SELECT grant on the source is still on it;
a restore or migrate rotates a target's database user, and the superseded user keeps the grant.

The first two are found from the user side — the grants this instance's proven users hold. The last two have a grantee that is nobody's current user, so they are found from the other direction: the grants on this instance's proven databases. Both directions require the host to be loopback (the only place this feature grants) and the privilege set to be exactly SELECT (the only shape it issues); a row found from the database side is revoked only when, in addition, the grantee is neither a proven user of this instance nor the database's own same-named user, and the grantee is on this instance's own record of former site database names. That record has three sources: deleted sites' surviving site records; superseded revisions, inert on this fork, which updates a site's single revision in place — exactly why the third source exists; and the names rotation writes into the hosting_migrate_source_former_names variable (90-day expiry, 200-name cap, both bounds moving a grantee from "revoked" to "reported", never the reverse). Restore records its name from its post-task hook, where the front end's record still holds the superseded name. Migrate records before the task dispatches, because hosting_migrate's own post-task hook refreshes the record to the new name first; a migrate that then fails leaves a still-live name on the record, which is the safe direction — a proven user outranks the record, and the entry expires.

A grantee outside that record — an imported site's original user, for instance — is warned about on every pass and left to you. Find such rows spelling-agnostically — mysql.db stores the escaped spelling (site\_0) for every grant the backend issued, so a plain Db = '<source db>' match returns nothing on exactly the collision-suffixed names this feature escapes: SELECT User, Host, Db FROM mysql.db WHERE Select_priv = 'Y' AND REPLACE(Db, '\\_', '_') = '<source db>'; Then REVOKE, and DROP USER only an account whose remaining grants are bare USAGE — an account still holding grants on other databases is one Provision keeps on purpose (a shared or imported user another site may rely on), so check SHOW GRANTS first. The sweep itself never drops a user, so a reclaimed grant can leave an empty user behind; dropping it is yours too. A monitoring or backup user you granted by hand is never a name this instance minted for a site, so the sweep reports it but structurally cannot revoke it.

The sweep:

TXT

drush @hostmaster hosting-migrate-source-sweep
drush @hostmaster hosting-migrate-source-sweep --dry-run

It revokes grants that nothing justifies, and where an intended grant is missing it queues a verify rather than granting anything itself, so the backend remains the only place a grant is ever created and every check above still applies.

How it decides what is its own. The front end's site records pick the candidate nodes: this instance's own non-deleted sites, never the front end itself, and only those whose recorded database name is a plain identifier — candidates are keyed by that recorded name, so a site with no recorded name, an invalid one, or one whose recorded name duplicates another site's drops out of the pass, each counted in the pass summary; the invalid and duplicate cases are also warned about with the node ids involved. The identity that revocation acts on comes from elsewhere. Each candidate must first prove possession of its database, by the same connection test that gates a grant, and it is the proven user that is then examined — the sweep asks what that user holds, and revokes there. Keeping those two apart is deliberate: a site's recorded database name is a cache that goes stale, and one a client can influence through an imported platform, so it is fit to suggest where to look and not to decide what to destroy. A site that cannot prove its identity is counted and reported, and neither examined nor touched.

An association is held to the same standard from both ends: the grant it justifies counts as intended only while both target and source prove possession. If the source cannot prove itself on a pass — its platform tree mid-rebuild, an unreadable drushrc.php, a database briefly refusing connections — the association is logged as unproven and the target's grant is revoked on that pass; a later pass queues a verify that puts it back once the source proves again. A source mid-rebuild during a sweep shows exactly that revoke-then-restore cycle in the logs.

Enumerating from users is also what makes the freed-name situation visible at all: once the source is deleted its database is dropped, so nothing remains to find on the database side, while the grant on that freed name survives.

Guards. --max-revokes (default 25) makes it refuse to act when more grants look unjustified than a real instance would produce, so corrupt or empty association data raises an alert instead of revoking everything. When the guard trips, the whole pass aborts — including the missing-grant half, and including --dry-run, which is checked after the guard. Inspect with --dry-run --force-revokes (still changes nothing), look at the association table, then re-run with --force-revokes deliberately. The associations live in the hosting_migrate_source table, keyed on revision, so read it joined to the current one: SELECT m.nid, m.source_nid FROM hosting_migrate_source m INNER JOIN node n ON n.nid = m.nid AND n.vid = m.vid WHERE m.source_nid <> 0; — rows against older vids are dead history, and source_nid = 0 means cleared. The bypass flag is deliberately not spelled --force: the queue command this sweep is armed on (hosting-migrate_source) accepts --force as its documented queue-lock bypass, and options are global to a drush invocation, so that spelling would let an operator unsticking a stuck queue disarm the max-revokes guard without meaning to. The sweep command refuses --force as an unknown option.

The sweep's reach is whatever it enumerates: this instance's own proven users, and the grants on its own proven databases. On the database side, a grant whose grantee is not on the instance's record of former site database names — a monitoring or backup user among them — is reported and left alone. In both directions, a grant whose shape this feature never issues — a wildcard, anything but a plain database name — is left alone and logged, as is a grant whose host column does not parse. On the user side a non-loopback row is filtered out in the query itself, so it produces no log line and is not counted, whereas a multi-privilege row at a loopback host still logs "leaving @u alone". Only the former-names skip feeds the "grants left to a human" counter; the shape skips are warned in the log but not counted, so for those the log is the complete list, not the number. Host-skips are counted: the summary reports them separately ("skipped on an unrecognised host"), and the "revoked" figure excludes them — it counts the rows the pass attempted, minus failures, minus host-skips, so it is exactly the revokes that actually ran.

Reading a pass. Every pass ends with one summary line — databases examined, associations intended, revoked (and how many of those came from former site users), revokes failed, skipped on an unrecognised host, verifies queued, abandoned, grants left to a human, and the three candidate-selection counters: sites without a recorded database name, with an invalid one, and duplicate names skipped — with the revoked count reported as 0 under --dry-run. A revoke that fails does not abort the pass: it is caught, counted, warned about and stepped over, because a target's own verify revoking the same grant concurrently is normal on a busy box. Two passes end early instead: MIGRATE_SOURCE_NO_FRONTEND when the front-end site cannot be identified (the sweep refuses to run rather than guess), and a plain ok-level line — carrying the candidate-selection counters — when the instance has no examinable site databases.

It gives up rather than retrying forever. If an association can never be satisfied — a target that has moved off a Drupal 8+ platform, a source that cannot prove ownership — the sweep stops re-queueing a verify for it after three consecutive passes and warns once. That bound is what keeps an impossible association from becoming a permanent generator of full site verifies on an instance nobody is watching. The count lives in the hosting_migrate_source_retries variable, keyed on the target's proven database user plus the source database; it is cleared when the grant finally appears, and pruned when the association stops being intended. To start over: clear the source, let one pass run (or wait one queue interval) so the record is pruned, then set the source again — re-setting immediately still runs the site's own verify and its grant attempt either way; only the sweep's repair budget needs the intervening pass. drush @hostmaster vdel hosting_migrate_source_retries resets it directly. A --dry-run pass never writes it. Because the key includes the target's database user, a restore or deploy that rotates that user also resets the count.

Scheduling: nightly, inside a queue pause. On BOA the sweep runs once per night per instance, from the nightly maintenance window, wrapped in a box-wide task-queue pause: the night runner takes the same /run/boa_queue_stop.pid marker the backup relocation holds — honoured by the dispatcher for every instance — waits a grace period so a dispatcher pass already beyond the gate cannot fork one more task, drains anything still running, and only then sweeps each hosted instance in turn. Nothing can revoke a grant concurrently with a sweep pass, so the concurrent-revocation race cannot occur at all. Each instance's pass is capped at five minutes, and only instances that actually have the module enabled are swept. If tasks are still running after the grace and drain, the night is skipped rather than swept un-quiet; a night is also skipped outright when another operation already holds /run/boa_queue_stop.pid; and /data/conf/disable_migrate_sweep_night.cnf disables the nightly run and, on a box whose nightly passes had switched the daily queue off, hands the sweep back to that queue once (see the control file's entry above). The feature's own hosting queue — Migration source grants (migrate_source), one whole-instance pass per run, daily by default, tunable at Hosting → Queues — remains registered as the fallback scheduler for plain Ægir installs; while the nightly run is enabled the BOA night runner switches that queue off per instance on every pass, so the paused nightly run is the only scheduled one on a BOA box. Nothing needs adding to cron, and a cron entry would double-run it. The command form is for an immediate pass or a --dry-run. One consequence of a scheduled run taking option defaults: on an instance where more than 25 grants look unjustified, every scheduled pass aborts with MIGRATE_SOURCE_TOO_MANY and never converges until an operator inspects and runs the command with --force-revokes. The sweep is idempotent, and a clean instance does nothing.

Behaviour across the other tasks

Clone deliberately clears the association. The clone receives a new database user with no grant, so a carried association would compose a connection it could not open.

Migrate keeps the association, and re-establishes the wiring. Every migrate — platform move and rename alike — runs a deploy that mints the site a fresh database name and user and drops the old database, so the wiring is re-made by the verify the Migrate task queues for the migrated site. Migrating a source is the rougher case: each dependent target's settings.php and grant still name the freed old database name until that target's own next verify, and nothing chains those verifies after a migrate — the sweep is what repairs them. Migrating either end to a different database server is a different matter entirely: the association becomes ineligible and is cleared (see below).

Restore keeps it, and re-establishes it in the right order. Restore deploys over the site, rotating its database name and user and regenerating settings.php, and queues no hosting-task verify of its own (the backend verify it runs cannot refresh the front end's record of the database name) — so the feature queues one, which grants the new user. Restoring a site that is somebody's source is ordering-sensitive: tasks fork and run concurrently, and a freed database name is globally reusable, so a dependent verified against the stale name could re-grant onto whatever site takes that name next. The feature therefore verifies the restored site first — refreshing its recorded database name — and only from that verify's completion chains one verify per dependent target. The follow-up round is one-shot per restore, so it does not cascade. Expect access through the migrate connection to return after the second round, not the first. The superseded database name is recorded at rotation, which is what lets the sweep reclaim that user's leftover grant on a later pass — for a site whose database user carries the database's own name. An imported site whose user differs is reported for a human instead.

Disable does nothing to it, by design. Disabling a source takes it off the web but leaves its database intact and readable, which is what an operator finishing a migration at cutover normally wants. Disabling the target also keeps the grant — and hides the task from that site's task list, which is the client's only control over it, so a disabled target holds its access until the site is re-enabled or you intervene. Say so plainly if a client asks, because disabling looks like it should cut access.

Moving either end clears the association permanently. Every verify re-validates the stored association before pushing it. A source moved to another client, moved to another database server, or deleted — or a target moved onto a pre-Drupal-8 platform — makes the verify clear the stored source back to none, with a warning in that task's log, and the grant is revoked in the same run; the sweep enforces the client and database-server bounds independently. Undoing the move does not restore the wiring — the site owner sets the task again.

Verifies this feature queues by itself. Deleting a source queues one verify per site that read it — that verify is what performs the revoke, and BOA runs no routine site verifies, so nothing else would. Deleting a platform does the same for every source on it, with no Delete task ever running. Restoring a site queues its own verify and then one per dependent target. The sweep queues one per missing grant. The sweep's verify and the restore-chained per-dependent round are skipped for a site with an outstanding delete or verify; the teardown verifies queued when a source or its platform is deleted, and the restored site's own verify, are skipped only for a site with an outstanding delete.

Restrictions

The task is offered only on Drupal 8 and newer targets, and the backend refuses to emit on anything older even if the context is set by other means. On Drupal 7 the emitted block would land before BOA's settings chain derives that site's fallback hash salt from its database settings, changing the salt and invalidating every outstanding one-time login link — and changing it back when the wiring is removed. Since migrations run from Drupal 7 rather than into it, nothing is lost.

Source and target must be on the same database server; a grant cannot span servers.

The grant is only ever issued on localhost/127.0.0.1. On a database server configured to grant at % (db_grant_all_hosts), or a remote database server whose grants sit at the web server's address, the backend logs no local grant hosts found for this site, skipping and issues nothing — while the task still reports success and the settings block is still emitted. That log line is the symptom to look for when the connection never starts working on such a topology.