AI crawler policy — per-class maps, rate limits, per-site flips

BOA classifies AI crawlers, indexers and assistant fetchers into mutually-exclusive classes by user-agent, and applies a default action per class at the Nginx edge, before a request reaches PHP-FPM. Classification is always by a distinctive UA token, never a bare vendor name — so GPTBot (training) and ChatGPT-User (assistant fetch) land in different classes, and a bare ChatGPT matches nothing.

The class maps are rendered into the master http{} config by server.tpl.php and enforced per request by the guard chain in Inc/vhost_include.tpl.php. The rest of the edge machinery those guards depend on — Cloudflare real-client-IP recovery, the secret-path deny, the shared reload lock — lives on Edge policy; the abuse/IDS layer is Abuse guard.

The classes and their defaults

The stance: block the worst offenders unconditionally, sort every real AI agent into a class, and make each class flippable per site. Training and the evasive user-fetch class are opt-in (off by default); search / user / utility are opt-out (on by default). Source: server.tpl.php:323 ($is_ai_training), :333 ($is_ai_search), :346 ($is_ai_user), :359 ($is_ai_evasive), :368 ($is_ai_utility), :378 ($is_ai_forged).

Default inversion — training is BLOCK by default

This is a behaviour change from the earlier policy. The old single broad $is_ai_crawler map allowed AI by default; the per-class policy blocks training by default and only allows search, user and utility traffic (each rate-limited). Operators who want AI training crawlers on a given site must opt in explicitly (see per-site control). For example, Amazonbot is classed as training because it is a bulk crawler that feeds Amazon's models, so it is blocked by default; a site that wants Amazon indexing can allow training for it (train-allow) or move the token into the utility class in server.tpl.php.

The $is_ai_crawler backward-compat shim

The old broad $is_ai_crawler map is kept defined as a do-not-remove backward-compatibility shim (server.tpl.php:303-318, which carries an explicit DO NOT REMOVE banner). The reason is a render-order trap: a barracuda upgrade re-renders this master http{} config before the per-Octopus instance vhosts. An instance vhost rendered by the previous Provision still references $is_ai_crawler, so an http config that no longer defined it would make nginx -t fail with unknown "is_ai_crawler" variable — taking every instance on the host down until each is individually re-rendered. Shared http-block map variables must stay a superset across versions: never remove or rename one a deployed vhost may use.

Evasive class — best-effort only

$is_ai_evasive exists for agents that are nominally user-triggered but ignore robots.txt and, when blocked, drop their declared UA and rotate IPs/ASNs to slip past a UA rule. Perplexity-User is the current member — Cloudflare de-listed it as a verified bot for exactly this. It is blocked by default, cleared per site by the evasive-allow flag. Because the block keys on the UA, it is best-effort only: once the agent abandons its UA it looks like an ordinary browser and the (fail-open) maps let it through. The real backstop for the rotating/undeclared traffic is the IDS/CSF layer (see Abuse guard), not this policy. Source: server.tpl.php:352-363.

Note the honest counterpart Google-Agent is classed as an honest user-fetcher in $is_ai_user (matched as Google-?Agent, server.tpl.php:348), not as evasive.

UA classification is forgeable — IP verification is deferred

Classification in this cut keys on the UA token alone, which any client can forge — a request can simply claim to be ChatGPT-User to land in an allowed class. The robust hardening is to gate each allowed-class match on the source IP belonging to that vendor's published, signed IP-range feed (OpenAI, Anthropic and Mistral each publish one; the fetch would mirror cloudflare_realip.sh), and it is deliberately not in this cut. Such IP verification would confirm genuine OpenAI/Anthropic traffic, but it still could not make Perplexity-User blockable by IP — that agent rotates outside its own published range, so it stays with the IDS/CSF layer, not this policy.

Forged opt-out tokens → universal 444

Google-Extended and Applebot-Extended are robots.txt-only opt-out directives — a real crawler never sends them as a live user-agent. Their presence in a UA is therefore proof of forgery and is hard-blocked (444) globally, regardless of UA class, with zero false positives. Source: server.tpl.php:378-382; the guard fires at vhost_include.tpl.php:218 (after the secret-path deny and the $is_cms_probe guard).

The $is_crawler / AI boundary

The scrapers / bad-bots hard-block map $is_crawler (444 at vhost_include.tpl.php:257) is separate from the AI maps. The former AI substrings (Amazon, bytedance, CCBot, externalagent, openai, perplexity) were deliberately removed from $is_crawler because bare openai / perplexity matched the openai.com / perplexity.ai vendor URLs embedded inside OAI-SearchBot / ChatGPT-User / PerplexityBot / Perplexity-User user-agents, and so wrongly blocked allowed AI traffic.

AI is now governed solely by the per-class $is_ai_* maps. Do not re-add AI tokens to $is_crawler — per the never-pollute shared-map rule. Source: the removal-rationale comment and cleaned roster at server.tpl.php:478-492. ($is_crawler itself — the generic scraper block — is part of the abuse-guard layer; see Abuse guard.)

Keeping the class maps current

The $is_ai_* maps are a point-in-time roster: vendors add and rename agents, so the token lists are only current as of the cut that shipped them. Re-check each vendor's crawler documentation periodically and add any new token to the matching $is_ai_* map in server.tpl.php, keeping its $ai_*_limit_key roster in step (per the per-vendor rate-limit rule).

A UA that matches no token in any map is not blocked by this layer — it falls through to ordinary, non-AI request handling. That is fail-open by design, not fail-closed: the maps act only on agents they recognise. Worth remembering when auditing logs — an AI fetcher that passed through un-throttled is usually just one whose token is not in the roster yet, not a policy failure. The backstop for genuinely abusive but unrecognised traffic is the IDS/CSF layer, not these maps (see Abuse guard).

Applying a map change is an edit to the $is_ai_* map in server.tpl.php, not to any deployed file: the roster is part of the Provision-rendered master http{} config, so a platform/site provision-verify re-emits it and reloads Nginx to pick it up — the render-and-reload workflow owned by Config templates. Do not hand-edit the deployed master config — Provision overwrites it on every render and a barracuda upgrade re-fetches server.tpl.php itself, so a box-local edit does not survive; the durable path for a new token is the BOA source.

Per-vendor rate limits

The three allowed AI classes are throttled by limit_req zones, but keyed per vendor rather than per client IP. The keys come from map definitions that emit a constant string per UA token (one constant per vendor) and an empty key for everything else — $ai_search_limit_key, $ai_user_limit_key, $ai_utility_limit_key (server.tpl.php:395-421). An empty key is not counted by its zone, so no non-AI traffic is ever throttled by these zones.

Source: zones at server.tpl.php:109-111. Keying per vendor (not per IP) is the correct primitive for an assistant whose single prompt fans out across many source IPs: a per-IP cap never bites the aggregate (each IP stays under the limit), whereas one shared key per vendor caps that vendor's total rate across every IP and vhost, and gives each vendor its own bucket so no one vendor can starve another's allowance.

Each $ai_*_limit_key roster must track its matching $is_ai_* class map — a token in one but not the other is silently un-rate-limited (or, for the class map, un-blockable). The two are kept adjacent in server.tpl.php for that reason.

Per-site control — policy.txt

The global defaults are flipped per site via a per-Octopus control file:

/data/disk/<oct>/static/control/ai/policy.txt

read by aegir/tools/system/ai_policy.sh. Each record is one line:

<site>  [train-allow] [evasive-allow] [search-block] [user-block] [utility-block]

ai_policy.sh generates a per-site fragment into the instance's own config/includes/ai_policy/<site>.conf — exactly the path the per-satellite vhost pulls via include $server->include_path/ai_policy/<uri>.conf* (the .conf* suffix anchors the glob to this site's own fragment; a bare <uri>* would also match a longer site whose name extends it, e.g. example.com vs example.com.au). The flag → directive mapping (ai_policy.sh:87-91):

A site with no record keeps the global defaults. Lines beginning with # are comments — a # anywhere on a line drops the rest of it, and blank lines are ignored — so you can annotate policy.txt freely, labelling which site each rule is for or leaving a note beside an override (ai_policy.sh:76-77). ai_policy.sh is a single global script that loops over every Octopus instance with an activated policy.txt. The control file is also catalogued in Control files & INI as a per-site control file, and the hosted-customer view of the same file is AI crawlers and assistants.

The guard chain (per request)

The relevant subset of vhost_include.tpl.php runs top-to-bottom early in request processing. Training and evasive are blocked by a composite guard (set the block to the class, clear it if the per-site allow flag is set, drop if still set); search / user / utility have no global guard because they are allowed by default, so a per-site block is carried directly by the policy.txt fragment.

if ($is_secret_path)  { return 444; }   # secret/config-path probes     (vhost_include.tpl.php:198) — edge-policy layer
if ($is_cms_probe)    { return 444; }   # foreign-CMS admin-path probes (vhost_include.tpl.php:210) — abuse-guard layer
if ($is_ai_forged)    { return 444; }   # forged opt-out tokens         (:218)

set $ai_train_block $is_ai_training;     # training composite           (:230)
if ($ai_train_allow)  { set $ai_train_block ''; }   #  per-site opt-in clears it
if ($ai_train_block)  { return 444; }    #  otherwise training blocked   (:234)

set $ai_evasive_block $is_ai_evasive;    # evasive composite            (:246)
if ($ai_evasive_allow){ set $ai_evasive_block ''; } #  per-site opt-in clears it
if ($ai_evasive_block){ return 444; }    #  otherwise evasive blocked    (:250)

$ai_train_allow / $ai_evasive_allow are defaulted to 0 before the ai_policy include in the vhost template, so without a fragment both classes stay blocked. The rate limit for the three allowed classes is applied separately in location / via limit_req zone=ai_search|ai_user|ai_utility. (The $is_secret_path guard is documented on Edge policy; the $is_banned, $is_crawler and $is_cms_probe guards that bracket/interleave this excerpt belong to the abuse-guard layer — see Abuse guard.)

Related

• Edge policy — the machinery this policy stands on: Cloudflare real-client-IP recovery (every guard above keys on the true visitor), the secret-path deny, and the shared Nginx reload lock that ai_policy.sh takes before its configtest+reload.
• Abuse guard — the IDS/abuse layer: $is_banned geo, flood chains, $is_crawler, scan_nginx scorer, the CSF ban pipeline.
• Config templates — the master Nginx config the class maps render into, and the never-remove-map-vars rule.
• Control files & INI — policy.txt and the other per-site control files.
• AI crawlers and assistants — the hosted-customer page for the same per-site control file.