AI crawlers and assistants on your sites

AI bots visit your sites for very different reasons — some crawl to train models, some index you so you turn up in AI search answers, and some fetch a single page because a person asked an assistant to read it. Your host sorts them into classes and applies a sensible default to each, the same on every site. You can change that default for any one site, from a small text file in your own account.

You don't have to set anything up for the defaults to apply — they're already in force on every site. Reach for this page only when a particular site wants something other than the default.

What the defaults are

For most sites these defaults are exactly right, and you can leave this alone entirely. The balance to notice: blocking the training class does not hide a site from AI tools — AI search indexing and person-triggered assistant fetches stay open, so you keep the visibility while skipping the bulk harvesting.

Changing it for a site

You set overrides in a control file in your own account:

static/control/ai/policy.txt

On a typical account that's ~/static/control/ai/policy.txt in your shell (the folder may not exist yet — just create it). Add one line per site: the site's name, then a space, then the changes you want. A site with no line keeps the defaults above.

You can combine flags on one line, in any order:

# static/control/ai/policy.txt

# A news site happy to be in AI training data:
news.example.com    train-allow

# A members-only shop that wants no AI bots indexing or fetching it:
shop.example.com    search-block user-block utility-block

# A blog that wants AI search but not the assistant "user" fetchers:
blog.example.com    user-block

# A research site that wants even the evasive Perplexity assistant allowed:
research.example.com    evasive-allow

As with the IP-lock control files, lines starting with # (and anything after a # on a line) are ignored, and so are blank lines — so you can annotate freely, labelling which site each rule is for. The change is picked up on its own within about two minutes — no restart, no support ticket. To put a site back to the defaults, just delete its line; the override is removed on the next pass.

What stays on no matter what

A few AI-related protections are always on for every site and aren't set from this file:

• outright scrapers and download tools — the mass-harvesting bad bots,
• forged opt-out tokens — a request arriving as Google-Extended or Applebot-Extended (those are content-licensing signals, never real crawlers, so anything wearing one as its identity is turned away), and
• credential and dotfile probes — requests fishing for .env, .git, and similar secret paths.

These are blocked on every site and there's no line you can add here to change them. If one ever catches a genuine request you need to get through, that's one for your host to look at — the same as any other server-level control.

What about robots.txt?

robots.txt still matters, and nothing on this page changes what your site says in it. But it is a request, not a control: it works only on crawlers that choose to read it, on their own refresh schedule, and only on what your file actually says. Three things are worth knowing before relying on it against AI crawlers:

• Stock files predate the AI crawlers. The robots.txt that ships with Backdrop and with Drupal 6 and 7 has a single User-agent: * block that keeps crawlers out of the admin, login and search paths, leaves all content open, and asks for Crawl-delay: 10. Drupal 8 removed the Crawl-delay line in 2015 and Drupal 9, 10 and 11 never had it. None of them names an AI agent. On an end-of-life core the stock file will never change again, and on any core it changes only when you, or a module, change it.
• Compliance is partial and uneven. Anthropic documents that its crawlers honour robots.txt and Crawl-delay. OpenAI documents that GPTBot honours robots.txt, that a change can take about a day to reach its search crawler, and that for the person-triggered ChatGPT-User fetches "robots.txt rules may not apply". Google documents Crawl-delay as unsupported. Independent measurements (TollBit, first half of 2026) found roughly one in seven identified AI page-fetchers on URLs the site had disallowed, and Cloudflare documented Perplexity crawling behind an undeclared browser user-agent from rotating addresses after being blocked.
• The file cannot single out what it does not name, and it cannot pace anything. New agents appear every few months; a file written last year does not know them, and a compliant crawler taking every page of a site whose file never mentioned it is breaking no rule. Crawl-delay is honoured by some crawlers and ignored by others, and there is no directive at all for "this much per vendor across all of its addresses".

That is why the policy on this page is enforced by the web server, per request, regardless of what any robots.txt says or when a crawler last read it: training crawlers are refused at the door, the allowed classes are paced per vendor (the thing Crawl-delay asks for and cannot enforce), and either default flips per site from policy.txt above.

Your robots.txt stays yours. On Drupal 6 and 7 sites the RobotsTxt module is kept enabled and its output is served as a static snapshot; on every core a sites/<yourdomain>/files/<host>.robots.txt file, which BOA never regenerates, is served ahead of the platform file — see Your robots.txt is managed too. Nothing here writes AI rules into it; if you want the sign to say the same as the door, add the lines yourself.

Publishing your AI policy at /llms.txt

Separate from blocking or allowing crawlers, you can tell AI tools what your site's usage policy is — the emerging convention is a plain-text file served at /llms.txt. Every BOA site already has the route wired: requests for /llms.txt are served from your site's own files directory, so publishing a policy is just uploading a file — no module, no configuration:

1. Write your policy as plain text in a file named llms.txt.
2. Upload it into your site's sites/<yourdomain>/files/ directory (over SFTP or the shell).

That's it — https://yourdomain/llms.txt serves it immediately. Two extras:

• Different policies per domain: a multi-domain site can also upload <host>.llms.txt files (for example shop.example.com.llms.txt) — the per-host name is tried first, and plain llms.txt is the fallback.
• Prefer managing it from Drupal? The contrib llms_txt module works too — the route serves whichever exists.

Related pages

• Restricting access to your site — the overview of the access-control family, and the default admin-page protection.
• Site IP lock and Login/admin IP lock — the IP-based locks, managed from the same kind of control files.
• Shell and SFTP access — how to connect so you can edit policy.txt in the first place.
• Managing your sites — the day-to-day site tasks in your control panel.