AI crawlers and assistants on your sites

AI bots visit your sites for very different reasons — some crawl to train models, some index you so you turn up in AI search answers, and some fetch a single page because a person asked an assistant to read it. Your host sorts them into classes and applies a sensible default to each, the same on every site. You can change that default for any one site, from a small text file in your own account.

You don't have to set anything up for the defaults to apply — they're already in force on every site. Reach for this page only when a particular site wants something other than the default.

What the defaults are

For most sites these defaults are exactly right, and you can leave this alone entirely. The balance to notice: blocking the training class does not hide a site from AI tools — AI search indexing and person-triggered assistant fetches stay open, so you keep the visibility while skipping the bulk harvesting.

Changing it for a site

You set overrides in a control file in your own account:

static/control/ai/policy.txt

On a typical account that's ~/static/control/ai/policy.txt in your shell (the folder may not exist yet — just create it). Add one line per site: the site's name, then a space, then the changes you want. A site with no line keeps the defaults above.

You can combine flags on one line, in any order:

# static/control/ai/policy.txt

# A news site happy to be in AI training data:
news.example.com    train-allow

# A members-only shop that wants no AI bots indexing or fetching it:
shop.example.com    search-block user-block utility-block

# A blog that wants AI search but not the assistant "user" fetchers:
blog.example.com    user-block

# A research site that wants even the evasive Perplexity assistant allowed:
research.example.com    evasive-allow

As with the IP-lock control files, lines starting with # (and anything after a # on a line) are ignored, and so are blank lines — so you can annotate freely, labelling which site each rule is for. The change is picked up on its own within about two minutes — no restart, no support ticket. To put a site back to the defaults, just delete its line; the override is removed on the next pass.

What stays on no matter what

A few AI-related protections are always on for every site and aren't set from this file:

• outright scrapers and download tools — the mass-harvesting bad bots,
• forged opt-out tokens — a request arriving as Google-Extended or Applebot-Extended (those are content-licensing signals, never real crawlers, so anything wearing one as its identity is turned away), and
• credential and dotfile probes — requests fishing for .env, .git, and similar secret paths.

These are blocked on every site and there's no line you can add here to change them. If one ever catches a genuine request you need to get through, that's one for your host to look at — the same as any other server-level control.

Related pages

• Restricting access to your site — the overview of the access-control family, and the default admin-page protection.
• Site IP lock and Login/admin IP lock — the IP-based locks, managed from the same kind of control files.
• Shell and SFTP access — how to connect so you can edit policy.txt in the first place.
• Managing your sites — the day-to-day site tasks in your control panel.