Robots.txt Generator

Choose what crawlers may do and get a valid robots.txt, including the AI crawler groups worth deciding on separately.

Free to use, no sign-up, and your text is not stored.

Site URL

Let search engines index the site

Let AI search crawlers in

These decide whether ChatGPT, Claude and Perplexity can quote you.

Let AI training crawlers in

Affects model training only, not whether you get cited.

Paths to keep out

One path per line, e.g. /admin

robots.txt

User-agent: *
Disallow:

User-agent: OAI-SearchBot
Disallow:

User-agent: Claude-SearchBot
Disallow:

User-agent: PerplexityBot
Disallow:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

Your first article is free.

Then one article a day, published for you.

See what AI says about youEarn backlinks to your siteGet a new article every day