SERP Forge · Free tool

AI Crawler Checker

See which AI crawlers a site lets in — GPT, Claude, Perplexity, Google and the training scrapers — read straight from its robots.txt.

Domain

Enter any domain to see which AI crawlers it allows or blocks — read straight from its robots.txt. No sign-up.

Verdict

How it reads the file. Each crawler is matched to its own User-agent group. Blocked = its group says Disallow: /. Allowed = it has a group that doesn't fully disallow. Default = no rule of its own, so the User-agent: * group applies. robots.txt is a stated instruction, not proof of what a crawler does — some ignore it.