AI crawlability
Can AI crawlers read your page?
Paste a URL. We fetch it as both a browser and a GPT-class crawler, parse robots.txt and meta robots, and tell you exactly what assistants like ChatGPT, Perplexity, and Claude see.
Frequently asked questions
- How does this checker work?
- We fetch the URL you paste twice: once with a browser user-agent and once with your selected AI crawler. We also request /robots.txt and parse the directives for GPTBot, PerplexityBot, ClaudeBot, and Google-Extended. The response shows HTTP status, content parity, and robots coverage side by side.
- Why does the AI fetch return less text than the browser?
- Client-side rendering. If your page needs JavaScript to paint content, an AI crawler that does not execute JS will see a near-empty body. Server-render the critical copy so the first HTML response contains your answers.
- Does a 200 OK mean I am good?
- Not necessarily. Some sites return 200 with a cookie banner or paywall wall that hides content from the bot. Compare the visible-text counts. If bot parity is under 70%, your content is not reaching assistants.
- Does this respect robots.txt for my own domain?
- Yes. The request is a standard HTTPS fetch that honors your server's response. We do not bypass paywalls, authentication, or Cloudflare challenges.