Glossary
What is an AI crawler?
An AI crawler is a bot that fetches public web pages so an AI search or answer system can discover, understand or cite content.
Last updated: 4 June 2026.
Why it matters
If robots.txt or meta robots blocks the crawler, the page may be harder to include in AI search answers. Site owners should decide which crawlers they want to allow and document that choice clearly.
What to check
- robots.txt does not accidentally disallow important pages.
- Pages meant to rank do not use noindex or nosnippet.
- Sitemap.xml lists the important public pages.
- The content is visible in HTML, not hidden behind fragile scripts.