Skip to main content

AI Crawl Control

With the rapid rise of AI tools and crawlers, many applications are now seeing requests from AI systems that scrape content to train models or automatically summarize websites.
Not all AI crawlers are bad — some are legitimate (for example, search engine indexing or verified AI services), while others may scrape sensitive data or consume resources without permission.

Cloudflare’s AI Crawl Control (formerly AI Audit) helps you:

  • Detect and classify AI-related crawlers and scrapers accessing your site.
  • Differentiate between verified AI bots and unknown crawlers.
  • Decide how to respond — allow trusted crawlers, block unauthorized ones, or challenge suspicious traffic.

This gives you greater visibility and control over how AI systems interact with your content.

Why It Matters​

  • 🌐 Protect your content from being scraped without consent.
  • ⚖️ Strike the right balance between visibility (good bots) and control (bad bots).
  • 🔍 Gain insights into how AI systems are accessing your application.

👉 In this session, we’ll explore how to configure rules to filter AI crawlers, separating the useful from the unwanted.

Control AI Crawlers​

  1. Navigate to Cloudflare Dashboard > AI Crawl Control
Why do I only see a few AI Crawlers?

If you only see a small number of AI Crawlers or Bots, that’s normal — it means Cloudflare has not yet detected significant active AI traffic on your website.

only-google-ai

You can uncheck Show active crawlers to display the full list of AI Bots:

show-active-crawler more-ai-crawler

  1. You can choose to Allow or Block AI Crawlers from certain providers disable-ai-crawler

  2. Navigate to the Metric tab, and you can see if there are any bots crawling your application domains already for content ai-crawl-metrics

Key Takeaway​

  • ✅ Visibility first → AI Crawl Control gives you a clear view of which AI systems are accessing your site. Start by observing the dashboard before making enforcement decisions.
  • ✅ Not all AI bots are bad → Some are verified and beneficial (e.g., search engine AI crawlers), while others may scrape content without permission.
  • ✅ Granular control → You decide whether to allow, block, or challenge each AI crawler based on your business needs.
  • ✅ Balance is key → Use Crawl Control to protect sensitive data while still enabling trusted AI services that bring value.

📖 For deeper details, explore Cloudflare’s docs: AI Crawl Control.