Many publishers ask the same question: should I let AI bots read my site? Here is the background and the latest news.
The imbalance
A TollBit report cited by Search Engine Land found Google sent publishers 831 times more visitors than AI systems in the second quarter of 2025. Cloudflare’s crawl-to-visit ratios, reported by Search Engine Land, were about 60,000 to 1 for Anthropic and 1,500 to 1 for OpenAI, versus 18 to 1 for Google. In simple words: AI bots take a lot and send little back.
What Cloudflare offers
- Pay Per Crawl launched in private beta on July 1, 2025. A publisher can set a price for AI crawlers, using the web’s “402 Payment Required” response. Each crawler can be set to allow, charge or block. Cloudflare also began blocking AI crawlers by default for new domains.
- Cloudflare says it has blocked 416 billion AI bot requests since the launch.
- On September 30, 2026 Cloudflare moved two new tools into beta: a Monetization Gateway (charge per request) and Pay Per Use (publishers are paid when AI companies report using their content).
- On August 4, 2026 it announced Wallets for AI agents. Coverage notes there is still no general availability or published pricing.
Google’s position
Google is running its own invitation-only AI payment pilot, reported by Search Engine Journal.
The big warning
Search Engine Journal points out that blocking or charging Googlebot can hurt your search visibility, because Google treats a 402 response as a client error. Separate Googlebot from the AI-only bots before you change any rule.
What to do
- List which bots hit your site, and separate search bots (keep them) from AI training bots (your choice).
- Decide on purpose. If AI citations matter for your brand, blocking everything also removes you. (Why each AI differs)
- Do not lock a permanent block into contracts. Coverage advises against it while the systems are still changing.
- Check your CDN settings. Cloudflare’s defaults may already be blocking bots you wanted.
