Breaking: Cloudflare Launches “Pay Per Crawl” to Let Sites Charge AI Crawlers
Cloudflare has introduced a new feature called “Pay Per Crawl”, allowing websites to charge AI engines for crawling their content.
This gives publishers more control, offering a middle ground between blocking AI bots entirely and allowing free access.
With this system, AI crawlers must include payment in their request headers. If not, they receive a “402 Payment Required” response.
Site owners can then decide whether to allow, block, or monetize AI access.
While this empowers publishers to monetize their content, it raises concerns…
As long as some sites provide free access, AI models may not feel pressured to pay.
The system may benefit only the top-tier publishers whose content is essential for AI training.
Credit to Chris Long for informing us!
