Cloudflare Accuses Perplexity of “Stealth Crawling” Blocked Websites
Cloudflare has published a detailed report alleging that Perplexity an AI-powered search startup, has been bypassing robots.txt and firewall restrictions by disguising its bots.
When access is blocked, Perplexity’s crawlers reportedly change their user‑agent to mimic Google Chrome on macOS, rotate IP addresses and autonomous system networks (ASNs), and operate as “undeclared crawlers” to access content on restricted sites
Cloudflare detected this behavior across millions of daily requests across tens of thousands of domains.
As a result, Perplexity has been delisted as a verified bot and Cloudflare has deployed stricter anti‑scraping tools specifically to block such stealth operations
Perplexity’s spokesperson responded by calling the claims a “publicity stunt” and suggested misunderstandings or misattribution in the Cloudflare blog.
Meanwhile, Cloudflare is pushing a broader model where AI firms must obtain explicit permission or pay to crawl content part of its mission to empower publishers and protect intellectual property in the AI era.
This situation reveals the growing friction between content‑hosting platforms, web infrastructure providers, and AI companies. It raises critical questions about site owner rights, data scraping norms, and how trust can be maintained in AI development.
Cloudflare is also rolling out tools like AI Labyrinth, which ensnares suspicious crawlers in decoy content loops to expose bot behaviors and deterunauthorized scraping
Source: https://lnkd.in/ekNndyU7
