At Burzcast, we believe in the foundational values of the open web: transparency, trust, and respect for publisher intent. That’s why we’ve chosen Cloudflare not just as a trusted infrastructure provider, but as a technology partner we actively recommend, implement, and promote through our consulting work and new media content. We’ve seen firsthand the benefits their platform brings in performance, security, and — increasingly important in the AI era — responsible traffic management.

So when Cloudflare recently published their findings about Perplexity’s stealth crawling behavior, we took note. Not just because we run multiple production websites protected by Cloudflare’s WAF and bot detection rules, but because we’ve witnessed a surge in AI-powered bot traffic, much of it disrespecting even the simplest of protocols: robots.txt.

Let’s be clear: this is not about being anti-AI. We are avid users of AI in our work. We build client solutions on OpenAI models via Azure AI Foundry, and we even use Perplexity Pro to research, brainstorm, and test next-gen user experiences. But there’s a world of difference between leveraging AI responsibly and abusing the web’s trust model for unconsented data harvesting.

Robots.txt Is Not Optional

The robots.txt protocol may be decades old, but its purpose remains vital: it’s a website owner’s request, clearly declared, for how bots should interact with their content. It’s not about “hiding secrets.” It’s about maintaining bandwidth, respecting copyrights, and preventing misuse. This isn’t a gray area or a loophole to “vibe-code” around. It’s a simple directive.

And that’s what makes Perplexity’s actions so egregious. Cloudflare observed Perplexity ignoring these directives, masking their bot identity, and re-attempting access from different ASNs and generic browser user agents, like ones impersonating Chrome on macOS. That’s not just careless — that’s deceptive.

Contrast that with how OpenAI handles robots.txt: their crawler, ChatGPT-User, reads the file and respects it. If it’s blocked, it stops. If it encounters a block page, it doesn’t try again with another user agent. That is the behavior of a company that understands and respects the implicit contract between web publishers and automated tools.

Perplexity’s Response Is a Distraction

Rather than directly addressing Cloudflare’s findings, Perplexity fired back with a deflection, blaming a third-party service (BrowserBase) and accusing Cloudflare of seeking “a publicity moment.” They didn’t deny the behavior. They didn’t clarify their policies. They didn’t explain why their crawler ignores robots.txt. And most worryingly, they didn’t say when — or if — they plan to fix it.

We are left with a troubling question: If Perplexity can’t guarantee their crawler plays by the rules, how can they be trusted with the content they gather and serve?

We Stand with Cloudflare

Cloudflare’s job isn’t to be popular. It’s to protect the web, and that means holding even their customers accountable. We respect that. We appreciate it. And we rely on it.

As a company that builds secure, performant, and ethical technology solutions for clients across the globe, we will continue to trust Cloudflare’s insights, and we urge AI providers — Perplexity included — to return to the basics. Respect robots.txt. Be honest about your crawler identity. Stop circumventing rules just because you can.

Because the moment we normalize deception in AI crawling is the moment we lose trust in the very web that makes these systems possible.