> That said, why does perplexity even need to crawl websites?

So you just came here to bitch about Cloudflare? It's wild to even comment on this thread if this does not make sense to you.

They're building a search index. Every AI is going to struggle at being a tool to find websites & business listings without a search index.