Well Cloudflare doesn’t even block Google’s AI crawlers because they don’t diffe... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		rzz3 18 days ago \| parent \| context \| favorite \| on: Perplexity is using stealth, undeclared crawlers t... Well Cloudflare doesn’t even block Google’s AI crawlers because they don’t differentiate themselves from their search crawlers. Cloudflare gives Google an unfair competitive advantage.

warkdarrior 18 days ago [–]

Google claims their AI crawlers have user agents distinct from the search crawlers: https://developers.google.com/search/docs/crawling-indexing/...

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact