I don’t disagree with you about robots.txt… however, what _is_ the right tool fo... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		catlifeonmars 18 days ago \| parent \| context \| favorite \| on: Perplexity is using stealth, undeclared crawlers t... I don’t disagree with you about robots.txt… however, what _is_ the right tool for the job?

hundchenkatze 18 days ago [–]

auth, If you don't want content to be publicly accessible, don't make it public.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact