Yes, but there's no way to say "allow indexing for search, but not for AI use", ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		foota 18 days ago \| parent \| context \| favorite \| on: Perplexity is using stealth, undeclared crawlers t... Yes, but there's no way to say "allow indexing for search, but not for AI use", right?

warkdarrior 18 days ago [–]

But there is: https://developers.google.com/search/docs/crawling-indexing/...

There is an user agent for search that you can control in robots.txt.

    user-agent: Googlebot

There is another user agent for AI training.

    user-agent: Google-Extended

foota 18 days ago | [–]

Wow, I had no idea this page existed, thanks for the reference!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact