No they're supposed to allow scraping and information aggregation. That's the es... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		spwa4 18 days ago \| parent \| context \| favorite \| on: Perplexity is using stealth, undeclared crawlers t... No they're supposed to allow scraping and information aggregation. That's the essence of the web: it's all text, crawlable, machine-readable (sort of) and parseable. Feel free to block ddos'es.

bayindirh 18 days ago [–]

Feel free to crawl paywalled sites and republish them with discoverable links.

Also after starting the crawl, you can read about Aaron Swartz while waiting.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact