Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Is the article really talking about crawling? Because in one of their screenshots where they ask information about the "honeypot" website you can see that the model requested pages from the website. But that is most definitely "fetching by proxy because I asked a question about the website" and not random crawling.

It is confusing.



Yea, now I feel like my comment might be misleading. The title mentions crawling; the article itself is talking about something else.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: