Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Adhering to robots.txt is merely a courtesy.

Much like a trolley drop off at your local shopping center car park. Some users will adhere to it and drop their trolleys in after their done. Others will not and will leave it wherever.

Your machine might access a page via a browser that is human readable. My machine might read it via software and present the content to me in some other form of my choosing. Neither is wrong. Just different.

Don't like it? Then don't post your website on the internet...



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: