robots.txt isn't even designed to stop recursive fetches. It is designed to ask nicely recursive fetchers not to recursively fetch. It comes from a time where site operators wanted their sites to be scraped by search engines, but not things like edit links and admin panels.