This is a hypothetical so give me a little rope here, but what if robots.txt wasn't a suggestion? What if it were binding (leaving aside for a moment how one would enforce / mandate / guarantee that)?
Would that solve the whole problem? Folks who ran webservers declared what they consent to, and that happens?
I think it's useful to just see if there's a consensus on that: actually making that happen is a whole can of worms itself, but it's strictly simpler than devising a good outcome without the consensus.
(And such things are not impossible, merely difficult, we have other systems ranging from BGP to the TLD mechanism that get honored in real life).
Would that solve the whole problem? Folks who ran webservers declared what they consent to, and that happens?
I think it's useful to just see if there's a consensus on that: actually making that happen is a whole can of worms itself, but it's strictly simpler than devising a good outcome without the consensus.
(And such things are not impossible, merely difficult, we have other systems ranging from BGP to the TLD mechanism that get honored in real life).