We have the same issue (billions of URLs). The newer bots that rotate IPs across...

		graybanana 17 days ago \| parent \| context \| favorite \| on: Perplexity is using stealth, undeclared crawlers t... We have the same issue (billions of URLs). The newer bots that rotate IPs across thousands of IP ranges are killing us and there is no good way to block them short of captcha's or forcing logins, which we would really rather not inflict on our users.