The difference is that if the NSA has physical access to my phone, I'm probably aware of it. It makes routine fishing expeditions across broad populations much harder and more expensive, as well as easier to oppose.

If they can fish remotely and automatically, accountability goes completely out the window.