Even if it isn't neatly in it's own tree, you can get away with rewriting commits and some tree objects. It's more complicated, but it can be made to work.
But this isn't something that needs to be nice. Files moving between libraries should be rare in well designed libraries, and splitting a subtree out of git is easy enough for when the need does occur.
Keeping a giant repository on the other hand makes it very easy to do stupid things and end up with tightly coupled code. You don't want that to be the path of least resistance.
Advocating a solution that's only simple if all of your code is well-designed all the time is a poor strategy. Everyone has times they don't see the big picture clearly enough, or early enough, to segment or group responsibilities well. Or, even if you see the current big picture clearly, it's difficult to predict how future changes in business goals will change the conceptual model you have for your code today.
Either isn't as nice as `p4 move ...` though.