The location providers on Android are not tied to any one app, but are system level services that apps register with for updates. Location that takes GPS can take longer to acquire fixes and Google Maps will attempt to use the most accurate fix possible, so if the GPS acquisition is slow, maps will be slow.

I totally understand your points, though. I use an older device with roughly the same specs, and when I was using stock 2.3 it wasn't very fast.