I think the graphics stack is a sore point on most or all platforms that have to support multiple GPUs from multiple vendors, at least the stuff in between userland GL calls and the bare metal.

In an ideal world it would be obvious where to draw the line between the GPU- and vendor-independent bits and the common code shared by everyone, but not only is that very hard to do, it can change drastically over the years. Couple that with the fact that platform vendors have to maintain these imperfect interfaces for many years regardless, and that vendors can have their own abstraction layers in between the top layer of their driver code and the bare metal of all their GPUs, and you have a perfect storm of too many layers getting in the way and imperfectly defined interfaces to many of those layers, which necessitates occasional layering violations at every level of the stack if you want to avoid some pathological performance bottlenecks for some GPUs.

At best you end up with a bunch of ugly, imperfect, but generally functional software that more or less gets the job done at an acceptable level of performance.

I would imagine that the situation with Linux just adds yet more issues to that bag of hurt. GPU vendors don't always want to help out, and are likely to withhold valuable technical information that they don't want to be public. The whole process of defining how many layers there are, what functionality exists at what layer, and what interfaces are used between the layers is almost by definition (for GPU driver models at least) a bunch of questions for which there are no good answers, which is also a perfect storm for Open Source development. Any solution anyone comes up with is always going to have some caveats and gotchas, which for open, egalitarian development models means endless arguments and bikeshedding in which everyone involved is correct in pointing out the weaknesses of the system as designed, but there are no alternatives that anyone can point to that don't have similar problems in someone else's eyes.

For example, back in the '90s, for right or wrong I was very optimistic about GGI solving a lot of graphics problems for Linux, and that was limited to the 2D stuff (I can't imagine how controversial it would've been had it included 3D as well):

http://en.wikipedia.org/wiki/General_Graphics_Interface

It didn't pan out that way, but so much energy was burned by so many people arguing about it over so many years. :( I don't have authoritative knowledge of the whole thing, so maybe the doubters were correct, but at the time it felt like a bunch of people shooting down an imperfect solution while offering no solution at all.

EDIT: TL;DR designing a graphics stack that ultimately goes down to the metal is a set of very hard problems, many of which have no perfect answers. This, coupled with secretive vendors and the level of specialized knowledge required to even contribute to stuff like this, makes it an even harder problem in the Open Source world. Not an insurmountable obstacle, but a really hard set of problems nonetheless, for both technical and human reasons.