At first I hated the 3D thing, but that was based on knee-jerk reaction. I'm not going to DL it to test as I don't want to install any unnecessary apps, but it looks at least worth playing around with.
My personal work arounds for navigation have been to create keyboard shortcuts with iKey (e.g. Control+D is my documents folder). Occasionally these shortcuts interfere with application-native shortcuts (FCP/Maya), but for the most part they work across the board. Additionally, I think voice control is woefully underused. I used to set up voice control sequences to launch frequently used files, but the recognition was so finicky that it ended up being more frustrating than useful. I still think that's a whole untapped approach.
I think Metadata navigation is going to be good and bad. Too much Metadata is going to give us the Google effect, where you end up with a gazillion files and you only need a specific one. Now Spotlight's filters will work around that to some extent, but it will be interesting to see it in action, if it gets too out of control for some of us.
On some level we still need to organize things logically/hierarchically for a variety of reasons. Many apps require proper relative pathing for file access (web design/page layout/video editing and more). And for archival, it's a lot easier to just burn a folder than trying to do some kind of metadata search for all related files for a given project. Course, soon as I say that, there's no reason a metadata tag couldn't be "Project Name", thereby mitigating my argument a bit.
In the end, I agree it seems like there could be a "better way", but I certainly don't know what it is. I think it would be fun as hell to develop a vaporware GUI. I had started one a while back that was based on the idea of anal-retentive cleanliness. There were no icons or menus visible on the screen. You had to drag to and screen area top, bottom, or either side and menus would pop out like the Dock does now. Kind of like how InDesign CS's new side pallettes work, except completely hidden until you roll over to that area. Not very well thought out, but it was clean!