Umm, actually, OS X and aqua work just fine without the BSD subsystem. I know, because I just installed a copy (10.2) without it yesterday. Just click the customize button in the install window, and uncheck the BSD subsystem. Install goes fine, system works fine, can't even tell the difference. Of course, i am not trying to use the developer tools on that install, nor am I trying to do much in the command line, which is what the BSD subsystem is needed for. Otherwise, the OS works just fine without it.
As for installing it only over an existing OS X install, It appears it can be done by doing the following (again tested in 10.2):
1) Insert the 10.2 install disk one
2) Navigate to Mac OS X Install (or upgrade, whatever) Disk/System/Installation/Packages/
3) Open and install BSD.pkg as you would any .pkg
I tried this just now on the aforementioned system that I installed without the BSD subsystem, and the install went off without a hitch. I don't know how I would test it to make sure it is installed (I suppose I could try installing the developer tools) but since there was no errors during install I don't see why it wouldn't have worked. Hope this helps!