First thing's first: uninstall Saft. The more add-ons you have, the more chances for instability.
I wouldn't dare uninstall PithHelmet, though, because IMO Safari is simply unusable without it. However, you should make sure you're using the current version (there are two "current" versions out right now; one for OS 10.3.9 and one for 10.3.8). Set the "Block mode" from "Faster" to "Safer", and try disabling "Fast collapsing" (under the Advanced tab of your site rules).
Aside from that, I really don't know. What sites does it tend to crash with? I personally have never had major stability problems with Safari, with the exception of a few nasty Java applets.
As for speed...that's complicated. Again, which sites are giving you problems? PithHelmet has been known to really, REALLY slow things down if it's not properly configured, but in my experience the latest version doesn't cause major problems.
Here's one tip for increasing Safari's speed (or at least it's perceived speed): open Terminal and type "defaults write com.apple.Safari WebKitInitialTimedLayoutDelay 0.25". That'll change the standard delay before displaying the page from the default 1 second to 0.25 seconds. This way Safari will display data from incomplete pages sooner (like Firefox does). You might even want to set it to 0.1. Different people recommend different values, and naturally YMMV.
Safari's web site icons cache can slow things down, too. Download
SafariSpeed to clear Safari's icon cache and keep it from making a new one. SafariSpeed also provides a simple GUI to the trick I described above, although it uses 0.000001 (or something like that) as its value, which I think is way too aggressive.
As for your problems with Firefox, beats me. One of the main reasons I don't use Firefox is because of its terrible scrolling, too. Keyboard-controlled scrolling is way too slow, and mouse wheel-controlled scrolling is too fast and completely non-standard. I use USB Overdrive to fine-tune my mouse wheel behavior, but Firefox doesn't play by the rules like everything else does. Grr. The other reason I prefer Safari is Firefox's all-around hacky interface. And I personally like PithHelmet much better than Adblock.