I have qmail set up and running on one of my servers. Using multiple domains, aliases, etc., not supported by OS X Server. You can get a distribution of qmail with OS X binary at osxgnu.org... it's a bit hairy to setup and install, but problem I ran into was with relaying... the port of qmail...