Okay, the absolute simplest thing to do first, which you've likely already done, is to make sure /etc/hostconfig contains the line MAILSERVER=-YES- rather than with the default -NO-. If you do have to change it to -YES-, then there's a certain process you'll have to kill -HUP before it starts running. I think inetd might be the process that keeps track of the services to run. If all else fails, just reboot.
The most common problem with sendmail on OS X is just a permission problem, and one which will show up after installing most packages from Software Update. Check /var/log/mail.log and see if it's complaining a lot about a group writable directory. If that's the case, then the problem is that the root directory is group writable. That's easily fixed with a quick "sudo chmod g-x /". Then sendmail should be able to start properly, and I think it can be invoked by typing "/usr/sbin/sendmail -bd -q1h". Again, if all else fails, reboot.
After it gets running, the only problems you'll have to deal with are relaying problems. By default, sendmail won't let you bounce mail off of it on the way to a different destination computer. To tell sendmail which hostnames to accept mail for, the hostnames should be listed in /etc/mail/local-host-names although there may be a special syntax for the entries (I haven't configured sendmail in quite a while.) You'll want to include localhost in those entries, along with any permanent hostnames associated with the mail server. Assuming that you're running sendmail just for local, experimental purposes, you shouldn't need more than localhost. Mail could be sent to me@localhost type addresses. Unfortunately, you can't just give it IP addresses and send mail to me@10.0.0.1 or whatever. Oh, and updating this local host list will probably require restarting sendmail.
Holy crap, it's amazing how much I'll end up typing when nobody makes me stop. He he. Let me know if any of this helps.
