For the other two parts of ffmpegX, download
the binaries, unzip it, and point ffmpegX to the two files in the resulting folder.
What exactly do you mean by "all hell broke out"? Did it crash, or give an error message, or what?
If I remember correctly, for ffmpegX to read a VIDEO_TS folder, you need to select a video codec based on mencoder, e.g. "Xvid [.AVI] (mencoder)". Oh, and ffmpegX
will not convert commercial DVDs; it will only accept VIDEO_TS folders from your HD, so you might need to first rip it to your HD using a program like Mac The Ripper.
As for Handbrake, it could be that your DVD is just not formatted the way Handbrake expects, and it can't find the main feature. You should be able to get around that by manually selecting the title and chapters in the top-left of the window. (Also, do not select a VIDEO_TS folder from a DVD. Either select the DVD itself, or a VIDEO_TS folder from your HD.)
Just to be sure, when you say "TS file", you do mean a "VIDEO_TS" folder taken from a DVD, right? Because technically, "TS file" is a completely different kind of video format (also known as MPEG-TS, usually with the extension .ts or .mpg).