This can be done, but it requires a little "experimental" software.
First, get ffmpegX. It's a very powerful video converter.
Then you'll need an add-on to let if read RMVBs. ffmpegX is based partly on mplayer/mencoder, and there is an experimental codec pack available for that at the mplayer web site (scroll down to the "Binary Codec Packages" section and download the OS X PPC one).
After installing that codec pack, ffmpegX should be able to read (and convert) RMVB files. In the Video tab of ffmpegX, just make sure you select a codec that uses mencoder, e.g. "Xvid [.AVI] (mencoder)".
I'm not sure it will work for all RMVB files, but I have successfully converted all the files I've come across since discovering this.