ffmpegX (and similar programs like iSquint) can convert some older .rm files, but not all of them, and not .rmvb files.
If ffmpegX doesn't work, I think your only chance is to use a screen capture like Snapz Pro X to record the movie onscreen as it plays in RealPlayer. It's messy and a hassle, but it can work.