It sounds like the file is not a real avi. A lot of movie files you get online will be misnamed ".avi" or ".mpg" even if the file is in fact an asf or ogm or something else.
In MPlayer, drag the movie into the playlist, select it, and click the Info button. That should show you the REAL type of the movie, and then you can change the name as appropriate.
If it's an asf/wmv, you can use Flip4Mac's WMV Player codec to play it within QuickTime. If it's an ogm/ogg or mkv/Matroska, then QuickTime can't handle it, and you'll need to stick with MPlayer or VLC.
In any case, MPlayer and VLC will probably play the file without the need to fuss around with the file name extension. QuickTime is peculiar in its blind pickiness with file types. It's ironic that the only truly Mac-centric player is the only one that exhibits such a classically 'PC' problem!