What was the error message in VLC? And are you using the latest version (0.8.6, released just two days ago)? FYI, I usually use MPEG4 video and MPEG4 audio in an mp4 container, since most of the other formats VLC can export to are not so QuickTime-friendly.
Edit: Also, if you did recently update, you might wanmt to run the "Delete_Preferences.app" that they include with the package. You'd amazed how many problems with VLC can be solved by deleting older preference files.
And after updating WMV Player, did you quit & reload QuickTime Player before trying to open the movie again?