The Mozilla project is actually in charge of the Java Plug-in development. If you are having problems using it, please submit it as a bug on Mozilla.org. They would highly appreciate it.
Apple released their Java framework API relatively late to Mozilla, and that is partially the reason for the somewhat belated support for Java in Mozilla. At present, I think only Mozilla and IE supports Java (in terms of browsers), hopefully this would change soon. IE's Java support has always been somewhat lacking - their version of Java is not Java2 compliant.
I find it somewhat amusing that the Java applet that is giving you the most problem is actually on MSN.com.
Instead of using the Java version of Yahoo! IM, you might want to checkout their OS X IM app, or 3rd part IM apps like Fire.