A tool that might be useful in finding at least malware or adware is FSE:
http://www.scsc-online.com/FSE.html
WARNING: READ THE PRODUCT PROFILE FIRST - don't run out like a dingbat and just buy it if you don't know what it does, and read the "experience level" required to use it! If you don't understand Unix processes, filenames and paths, permissions, etc you won't have a clue what this thing is doing.
FSE is a file system events monitor. We use it during the installation of packages to track what's being installed on a system. I'm sure this has been mentioned in this thread, but malware/adware typically hides itself behind fake and often official looking names, like plist files, libraries, etc. etc.
There's a fairly well known (but somewhat questionable) product on the market that offers a "demo" version of what it does. When the demo period expires it demands credit card info. If you go ahead and click on the option to buy it and then opt out, the thing more or less creates a primary window demanding payment and won't let you access anything else on your system. Most people end up hitting and holding the power button.
We tested this as follows:
1. Monitor the installation w/FSE and let it record all file activity.
2. Set the computer to a manual date in the future so the expiration date of the demo with the adware/malware is over.
3. Reboot the system.
4. Start FSE in full logging mode to record everything.
5. Launch the demo program with the adware/malware.
6. Duplicate what a user is doing to create the problem.
7. When the system locks, push the power button and hold it to shut the system down.
8. Reboot.
9. Look at the FSE log file.
Typically the log file will show that when the "adware/malware" phase of the program kicks in, it starts doing things to its official looking binaries. They're typically one of the following:
1. An actual binary
2. A compressed application that it uncompresses and installs unknown to you
3. Launches the binary that locks up the system.
From this information, you'll be able to track down the offending binaries.
Some observations and warnings about using FSE are as follows:
1. This is NOT a tool for amateurs. If you don't understand Unix command line file paths, processes, commands, etc. you'll be sorry.
2. It's a direct interface to the kernel and it WILL bog your system down, probably to about 50% of its speed.
3. Avoid using it during Spotlight indexing. The FSE display is just an indicator. Spotlight can create files that are 10's of MB in a few seconds.
4. Don't leave it running for a prolonged time. Your log files will be gigabytes (that's not a joke).
FSE is log file oriented. You will also need to be able to read the log files it generates and understand them.
We find the product useful, but you've been warned!