I found the problem! Its a little file created by Stuffit in the system folder. Boot into OS 9 and use a tool such as resedit to unhide a file named " Aladdin Transaction Info ". Once you have done this, go into your system folder and throw that stupid file in the trash and empty it. Now you can go back into OS X and run X Optimize or 'update_prebinding'. If all went well, you will not get any more error messages and the optimization will not loop for hours and hours. If it still does not work, follow these instructions from febo:
Hi all,
I just found a solution to the 'out of memory' problem when using update_prebinding.
I used the command 'fs_usage' to monitor the disk activity of update_prebinding.
It turned out that there was a file with rather exotic characters in it's filename in my OS 9 Systemfolder. When update_prebinding visited that folder it reread it's contents again and again... until it ran out of memory. I deleted that file, reran update_prebinding and it worked just fine
So here is what I did:
1. open two Terminal Windows
2. become root in both windows using 'su'
3. run 'update_prebinding -verbose -root /' in one window
4. run 'ps aux | grep update_prebinding' in the other
5. look at the line that does _not_ have the word 'grep' in it; note the number in the second column. This is update_prebinding's process ID. Lets assume it's 208.
6. run 'fs_usage -w 208' where 208 is of course the process ID you found in step 5.
7. wait until the point you remember update_prebinding stopped to make any progress.
8. Watch the output of fs_usage closely. When it starts to show only blanks where previously filename fragments where shown it's time to...
9. press ctrl-c in both the update_prebinding and the fs_usage window
10. scroll up in the fs_usage window until you see the last file or folder name in the middle column (it may be only a fragment, as fs_usage only displays the last 28 characters)
11. Go to that folder and examine it closely. The file that I had problems with didn't show up in the Finder but in the Terminal using 'ls -la'. Except for some questionmarks in the filename it also contained the String 'Aladdin'....
12. Now comes the tricky part: get rid of the file. I managed to remove it by using a visual ftp client, logging in on my own machine and hitting delete. I guess there are better ways - but I couldn't explain to Terminal (or 'tcsh' to be precise) what file I wanted to delete. Be careful: the file might be vital to your system. So you might just want to rename it instead. Or not even touch it at all. But this is for you to decide. Don't blame me!
13. run 'update_prebinding -verbose -root /' again.
Good Luck!