Wow. I have a whopping 10 non-permanent items on my desktop, and even that kind of bugs me.
If you can work like that, kudos to you. If it's something that bothers you, but not quiiite enough to actually clean it up, perhaps some AppleScript folder actions to automatically sort files for you as they're placed on the desktop would make your life easier. I use a folder action to automatically sort my web downloads folder. It keeps me sane.