Even if you aren't using an application, if it was a window on the screen or in the dock, it's going to get redrawn. Whenever the screen gets updated the OS is going to ask the app to redraw it's window. Therefore it will hang around in memory, or get swapped back in from disk.
OS X's...