You also need to have enough empty space on your hard drive. Mac OS X - without Photoshop - likes to have 15 % or more of the hard drive size free at all times. So a 100 GB HD = 15+ GB free for maintenance.
If you run huge files in Photoshop or video editing programs etc, those will create huge cache files. If you have that 100 GB HD and it has that 15 GB free, if you have big photos open PS could suck 10 GB for the caches, leaving very little space for the other things the system needs, causing slowness.
What is your hard drive size, and how much of that is empty? If you need more space, upgrade the hard drive or get an external HD for files you don't need access to all the time.