Well - all emails are also in the mail server, all chat logs are in the chat providers' servers and so on. So it's not just the caches in your ~/Library... To keep a bit more control of those, you'd have to set up your own mail server and so on.
A few things make always sense (even if you don't go with open firmware passwords or other features): lock your screen when you are not on the computer. Whether it's for a two minute break, or sleeping overnight, others shouldn't have access to the account with your data.
(Even if I trust the others home to have access on my computers, I still lock the screen to control the access. Even if it's mostly a "cats' loc"... so even today when I logged in to my server I could see at least one of my cats had walked on the keyboard trying to log in. I definitely don't like the idea of the kitties setting up any odd services or doing any weird stuff on my server...)
With Filevault, Mikuro nailed most the good points for using it.
But if you use it, you'll still need some backup plans. You'll need regular backups of your system and home folder. You'll need way more empty space in your system any given time you want to disable Filevault (at least as much empty space as your home folder takes space). And the plans for the catastrophes in case something goes terribly wrong with the FV sparseimage and it gets corrupted. (command line acrobatics and mounting it from another user can do it... and if unprotected access for other users to log in to your computer, that could be one weak point in the chain of protection. Don't give access to others, lock down all users, ideally with login asking for username and password etc)