That's the normal behavior of OS X. As you keep using more computer, it caches more and more pages, and even programs. At some point you start swapping files to and from the hard drive. After a restart, there is nothing in memory, so it can fill up again.