In OS X, memory management is all automatic. If a program needs more memory, it gets it (up to a limit of 4GB, anyway). If there's not enough RAM available, virtual memory is used.
There are a few applications, like Photoshop, that allow you to set a maximum amount of memory they'll use in their preferences. But I don't think this is really useful; it's more a throwback to the earlier days of OS 9, when this was necessary.