Yes it does have to be filetype by filetype, its a contextual menu.
If things are nto openign in the correct program you can change the default for a filetype by doing 'Get Info' on one of those type of files, selecting the program you want them to open in and hitting 'change all'.
If you have files you need to open in multiple different programs, why not drag them to dock icons rather than use open with, maybe that suits you better.
I think the only way to 'clean' the open with menu is to uninstall programs. That said I really don't think most people use it enough to get irritated by this kind of thing, maybe yours is a special case.