There would be no performance loss to OSX if you've got thousands of fonts.
There will be a loss of performance for those apps which let you see all the fonts before you apply them. Which is where your performance loss comes in.
So, Yes, you will be able to have thousands of fonts, but I recommend removing those fonts that you wont ever use.
Hard Choice, but the delay would be there for any platform.