I get the gist of your question, but maybe not.
An "FTP Server" is a binary program such as ftpd or proftpd that runs on some server platform such as Unix, Windows NT, or Mac OS X. Likewise these platforms can all run mail servers, database servers, X-Windows servers, you name it. So the minimum anyone needs to provide file transfer to the whole universe is some space on a server that supports FTP. Most ISPs are able to provide this kind of service for uber-cheap.
Okay, so you probably knew that...
Now then, when you describe your notion of a chat suite composed of a client application and a server application I think immediately of programs like AOL instant messenger, ICQ, and Yahoo Messenger. Each of these programs connects to a central server in order to log you in and find out who else is connected. Users that know avout each other (i.e., "Buddies") are then able to connect directly to one another (peer-to-peer) and chat, transfer files, etc. Thus these chat applications act as both client and server once they've established that peer-to-peer connection.
The central login servers for each of these chat clients are most likely running on some flavor of Unix. Under Linux / Unix there are many ways to create a login interface such as those these chat clients use. For example, one could create a script in PHP that receives an HTTP request and checks the request against an SQL database to confirm a user's login and password. The PHP script would then return the equivalent of "yes" or "no" to your client's request to log in. (All of this should of course take place across a secure connection such as that provided by the popular Secure Socket Layer.)
So, if you decide to write a server-side application of your own you'd need to write it in a language supported by the server on which it would run (or so they tell me). For the most portability you could write it in perl, PHP, or as a shell script that would run under sh or bash. You would need some kind of accessible storage to keep track of users. MySQL is a fine solution under perl or PHP, whereas a shell script would make use of certain command-line utilities that allow you to create databases in the native filesystem.
It's pretty straightforward to install MySQL under Mac OS X, and I believe that the Mac OS X build of PHP includes MySQL support Right Out of the Box. Setting up support for perl is also pretty straightforward once you know all the ins and outs. None of this is trivial by any means. I've been a web developer for three years now and have only been doing this kind of administration stuff since Mac OS X was released. Fortunately there are lots of great articles about the magic combination of Mac OS X, Apache, PHP, and MySQL. A search on the latter three terms here in these very forums should give you lots of places to start. The articles on StepWise, O'Reilly, and Apple are particularly good.
Now put away your web browser and hit the books!
