Out of the box, on identical hardware, Server would be slower than Client for client-oriented tasks. I haven't worked with the new version of Server (10.x) much because there IS a client version out, but one of the first things I did when configuring Mac OS X Server 1.x as a client for people was edit /etc/hostconfig. The last line would have "SERVER=-YES-", which slows down the system from a user's perspective, but helps from speed up serving tasks. I edit the line to be "SERVER=-NO-" for systems that would not be used as a "server".
If this is the same for Mac OS X Server 10.1, then it would seem slower than Mac OS X 10.1 even with all the server apps not running. Otherwise they are the same basic operating system, and configured the same would run the same (only one would cost WAY more than the other).
The main reason for setting up the old version as a client was for web developers who where working with Web Objects (and didn't want to work with Windows NT), but you can now run Web Objects developer tools on Mac OS X client so people haven't felt the need to work with Server as anything else other than a server lately.