While I have had no problems, this had been mentioned on the Fink-users mailing list. Someone has posted a solution that might work (not having the problem, I can't see if this works):
XDarwin 1.0a2 wants to create a socket the first time you run it, but this can only be done as root. So try typing
sudo startx -- -quartz
in the Terminal and giving it your password. Hopefully, you can then quit your X-11 session and run it as a normal user forever on then.