The bulk of X Windows appears to be kept in /usr/X11R6 and /etc/X11. If you don't have anything in those directories that is necessary for your system to work, uninstallation could be as simple as removing those directories. Someone tell me if I'm forgetting anything, though.