the X windows system usually resides almost completely in /usr/X11R6, so doing a 'sudo rm -Rf /usr/X11R6' should kill most of it. there's probably also stuff in /etc/X11, so also do a 'sudo rm -Rf /etc/X11'
if you used fink, i think the same procedure applies for the most part, except fink prefixes everything with /sw. i'm not sure if this is still the case, as i haven't used fink in a loooong time.