Actually, I figured out how. You should copy the file xinitrc from /etc/X11/xinit to your home directory and rename it .xinitrc
Then edit the file and put a # in front of the line that reads
xterm &
This will prevent X11 from running xterm when it starts up, as the xinitrc script is run every time X11 starts.