Basically, rmdir only removes an empty directory; this means you have two choices, either remove everything in it, then rmdir it, or use the recursive flag of rm,
sudo rm -r non-empty-directory
Have a look at AdmiralAK's Unix Basics thread for a good intro.