I do not see how leaving the "TV-out" setting on would harm any headphones -- the video pickup area on the headphone plug is different from the audio pickup area, and if headphones are used that do not have a video pickup area, the video signal will simply be ignored and will not cause damage to the headphones. Apple knowledge base doesn't have any information stating that you should not leave TV-out on when not using the iPod video for video playback on a TV.
I do not have a video iPod to test this with, though, so I could be completely wrong -- but it seems silly that leaving the TV-Out setting on would damage headphones. It certainly doesn't happen on the iPod photo models for video-out.