I have had the same problem, only in my case its when I ssh into the OS X box. Basically it works fine, but if I leave my desk for a few minutes, when I come back ssh on my linux box is unresponsive and requires killing.
It happens with both openssh and ssh.com on linux, so I assume its a...