"Dual booted with windows" means BootCamp, am I right?
It's most probably a memory problem, if you even can't start or operate TM. Get more RAM! ;)
I haven't seen this behavior, yet, but the way you got rid of it (killing applications with considerable memory demands like PS) lets me think in...