I've had this exact problem for a very long time and for some reason the problem solved itself when I swithched my logon account to a domain wide admin account (win2k AD).
Have you started the mac agent in debug mode?
Maybe you'll see something in the logs.
I also remember seeing an article at symantec about BE 10d using the NDMP port (10000) and one should make sure that particular port isn't used by another application. This can be done by looking for the NDMP entry in the services file.
Hope you'll find some luck!