I've been pulling my hair out with this same problem.
I seemed to have solved it though. After searching a few boards for answers, here's what I've found:
1. Normal Apache Allow directives don't work. Due to the port swap REMOTE_ADDR always has the ip address of the server instead of the ip...
