Yea, you have to have X11 installed on your Mac. You said installing another X11 on your laptop, what do you mean? You already have it installed and don't want to install it again, or you don't want to have it installed on the server and the laptop?
Anyways, I can't really find a reason NOT to...