Did you make sure that this was a Mac comaptible AGP card? ATI and other vendors have cards that officially support the Mac.
It sounds like you have a PC version of the 9800 Pro. This means that the firmware was only programmed to be recognized on a PC. If there is no microcode to support the Macintosh hardware, it will fit in there but it won't know what kind of system it's in. The ones that do support the Mac are usually coded to support both the Mac and the PC, hence the higher cost.
Now, you can "flash" the firmware of the PC video card, but it's not supported and in some instances is illegal. If it does work, it doesn't work very well and can cause anomalies in the video of crashes in the system. There have been cases where people have tried to flash a PC video card to support the Mac and because there is not enough space in the firmware to support the Macintosh flash it ends up not working at all.
You're best bet would be to return this card and ask specifically for one that is OFFICIALLY supported on the Mac. Check ATI's website for details on which model. support both platforms.