Tiger should not slowdown your computer. How much RAM and free harddrive space do you have? As for downgrading back to Panther, you have a problem because this is the OS that shipped with your computer. It is a problem if you have only a System Restore disc anyway. If you have a bootable MacOS X 10.3 installation CD, then you can do an archive and reinstall. Just slip the CD into your computer, reboot while holding down the [Apple]-[C] key. Select Archive & Install. If you have a System Restore CD, then you will probably no be able to use a retail MacOS X 10.3 CD. This is because you cannot use a version of the OS that is older than the one that shipped with your Mac. The System Restore CD can only return your computer's software to its as shipped state.
My suggestion to you is to look into actions that are less disruptive than downgrading your OS. Run fsck -fy in single-user mode. Download, install, and run OnyX. You don't burn your house down because a faucet is dripping.