Apple usually ties their hardware to the operating system that shipped with the system... I would think trying to install 10.3 would be problematic at best.
For one thing, you would need a disk that has 10.3.7 on it (at least) because that was the first version of Mac OS X that shipped on the Minis. And a Panther CD set before that may not recognize the hardware.
You could give it a try, but I wouldn't bank on it working if you are trying to install off installation disks before 10.3.7... and considering that Apple doesn't rewrite the generic installation disks all that often, odds are that the boxed version of Mac OS X v10.3 isn't going to recognize a Mini.
And if you haven't guessed... Apple will not support any operating system below the version that shipped with your system. If you run into trouble doing this, Apple isn't going to help you.