Price would be a concern. If you look at the iPod, it's a great device, but the price does scare some people off.
That said, it would be tough for Apple to corner enough of a market to make it work. If it went upscale, it would have to compete with the low-end spectrum of Macs now.
If it went the cheap route, it would have to compete with everything from basic pdas such as the zire to upscale models such as the Tungsten T and Sony Clies. Plus, if it was cheap, it would be bare-bones and that might soil the image of Macs and potentially scare off buyers in that market.
I doubt we'll see an Apple pda for some time. More likely, it'll be some future intergration of devices such as the iPod and/or pda.
An iPod could be given a touch screen and some stripped-down programs, kind of like an iLife lite. And with better support and more choices in programs and Mac compatibility, existing pdas can work in better concert with Macs, such as the bluetooth function.
While an iPDA or iPhone would be cool, it doesn't seem to make sense, imho, for Apple to re-invent the wheel in an already crowded market. All it needs to do is better bridge the gap between pdas such as Palm and Sony and mobile phones such as Nokia, SE, Samsung, etc.