I believe the underlying mechanics only separate into the following categories...
Physical damage (type)
Magic damage (type)
Spell damage (source)
Basic attack damage (source)
"Physical Spells," as you have stated, do not exist. Also, RNG for spell damage is highly unrewarding to the user.
I would not buy this item on anyone, period. I consider flat ArPen to be a waste if you're making a strong investment into AD scaling damage abilities, there are easier and faster ways for me to access cooldown reduction in large amounts, the hard AD benefit is low, and there's no way I'm going to try and roll dice for extra damage.
That's ignoring the fact that, with the current mechanics, you cannot separate physical spell damage from magic spell damage. It's not like most AP casters would normally spend that kind of money on such an item, but a well fed Veigar would pick it up for the spell crit and CDR.
On anyone in the actual category to benefit from this item as you intended, the suggested spell crit is not OP. In fact, it's pretty weak.