To some degree, I agree... I've seen people shoot magnum loads at clays, and that's the peak of stupidity. But the recoil from normal, cheap 12ga target shells doesn't bother me nearly enough that I'm going to spend money on a 20 gauge. That said, if I had one, I'm sure I'd use it sometimes.
I have shot a 20 gauge, and I absolutely don't think there's anything "wrong" with that gauge. I just don't think that I would personally benefit from one, at least not to the extent that it's worth spending the money. Reduced recoil loads are fine for new shooters, and you have the option to use extremely powerful shells that simply don't exist in 20 ga (though I can't think of any realistic uses for some of the 12 ga super-magnums).