First, jmr40's post was full of BS. A 9mm feels much milder than a 357 Magnum because it is much milder.
Second, when you compare two real cartridges like the 9X23 Winchester and 357 Magnum that push the same bullet to the same speed, you have to look at how they do that. One of the more common ways the 357 Magnum pushes 125 grain bullets to 1450 fps is to use a large amount of slow powder, like Winchester 296. Charge weights can run up to 21-22 grains. This powder also produces wonderful fireballs that add to the noise and spectacle of the muzzle blast. But the cases are pretty full at this weight. That much powder takes up a lot of space.
The 9X23 Winchester's smaller case can't use 296 very well, so it has to use somewhat faster powders that are more efficient, but at the same time produces higher pressure than the 357 Magnum does. Still, factory 9X23 ammo uses about 10 grains of powder, based on the ones I've pulled and measured. The 9X23 has plenty of recoil in it's own right, because there is no free lunch. But the lesser powder charge, and that fact that it is in a semi-auto pistol, makes it feel like less than the 357 magnum in a revolver.
The gunpowder, and how much gunpowder is required to get the same velocity, matters. If you compare gunpowders that use different amounts to push the same bullet to the same speed, the one that requires more weight produces more recoil. That's because the weight of the gunpowder is part of the ejecta that comes out of the barrel, and that increases recoil force. This is well known, and formulas that calculate recoil require the weight of the gunpowder.
https://en.wikipedia.org/wiki/Recoil#Including_the_ejected_gas
http://kwk.us/recoil.html
I hope this helps you to understand some of the differences between cartridges and recoil.
And if you want to get an even better understanding of how important powder selection can be in high performance, check out the link below. It is possible, when selecting the right powder, components, and overall length, for a 38 Super to match the ballistics of the 9X23 Winchester and 357 Magnum with a 124/5 grain bullet, and still stay within the 38 Super's normal pressure limits.
http://www.shootingtimes.com/editorial/super-powders-for-the-38-super/99160
For another explanation of how gunpowder contributes to recoil, see this article:
http://www.shootingtimes.com/editorial/gunpowder-contribution-to-recoil/328788