I've shot the Ruger SRH .454, Ruger SRH .480 and the Smith .500.
The .480 was unpleasant and the .454 wasn't any better, but not worse. Both had a lot of muzzle flip to me. That said, I hate the SRH grips. So, with good rubber Hogue grips (I had them on my .41 Redhawk), I'd think it would feel much better.
My favorite was the .500. Great grips, more of a push back than snap, fairly comfortable to shoot. Much better than the .454 or .480.
If I could only pick one, I'd try to find a deal on the .454 and try good grips. It seems more versitle with light cowboy loads all the way up to heavy bear loads. There's also a way to shoot .45ACP in it with moon clips, although I hear accuracy is compromised. If it doesn't work out, you could resell it and probably not loose a lot. I see some used .454's around here for decent deals, but like one store told me: 'pretty much the .44 and .357 are standard, so we can't give good trades on anything but those.
The .500 at this point is expensive and so are the cartridges. Even to reload for the .500 it'd get expensive.
The .480 seems like a waste of time. You can only shoot .480, the components are limitied and the factory ammo is still pretty expensive. So if you are big on the .480, I'd check into a Freedom Arms .475 and you can shoot the .475 AND .480.
Hope that gives you something to go on.