A major disadvantage with most cap and ball revolvers, of suitable power for deer, is the revolver's sights.
Hunting requires very accurate bullet placement for a quick, humane kill.
The Walker and Dragoons have primitive sights.
Yeah ... I know ... someone will follow me and proclaim that their Walker gets 2" groups at 25 yards.
Fine. Great. But that's at the target range, where you can take your time, your heart isn't pounding, adrenalin is coursing through your veins, etc. Hunting deer from a blind is most conducive to accuracy, but if you're roaming through the woods you also have fatigue to contend with.
The Ruger Old Army is the best cap and ball revolver for deer-sized animals, but like any hunting you should be well-practiced with it before hunting.
Run 100 yards, pick up the revolver, quickly put five rounds in a 4" circle at 25 yards while standing upright and holding the revolver with both hands. Once you can accomplish that, you're ready to overcome Buck Fever, fatigue, exertion and adrenalin common to hunting.
A real sportsman has an obligation to kill his prey quickly and humanely. That's the foremost consideration. If you can't assure that, then don't hunt.