Yes, they have much to do with the hands. The short radius makes it more difficult to register, and so control, movement in the hands. The increased muzzle-flip also means that you have greater distance to return to the consistent aiming point, which is more difficult to maintain because it is less easy to observe changes relative to the sights because they are not as observable. The shorter sight radius does make it less easy to shoot, regardless of whether or not it has inherent accuracy. Why is it, then, that the vast majority of people shoot rifles more accurately than pistols (compare, say, a single shot pistol to a single-shot rifle) if this is not the case?
That is not to say they cannot be shot accurately, but it does not mean that a 6 inch Colt and a 2 inch Colt will be equally easy to shoot.
