With modern bullets I think it is essentially "six of one, half dozen of the other".
The Colt has a slightly larger diameter, which should hit game slightly harder, but with modern bullets, I believe the difference is, well, slight.
It is true that the .44 can be loaded to higher pressure, but what of it? With heavy-for-caliber hunting bullets in the 300-330 grain range, both cartridges produce about the same velocity.
Yes, the .44 Magnum guns can be used with the .44 Special. The Colt can be used with Schofields to the same effect. And both cartridges are easily loaded down with Unique or Trail Boss.
I do worry about Ruger-only .45 Colt loads finding their way into my weaker guns. I now use only LBT shapes in my hot loads for easy identification. Problem solved.
I can easily find just about any shape, weight, and construction for bullets and moulds in both diameters. Brass for both is plentiful and about the same price. I find no difference in brass life or ease of reloading. And recoil between the two is identical, at least in my hands.
So I use and enjoy both. I suppose my preference is for the Colt, but for reasons of the heart rather than the head. From a strictly rational point of view, I'm honestly not sure that there are any differences worth worrying about.