I made the same choice. My first hunting rifle was an HK91. No possible way was I going to use the skimpy 5.56 as a deer caliber, and most of the professionals and authorities agreed. It was even illegal to do so in Missouri, and only recently changed.
I changed before they did. The .308 is a fine cartridge, in it's place. In battle, that's in a crew served machine gun or precision team doing long range work - not the soldier on patrol. He only needs lethality out to 500m.
Hunting, the cartridge fans tend to ignore documented history. The .30-30 is still the top selling medium game cartridge in America, and has been for over 100 years. That's neither a long range or exceptionally powerful round. In fact, it's a black powder conversion forced to use round nose bullets because the typical gun has a tubular magazine. For all that, Americans have likely taken more game with it than any other. The power and range was obviously acceptable. .308, maybe not so much.
With that in mind, and having carried both a M16 in the field over 22 years in the Reserves, and an HK91 hunting, the extra power does get cumbersome. It gets to be a drag when slowly still hunting for miles over wooded terrain. Those who quickly hike in to a stand and stay in one place for the morning may not notice the weight of the .308 resting in their lap, but carry it in your hands at the ready for 8-9 hours of shooting light, trying to kick out bedded deer hunkered down hiding from dozens of noisy hunters, and it's not the #1 choice.
Switching to a Rem 700 in .30-06 brought on disadvantages with the action I couldn't accept, and the Win 94 couldn't overcome them, either. They were lighter, but obviously less user friendly. Much less the heavier recoil of battle rifle calibers being harder to regain a sight picture, the manual actions slow down getting the trigger finger into action. If I want that, I can take a black powder rifle. It's really about the same.
An AR15 in 6.8SPC doesn't need to shoot thru trees to get a deer, and the rapid second shot targeting will allow another if it does happen. 6.8SPC and intermediate cartridges like it have more than enough power to carry 1000 foot pounds beyond 350 yards, which is the average maximum effective range for hunting OR combat. The BS of TEOTWAWKI aside, a lot of hunters don't use battle bore cartridges. The disadvantages are there, and the extended power isn't often needed. If it's elk, moose, bear, antelope, or you're on a safari in Russia, maybe. Keep in mind, the Great White Hunters of Africa kept smaller guns and used them more often on the light game they shot regularly for meat. Unless they were contracted to eradicate rhino or elephant, they didn't use large guns unless necessary on dangerous game.
The reality is just like your parking lot at work - men tend to buy and use things that express what they want others to think about them, not what they need. All the short magnums and 12 power scopes aren't being used on rams in Kyrgyzstan. If anything, they're all about impressing their buddies.