Sometimes, but not always. If I'm going to hunt short range, brushy country I usually carry my 1894 30-30 and my SAA 357 or 38-40. If hunting open spaces, I leave the handgun at home. Always carry my Ruger Single Six or Colt 1911-22 when I go fishing, but that's for the snakes or occasional varmint I might see along the way. Same with woods wandering; I've come to prefer carrying a lightweight, small caliber handgun when out looking for mushrooms or scouting new hunting spots in the spring vs. the short carbine or shotgun I used to drag along. Leaves both hands free to do whatever's needed, from carrying a fishing rod to swatting 'skeeters.
Last fall, I carried only my 38-40 SAA while hunting with my little brother. It was the first time I'd ever gone deer hunting without a rifle, but I fully intended to not fire a shot unless absolutely necessary so he could fill his tag and we were successful. I will say, a 7.5" SAA isn't really woods friendly but I didn't feel under-gunned at all.
Mac