I hunt both, depending on where and when.
When I lived around Flagstaff, we'd hunt deer in the mornings til about 10:00, then switch to out 22 rifles which we had slung on our backs, and shoot squirrels. around 2:00 or 3:00 when the deer started moving again, we'd switch back. We used CB caps in our rifles so as not to spook deer. Good if you can get clean fairly close shots. short hollow points kill better, and still dont kmake as much noise as LR loads, but do make a bit more noise than CB's. We used 22 pistols sometimes for this also. In a 6" barreled K-22 the CB's and short HP's don't make enough noise to need ear plugs.
I also load small game loads for my big rifles. A round ball load or collar button load in a 45-70 kills small game well at close range, but doesnt ruin meat or make much noise if the loads are kept light enough. Same for round balls or 100 gr cast bullet loads in 30-30. I keep velocity at or below 1200 fps. These loads don't make much noise, and shouldnt spook game very far away.
Also load light jacketed loads for 30-06, 338, and will work one up for 375 H&H. I use an easy to identify bullet, like a flat point, so they can't be confused with a full power load and used on a large animal.