Started hunting in the Georga woods more than 40 years ago with a 30-06 bolt rifle. In that time 99% of the hunters I meet have been using bolt rifles in 30-06 or similar chamberings with low powered scopes on them. In 40+ years I could count the number of lever actions actually used by hunters on 1 hand and have fingers left over. The other hand would account for about the same number of various Remington semi autos, every one in 30-06.
In my opinion, it would be the scope that would limit you in the woods;
The only problem with scopes is using too much scope. Anything 4X or greater is a handicap. But a low powered scope in the 1X-3X range is faster on target, helps you see your target far, far better than irons in dark woods, and are more dependable. I've literally seen more iron sights fail than quality scopes.
The 180 gr bullets are much heavier than needed, but if they shoot accurately in your gun there is no real reason to change.
A scoped flat shooting rifle is far better for thick woods hunting than the "traditional" guns. There is no such thing as shooting through brush. ANY bullet that strikes even a tiny branch or leaf is going to be defelcted. The secret is to shoot through tiny openings in brush. A scope helps you see those openings in poor light.
Most people don't realize how helpful a flat shooting rifle is even at close ranges. My bolt rifles are all zeroed at 100 yards. From 50 yards out to almost 150 yards the bullet will never be more than 1/2" above, or below my aiming point. This helps tremendously in getting bullets through tiny openings even at 50 yards. A lever action 30-30 will at times be 2-3" above or below the aiming point at the same ranges. More than enough to hit branches in between you and the target at quite close ranges.
Rifles with arched trajectories are much easier to use at long ranges out in the open. If you know the range you can always aim high and hit the target. Even if you are talking about aiming high in feet, not inches. Try that in woods and your bullet is going to be deflected.