Getting into a physical altercation while carrying a weapon is just stupid. You are using deadly force either way. You don't know what the guy you are fighting is capable of. There's many cases of normal sized men being completely disabled by one punch.
Grappling with someone when you have a firearm on you is a great way to give a gun to the other guy.
Walk away. If you are prevented from escaping the situation, you're being threatened with eminent and grave bodily harm. If you are prevented from escaping the situation, you're justified in using force. Force is force.
I'm not saying you shouldn't be in shape and able to fight in a conflict. Part of an armed conflict may involve HTH skills. This, however, should be something you try to avoid.
Carrying a weapon means you have to make decisions wisely. If you can't just walk away from someone taunting you because of your ego, you lack the maturity to responsibly carry a firearm.