Living on an isolated farm with a couple of miles of riverfront, I've had my opportunity to confront my share of trespassers. The first thing I'd point out is that if the offense is mere trespassing making any kind of threat with a firearm is going beyond your legal right. Trespassing involves usually a small fine for isolated offenses, although the law in Virginia allows a year in prison and a $2500 fine. I don't know of a single case where this maximum penalty was applied, and if it was I suspect a good deal more was going on than mere trespass unless it was repeat and aggravated offense.
Sometimes a quiet word is all it takes to solve the problem. The person leaves, you go on with your day, and it's over. Sometimes too, you'll run into people who are lost, had vehicle trouble, trouble on the river, and who genuinely need help.
I've learned to be careful about who I approach. We have plenty of people who resent the idea of private property and a quiet question about why the person is on your property may well elicit extreme anger. More than once I've felt myself being assessed as in whether it would be best just to scream obscenities at me, simply ignore me, or attack me. Clearly, if that happens, I have the right to defend myself. Public perception of what then happens may not go in my favor. A large proportion of the public is going to view the incident as "he just wanted to hunt, fish, camp, or whatever he was doing, and you hurt or humiliated him"--skipping the middle part where he attacked me. Again, the very common resentment of private property ownership rears its ugly head.
If I run into somebody vandalizing property or stealing, I can and will confront them, taking whatever precautions the situation allows, but if it isn't a fast-breaking situation, I've long learned to call either a game warden or a deputy sheriff. The same person who shows extreme anger toward me will usually meekly follow an officer's commands, and when they pick up a citation for trespassing and whatever game and fish violations they're committing, word gets around, and it's usually a long time before the problem crops up again.
Most of the time, these people are dumb enough to resent the officer and not me. It doesn't always work. Some years ago, back when my father was alive I saw a gang of people fishing illegally, littering, and creating a general loud nuisance on the upper end of the farm. I called the game warden, and went back to work. He wrote them tickets and afterwards they found my father at the house, and demanded he call the warden and tell him they had permission. When he didn't do what they insisted, the preacher, with his authority with his church, condemned my father to hell. We had a good laugh about him paying for my actions.
The officers I deal with now have learned not to send trespassers back to me, but there's no way to guarantee that won't happen. Often, right after a person is written a ticket, he will hunt me down to offer an apology. I have to admit I'm usually not too nice because I tend to suspect what ends up as an apology and plea to intercede with the officer's decision to write a ticket would have been out and out intimidation had I appeared frail or vulnerable. For this reason, a lot of older landowners don't feel they can dare protect their property.