Yes, they do try to hurt him. 1st they forcibly stop him, then a guy, directly to the riders left, repeatedly tries to punch him in the head.
I'm guessing the rider started repeatedly yelling "I'll shoot you *@!#!..."
So they sorta move off worried he might have a gun. Then they see he really does have a gun, and take off runnin'