I live in an interesting place. The North end of the county, where I live, is mainly agricultural and pretty conservative. A small number of (apparently) genuinely needy people stand on streetcorners with signs, but almost never approach anyone.
But if you drive 20 miles South to the county seat, it is a very liberal college town. The panhandlers are aggressive and will occasionally challenge you if you tell them "no".
I always give them my standard reply: "I gave at the IRS". That usually confuses them long enough for me to walk away, while they stand there with a stupid look on their face.
The way I see it, yeah, I have money in my pocket, but I also have a mortgage of a couple hundred thousand dollars. So the bums actually have more money than I do. All of the money in my pocket is spoken for.
Besides, like I already implied, the government has seen fit to take a significant portion of my money and give it to other people. Until that process slows way down, or at least becomes more equitable, I will not give money to ANY charity.