I do tend to agree that wearing a 'gunny' t-shirt can draw unwanted attention when out and about, the bumpersticker equivalents may slightly increase the chance of your car being broken into (by a guy hoping to get a gun) and there is a slight chance that an armed badguy would put you at a igher priority seeing your shirt and thinking you are armed. (What about the chance an armed robber sees your gunny shirt, decides you probably CCW, and goes to rob a different 7-11?) To me the biggest one is looks and hassle from average folk who think differently than me on the issue. I tend to not wear overtly political, religious, or philosophical shirts just for that reason.
I am, however, fond of shirts taht let those 'in the know' recognize your stance, molon labe is a great example of that.
Now, to those who fear gunny Tshirts drawing unwanted attention from armed badguys, what YOU need is the equivalent of those gag shirts with the arrow and "I'm with stupid" text, one that has a big arrow pointing to the left and the words "Hey, I think this guy is packing heat!" That way you throw the badguy off!