It all depends on where you live. I've lived in suburbia(polos, khakis), at the beach(flip-flops, shorts, hiwaiian shirts), in farm country (overalls) and in podunk(camo boots, boonie hats/camo ball caps, camo pants). It's all a matter of where you live, not if you're a gun-nut or not. I'm currently a river/swamp guide and wear every variation of BDU pants due to mud/dirt, pockets for garbage, and quick-dry, and a boonie hat for sun-burnt ears and ticks. I go to Wal-Mart however I want and still fit in here, but wouldn't dress like that in any of the other places I've lived. You just gotta assimilate in your community and wear practical clothing for whatever you do.
When I visit the other areas, I dress accordingly.
I've only lived in the Carolinas, but always dress to not stick out and have the closet to prove it. It's just "locational camo" to keep a low profile and draw no attention as being an outsider.
Armed, legal, and invisible are good things, wether wearing a hawiian shirt or realtree.