I used to be much more secretive than I currently am. Some friends, family, and co-workers know that I have a carry permit, and even to those who are anti-CCW, it doesn't seem to matter. Granted, nobody knows when, where, or what I'm carrying at any given time. Moreover, I just don't seem like someone who might be carrying anything--my clothes, grooming, and attitude show only an upwardly-mobile twentysomething.
Ultimately, the element of surprise is still intact--the random bad guys most likely to assault you will have no idea what's on your waistline.