I served in the US Army from 6 Dec 74 till 1 Nov 03. I haven't put on my uniform since I retired in November of 2003. I have worn BDU pants on the range and on some work assignments. (I'm currently a police officer).
Army regulation 670-1 covers wear of the uniform. Civilians can legally wear BDUs or other uniforms as long as they remove any distinctive items. Distinctive items include buttons on the dress uniform, the US ARMY patch over the left pocket, shoulder sleeve insignia (unit patches), distinctive unit insignia (unit crests), rank insignia, decorations, skill badges and headgear.
Now I doubt you'll ever get a US Attorney to prosecute anyone for this, unless they are pulling a scam and defrauding someone or wearing the MOH, but it is a violation of federal law.
I see no problem with someone wearing a t-shirt or sweatshirt with a unit insignia or motto on it, heck, my wife and kids have a pretty large collection of those.
I do have a problem with people (military or civilian) wearing enough parts of the uniform that they look like they are trying to impersonate soldiers. I have been know to approach those people and ask them for their military ID. If they produce one, I contact their 1SG, if they can't I remind them of the federal law about unauthorized wear of the uniform. There is just something wrong about someone with hair halfway to their shoulders, a three day growth of beard, and their hands in their pockets, walking around in full BDUs with patches on them and muddy boots and an open jacket.
I doesn't take any time at all to remove the patches from a BDU jacket if you want to wear one.
Jeff