Depends on the law in your jurisdiction, the attitude of the police and the attitude of your local prosecutor.
I can easily see someone being stopped by the police and questioned/hassled because they are wearing an empty holster. The obvious question is, "Where is the gun?" I can see that fact holding up as "reasonable suspicion" to justify a simple "stop and question" with the accompying "Terry frisk." Hopefully, it would end there.
As far as someone actually being arrested and charges being filed, my layman's opinion is that it would be unlikely, absent any specific law against wearing a holster. If there's no gun, what would they charge you with? "Brandishing?" Brandishing what? I supposed you could imagine someone being charged with "Disturbing the peace" or some such, but I doubt it would stick past the initial arrest.
Ultimately there is no real answer until the situation comes up.