did something similar yesterday, I had to travel from the great carry state of PA, to the rights respecting Virginia (Dullas Airport to be exact) Since I knew my destination thougth they were above the law, and I had to travel through the gun unfriendly state of MD, I chose to leave the carry piece at home (don't worrry, I wasn't disarmed). I wasn't in the mood to remove my pancake holster so if someone looked at my belt line they would have seen the top of the holster. I wondered if anyone would have said anything, or if anyone noticed.
As others have said, sometimes there are legitimate reasons, some times there are errors. I think it would be an error on th ecaution side to discreatly approach the individual and let them know. If it's legit, no biggie, but if he just came out a men's room, you never know, you may have made a friend/ range partner, and kept a firearm out of the hands of a bad guy.