I'm with the other couple of posters that suggested this guy made the same mistake you did.
He showed up, got out to check the hours like you, and saw you walking away eyeing him.
He felt embarrassed under your close watch, and instead of cracking a verbal joke, tried to play it off with an exaggerated grin.
No harm, no foul.
And, no, it wasn't me.