They don't have to prove where you were or were not going in order to arrest you. Maybe they don't bring charges, but I'll bet it isn't terribly difficult for an officer of the law to articular probable cause that you were not abiding by that law.
Heck, it could be as simple as you were pulling out of the parking lot of a restaurant after dinner when they pulled you over.
Heck, it could be as simple as you were pulling out of the parking lot of a restaurant after dinner when they pulled you over.