To a point, but a little isn't going to affect much in the way of accuracy unless you're chasing bughole groups.
I did an experiment once with a .308 bolt rifle by loading 10 rounds normally and another 10 in which I checked concentricity and carefully straightened them. The rifle was a tack-driver, so I knew that it would be a good test platform.
At 100 yards, the difference in the 10-shot groups was just under a quarter-inch. An additional .125 inch dispersion from POA in any given direction isn't going to make any practical difference on a deer clear out to the limits of the rifle, and to especially my limit as a field shot.