POI, not so much. But group size, yes, it can matter, but the 0.003" to 0.005" variance you are seeing is probably the variance in the bullet tip, and it's not what matters.
Bullet tips are not very consistent, which is why Walkalong and krochus say not to worry about it, and you will only frustrate yourself. If you vary only 0.003" to 0.005" you are doing great. If you really want to know, as krochus said you need a comparator, which uses the ogive to measure from instead of the tip, which is way more consistent.
When loading rifle rounds especially near max follow the load data for COL (aka COAL) for the bullet you are using as a baseline and never go shorter than that. Now don't drive yourself crazy, measure a handful of bullets and pick one in the middle to use for your "dummy" round, go slowly and get it seated to the listed COL. Then without touching your seater, load the rest of your bullets.
Now if you make any adjustments to your die for a different bullet or whatever, you can always get back to "zero" by placing your "dummy" round in your press, backing out the bullet seater, raising the round up into the die, and then screwing in the seater until it touches the bullet.
What is more important to group size, as ranger355v alluded to is seating depth. The best seating depth for your load/bullet/rifle combination will be unique to your gun. There are a few ways to go about this, but instead of typing, just google it, and pick the method you want to use.
However before I start varying my seating depth, I start by working up a load with the bullet 0.015" to 0.020" off the lands, but never shorter than the listed COL. Once I find the powder charge that gives the best accuracy, then I start adjusting the seating depth to see if I can improve my group size.
If you can't get 0.015" to 0.020" off the lands with the chosen bullet without going under listed COL, then you have to reduce the load and work up cautiously.
I say 0.015" to 0.020" because the variance is going to be less than that and you won't get into trouble by jamming the bullet into the lands, creating an overpressure situation.
Your most accurate load might be less that 0.015" off the lands, but that's not where you want to start. JMO.