The question is can you shoot the difference when using same headstamp brass, same primer, and same charge? Many of us can't actually tell the difference at 100 YDS. I probably could not do it realistically. If you can I applaud you.
1.Same alloy, cast at the same time? Light bullets may have air pockets in the bullet. Or the mold cavities are very different in size? Accuracy would be poor @ 50 yards.
2. If cast at different times, different alloys, accuracy will be poor also. The 164.5 would have more pure lead, softer.
The 160.2 would have more antimony & be harder. Changes pressure when fired.
The as cast diameter must be large enough to size down.. Do measure diameter after sizing. Out of the same die, harder alloys will be larger in diameter. Soft alloys may produce undersize bullets, out of the same die.
I keep my cast in different lots & watch for point of impact changes between lots.
Hard to tell the difference these days, old age catching up to me.
On cast bullets less than 200 grains with 4 grains of variation, highly suspect. I'm casting two to three tenths at 156 grains at a 154 regulated mold. I bet you see a difference at 15 yards but heck do you.