Hey Pyro! I have that problem with the S&B FMJ and the Fiocchi FMJ. I can't remember if I had that issue with the truncated FMJ from Winchester, but that Winchester doesn't work well in my gun anyway. Back when I got my 10 round mags, the NAA website had a statement about recommending the 10 rounders only for target practice. The website got upgraded since then.
One thing I didn't realize when I got my Guardian was that .32 ACP is semi-rimmed. If you have a tall enough stack of ammo, the rims add up to an angle change of the ammo. In my case, the mag follower will not change angle but the top round in the mag will.
.22LR rimmed ammo can do the same, but their magazines are made to compensate for .22LR by having a highly angled magazine or a curved magazine. Some of the more vertical .22LR magazines have cut outs at the top to allow the rounds to somewhat double-stack on the tail end.
Anyway, it is something to know about the .32 ACP round which can help you understand how it may or may not affect your Guardian.
From reading everyone's posts here, you can see that we have differing results with different ammo. All you can do is test a few types and see what works best in your gun.
I do like my little Guardian. It just requires a little more getting acquainted time than other pistols.