When you decap, there is a small amount of debris left in the primer pocket from the burned Lead Styphnate (priming compound). If you look at it under a microscope, you'll see it's crystilline in nature. This is probably what's getting on your priming stem.
I clean all my primer pockets, which some people do, and some don't. It's up to you how you do it, since it's your shop.
Hope this helps.
Fred