You are lubing them, right? I always lube the inside and outside of the necks, and it's always been as smooth as silk. What kind of lube are you using?
The first two considerations that come to mind for me is, do you have the die adjusted correctly, and are the necks properly lubed. I would personally look closer at my die adjustments before thinking the expander needs polishing or a possible defect die. I've had exactly zero contact with defect dies, and I have never run across a rough expander, and some of my dies are 40+ years old, some are very new. I'm constantly hearing, oh polish your expander, or your expander is too big take some emery to it. I just don't think that quality control can possibly be that bad, especially with dies, or I would think I would have seen some evidence of such.
GS