Everything matters when it comes to reloading, but how much depends on how much effort you feel is necessary to suit your reloads. I sort by headstamp (just outta habit, really doesn't make much difference) and in 30 years I've never measured a flash hole. I did "deburr" some in my .308 brass but I was just experimenting with the process. I haven't cleaned a primer pocket for any handgun brass in 29 1/2 years, and can't remember if I cleaned any pockets in my Garand or 308 brass or not, it just makes no noticeable difference in my reloads. But, if you wanna "resize" or ream flash holes it's really OK. If it doesn't suit you reloading/shooting needs then don't, it's OK too...
BTW; I haven't found any too small to deprime flash holes, yet...