Well, over time and multiple loadings, it can be a problem in that it may influence case life, since you're in essence removing a thousandth of an inch of brass or so each loading if you don't correct the issue. I wouldn't be too worried about it at all though. Chances are that your cases will die off from other natural causes before the shaving ever becomes a concern. It wouldn't hurt to trim them to a uniform length at some point though, if you plan to get a lot of firings out of them.
Remember, at least you said they are shaving consistently. In the end, I think the most important thing if you're loading for precision shooting is that all the cases in your batch are consistent.
I have 200 or so pieces of .45 Colt brass that I use exclusively for my cowboy loads. These pieces of brass are some of the most diverse, random pieces you can imagine. They are a mix of maybe 4 or 5 different lots accumilated over time, as well as the picked up brass that belonged to other shooters that accidentally made it into my pile. I have probably as much as .005 to .006" variation between all of them, which means it's virtually impossible to get a consistent seating depth of crimp on them -- unless I actually took the time to sort them and trim them. I basically just set my seater die to a healthy medium and have at it. Some of them seat high with a light crimp; some seat too low with a heavy crimp; some seat perfect.
But I don't care. They are for pipsqueek cowboy loads, where the hardest shot is maybe an 8 inch diameter steel plate at 30 yards max from a rifle? I'm not too concerned as long as they safely chamber and go boom every time. "Close counts in horseshoes, hand grenades, and cowboy handloading."
I have other lots of good brass set aside for my "accurizing" endeavors. All just depends on your application.
Have fun shooting em. =-]