The lower does matter less, as mentioned above, but do make sure you get one from a reputable vendor/manufacturer. I've seen some lowers that were just trash. Pin holes out of spec and/or in the wrong places, nasty machine work, horrible finishes and I even ran across and outfit once that was selling receivers made of 6061.
Now the upper, opinions are going to vary. Personally, based on my recent addiction to precision ARs, I'd have to say that the upper does matter. Getting one that is on the tighter side of spec helps with accuracy by keeping things more consistent.
If Service Rifle is what you want to do, maybe your should give Compass Lake a call. Friends into that sort of thing speak highly of them.
If by "building" you mean that you are going to build one from scratch, I'd go with something like a Daniel Defense upper receiver, a White Oak (if you're on a budget) or Krieger (if you're not on a budget) barrel, BCM bolt group and a Geissele Hi Speed trigger.