"Everyone" seems to have the idea, and freely espouse information regarding same, that receivers "don't matter". I think they do.
I think that the lower doesn't matter as much as the upper, from a re-design perspective. They are, after all, and as Derrick Martin characterized them in The Competitive AR 15 first-edition, "cases." The lower receiver doesn't do much work, but it holds some parts that do, and that most importantly is the trigger. Quality always matters.
Whenever I've been asked by manufacturers to pitch in a couple cents on what I'd like to see done I usually include "a better upper" on my list of recommendations. A "better" upper is one that's stronger and stiffer around the threads (longer threaded area and finer threads would be good too), has a larger ejection port with no cover, better height on the rail, no forward assist appliance, and closer tolerance where the barrel extension fits. The upper is a part where materials changes might make it even nicer.
Areas of concern (those that can make some difference) is the parity between upper and lower mounting holes, and then with one another. Folks, the "fit" is in the pins, not the receivers. It's the receivers that perhaps interfere, and no doubt destine this fit, but the gap between upper and lower is not at all the "fit". What matters here is actually that there is a gap. The upper and lower shouldn't bear on one another. That causes binding and binding can indeed hurt accuracy.
Ideally, the receivers will fit together equally if we were to compare "tensions" in the front and rear mounts assembled. I'm not sure if this is measurable but someone with enough creativity and a dial indicator could probably compare movements front and rear when only those pins were installed. There has to be some room for give in the upper and lower joints. An idea that I do know has merit is to line-bore the takedown pin holes. That's a metalworking tactic whereby the parts are set in place, where they should be of course, and then the holes are drilled through both parts at once. That would make the upper and lower "fit" as good as it could be. RND line-bores its receivers, but I don't know of any other doing it.
How does fit matter? Other builders re on record saying things to the effect that "the more they rattle the better they shoot". I think that's a way to say it's not at all critical rather than it is an unyielding truth, and I'll never buy the idea that they shoot better when they are loose enough to rock. I an tell you that unyielding truth regarding upper and lower looseness says that benching a rifle and shooting it from the mat are two distinct things, and they are accordingly more or less influential in what they reveal.
Noticeably loose fit in the receivers which, to me, means they can be twisted out of sync, will mean orientation changes from the shoulder, in the shoulder. Whether this gets transmitted to the barrel is hard to judge, but if the barrel (which is firmly attached to the upper) is that free to shift, then it may also look elsewhere. Pressures the shooter puts against the rifle are as influential on an AR15 as they are on any target rifle. That has to do with the way the whole "pulse" transmits during firing. The proactive means is to put the pressure and position the same on the gun each shot. I can tell you (and boy do I know) that altering pressures and orientations intentionally sho will move a group. Altering them unintentionally is, using reverse physiology, a huge key to escalating one's classification. Anyone who doubts is encouraged to fire three 10-shot runs using one or two notches different in sling tension or using two different buttplate orientations on the shoulder, or, debunking another "myth", more or less pressure on the piston grip or forend. It's the unintentional alteration of pressure that's my big question on receiver fit. These pressure point changes don't affect a rifle being bench fired nearly as much, if any.
Fit will get worse with rifle use before it ever gets better. I think that steps should be taken to preserve fit, whatever quality it might be, and that's primarily in the receiver pins. If there's a booster of any sort bearing upper away from lower then push very firmly down on the upper receiver when replacing the back pin. If the upper has been removed all together from the lower it's probably best to install the back pin first before putting in the front. Keep both pins greased. Heavily.