when you say "first groups" what are you talking about? you say "after 5 or 6 shots" it tightens up...
What I think you mean is your first group (singular). And it's not at all uncommon for it to have some spread. In fact, it would be very unusual not to. The reason is that the first shot from a clean, cold bore is usually different than all the others.
This theory is simple enough to test, and it's easy to separate the 'clean' variable from the 'cold' variable.
Next time you go to the range, put a few fouling shots through it. Then shoot something else for a half hour or so. Now, shoot a string of 5 shots, but wait at least 3-4 minutes between each shot.
Next, try another five shot string, this time cleaning the bore between each shot.
depending on what kind of gun it is and what you do with it, it can be important to know what the first shot is going to do.