Hi vzenmn
You did not mention what sighting system you use?
My results were with a scope and a brand new #4 barrel with the chamber cut back a little to get the extractor cut-out to line up. This resulted in a tight chamber and of course, reduced bullet jump.
Have you checked your muzzle crown?
Even a new barrel can have a lousy crown (some well meaning folk run a countersink in their muzzles). Also, a used barrel may well have muzzle tip wear from a pull through or cleaning rod or even light corrosion and or dust, which is basically invisible to the naked eye. If the barrel doesn't 'look' brand new, then chopping off 10 to 15 mm and crowning the muzzle properly can make a huge difference - like 3-inch groups down to 1 inch. Flyers are also an indication of muzzle wear.
Have you examined the appearance of the bullet hole on the target?
This hole should be perfectly symmetrical. Just the slightest eccentricity (and I refer to the 'petals' formed in the centre of the 'ring') will indicate bullet yaw, which means muzzle wear.
Try using round nose bullets of the largest diameter you can get. This gives minimum bullet jump as well as longest bullet bearing surface, all of which helps with accuracy.
Are you testing with starting loads or max loads?
Too light a load may not give enough velocity-spin to properly stabilize the bullet. (I never found any difference in group size due to load).
It is also important to ensure that there is no case deformation from resizing or seating the bullet - the neck can easily get pushed out of line - one of the hazards of neck sizing only.
I have re-barrelled three Lee Enfields - two Mk1's with #4 barrels and one Mk3 with an Mk3 barrel. The Mk3 shot well with any cartridge or loading, to the same point of aim. The #4 barrels shot better.
My load data;
CCI primers Powder - MR200 Lot 180 from Musgrave
150gr Hornady SPSP 40.9gr 2690 fps
180gr Speer RNSP 38.6gr 2395 fps
I used the same powder in my 223 Mini-14, also with good results.