In addition to Dave951's info on Minies, perhaps there is another factor at play here,
in regards to the flyers.
GIVEN -
Every barrel on any type of firearm or firelock when shot, whips in a sine wave manner, vibrating like a sideways 'S'. A
node is where the barrel is at the point(s) of the LEAST movement.
That is when you want your shot to be fired. There may be several nodes identified as you increase your powder charge; where I had one fire lock with 3 nodes, @ 55-grains, 75-grains and 105-grains. To test this, you randomly shoot 3-shot groups at a bullseye for each charge, beginning with your starting load and say working up in 5-grain increments for 6 different charges. Then repeat with another set of 4 to 6 up to what you think may be near your max charge.
For best results,
do not shoot all of 1 charge at once, but switch it up, randomly shoot 1 charge, then another higher, then another less, etc. - only 1 shot @ each - until you have the 3-shots for each charge on the 6 targets. By NOT shooting all of one charge at once, 'statistically' you have normalized the data by eliminating your bias and spreading any human-induced variation across the spread of the total of all of the shots; say like you could begin to shoot poorly towards the end - when tired - as that could adversely influence or bias the last group's results. Statistically this method also significantly increases the confidence you can place in the results.
Those shots with the tightest group denote the range around a node. To fine tune, then experiment with a set of 2-grain increments starting say 4-grains below an identified node. I do similar for my modern 1,000-yard rifle, but this also works for black powder as I've proven it many times, like that above for that 54" barreled musket that had the 3 nodes!
If your powder charge has the barrel MOVING when fired, that can be a source of flyers and depends on other factors such as heat in the barrel. Pay attention to when those flyers occur - are they purely random or only after already shooting a few shots? If that was a longrifle, with a pinned barrel, then I'd advise one to remove the barrel and change the pin holes to be slotted, especially out towards the muzzle where the most expansion from heat will occur. Perhaps the forward barrel bands on that martial arm are too tight? Also check the bedding.
Off-topic - But I think you're running into a situation where you want the heavier charge, and maybe using a solid-base slug so as to not blow out the skirt, but heavier bullets needs a faster twist, and me thinks that perhaps the barrel twist on that arm is the limiting factor. Bullets shot from non-optimal twist barrels are known to be a source of random flyers.
