If you are referring to lead bullets (you didn't say) it could very well have to do with bullet hardness. A harder cast bullet requires more preasure to properly obturate. I had exactly the same issue with a .44 Mag and purchased cast bullets. If I cast softer bullets (less antimony and tin), then I could get the good accuracy with light loads, otherwise, with the harder "store bought" stuff, I had to stay up pretty high on the power scale. The bullets sold as "Coyboy" are softer and usually obturate at the lower preasures. I got the softer bullets by adding pure lead to wheel weights, thus lowering the ratio of antimony, which of course is the primary hardening agent. But, remember, if you get some softer bullets, be forewarned that if you load them hot, you'll likely get a lot more leading. So, try some softer bullets for the lighter loads, but use hard bullets for hotter stuff. Could well address your issue.