It can get more complicated. However first thing is to make sure all action screws are tight. Grab the front and rear sight and try to twist them. If they move easy, then it is likely your sights are loose and shifting with every shot.
If these are OK you are going to have more fun figuring it out.
First thing I do is check the bedding of the rifle in the stock and the bedding of the barrel in the channel. To check the bedding of the action in the stock I "glassbed" the action". If you want to know how to glassbed, google it and I am certain someone has a page on this. As for bedding of the barrel in the channel. The barrel should lightly touch the stock at 60C on the barrel the length of the final step down section of the barrel. The barrel should not be touching the stock on the sides of the barrel. Only on the bottom. The barrel should have some clearance on the inside of the barrel band. You should try to check the barrel clearances before bedding and align the barrel so you have to relieve the minimum amount of wood.
About 20 years ago, I did all this on a Nazi K98 that the Yugo's rebuilt. It had a beautiful brand new barrel and the bluing was upper 90's. After making sure that the rifle was mechanically tight, and getting the bedding correct, I still had a had an inaccurate K98. Bedding my rifle resulted in having round groups, but they were still large, like 6 or 8 inches at 100 yards.
I was totally baffled. :banghead: But one day, I decided to stick a dial caliper in the muzzle and measure the distance between grooves. The groove distance should have been .323 (please forgive if my memory is faulty)but instead it was .327". I had a brand new Big barrel!!
Since I was shooting standard 8mm ammunition, I was hosed. The only way to tighten the groups was to get a tighter barrel. And I ain't planning on doing that.
I hope your rusted out barrel is not the source of inaccuracy, but your groups should not be as big as they are at 25 yards, even if you were firing a smooth bore musket.
You may be flinching, it never hurts to practice, always wear your ear and eye protection.