If you're hitting 4 of 6 on a B21 silhouette at 100 yards right out of the box, I'd say you're doing pretty good.
Before you change ball size you might want to play with the charge a little. Typically, heavy loads in a CB revolver seldom produce the best results accuracy wise. Although 47gr isn't heavy for a Dragoon historically, it might be more than you need to get the job done.
Don't be afraid to experiment and drop the load to maybe as low as 30gr in 5gr increments. The problem that will crop up doing this, is that the ball will get farther and farther from the forcing cone. To rectify that problem, use some sort of filler to bring the ball back up to where it should be (cornmeal/cream of wheat).
There may also be an added benefit to the filler. I haven't chronographed it yet, but I'm thinking that the filler just might compress around the ball and make a reasonable "gas-check" since it won't all burn before exiting the barrel. Since I haven't tested it yet, I can't say whether it helps or hurts but it's something to think about.
Have fun punching paper at 100 yards. I've done it with a Dragoon, but I'm usually too lazy to walk that far. If you get good at it, you'll impress the snot out of the rifle shooters with your Horse Pistol.