It really depends on each individual barrel.
Of course the patch thickness matters too, but some barrels just won't like loading with .495 balls. They do usually load noticiably tighter even if using a patch thickness that is .005 smaller than when using .490 balls. That's simply because there's not as much patch material to compress to fill in the rifling grooves.
It boils down to a compromise between the accuracy desired and the amount of loading difficulty that you want to encounter with your ramrod.
For the typical .50 caliber bore, start with a .490 ball and a lubed .015 patch. Then based on the performance and loading difficulty, you can decide whether to try the .495 balls with the same patch or a thinner lubed .010 patch.
Some folks might try a .490 and an .018 pillow ticking patch first before switching ball size because it's recommended to just switch one component at a time to properly evaluate the effect.
But obtaining shooting experience with that rifle will help you to judge for yourself what you should try next. A tighter load might require swabbing more often between shots. But that can also depend on how much powder is loaded.
By starting at the lower powder charge of ~50 grains at 50 yards and then increasing it in 5 - 10 grain increments, you'll find the powder charges that shoot the most accurately for that combination of patch and ball.
Then repeat the process for whichever new combination that you eventually decide to try.