If you're never going to shoot over 500 yards, then the 24" offers no real advantage in practical terms.
Yeah, more velocity will help fight the wind, flatten the trajectory a bit - but it's not that significant. Key here is, IF you are looking for handiness. If handiness is not a factor and this is going to be a rifle shot off the bench as a range toy, go for the 24" as most published reloading data is based on a 24" barrel. As others have stated, the extra 4" will add a little weight that will add a small bit recoil reduction. That won't be a significant difference either.
In my opinion, need long range shots? Absolutely 24"
Never going to shoot far and want compact rifle? 20"
Won't shoot far, but don't care about compactness? 24"
If that makes any sense. That's how I would judge it.