It's a matter of what your maximum tolerance for accuracy is. If you're happy with "five shot clovers", then when those clovers start to open up to 5 separate holes, get a new barrel.
On the other hand, if you want them all in .400" or less, you need a barrel now.
Hart, Shilen, Krieger...