How much are you looking to spend? (It would be helpful to know a little more about what you consider "factory" to mean. Reading between the lines, it seems like you'd be happy with anything that didn't start life as one factory's gun only to be modified by a gunsmith down the road. Is that right?)
We can talk about semi-custom makers Wilson, Brown, and Baer, all definitely above Dan Wesson, but then there are a few full-custom 'smiths who will scratch-build (meaning they're not modifying an existing gun) something even nicer.
Baer is pretty easily the best value, to my mind. Wilsons and Browns are more refined, and Wilsons would get my vote as the all-around "best" of the three, but I don't think they're particularly good values. Wilson is the most popular and well-known of the semi-custom brands, and you're paying quite a surcharge for the name alone. Wilson does have great CS, which has value, but by the time we're talking prices for anything but the most basic Wilson, I'm going up a level to a full-custom. Baer takes care of his customers as well; I believe Brown ultimately does too, though anecdotally I've heard from a few who have found their interactions with Brown CS to be less pleasant. Bottom line, though, is that you can get a gun from Les Baer that is guaranteed to shoot a 1.5" five-shot group at 50 yards for around $2,000, and it will be built like a tank. That's a better accuracy guarantee than you'll get from Wilson or Brown at any price level. A Baer will be the tightest-fitting of the three brands (some would say too tight), because that's just how Les insists on building the guns.
It sounds to me like an SVI/Infinity would actually be right up your alley. They're a very clear level above anything from a semi-custom shop (even a Supergrade from Wilson), and they machine every part in their guns in-house. The precision of the fit is just incredible. They're very competitive price-wise, too. The base gun starts at $3,800, so I would have no interest in any Wilson or Brown that began to approach that price.
If money is truly no object, then I personally don't think it gets better than Stan Chen. But you can expect to pay $9K or more, and you'll have to wait for several years to get the gun -- if you can even get on the wait list in the first place.