I use factory ammo for personal defense in my pistols (200gr Speer Gold Dot). Personally, I do so for a greater guarantee of reliable function. I've never had a handload fail in my reloading career, but I also haven't taken the time to chronograph most of my pistol loads, and I have shot far more factory ammo than I have handloaded ammo over the years.
I'm sure that I'd be fine using handloads if I made a significant effort to do so, but I'd want to be sure that I made those pistol rounds to very exacting specifications (checking velocities over a chrony, primer seating depths, etc). I just don't put all that much effort into pistol loading, as I typically load for plinking or IPSC matches (short range and fast shots). I've also found that the factory defensive loads are of high quality, and function reliably. Plus, the cost difference is only marginally significant, since I don't normally shoot through my defensive loads.
As far as the legal concerns of using handloads, I could see this being a major problem for a police department, or other public safety organization (we face scrutiny on everything, given our role of serving the public). Otherwise, the idea that handloads would be a problem (legally) in a self-defense situation involving a private citizen has always struck me as internet mythology. Simply put, if you pull the trigger on a person, the shoot is going to be judged on the circumstances, not the ammo that you were carrying.
If anyone claims otherwise on the subject of handload legality, I'd love to see some documentation of any court case where the ammo that was used was decided to be the determining factor regarding the legality of a defensive shooting! Simply put, the legality is the least of my concernes when it comes to carrying handloaded ammo for defensive purposes!