My thoughts and experiences...
When I bought my Kimber Custom CDP, I had the same question. I decided to go with the shorter, simply because it was going to be my main carry weapon. When I was testing them for comfortability, I did not find a significant difference in the two.
If I was going to do it over again, I would have bought the 4". Although my 3" is very accurate itself, I personally will shoot a 4" more accurately, on average. My buddy has a Colt Govt. model which I can shoot much better. However, it is important to note the wieght difference between his Govt. and my Kimber CDP. My frame is aluminum vs. the Govt.'s all steel frame.
Obviously, the most significant difference that - I personally - do not like is the shortened distance between the front and back sights. While I have learned to shoot my Kimber CDP and other "shorties" more accurately than I could previously, I feel more confident with a 4" and can shoot a tighter group of follow-up shots in rapid succession with one.
Controllability and accuracy are the name of the game, as far as I am concerned, when choosing a defense pistol. However, you have to personally weigh what you want out of it. It is a give and take issue - ON AVERAGE - most people will shoot more accurately with a longer barrelled gun.
Do whatever it takes (well almost) to test out both models by shooting and carrying them if you can. Personally, I don't buy/sell/trade guns that often - mostly due to a lack of funds and time. So when I make a decision to buy a gun it is usually one that is long term! Don't get into a hurry about it, unless it is absolutely necessary.
Good luck!!
BTW - I am 5'11" and weigh about 175. I have a medium build with no spare tire to push the grip out and the barrel in. The "spare tire" has been brought up as an issue by others when choosing a gun for concealed carry.