Started with a Springer subcompact XD in .40, great gun overall and I had no real problems with it, but I fell in love with 1911s and sold it to finance more expensive toys. I'd probably still own it and maybe occasionally carry it if I didn't need to sell it.
I also supplemented my XD (and later the 1911s) with a Sig P238 and S&W 442 for pocket carry, still spend most of the time carrying a 1911.
I can't vote twice, but I'll relate the experience of my friend who started carrying about a year or two ago. He started off with a S&W M&P in 9mm, awesome gun, but full sized guns are often too big for many people to carry on a regular basis. So guess what? He rarely if ever carried for a while, until he went to the other extreme and bought a S&W Bodyguard .380, which he does carry on a pretty regular basis.
Who knows, he might have saved himself the trouble if he split the difference and bought a compact M&P or similar mid/subcompact sized handgun to start off with. So just my 2 cents, I wouldn't start off with a full sized, but I wouldn't jump straight to the pocket gun either.