I take my holster with me to try on pants. To begin with, once I settled on a new size I simply replaced my currently owned pants with the new size as the old ones got worn out.
I think it was maybe a 1 to 2" waist size difference... not sure exactly. I now wear the same size pants with or without my IWB holster, as long as I'm wearing a belt everything fits fine without looking awkward. My IWB carry thickness ranges up to 2x layers of leather, 1x layer of kydex, and a 1.2" thick handgun. Then again, I've always worn relaxed fit pants and I don't partake in the fad of men wearing skin-tight pants so you have to account for whatever is your own personal style. Obviously, YMMV.