Carrying IWB takes a little getting used to, and you have to find your sweet spot, and give your body a chance to get used to the gun being there. At first, it will seem uncomfortable, but once youre used to it, the only time you notice it, is when the gun isnt there and things feel off. I've carried Government Models, Commanders, High Power's, P220's, P226's, P229's, Glock 17's, and a few more, every day for about 35 years now, so I know it can be easily done, if youre wanting to do it.
A good gun belt and a good holster are a must, and make all the difference in the world, and will usually give you totally different experiences. This isnt a good time to want to scrimp and be a cheapskate. A good belt and holster are an investment, and well worth the money. Like anything else, you do get what you pay for, and only rich people can afford cheap stuff (which you usually dont see them buying either, by the way) so spend more than you can afford, you wont regret it.
With proper gear, weight isnt an issue, or at least, not nearly as much of one. I know some will tell you you have to wear pants a couple sizes larger to accommodate things, but I dont agree, I've always worn my regular size, and while snug at first, the material gives, and the pants hold the gun more firmly and things dont move around, and it tends to carry the weight better. On the rare instances I should decide to wear an OWB or no gun at all, my pants still fit properly too.
If your trying to carry a full size gun in sweat pants with a Fobus paddle holster, youre going to have a totally different experience than you will if you carry the same gun in a Milt Sparks VMII with a proper belt, wearing a pair of properly fitting Carhart's. I'll guarantee that.