Here's the logic behind the gunbelt. Take a look at any police officer, and you'll notice a HUGE belt that is very rigid. Obviously, they've got to carry a variety of items at all times.
I'm a big fella, and I picked up a nice belt from Eric Larsen (hbeleatherworks.com), along with a holster last year.
After wearing it for a few days, I came to a conclusion really quickly: A big guy needs a big belt, whether he's carrying or not. I'm so happy with this belt, I've worn it every day for the last year. I'll never buy a regular belt again.
Normal store-bought belts just don't work for big guys. Now that I have a sturdy belt, i don't need to hitch up my pants all day long. Make sure that it fits nice and snug, and you'll be really happy with it, pistol or not.
<Edited Eric Larson's URL>