Try shifting the holster just a little further back. I'm assuming you carry at the 3 o'clock position. Try 3:30-4.
Or, depending on what kind of holster you're using, it could be sticking out a little too far for you. A paddle holster such as Fobus or Serpa is going to stick out much further than a pancake.