I was having the same thing happen with some slow powder.
You can increase the bullet weight, or increase powder charge for some effect, but the best way to deal with it is to move to a faster burning powder. You could try hotter primers instead with that load, as they will also jump the pressure up. With the 300 whisper, there are a lot of medium and fast powders that will work, though, so I would just switch.
Most powder should not leave much behind at all if it is burned under the correct pressures, and you also don't want excessive barrel fouling for the sake of accuracy. Even with a silencer, the powder should be used fully and not leaving large grains, as it should be burned by the time it reaches the silencer.