Using standard soft buffs is exactly how good guns get butchered when being refinished.
Professional polishers use very large diameter HARD polishing wheels that are usually about 6 to 8 inches wide.
They also have special shaped buffers for areas like inside the trigger guard.
Metal polishing is both an art and a skill that requires development and constant practice to maintain the skill.
I'd recommend forgetting the buffs and use very fine abrasive sanding cloth and finish with a metal polish.
Nothing looks worse than a gun with a bright, shiny blue finish, but the metal UNDER the blue has rounded off edges, dished out holes, faded out stamps, and waves and ripples in the flats.
Nothing screams "AMATEUR BOTCH JOB" louder.