Cast iron lead dipper...?

Get a Dremel tool and sanding drums. That should work well.

I would try sandpaper wrapped around a wooden dowel, followed by steel wool.

If you keep the ladle hot by letting it sit in the molten lead, accumulation should be minimal even without smoothing.

Just cut a small notch in the spout part. Use high heat. Let dipper sit in the alloy/lead a while.

You may not ever get cast iron to a smooth surface, but the above answers will do as good a job as possible. I would use crocus cloth working down from course to fine (sandpaper is usually used on wood or cleaning for painting). If the cast iron is of good quality it won't be too porous and you can get a good smooth surface. And the posters above are right about keeping the ladle hot...

Drill it. Place the dipper in a vice. Choose the size pour spout you want. Pick the drill bit that matches the size and drill 1/4 to 1/2 the thickness of the ladle at an angle. Need help? take a piece of strap steel. Drill the angle you want trough it. Clamp the guide to the ladle. Drill the ladle. More help? Send it to me with directions and thoughts and I'll mill it for you.

Before you sand, Dremel, or otherwise grind anything down, first make sure you're letting the ladle sit in lead a while to get really hot. That may be all the difference that's needed.

You mention it needs to pour "quicker"- your ladle isn't aluminum, it's cast iron and could be acting as a heat sink, causing your alloy to cool down too much and affect the flow.

I use a small round "rat-tail" file and various grits of wet-or dry paper. Then used various files and tools to scrape out the insides....mine is 40 years old and had gotten ratehr cruddy in there. Smooth off the tip that presses against the mould sprue hole also.
Letting it sit in the melt while heating and keeping it in the melt while you open the mould helps a lot in keeping it free flowing. If you take it out for any reason, submerge it in the melt and let it come back to full heat. Otherwise the lead will freeze and plug the spout when you try to fill a mould. And make sure to use enough heat to keep the alloy flowing from ladle into mould without hardening before proper fill out.
Gary