Before you sand, Dremel, or otherwise grind anything down, first make sure you're letting the ladle sit in lead a while to get really hot. That may be all the difference that's needed.
You mention it needs to pour "quicker"- your ladle isn't aluminum, it's cast iron and could be acting as a heat sink, causing your alloy to cool down too much and affect the flow.