A Delrin (trademark name IIRC) rod turns down nicely in a lathe and would make a fine drift.
.
You could leave the size as large as you desire for your hand to hold and sufficient surface area for the hammer blow.
Same for making your own light duty punch set.
Delrin also makes for an excellent spacer material for my M-16 that allows me to use the carbine buffers
in the rifle length stock without the carrier key impacting the lower receiver ring.
+1 for getting the sight close to fitting before having to wale on it too much
with either Delrin or brass drifts.
Removing metal from the bottom of the dovetailed sight effectively narrows the angled width of the dovetail by lowering the sight in the receiver.
Think about this one for a while and it will make sense
Taking the metal from the bottom of the sight (running the sight across a piece of sandpaper or stone on a hard flat surface)
allows one to control (with much better precision) both the amount being removed and the uneven,
unsightly marks that you get by working the dovetail angles.
JT