A few notes.... a drill press is marginal for doing this work. As you noted, a lot of chatter and slippage. The drill chucks just aren't built to take the side loads that a mill is designed for. You need to go very slowly and take very shallow cuts when using a press.
Not sure what you paid for your lower casting. I applaud your tenacity, but I'd likely have chucked it and started over. Tactical Machining currently sells them for 29 bucks when you buy a three pack.