Since you reload, any of the classic bolt action deer rifle cartridges (243, 260, 270, 308,) is what I,would go with. Or the new 6.5 Creedmore which seems to be really versatile. If one of those classic deer cartridges cost much less to reload because it uses much less powder per shot then I'd probably choose that.
Then again, you mentioned lever actions. I could see going with a 38/357 one. Would be a hoot to shoot, economical, and good enough on deer.