Backing a 30-06 off to lower velocity is a fine move. As rcmodel points out 30-30 bullets have killed many deer and it really doesn't matter what case/gun combo they are fired from. However, give up on 4064 for reduced loads. Move to a faster, easier to ignite powder, that'll be less position sensitive in that large case. I suggest about 35 grs of 3031, or about 30 gr of 4198 or perhaps about 28 grs of 2400, using a 150 gr jacketed bullet.
Remember also that the trajectory will be definitely more hook shaped, like a 30/30. Take that into account, shoot enough to understand where the go and how fast the bullets fall, I think 150 yds should be doable.
I shoot an 8mm'06, which is basically a .32-06, uses .323 bullets instead of .308. I shoot cast 175 grs with 16 gr of Herco, mv 1650 fps, I can keep 5 shot groups on a piece of standard copy paper at 300 yds bench rested. I am quiet sure it would completely penetrate a deer at that yardage, and the deer would die. Although I doubt it would be a dramatic Hollywood style kill knocking the deer backward through the air, but the deer would die a short, very short time later.
Remember, lots of people in the 1800s and 1900s ate deer that were taken with bullets running between 1000 and 2000 fps driven by black powder.