Some of it depends upon the skill of the user. I took a running deer at a little over 50 yards last year, using a 12ga 870 and rifled slugs. A stationary deer at 75 yards would be doable... maybe even out to 100 yards if the conditions were right. In order to do that, you need to find the right ammo for your gun and practice. Personally, although I can make 100 yard shots at the range, in the field I won't go over about 75.
As far as buckshot, I've not done a lot of work with it. Personally, I consider it more of an "inside the house" load, and it's quite effective at those ranges. Much beyond 25 yards, and I'd really prefer slugs.
With birdshot (at actual birds), I like to keep my shots within 50 yards or so. However, on clay targets at tournaments, it's not too unusual to see them out to 60 yards and be entirely hittable.
There's not a huge difference between 12ga and 20ga, but I'd consider the optimal range for birdshot and a 12ga to be around 50 yards and under, while 20ga would be a little less (maybe 40-45). Bob Brister mentions some of this in "Shotgunning: The Art and Science", but I can't remember exactly.
Like I said, it depends upon your skill. It also depends upon what your intended target happens to be.