If you are sighted in for 100 yards, the bullet is falling at that range. It reaches maximum ordinate of around 0.87 inches at 85 yards and then begins downward.
Now technically, the bullet is always falling -- as soon as it leaves the muzzle. When we zero a rifle, we ajust the sights so they point down. In order to aim at the target, we must elevate the barrel, so that it points at an upward angle.
So even though the bullet is falling when it leaves the muzzle (that is, there is a downward component in its trajectory) its velocity allows it to rise. Eventually gravity overtakes the upward vector imparted by the launch angle, and the bullet begins moving downward.
Added: At 50 yards, a .30-06 (this is one of my handloads, with a 180 grain bullet at 2,800 fps) will be about 0.16" low -- assuming a scope sight with line of sight 1.5" above the bore line.