...the main danger is bits of lead being sheared off when the bullet is launched out of the gun.
Actually lead exposure mainly comes from hot gases vaporizing exposed lead at the base of the bullet and cleaning guns without using gloves. Inhalation of lead vapor as well as oral ingestion are the two major routes for lead to enter the body. Jacketed rounds with covered bases offer virtually no lead exposure to the shooter. You are correct lead exposure from the primer is minimal.