I spent the first years of my career with the EPA, all of my degrees are in public health. Take two seconds with Google and educate yourself about high blood lead levels especially those exposed to lead metal fumes i.e. smelting, making caming, etc. As a Nation our blood lead levels have dropped drastically since leaded fuel was banned. People screamed about that one too. Take it a step further and read about the blood brain barrier in children and how and when it develops.
That whole, "what an idiot, bet he ate paint chips as a kid" comment we throw around? 100% true.
The unfortunate thing about many public health (EPA) regulations is that they are forced into being to protect the stupidest 1% from themselves. The rest of us are stuck with government regulations that should just be common sense practices. I don't agree with all of them, but consider the lifecycle of the material, yes lead comes from the earth. But in extremely low concentrations, what are the effects when we concentrate it and dispose of it improperly in high concentrations? I'm just trying to point out there is more to the story than just depriving my fellow shooters of inexpensive raw materials.
Full disclosure, I don't cast my own, but I grew up casting fishing sinkers about once a year from wheel weights. At my current rate of fishing I have enough now to last lifetimes.