I do swage bullets, and my experience is that anything other than pure lead increases the pressure required to get good bullet fill out in the swaging die. Even alloys with a bhn of 12 (wheelweights) is much harder to swage than pure lead with a bhn of 5 or 6.
If you tried to swage something as hard as silver with the commonly available swaging dies and presses, you would risk cracking a swaging die, bending parts of the dies, or breaking the linkages on the swaging press. The pressures required would be above the design limits of the lead swaging dies.
These assumptions aren't based on scientific knowledge, just the knowledge gained from pulling the handle on my swaging presses.
Hope this helps.
Fred