My daughter and oldest boy started shooting .22 handguns at 3 years. They were interested, I was doing most of the "shooting" (actually supporting and aiming the gun while they pulled the trigger) and they loved it. My daughter also wanted to hold the grip and pull the trigger on the AKM at that age, but my son didn't like the concussion.
Now that they're older, they shoot a Savage Cub .22, and still shoot the .22 Bearcat and Mk II.
My daughter, now 7, does pretty well handling my 629 with .44 Specials. Her stance is good and she doesn't seem to notice the (mild) recoil.
She's still a bit too small to properly fit the length of pull on the Cub rifle, though.
A lot of this "shooting" is still sitting in daddy's lap, with daddy's hands physically on the gun the whole time, but they love to do it, and I do too.
The real value of all this, I think, is conditioning the safety rules, religiously, from such an early age. Don't child-proof your guns -- gun-proof your kids.
-Sam