The balance between freedoms (there really are no such things as "rights") and restrictions is a delicate one. If everybody were self-responsible and highly competent, a completely deregulated society would be both functional and stable. Unfortunately, that isn't the case at present - so where do we draw the line?
The problem with age restrictions is that, again, not all people are created (or raised) equally. I've known some 18 year olds whose poise, integrity, and competence would easily exceed that of many people twice their age. I've known others who should probably still be sleeping in a crib (at which point they're probably a lost cause as it stands). The effective solution is to implement a meritocratic approach: do not simply deny people an opportunity for superficial reasons, but do make them earn it.
I've always favored a "universal age" approach to issues such as these. Once you are a legal adult (18, suppose), you ought to be granted all of the powers inherent in that distinction - or, at least, the open opportunity to acquire those powers. If it were up to me, everything would be affixed to that age - driving, drinking, and voting, among others. Powers of a particularly great magnitude (e.g. driving, CCW, even voting) can then be delegated according to competency. I'm confident that most 18 year olds, given an appropriate upbringing and education, would have little trouble meeting what I would consider reasonable standards necessary for the acquisition of said powers. For what it's worth, I think it's absolutely absurd that we'll put a 16 year old at the controls of a 4000 lb vehicle, but won't let them buy a damn beer for another five years - really, it should be the other way around if anything!
The "irresponsibility" argument is spurious. As the military example demonstrates, training can eliminate or mitigate many of the potential problems that can arise from granting such power to supposedly "immature" people. In fact, that applies to anything or people of any age - I drove/raced in sanctioned motorsports competitions as a teen and was a far more competent (and responsible) driver by 18 than the vast majority of people are at any point in their lives. If it's such an issue, why not simply mandate additional training/competency requirements for younger CCW applicants? Biologically, most people have reached a state of near-complete maturity by 18 - the rest is simply a consequence of environment.
Really, we reap what we sow - if we want to create a restrictive nanny state which assigns restrictions in an arbitrary manner, we will also create people who function best under such conditions (i.e. helpless and incompetent). As somebody who spent some years studying the Classics, I can't help but note that the average Roman noble was expected to speak numerous languages, possess an almost encyclopedic knowledge of Greek literature, and command armies by the time he reached his early 20s (I also should note that the Roman government did not even have a sanctioned police force, let alone any sort of regulatory agency). Were they really any better than we are?