When people say "milspec AR15", they mean milspec other than the lower receiver and trigger parts that must be changed to comply with NFA. It would just be a pain to write that out evey time.
There are definitely plenty of milspec parts that are not the best ones on the market. Just a few examples:
-KAC rails have long since been surpassed functionally by numerous other designs.
-Ambi selector is better than one-sided milspec
-A2 grip is probably the least ergonomic AR grip ever conceived
-All the military spec optic mounts (for Aimpoints, ACOGs) are inferior to multiple models in the civilian market.
-M4 milspec stock probably has the most rattle and least overall function of any carbine stock on the market today.
There are several (what amount to) construction techniques that are functionally desirable from a reliability standpoint that got popular due to being implemented on the military rifles. These are the kinds of things you learn about when building accurate-as-possible military clone rifles.
But the actual answer to the original question: You don't "need" a "milspec" AR to serve self defense role at home or in the trunk of your car. What is the *max* amount of ammo you are going to be shooting in such a situation, which is already incredibly remote? 30rds max? 30rds would be a very high estimate in such a situation for anyone who doesn't harbor Red Dawn wet dreams. Any functional AR can easily exceed that number by an order of magnitude with no cleaning or maintenance.
The desireable milspec construction techniques are really only a tangible positive on rifles that may see thousands of rounds and/or will get shot till they break. If you are going to be shooting it that much, there are few that will argue that a milspec upper shouldn't be on your shopping list.