Don't discount teenagers, visitors, nephews, etc.
Here is my $.02, based upon a lot of stuff, but also on the fact that my 16 year old nephew (who was gun safe and very familiar with firearms) was with friends at his home and blew the back of the head off one of his friends with a 9MM.
Look, be safe... You can buy a safe at Harbor Freight tools to keep your ammo in for $40.00. Quick access, digital keyboard.
Also, do you really want to make it that easy for a burglar to put your firearms and ammo together? What if you are arriving at home at that minute? I really don't relish getting shot, but getting shot with my own gun would POBAR me.