State law governs the amount of small arms primers that may be stored in a residence so the exact number can differ from state to state depending on the code. Most, if not all, states have adopted the fire code recommended by the National Fire Protection Association. The NFPA code is not a federal law but is insteada a recommendation that most jurisdictions adopt as part of their state fire code.
The SAAMI document on primer storage references the "RECOMMENDATIONS ON STORAGE AND HANDLING Issued by the National Fire Protection Association" which says this:
11-5.4 No more than 10,000 small arms primers may be
stored in residences.
11-5.5 No more than 10,000 small arms primers may be
displayed in commercial establishments.
Some believe that this applies to loose primers still in their packaging as well as primers in loaded ammo. In the end it will differ by state. The other major concern is to talk with your insurance carrier and make sure that they do not have a limit or exclusion in your coverage that could cause them to reject a claim. The danger from the primers and powder can also be mitigated by storing primers in their original packaging in a powder magazine constructed of wood. These rules are rarely if ever enforced. I cannot find a single innocent where a Fire Marshall came into a home to site someone for this offense.
The general consensus is that the "limit" for powder is 50#.