Yep,
Local zoning is the biggest hurdle. Assuming you want to be a "kitchen table" FFL and not invest in a storefront etc, it's best to live in a smaller town where you can be buddy-buddy with local government and the zoning board is all members of your bowling leauge, or even better an unincorporated area with no zoning oversight whatsoever.
From the ATF side of things, you have to be "actively engaged in the business". I don't know that means you actually have to turn a profit per-se or be a dealer full time, but you do have to be selling firearms on a regular basis, running a table at a gunshow etc. If they think you've gotten an FFL just to enhance your private collection on the cheap, and set up a few friends with free transfers, they'll pull your license.