There are quite a few details to pay attention to, with flintlocks, but I assure you they are 100% reliable. If. If the are cleaned properly, the flint is sharp and positioned just right, the frizzen is hard, the flash hole is positioned correctly, and more.
It does look like your flash hole is very low in the pan. That can be fixed by making your pan deeper. However, that should only cause hang fires and not miss-fires. It looks like your pan and lock are not tight up against the barrel. That should be corrected. Sometimes priming powder will accumulate in the lock, which than can ignite, blowing your lock off. Don't want that.
If the frizzen gets dirty, it should be wiped with a clean cloth. Giving it the finger (!!!!!) can/will just make it worse. The first time you load the gun, it should be totally dry and clean, no oil in the breech, and the frizzen should be wiped off with a clean cloth with some alcohol on it. Just some rubbing, or denatured alocohol, not your best Scotch.
As long as everything else is working right, the grain size of the powder should not cause any miss-fires, or make "much" difference in ignition time. 1fg could possible cause a noticeable slight delay in ignition, compared to a finer grain. But miss-fires, no. I've lately moved from 4fg to 3fg for priming both my flintlocks.
I get better, faster ignition when I load some 1fg first, then the remainder of the load 2fg. Or just 2fg. (.62cal) in another gun, it could be just the opposite. But the only difference should be ignition time, not whether it fires or not.
Homemade powder could be "suspect". Or, a suspect. I'd learn to load and shoot the gun with some Goex at the least, or some Swiss, just to eliminate that as a variable. It could be that it is very dirty burning. ?
The size of your flash-hole should not be so small that it clogs easy. A very small flash hole can make the gun "prone" to miss fires, once it gets dirty. When a flintlock does get very dirty, (way way after four shots) and it fails to fire, the problem usually is crud in the flash hole blocking it. A little pick, like a nipple pick, is good to have when you are firing many shots. Judging from how the flashole "seems" or looks to be so low, it may also be very small. Need a better pic.
A little rust in and around the pan is okay.
I have a very expensive gun, and I don't sweat any rust in that area. I'd rather keep the pan and area around the flash free of oil, and get a little rust, rather than having my powder contaminated by oil. (never over-oil your lock) Who made your Baker? Have always wanted one. Locks that are not of very high quality, can be troublesome on a flintlock rifle. Or smoothbore.
A couple of clarifications. Could you take a good pic of the location of the flash hole? Also, when the gun refused to fire, was it the pan that refused to flash, or did the gun not fire, even when the pan flashed?
A flintlock is a reliable weapon. The caplock was an improvement for military use, for several reasons, but a cap-lock is not more reliable than a flintlock. Just have to have all your ducks in a row.