The only way you could possibly get by with this is if you are shooting BH209.
If you are shooting T7, Pyrodex, Goex, real BP, or anything similar, you will have major problems.
This being said, I clean mine after each shot, and with the ones I've shot the first shot from a clean bore has always shot to nearly the same POI as the rest.
I usually either swab between each shot when sighting them in, but I have shot 3 shot groups before swabbing. I wouldn't just shoot and shoot with it super dirty sighting it in, and sight it in a a dirty barrel. As long as you swab between every shot or two when sighting it in, you should be fine hunting.
As for the rust, stainless holds up better, but will still rust. I had a blued muzzle loader one time that I shoot Pyrodex in. I shot it in the morning 3 times around 9:00AM. The evening I took it out and shot a deer with it. I spent the night skinning the deer, and getting it processed. That night it was 9:30-10:00 when I got around to finally breaking it down to clean it. To my surprise the gun had already started rusting. I got it mostly cleaned, but that just shows you how fast the rust starts. Imagine that for 2 months. I've got a buddy that never clean his, and the barrel is literally full of rust. It looks like it's set underwater for years. That said, his also isn't accurate at all, and he is the type of guy that says he just "shoots" out the rust. What scares me is someone let him borrow their muzzle loader and it's still sitting at his house uncleaned since they shot it last year. I'd never loan him any of my guns.
Now, my experience with SS ones is that they don't rust as quickly, but I'd still not let it go more than a day. If you do I suspect you will start seeing it rust.
If you use BH209, it acts similar to a smokeless powder and you clean it like you would a rifle. IMO you'd probably be okay to leave it for a month or two is this is what you shot in it, but that's just my opinion, and this is the only instance I think you might be okay. I still wouldn't do it, and I'd still clean it after shooting it though.