You'll have barrel walls of 7/8" with a projectile width of 1.25" then? Will it be screw or otherwise breeched, or will you have a solid breech? The reason I ask is that for safety reasons, you'd want to work up pressures slowly. That means you might have a stuck ball in the bore. One option to relieve that problem is to use a lead ball that you could tap into and screw out from the muzzle if it became stuck.
I'd use blackpowder or pyrodex. The pressure curve is longer and the overall pressure lower, so it's a bit safer than smokeless to experiment with. I'd start really low something like 20grs of BP, expect a stuck ball, and make SURE that that ball is ALWAYS rammed right up against the powder. An airgap between the two could walnut your barrel.
It's federally legal as long as the projectile doesn't explode, and it doesn't use fixed ammunition. Check your state laws, they may differ.
Where are you doing this? Wherever it is, you'd better have a LONG fuse, and something solid and unvaluable to stand behind, as well as a good base for your cannon so it stays pointed in the right direction and doesn't become a projectile itself.
Those mortars are really fun. Cheap projectiles such as golf balls or cans keeps it simple.
BTW, for legal reason I DON'T SUGGEST YOU DO THIS