Open bolt has a fixed firing nub on the bolt face, and is held rearward by the trigger/sear, and strips a cartridge out of the magazine, chambers it and fires all in one motion. It keeps cycling until the trigger is released, catching the bolt rearward once more.
The advantage is that it's easier to build/design, and the open bolt allows the barrel and chamber to cool fom both open ends. Also since cartridges are held in the mag until fired, there's no chance of cook-off in a hot chamber.
The disadvantage is that open-bolt guns often require heavy bolts, stiff, hard to cock recoil springs, and since the bolt jumps forward on firing wiggling the gun a bit, the first shot is not as accurate as a closed bolt design.
Closed bolt fires when the bolt is closed with a firing pin being struck by a hammer, much like most rifle designs. A cartridge can be held in the chamber until ready to fire.
The advantages are that the first shot, securely chambered is more accurate since until firing, the only parts movment is the trigger and the hammer falling. They can be made lighter since the bolt can be smaller as it locks into place against the reciever, or the chamber end of the barrel depending on the design.
The disadvantage is an unfired round can be cooked off since it's waiting in the chamber instead of the magazine, if a long burst has just been fired. It's also usually harder to design, and more complex to manufacture.
A submachine gun is usually defined by being chambered in a pistol cartridge, or at least one that's smaller than most rifle rounds. But there's some overlap, as some very short versions of cut down assult rifles are also sometimes considered subguns too.
Legally though, from an ATF, and U.S. private ownership standpoint, anything that fires more than one shot automatically with only one pull or holding of the trigger is legally considered a machinegun with all having the same regulations up to .50 caliber. (over .50 is a "destructive device" classified with rockets, shells, grenades etc.) There are no legal or registration distinctions between a full-auto pistol, subgun/carbine, rifle, or crew-served machinegun on a tripod. They're all treated the same.