With the standard sized flash hole and the explosive force of the primer, firing the primer off will cause the primer to actually back out of the case......setback (in the case of your rubber bullets it will cause the cyclinder of a wheelgun to bind right away). This normally occurs, but with a normal powder charge, the case will be forced back when the main powder charge ignites and basically reseat the primer against the breach of the slide.
I believe the instruction included with them rubber bullets will tell you to use a specified sized drill bit and to just drill the flash hole bigger......I don't know what sized drill though. That way the full force of the primer blast will go thru the bigger flash hole unimpeeded to push the rubber/plastic bullet out the muzzle.