Did you get split cases? The .38 S&W case diameter is slightly larger than the .38 Special case diameter, and a common problem with "converted" revolvers has always been that the .38 Special cases will split in the oversized chambers.
Another problem is that the bore diameter is wrong for the .38 Special. .38 Special is .357". .38 S&W is .361" (9.2mm). Thus, when you shoot an undersized .357" bullet through that .361" chamber throat and barrel, you may not get a good seal in the bore if the bullet doesn't obdurate sufficiently to make tight contact on the whole bore.
Those "conversions" consisted of using a .38 Special chamber reamer to lengthen the chambers, shortening the throats. This was done to allow shooting in the U.S. of the much more common .38 Special, but it was really an ill-considered thing to do to those guns.
Your comparison of shooting .22Lr through a .22 Magnum cylinder is spot-on: the .22LR case is undersized in the .22 Magnum cylinder, so it didn't get a good gas seal, and quite possibly split those cases, too.
If I were you, I would not shoot that gun with .38 Special ammunition, and I don't think I would shoot it with .38 S&W either, if you can even find any. What I would probably do is put it in a shadow box on the wall, or sell it. If I really wanted to shoot it, I would try to track down a replacement cylinder from another Victory revolver that was never mucked with. Or get another complete, un-mucked-up revolver to shoot and keep your current one for spare parts.