They work, but they don't work as well as factory formed heat treated rounds.
The art of expansion has come a long way in the last few decades and the better commercial rounds heat treat different sections of the jacket to cause the jacket to fail at predictable locations. That achieves the uniform expansion which results in something that looks like an opened flower when shot into water (water resists differently than random tissue and bone, so the water results are a little more perfect than reality.)
FMJ is the opposite: closed at the nose, but open at the base;
Not all FMJ are open at the base, and there is TMJ which are always closed at the base.
They can leave the jacket in the bore.
If the round is open at both ends it is possible. The X cut done by some in the past reduces that because the prongs still going over the top hold in the lead. While simply drilling it out or cutting the top off a bullet open at the base makes it a real possibility.
The result is usualy not the same as a modern hollowpoint.
It will reduce tissue flow around the projectile, which does increase tissue damage, increases resistance, and will reduce penetration. In that regard it is more like a softpoint in performance.
With jacketed bullets it causes the jacket to peal away and fragment to the depth of the cut which would increase wounding to the depth those petals remain attached.
Tissue cannot flow well around a random changing jagged shape. It can flow around a smooth rounded projectile readily. The result is the jagged shape results in resistance and damage to tissue that would have simply flowed around a FMJ round. This reduces penetration and increases wounding, though not to the extent of a modern hollowpoint.
soldiers would sometimes use bayonets or knives to cut "X"s into the nose of .30 Carbine round
This is the very definition of a "dum-dum" round. It would be a severely prosecutable offense in the U.S. military.
Certainly not something the military would be friendly about.
They are not "dum dum" rounds however.
It turns out that the actual "Dum Dum" rounds had the entire top simply cut off rather than an X cut into them. Creating a jacketed soft point that was also open at the base. This was very unsafe as the jacket became a tube open at both ends. That could result in the lead and jacket seperating leaving the jacket in the barrel with simply the lead shot out as a projectile.
The next round would result in disaster. That was a primary reason the practice at the time was frowned upon in official channels (before Hague.) Simply cutting an X leaves a tube where the lead has four prongs holding it in at the front. This is far less likely to result in the lead being shot through the jacket, though if the center of the X is very large it is a possibility. As a result the X is much safer for the user. The X also gives an indentation for tissue to enter, creating resistance that both smashes more tissue, reduces tissue flow, and helps to push the four prongs of the X apart which would be absent on just a soft point. A soft point would allow greater unharmed tissue flow, but the soft lead would also expand consistantly increasing the diameter of the wound channel more consistantly in a predictable manner.
Finaly the random hand cutting on the round is going to be aerodynamicly imperfect resulting in a much less accurate projectile.
So a round with an X will do more tissue damage than a FMJ, likely less than a modern hollowpoint, and will have levels of penetration in between both while being less accurate than either.