Given that you're talking about two pretty effective and relatively similar bullet designs the downsides are much smaller than many ammo-mixing schemes. The really poor ideas are the ones that start out with ineffective rounds like shotshells in order "scare" an attacker away.
With this combination the main downsides will be the difference in POI between the rounds and the fact that in a firefight you're not likely to be cognizant of what round is next out of the gun. However, given that you're talking a 642, you're not likely considering 25-50 yard shots, so the POI issue isn't as big of one either. The other possible downside is inconsistent recoil making it theoretically harder to keep a consistent rhythm of accurate rapid fire.
So far as penetration through heavy clothing goes, the LSWC will pretty much always win on penetration. In fact, it's probably pretty much the best penetrating handgun bullet design. However, the thing about hollow points is that against heavy clothing their penetration tends to increase, while their expansion decreases.