The .38 really isn't that "much larger" to begin with; calibers are almost identical. What sets the .38 ahead of the .380 (in similar-length barrels) is bullet weight. There just aren't too many .380 rounds using bullets of over 100 grains, let alone 135 or 158 grains.
But, in typical "self-defense" guns (CC pieces), the .38 loses some of its edge when the bullet leaves the barrel before its powder charge is done pushing. (This issue, however, is addressed in "short-barrel" loads, like Speer's 135-grain Gold-Dot round.)
Being designed around short barrels to begin with, the .380 already had been "optimized" for those little guns, wasting little of the charge's ability in guns with +/- 3-inch barrels.
So, in short barrels, one has the advantage in weight, and the other in velocity. The impact-energy tables tell which advantage is greater, but I'm too tired and it's too late in the day for me to look them up now. My money is on the .38 in most bullet configurations.