Well, as others have said, it depends.
It's fairly well known that .50 BMG can penetrate engine blocks and lightly armored vehicles, but I'd guess (I've never shot one) that even a round moving that fast with that much mass has to be slowed down by going through steel.
From Chuck Hawks...
At a more practical 400 yards, a range normally unheard of in pachyderm hunting, the .50 BMG (750 grain bullet at a MV around 2800 fps) can deliver 10,000 ft. lbs. of kinetic energy!
An incredible amount of KE, especially for 400 yards, but not something otherworldly. A 300 gr. .375 Ultra Mag, at 400 yards, packs in excess of 2000 ft. lbs. of energy, according to Remington. 2000 ft. lbs. will slice through nearly every body armor system in the world, and will penetrate unarmored vehicles pretty easily.
Thus if it hit something solid, it probably would penetrate a few houses but be stopped. Unfortunately, buildings today are mostly built with wood or concrete block, and neither of these will do much to a projectile like that. Precast concrete walls, on the other hand...
The threat is exaggerated, obviously. Sure it could penetrate 10 houses if you got a million dollar lucky shot, but it's not some magical exploding death bomb - it'll make a .50 hole in a bunch of walls and pipes, but it most likely won't kill anyone.
Regardless, .50 rifles are relatively expensive, always heavy, and usually fairly loud - three characteristics criminals and terrorists are unlikely to live with.