It is not always obvious what is harmful to another.
Oh, it gets worse.
I hate freeway noise. But I love being able to get to work in 10 minutes.
If a freeway is built near my house, and my commute is shortened from 30 to 10 minutes, but I also have to tolerate some freeway noise, is that harmful to me, or helpful? Who decides? Is that an infringement on my property rights, or does my use of my property actually benefit from it?
Maybe my neighbor and I agree about noise and commute time, but I place a slightly higher premium on total quiet, and he places a slightly higher premium on commute time. Then what? Sometimes a simple philosophy can't solve the problem. Furthermore, privatizing the freeway doesn't really solve it, either. We'd just sue the freeway company, and each other, and the company would then sue us, for infringing on each others' property rights.
And while we're all in court with 25 lawsuits per person, Mexico could invade and take over, thus ending our libertarian paradise.
So we need something like the Rule of Law.
Now I'm not saying that libertarians -- and I consider myself to be one in general -- have no answer to this. I am, however, saying that the answer will probably end up looking a lot more like our current system than a lot of us want to admit. Either that, or it will look a lot more like the anarchy we say we don't want.
There's nothing anti-liberty about the Rule of Law. In fact, no society has remained anything close to free without it. Democratic, perhaps, but free, no.