A process server has no more right to demand any person show themselves than any other citizen. Laws vary on how the process server may perform his job depending on the state, but the bottom line is that they can bugger off is you tell them to bugger off.
And if they DON'T leave...call the police, in my opinion.
All this is if you are NOT the person for whom they are looking for. You are under no obligation to open the door either way (whether you're the person they're looking for or not) but if you truly AREN'T the person they're looking for then as far as I'm concerned, they can go about their business performing their alternative research and delivery methods to their heart's content.
(NOTE: If you ARE the person they're looking for, avoiding a process server doesn't help you at all. Just answer the door (safely) and get it over with. In the long run it doesn't do anything for you to avoid it, and it may eventually get ugly if the court decides enough is enough and issues a bench warrent for your arrest.)
If they decide to be stupid about this and simply leave the papers there "for you", sight unseen, then they're violating the terms by which a process server is supposed to abide by. In that case, after you're sure they're gone, call the court involved and let them know exactly what happened. I shouldn't imagine that they'd be very pleased that a process was not served properly.
In part, SC law has this to say about proof of service:
"The person serving the process shall make proof of service thereof promptly and deliver it to the officer or person who issued same. If served by the sheriff or his deputy, he shall make proof of service by his certificate. If served by any other person, he shall make affidavit thereof. If served by publication, the printer or publisher shall make an affidavit thereof, and an affidavit of mailing shall be made by the party or his attorney if mailing of process is permitted or required by law. Failure to make proof of service does not affect the validity of the service. The proof of service shall state the date, time and place of such service and, if known, the name and address of the person actually served at the address of such person, and if not known, then the date, time and place of service and a description of the person actually served."
BOTTOM LINE: Like any other person you don't personally know, opening the door is ALWAYS to be cautioned against until identity has been proven or circumstances allow you positive control over events (such as during day time, lots of neighbors out and about, other people present, and whatever other mitigating circumstances there may be for your safety).
Obviously some strangers can adequately identify who they are by the authority they represent, such as an officer in uniform with badge and ID. Some process servers will be LEO, for example. Others you might be able to confirm by having them slip ID under the door which you can use to call the appropriate court or law firm to confirm.
But if you don't know them, and they admittedly aren't looking for you, don't open the door...especially if they get rude or belligerent. Call the police, tell them what's going on, that the person won't leave and you don't know for sure who they are. Let things fall where they may then.