You opened the door, unarmed, and stuck your single most vital part where he could get a good bead on it. There's no safe way to do that, so don't do it. Having a firearm behind your back is fine, but may not help if he's fast enough and knocks the door down on top of you--or just shoots through it.
If a stranger who looks out of place is knocking, ignore him. Keep a firearm ready. If he's peaceable he will leave. If he's coming to do harm he may try to come through the door, in which case you want to be down the hall with a rifle pointed at him, NOT standing with your head cocked around the door.
I'm not sure I understand why people would open the door and presumptively jam a loaded gun in the face of the stranger. THAT could get you into trouble. JUST DO NOT ANSWER THE DOOR FOR STRANGERS. Don't get in a @#$@ conversation with them, don't engage them, don't threaten them with a gun, don't call the police. Just ignore them and be ready if they try to break in. Say nothing. You are under no obligation to answer your door to odd strangers.