Of the rounds I have fired, when I was forgetfully without hearing protection, the .32 ACP Speer Gold Dot, and .44 Special Winchester Silvertip have been the most tolerable, in that order. These induced only a slight ringing of the ears, which soon went away, though to be clear, I do not recommend shooting without ear protection. Next on the "tolerable" scale, but notably harsher, was the .45 ACP, exact load now long-forgotten, but it was not +P. The .44 Special Silvertip is/was a notably mild load for that particular cartridge.
My preferred home-defense handgun cartridge, for fighting indoors, is currently .45 ACP. (Of course, long guns, if practicable at the particular moment in time, are preferable to handguns.) My chief wants me to carry a .40 double-column-mag autoloading handsgun* in my duty rig, so I am not necessarily able to always have my preferred handgun in-hand. The .40 S&W is obnoxious even with hearing protection in place; I will sometimes use both earplugs and a headset when firing .40 indoors. Fortunately, I have not (yet) been blessed with experiencing the report of .40 without hearing protection.
*Handsgun = firearm that resembles a handgun, but has a grip frame so big that I cannot grip it securely one-handed in spite of my large hands and medium-length fingers. If there is a chance of being within arm's reach of a bad guy, I prefer to have a handgun, rather than a handsgun, though there are handsguns I actually like.