There are two advantages to physically polishing brass. First, it can remove minor stress cracks in the surface of the brass shellcase and prevent the stress cracks from spreading. If you are not having any problems with cracked shellcases it might not make enough of a difference to make it worth the time to polish. But if you have had cracked shell cases it might be worth the time to physically polish the shellcases, especially if you are reloading self defense ammo.
Second, physically polishing the brass shellcases will reduce the force required to cycle the slide in a pistol. Not by a huge amount, but every bit of friction reduction helps.
There are plenty of scientific papers to show that ammonia can attack copper and brass and cause stress cracks. How severe the stress crack issue is probably depends on your polishing process. Do you do anything to ensure that all traces of the ammonia are removed? If after a chemical cleaning with an ammonia based product, you physically polish the brass, you will likely mitigate any potential stress cracking. How many times do you reload your shell cases? The more times you fire that shell case the greater the chance of a stress crack running completely through the brass shellcase.
I remember a friend from years ago that polished his brass shell cases in a vibratory polisher using walnut shells.