Actually, it does make a difference. I can see the differences between the lenses and coatings between a $500 scope and a high end scope.
Do I own any high end scopes? No. I'd love to, but I don't have any needs for one. I got to hunt one year with a Swarovski scope, courtesy of a Swarovski rep. The 30mm tube, 56mm objective lens and the top end glass and coatings would have allowed me to hunt well past legal hunting time (1/2 hr past dusk). There are some who pay for that ability, and I would if I needed it.
Currently, my most expensive scope is a Vortex Strike Eagle, right around that $500 mark. I also have some low end scopes, a Bushnell 3-9 that came with my Axis II, and I still have it on it several years later, despite the fact I keep saying "I should really put a good scope on that Axis sometime...."
So while I can see the deficiencies of a $500 scope compared to a top end scope, for my uses- they suffice.