Thankfully, that's almost always the case. Recoil trouble wears things out fast and breaks small parts. Pressure troubles cause really major problems. But that doesn't mean that recoil trouble can be ignored. Beat up a gun and it will break--probably when you need it most. You're better off shooting it with loads that don't cause accelerated wear.
If you really need more performance than you can get without beating up your gun then either get another gun, or get a heavier caliber.