I have two factory refurb Nikon scopes and a pair of binoculars. It seems that if you're going to experience any problems with the device, you'll likely encounter them within the first 90 days (just IMHO). FWIW, I've had the refurbished binoculars for over a decade without any issues.
I personally LIKE buying refurbished products (gun stuff, laptops, networking gear, etc.). Aside from the monetary value, I have the sense that the products are double tested -- once at the factory-new production line, and a second time before it goes on refurb sale. Thus, it seems more likely that any QC issues (if any) that escaped the first time would be caught the second time around.
Also, every factory-refurbed product I've ever purchased has shown up literally pristine -- I would never have been able to tell it wasn't "New" except for a different packaging.
.