You can negotiate who pays return shipping with seller. And you can try to negotiate transfer fees with the FFL holder. In most cases, the FFL has some minimal amount of work to do regardess if you actually take possession of the gun or not. So, it would be fair to compensate him for his time.
If you take possession of the gun, I believe you can only ship it to the manufacturer, a gunsmith, or a FFL. So depending on who the FFL is, you probably can't ship the gun directly to them, and would have to go through a FFL.
The way I see it, if you take the gun from the FFL, that's one transaction, and if you then send it back, that's a second transaction, and I would understand if the FFL wanted to charge a second fee.
The FFL I deal with, examines the gun when he recieves it, and if there's any question whether I would keep it or not, he doesn't do the paperwork to transfer it to me before I see it. (I tell him before hand what I expect to recieve: NIB, like-new, used etc.) I would never take possession of a gun I didn't intend to keep. It doesn't save him any significant work, so I'd pay him the same fee either way.
So far, everything I've bought online has been exactly as described. Maybe I've been lucky, but every transaction has gone smoothly and I've been satisfied with every firearm.