I have a 17 Remington AR-15. Shoots great. Mine is 24" heavy barrel rifle. Not exactly a "walk-a-bout" hunter". Be cautious if you build a thin barrel rifle. I have a short, thin barrel bolt gun in 17 Remington that heats up and shifts point of impact in just a few rounds.
Cost wise, except for the cost difference of powder, the reloading cost between 17 Fireball and 17 Remington would be small. Bullets and primers are the same. Case cost, if you can find any would probably be similar cost due to the low volume sold. (I have not bought any 17 Remington cases in years so I am not current on availability). Both can be formed but the parent cases may not be any less expensive and then you have the time invested re-forming them.
Although I have not looked in the past few years, 5 to 10 years ago, I never saw a cataloged 17 Fireball or 221 Fireball AR-15 barrel for the AR-15. I'm not saying some obscure shop or a gunsmith would not cut one for you, they just are not mainstream. My 17 Remington barrel is a Shilen barrel that I bought through Midway at the time.
As Varminterror said, feeding the shorter Fireball cartridge may require additional modifications to the magazine.
After building the 17 Remington AR-15, I considered using it for prairie dogs. It would be good medicine but I found the longer range a bit lacking. I ended up building a 204 Ruger AR-15 as it shoots flatter, longer than a 223 Remington or the 17 Remington. If you are looking for minimum pelt damage, the 17 caliber rounds would be a benefit.
So, don't let this be discouraging, but do some investigating on the ballistics of the various cartridges and choose what will serve you best. Sometimes, the cool factor is reason enough.