I don't have swivels on my 10-22, but I don't often carry it hunting. The holes are still there, so I can add them anytime, but can't remember the last time I had a sling on it. There are swivels on a couple of rimfire bolt actions and I've used them occasionally, especially, on my .17HMR CZ. the most over the past few years.
I carry a .223 bolt action rifle the most, always with a sling, when walking the roads and fields out back. Since having the .17 HMR, I tend to carry that with a sling a lot more than the .223, because I tend to plink with that rifle more than the centerfire and it will put down varmints out to 100 yards or so.