I've had dealings with Oly, they are right down the street, as in just a few miles away. I'm an AR nut, I build all my own, and to have them that close by and that I haven't been there in nearly ten years should tell you something. Rainier Arms, several miles away, are light years better, and they cater to AR builders and nuts. You'd think having an AR factory close by would be the BOMB, but it isn't. Not this one anyway.
Not to say I haven't had an Oly that did well. I built it myself and I'm not sure exactly how many Oly parts went into it, but most of the big ones save the trigger group were. I can say their hammer and trigger is atrocious, and a lot of their other parts are very poorly designed too. The bolt carrier looks like it was roughly hand made with a file and a dremel tool! The barrels? I've seen a couple perform okay, but most didn't. A bull barrel that shoots 2MOA!? Come on. This particular M4 barrel was okay though. The rifle was fairly reliable and tough enough for range or police work, but not military work, not by a longshot.
Their receivers are okay, they are better than a lot of the El Cheapos out there actually, but not nearly as good as a Noveske (they actually ARE nicer, you can tell most during the build). And the extra money you spend on that name will make any rifle you build worth more, trust me here. A $2000 parts rifle built on a Noveske can be a $2000 rifle, but one built on an Oly for some reason is hard to get rid of for anything over $1000. They ran that company and that name into the ground, refuse to make changes (we asked, they refused even though they could have gotten a military contract to make SDM upgrades for M4's ten years ago for Stryker Bde(s)). Said their design was perfect, no need to change. Not even their gas blocks that blow off after 2000 rounds. Buy 'em as is or blow off. We blew off.
So if I were you and I wanted to upgrade that rifle, I'd do one of three things. I'd cut losses and sell it now if I ever intended on selling it. Oly maintains a site for Oly fans. They don't understand why Oly has such a bad name over there. May be a good place to list it. Local sites do well too, but you won't sell it here unless you give it away.
Now if I were to keep it and not worry about the name on the side, I'd change the barrel with a good quality barrel, I'd put a Geissele trigger of your choice in it, and I'd change that junk bolt carrier with a JP Ent. "Tactical" bolt carrier. This will give you a fine rifle as good as any other fine rifle save the name for the most part. And if you ever intend on getting rid of it, take these parts back out and put the Oly stuff back in. You can put your new parts on a matched Noveske receiver set with JP small parts and have a very nice and expensive rifle plus the Oly one --or sell the Oly later to cover the cost of upgrading your PARTS to another receiver set. You may even get an optic out of it.
Finally, you could just keep it how it is, not invest anymore in it, and just shoot it until it fails. Use it as a practice rifle. A beater. A truck rifle. But I wouldn't invest much in upgrading an Oly unless you plan on doing what I mentioned above. See, been here, done that. I just sold the Oly for $750 not that long ago, an M4 type with their rail. That rail is junk too by the way, most stuff doesn't fit.
I have an M4 now I built to be a copy of the SOPMOD M4 i was issued in the army. Only difference is the 3gun trigger (which I would have preferred over the burst trigger or auto trigger any day) and AAC Brakeout. It has an LMT upper, and a Noveske lower for now, but it will be the LMT SOPMOD later when the Noveske goes to the .300BLK I'm building. Nothing wrong with it, but it just has more "value" if most of the major parts match. Anyway, to say that Oly M4 was anywhere close to this M4 I have now would be a grossly uninformed claim. If informed, then just stupid. The Oly M4 was a decent shooter, maybe a good truck rifle. The M4 I have now, it is one of the best, maybe THE best I've ever had, and I've had a few.
Lots of ways you can handle this really. These are just my opinions based on how I'd do it or how I've actually done it. Take it for what it's worth. Good luck!