The upper (as of my knowledge) is infact what determines the caliber. The .204 and .223 use the same bolt face, mags, and therefore should be able to use the same lowers. I know that you can get an upper in .204 for your .223 ar, so I am pretty sure you can get it the other way around as well.
The only lowers determined by caliber are those of short action catridges like the .243, .260, 7mm-08, and .308. These use a much larger mag well as opposed to the .223 cartridge.