What type of front sight are you using? Is it a standard fixed front sight or one that mounts to a handguard?
Usually if you have to have the rear sight to one side or the other, then the fixed front sight is canted (if equipped). If you are using a front sight that mounts to the handguard then there is a good chance that the handguard is not installed true to the upper receiver rail.
To answer your question, as long as you can zero it and then verify that it is on target at 300 meters/yards, then there is nothing to worry about.