It sounds like the bolt is trying to extract before the brass has shrunk away from the chamber walls. As a result, the brass is still "sticky" and your extractor is slipping off the rim. The rifle works for the first magazine because there is less heat build-up so the brass shrinks away from the chamber faster. As the heat builds up, this process becomes less efficient.
There are two basic issues here:
1. Your extractor is too weak for the current timing of the rifle.
2. Your timing may be less than the optimum.
First step I would recommend is buying a $6 extractor upgrade kit. This will give you a stronger extractor spring and a O-ring to help prevent extractor lift. You may also want to look at the extractor itself and make sure it looks clean and in good shape - not rounded or worn.
Second step would be the timing... typically, timing issues are a result of improper gas port and/or chamber specs for the type of ammunition being used. I would bet a few gallons of gas that your chamber is not cut to 5.56 NATO specs. If not then, hitting it with a 5.56mm chamber reamer will improve reliability but may reduce accuracy.
An easy way to help with timing issues is to use a heavier buffer and/or a stronger action spring. Given what you are describing here, I'd try an H2 buffer, which should run under $40.
If you are on a budget and have the time, I'd try the extractor upgrade first. If you continue to have problems, go with an H2 buffer. If you still have problems at that point, you'll want to gauge the chamber and mic the gas port to get a better idea of what is going on. Alternatively, since it is a new AP4, you can just send it back to DPMS and let them deal with it - just be sure none of your fixes void the warranty if you decide to go that route.
