I forgot about the bolt myself. Sometimes the retaining pin is hard to remove. A punch and a hammer should do the trick though. If you drill a hole in a board, you should give the retaining pin enough room to start sliding out.
Oh an addition to this, whatever you do, do not oil the pin! Just...