It depends on the shooter, gun, and the situation.
I have a 1911 with a great feeling trigger, but I will pick my M&P 40 for combat.
There is not much of a speed difference in how fast I can pull the trigger between the two.
Sure, the 1911 have a lighter trigger, but that does not mean I can just jamm on the trigger and expect hits.
Now, that does not mean I think M&P is the "fastest pistol." I am not aware of any pistol that cannot keep up with a speed of human trigger finger speed, although I do think heavy and long DAO trigger will slow a shooter down the most.
People who advocate manual firing inhibitor like to say it does not slow them down because the manipulation is done during the drawing motion or transition from ready to firing position.
Problem with that argument is that it is not always true.
Each of them have different doctrine of how manual firing inhibitor is used, and depending on the doctrine it can be a problem , especially in regards to situations where a user has to frequently switch between engaged and disengaged position in quick succession.