I've tested a few thousand direction reversals with my Avago optical encoder (brushed motor) and haven't had mis-counts (the encoder has an extra index channel, so it's easy to verify).I guess I was mainly wondering whether in some circumstances, encoder jitter (as described on pg 811 of the MK20DX128 manual) might affect the accuracy of the encoder library count, even if the code is working as intended - as I think you've clearly demonstrated. In other implementations of quadrature decoders, I've usually seen some kind of input filtering (eg. the digital filter on pg 750, or the RC filter used in the Microchip app note). It wasn't entirely clear to me whether this jitter would just be reflected as a fluctuation in the count, or whether it would actually cause a mis-count, i.e., to lose the actual position.
From what I understand about the Cortex M3/M4 interrupt system, you will get an interrupt for both pulse edges, even for an extremely short pulse (too short for the interrupt to finish processing the first edge). So unless you have noise on encoder channels A and B at the same time, there shouldn't be any count error (the short pulse will simply be an up/down fluctuation in the count).
So, if you get a reasonably clean signal to the Teensy, you should be just fine.
@Will: I think at this point it's quite clear that your hardware setup is screwed up. Looking at Pauls video, the AMT encoder puts out a very clean signal and the T3 can definitely count correctly.
Brushed motors can generate huge amounts of noise. You should get out a scope to see what's going on with your setup.