Hey Leeman,
your polling rate will depend on the way you wire up your switches and how you code the polling.
I would not worry too much about that now. I built a Midi Keyboard with 44 Keys, using an Arduino Leonardo (also a 32u4 chip, like the Teensy 2). I wrote a rather simple program using two nested for loops and digitalRead() and digitalWrite(), and I did not notice any Latency.
I would start by building a test circuit with just a few switches and similar simple code and test how fast it is. If it is too slow for you, then you can optimize your code, e.g. by manipulating the port registers directly.
Regards, Flophi