I haven't got the hardware to test myself at the moment but only AudioMemory(8) is jumping out at me. Try something higher like AudioMemory(20).
If it is a memory type issue, maybe check out Examples > audio > MemoryAndCpuUsage
What Teensy board and CPU speed are you using?