Sorry for yet another reply, but I was able to get the teensy working with the SD card, but only by using the breakout board. I just wired it up to SPI0 (figured out that it wasn't on SPI2, as you can tell) and it worked.
Moving the card to the internal SD card slot fails with the 0x20,0xFF...