What SD card are you using? Good concurrent playback requires a good SD card. I have been using Lexar 1000x 32GB and playing 3-4 samples concurrently is possible. The Lexar 1800x cards would be even better, but they're expensive.
Also, I would recommend Greiman's SDFat library and using SDIO and DMA if possible. SDIO and DMA support haven't been added for Teensy 4 yet though.