UART Serial may be more competitive with SPI than you're assuming.
On Teensy 4.1, the maximum baud rate is 1/4th of the UART's base clock, which defaults to 24 MHz. But the hardware can be programmed for an 80 MHz base clock, so at least in terms of raw hardware support in the chip, UART baud rates can reach into the SPI range. Even though not as high as the 30 MHz SPI maximum (or maybe 50-60 MHz if you don't care about NXP's official specs), each UART port has dedicated bandwidth. The total available bandwidth can actually be higher with serial, unless your design artificially limits communication to happen only on 1 port at a time, as you would be forced to do with SPI.
Serial gives you 2 important advantages over SPI.
#1: Wiring is point-to-point and high speed RS422 transceiver chips exist to make it very reliable over longer distance. With SPI, you must connect all devices to the same 3 wire bus, plus chip select signals. When you scale up to several devices and you wish to run at high speed, signal quality becomes much harder, especially if the devices are separated by any substantial distance. The asynchronous nature of serial means you can easily add buffer chips, but with synchronous SPI any buffer chips eat away at your timing margin, which is already tight if you're running near the maximum speed.
#2: Serial provides hardware flow control with RTS/CTS signals, and the existing software gives substantial buffers for transmit and receive. Perhaps you will write all of your code so efficiently that you won't need flow control or much buffering. But that is a difficult software timing challenge, especially if your code may be expanded or maintained in the future. RTS/CTS flow control makes the software side much easier and far more robust to future design changes which alter the overall timing of receiving data. One minor gotcha is you'll need to use the XBAR pins for many CTS signals, which are inverted from normal CTS, so you'll need to add hardware inverters to make it work. Still, I would highly recommend using RTS/CTS flow control, unless you know your communication will always be so very simple that your software complexity will remain very low.