It all depends on how many channels of audio you want to transfer!

Using the standard 48kHz / 16bit format of the audio library, you would need at least 1MHz bit rate per channel (~768kHz + some overhead for packet framing, start and stop bits, etc).

Paul lists up to 4MHz transfer rates for T3.6, so I'd assume the T4s can handle this, also.

You'd have to re-write the physical layer as UDP/IP is quite different (packet vs stream) from serial. Serial should be fairly easy to work with.

I'd advise providing transmit/receive buffers of at least two full packets, so that you don't have to deal with buffer overruns. And buffers add latency...

see Serial1.addMemoryForRead(buffer, size) https://www.pjrc.com/teensy/td_uart.html

I hope this helps.