Pretty sure the chip supports two USB channels. Only one of them is high speed though.
I assume you mean you want to be USB host to two devices? Really not sure what you mean by 4in 4out??
You can stack two audio adapters on a single Teensy for 4 audio inputs at 16 / 44 and 4 audio outputs at 16 / 44. That gives you your 4 in 4 out. I have not tried it yet though, nor have I spent much time with USB audio, but the bandwidth is sufficient so it is possible.
Starting with the Audio adapters is a good idea; it'll make it easier to get you're project going. You'll be stuck using something like I2S becuase you dont have enough physical DACs for four channels. Note aparently four channel audio has not yet been tested with teensy 3.5/6.
You've plenty of usb data, I imagine the bottle neck for fidelity will be the I2S chips you use.
If you haven't the audio GUI is a great place to start; it also has helpful notes explaining the limitations of different libraries (can't operate in both master and slave I2S at once for example). If you're wanting the highest fidelity audio, would you consider using a pair of teensys? (maybe 3.2 to drive the price down)
Eight simultaneous channels (4 input and 4 output) at 44.1 kHz 16 bits would use 706 kbytes/sec. That's most of the isync (host controller's periodic schedule) bandwidth of 12 Mbit/sec USB. In theory, it ought to work. But you can forget about 24 bits on all 8 of those channels! There just isn't enough bandwidth for more than 16 bits.
Secondary USB: The Teensy 3.5/3.6 have provisions for a second USB channel that can act in host mode (and the 3.6 can run at 480 Mbit/second) -- this of course depends on other software being written to work with the secondary USB;
Would that still have the bandwidth limitation? Can we use USB 2.0 "High Speed" mode with the Teensy 3.6?
The 480 Mbit/sec port would not have the 12 Mbit/sec speed limitation.
But there's currently no working software support. In theory, you can use it... but at this early stage "can" involves writing your own USB stack. Since this is a very complex modified EHCI controller, that's no small task!
A stable and usable library for the EHCI controller is going to take time. EHCI is powerful but very complex.
Even when stable EHCI support is available, whether USB audio, and multi-channel USB audio support is written remains to be seen. Many other things will be much higher priority, so unless someone (other than me) takes a particular interest in actually doing this work, odds are slim it'll happen anytime soon.
Did you ever actually use USB audio on Teensy with the Design Tool and Arduino sketches?
I'm curious to hear how XMOS's system compares...
Higher channel count support isn't going to exist for a long time, unless someone else steps up?