If you use Teensy 4.0, and if you get the latest audio library code from github (or wait a few days and get 1.52 beta which will have it), and if you're able to work from help here and comments in the code (but not yet nice documentation written on the website or design tool) then you could connect 3 audio shields. You could even connect 4 or 5 of them for 8 or 10 channel input, if you don't need all of them to have working outputs.
Connecting more than 2 audio shields (Teensy 4 only) also requires special with for the I2C communication to control them. Only 2 can connect to the same I2C bus (SDA & SCL pins) to configuration setup, because the SGTL5000 has only a single I2C address pin. The audio library's code to configure the hardware assumes they're connected to the main I2C pins (18 & 19). You'll need to copy & modify that code to configure the 3rd audio shield.
On Teensy 4.0, the built-in ADC pins are not yet supported for audio input. They can only be used for pots. They do work for (lower quality) audio on Teensy 3.x, but using the ADC pins for audio means the ADC is always busy for audio signals and can't be used for reading pots.
With either Teensy 3 or 4, you could use TDM instead of I2S, and connect a CS42448 chip which gives 6 inputs. TDM support many more channels than I2S.
https://oshpark.com/shared_projects/2Yj6rFaW
(click the "Description" button on that page to see all the info)
This might be the best solution for you. The only download is currently nobody sells a CS42448 shield. It's available only as a DIY soldering project.
On Teensy 4.0, it's even possible to connect two TDM chips...