Why not use multiple Teensy and audio boards?
For this project, wouldn't be more appropriate to use 3 sets of Teensy + audio board, keeping them in sync with digital communication, maybe with a fourth Teensy acting as master for the cluster? Considering the price of the boards it would not...