I see from this that the each step in an audio chain should take about 2.9 milliseconds.
This is regardless of how fast or how little the Teensy does the work.
It will wait 128 samples, before updating again. And if the work is not ready it should cause clipping like it would on a pc daw.

I read the average audio hardware latency to be about 7ms.

So that 2.9*3 = 8.7 ms
counting on the input and output adding delay, I am guessing 10ms total would be about right. I should consider that the upper limit of whats acceptable.

The instructions however do offer up the option to lower the samples from 128 to some other number.
Maybe this number can be arbitrary but maybe not. Considering it is a power of 2 I am guessing not.

Changing it to 64 samples per update should shift the update time to 1.45ms. So a theoretical 6 steps in the audio chain for the same time period.

32 samples seems less plausible but it would allow 12 steps.
This assumes that the teensy is able to do the work in that time. Load times would probably also become more of an issue. And it is probably less efficient in all sorts of ways, otherwise lower sample size would be standard.

Anyway is this roughly accurate?

The other question I have that is slightly related is, how much latency is added from sending i2s from one teensy to a second teendy? I am guessing it is something like 3-5ms. And what about teensy sending i2s to the dac?