The double buffering design, in hindsight, was a mistake on my part. Eventually it should get removed from all the output objects.
In the earliest days of the audio library, back in late 2013, I was overly cautious about so many things. Double buffering seemed like a good idea. And back then, so many other aspects of the system weren't stable, so some mistakes I made with other things seemed at the time to confirm the need for double buffering.
In fact, double buffering isn't needed at all. Usually isn't never even used. Typically the 2nd pointer always remains null. So AudioMemory(1) might work on Teensy 3.x & 4.x too.
I have known about this for years, but the only harm is a few extra bytes of memory and some unnecessary code, so it's been pretty much at the very bottom of my long list of issues. The only case where double buffering does come into play is when something anomalous happens (exactly what, I haven't investigated... remember, low priority issue) and the 2nd pointer ends up getting an allocated block. From that point forward, the interrupt always end up using data from the 2nd pointer. So the net result is an extra 2.9ms output latency is added forever (until reboot). If that anomalous thing happens again, you get an audio dropout or repeat, because there's no 3rd buffer pointer.
So if anyone cares about this, I would love to merge any well tested pull requests which remove the 2nd buffer. But please don't send a PR which hasn't been tested on real hardware. The last round of changes for cache management added bugs which made their way to users, because not all were tested. I want to avoid that when/if the buffering is changed.
@Frank - if you need any hardware for testing, just email me and we'll get you whatever you need. (I might have new CS42448 boards in the works....)