What I generally prefer is to use stacking headers. Adafruit sells 2 14-position stacking header + 1 5-position stacking header that is just right for the Teensy (or the Adafruit Itsy-Bitsy which uses the same 14-pin side pins + 5-pin rear layout as the Teensy 4.0). For breadboard and prototype usage, you would use the 2 14-position headers, and not use the 5-position header (or use the 5 position header, and cut off the extenders, to convert it to a normal 5 position female header). At the moment, Adafruit is sold out, but Digikey has some:
The nice thing with stacking headers is you can stack multiple boards.
Alternatively to the 14-position stacking headers, you can get longer stacking headers and use diagonal cutters to cut the header after the 14th pin. I generally first pull out the 15th pin, and then cut in the empty space. Then I use a rotary tool like a dremel to file it down. Every so often, I cut it too close to the last pin, and I have to throw out the header, and make another.
Note, Sparkfun sells a stacking header set for the Teensy 3.2/4.0, but it is not useful for the Audio shield. This is because the Sparkfun headers are 2 13-position headers plus 1 7-position header. Unfortunately, the 5 rear pins are not in the audio shield.
If you don't need to separate the Teensy and the Audio Adapter, you can get these pins from PJRC that allows you to solder the Teeny on the top, and the audio adapter underneath (or vice versa):
Or alternatively, get longer male pins, solder one board at the top, and the other in the middle, and you can put it in a breadboard or prototype board.
Here is a picture of a Teesy 3.1 with the old Prop Shield, both with stacking headers attached: