Thanks for the comments, I ordered a Teensy 4.1, so I should be good to go! I'll probably try out those mods to make it more efficient at a later stage.
After a week of debugging some odd behavior I'm just now realizing how CPU intensive the ladder filter emulation is!
I'm using a Teensy 3.5, and I boiled my code down to a very simple 3-voice poly synth with output to the onboard DAC (very...