Search:

This is how I allocate my poly-synth
https://github.com/manicken/teensy4.0polysynth/blob/ba1289cf68112bb3cd0cc6b4471ddb2dce836c95/src/Synth.h
line 88-122
the sustain part is
line 123-139

then...

but then when I look at the AudioSynthWaveform source code that should not matter,
as the begin function don't do any special things except setting the values,
which if not changed should not...

Aren't doing Osc1.begin in the main loop a little to fast changing/starting the waveform?
You should only call the .begin function once when the waveform change command is received.

How do you do the note on/off?
Are you using a AudioEffectEnvelope otherwise try that.