The problem went away, as soon as I removed the delay from the design.Although the USB had no real problems (besides the usual clicks), the SPDIF went bonkers. Without delay: no problem. Thanks Frank.
(I'm aware,that my information was a little vague, but I didn't want to paste 30000 lines of code here). My program is not only a 6 voice synth, but also a sophisticated MIDI controller.)
Anyway, good to know, it's not a HW issue.