Okay, two better workarounds have been established.
Again, restarting PulseAudio does work, the issue is that this also disconnects any playback sources, so they have to be restarted (browsers, Spotify, etc); very inconvenient.
Workaround 1 unloads every sound card and then reloads them all...