It should be fine to use a slower I2C clock.
You could also slow the I2C clock after the audio board is up and running. I2C is only used for changing settings (like volume), so if you don't call any of the sgtl5000 control functions after it's running, there is no more I2C communication to it.