I had the buzzing sound on headphones too after wiring everything together on the breadboard.
My setup added another SPI device in addition to the audio board by directly translating the 3.3v...