I2C problems with Teensy 3.6 and WM8731 codec

Status
Not open for further replies.
lol, of course it's fun. but it requires fiddling as soon you change a little detail..
Maybe I play with my cs42448 and use longer wires to see if I can reproduce the problem. Have to read the datasheet first, if it is a "slow" I2C device, too... It would be a chance to use my new o'scope :)

The words "longer wires" are crucial - there it is where the root of all problems lies. The I2C standard had been invented by Philips to allow communication between ICs which would all sit on the same PCB or on different PCBs with a fixed and defined wiring, i.e. inside a TV. That was in times where 100kHz was still respectfully considered being RF or at least "high frequency", and where everything beyond 1GHz was still in the metaphysic domain. I'm pretty sure that the Philips engineers didn't think of hobbyists breadboarding i2c communication with "flying" Dupont wires, sometimes adding or removing ICs to or from the bus, and all that besides MCUs which are now clocked much higher than the "usual" 4MHz...
 
I've been working on this problem for the past few days. I bought a cheapo logic analyzer ($12 on Amazon) and it has been amazingly helpful in working out exactly what is going on!
Here are my conclusions:
1) With the resistors the unit is rock solid with the Wire library (with the exception below). I have run it for 24+ hours without error on its own with a Teensy 3.2 and a Teensy 3.6, and also when it was sharing the I2C bus with a SI5131A clock generator that was continuously generating a frequency sweep from 10 to 15 MHz - thus keeping the bus busy. No problems.
BUT
2) with the Audio library I2S functions AudioInputI2S and/or AudioInputI2S active the results were a very mixed bag. With a simple sketch of an audio I2S pass-through, such as I posted in the first post of the thread, there were occasional NACK errors. Now the AudioControlWM8731 control does not recognize the I2C status reported by Wire.endTransmission, and any hard error can cause failure. So I modified and included the BALibrary, which repeats a I2C writes until the transmission is error free. This can repair occasional errors, and with this the pass-through code works most of the time.

However, with a more complex Audio library based program (such as my SDR and music software) the WM8731 code generates NACK errors on EVERY transmission!!! This is where the logic analyzer became invaluable. You can see the lack of ACK in each data packet. Go back to a simpler program and the problem disappears. So I removed Audio blocks one at a time and found that inclusion of objects AudioInputI2S or AudioOutputI2S caused the problems when the codec.enable() function is called.

Then I stumbled on another thread in the forum about exactly the same problem with the AK4558 codec here here. They also found the AudioInputI2S and AudioOutputI2S objects to cause problems with I2C. The conjecture there is that the fact that both include a call to begin() function in the class constructor is the culprit, and Blackaddr suggested taking the begin() function out of the constructor and forcing the user to call begin() after all initialization is complete. Unfortunately the thread stopped at that point.

So I made my own version of those two Audio objects with empty constructors. The begin() function in each is already declare public. I modified my code to call codec.enable() near the begining of setup() and at the very end called myInput.begin() and myOutput.begin() at the very end of setup(). And that fixed it!!! The logic analyzer showed all NACK errors disappeared and the system worked.

I have no idea what the interference between the I2C and the audio blocks is, or why the SGTL5000 is robust to the problem. Apparently it's not unique to the WM8731, it seems that the AK4558 has the same problem!
 
In the thread linked above, I had reported a trick for disabling and renabling the BCLK of the I2S bus. You can disable the bus, do your I2C config, then renable the I2S bus.
Code:
// On the T3.6:
CORE_PIN9_CONFIG = PORT_PCR_MUX(1); // disable the clock to stop the I2S
// ...configure the codec...
CORE_PIN9_CONFIG = PORT_PCR_MUX(6); // turn the I2S clock back on

Alternatively if you do not need I2C and I2S running concurrently, then using custom I2S objects to defer their startup can solve the problem as well.

For me, this was not acceptable because after the codec is configured and audio is running, I still need I2C to adjust codec parameters like headphone volume so I had to go the route of retry-on-failure. If you need simultaneous operation of I2C and I2S on the WM8731 and failures (and thus retries) were not acceptable, then I never found a solution using only typical pullups.
 
I'm working with this today, using a Teensy 3.6 connected to the MicroE-506 board, which has a mic input but the line in signals aren't connected on that board.

I can sometimes reproduce the problem. Here's the code I'm running, which plans an octave scale of beeps. No input signals needed.

Code:
#include <Audio.h>

AudioOutputI2S      out;
AudioSynthWaveform  wave;
AudioConnection     c1(wave, 0, out, 0);
AudioConnection     c2(wave, 0, out, 1);
AudioControlWM8731  codec;

float freq;

void setup() {
  AudioMemory(12);
  codec.enable();
  codec.volume(0.5);
  wave.begin(WAVEFORM_SAWTOOTH);
  freq = 220.0;
}

void loop() {
  wave.frequency(freq);
  freq = freq * 1.08333333;
  if (freq > 440.0) freq = 220.0;
  wave.amplitude(0.95);
  delay(250);
  wave.amplitude(0);
  delay(1750);
}
 
Paul - Blackaddr and I found that the problem was much, much worse when I2S input/output is running... Like you I was able to run Audio stuff - with a few errors - without I2S...

The reason for trying the WM8731 was that the SGTL5000 Audio board is still driving me crazy (after 2+ years) with random one-sample delays between the L/R channel in I2S inputs. It's not noticeable in normal music/audio work. This problem occurs either on program upload or power on/off (more prevalvent on upload) and once the problem establishes itself its there until a reboot. It completely destroys phase integrity in quadrature signal processing. Others have had the same problem in the context of SDRI submitted a bug report at the time, with no response. My conjecture is that the L-R frames in the I2S packets are getting confused either in the codec, or in the DMA software. I thought I'd try the WM8731 to see if I could help isolate the problem.
 
Hi Derek,

unfortunately I can confirm that the WM8731 (together with an STM32F4 / STM32F7) has exactly the same problem of an occasional random one sample delay. You are right, that is not very nice in the context of processing I & Q audio data, which has to be exactly 90 degrees out of phase . . .

We called this phenomenon the "twin peak", because in the context of Software Defined Radio, you can observe a second peak in the spectrum display whenever this one sample delay occurs in the codec.

https://github.com/df8oe/UHSDR/wiki/IQ---correction-and-mirror-frequencies

The phenomenon is thus NOT restricted to a single model of codec. Additionally it is NOT restricted to a single production batch of codecs of the WM8731, because users of our SDR software have observed this phenomenon in several HUNDREDS of codecs of different age and origin.

Bob Larkin has found a nice solution using cross correlations (reported here in the forum) to easily measure the phase difference between I & Q and -whenever this phenomenon occurs- to reset the codec until the problem is gone.

https://forum.pjrc.com/threads/4233...ime-processing?p=175849&viewfull=1#post175849

I have used a somewhat different approach and built in an algorithm by Moseley & Slump to correct I & Q phase and amplitude imperfections which simultaneously allows to check the correct 90 degree phase difference.

We have not found any other means to circumvent this problem than to reset the codec and the hope that the delay is gone . . .

All the best,

Frank DD4WH
 
After much fiddling, I'm pretty sure the problem is the WM8731 chip appears to be highly sensitive to ringing on the MCLK signal, maybe other pins too. Teensy 3.6 appears to have *much* higher bandwidth drive than Teensy 3.2. The result is pretty bad signal quality problems on MCLK.

To cut to the chase, adding a 220 ohm resistor seems to "fix" the problem.

DSC_0381_web.jpg

You can see in this photo, I've put the resistor in series with the MCLK signal. I also added a ground wire, which allows me to touch my scope probe without the normal (~3 inch) ground clip to get a good high-bandwidth measurement of the signal.

Here's how MCLK looks without the resistor.

file.png

With the 220 ohm resistor, things look much better.

file2.png

The I2C communication works much better when the WM8731 chip gets this clock signal, but still not as good as Teensy 3.2. I suspect there may also be ringing problems on BCLK, LRCLK and TX...
 
Personally I find this signal very strange for open collector logic. It is hard to believe that a pullup could produce an overshoot of what appears to be a little over 1 volt. And viewed from a distance, it looks like the rising edge ( produced by a passive pullup ) has a faster slew rate than the falling edge ( active driver ).
 
That's MCLK, not SDA or SCL. MCLK is a normal digital push-pull output. So are the other I2S outputs, BCLK, LRCLK & TX. I2S never uses open collector signals like I2C does.

I spent quite a lot of time looking at SDA & SCL, thinking perhaps Teensy 3.6 was doing something wrong on I2C. After much digging into this, I'm convinced that's not the case. Ultimately I ran Teensy 3.6 at 96 MHz, the same speed as Teensy 3.2. Both transmit virtually identical SDA & SCL waveforms. I even imported scope screenshots and overlaid them in the Gimp. The waveforms the WM8731 is receiving are identical. Yet it almost always (but not 100%) responds with ACK in the 9th bit for Teensy 3.2. After receiving the identical waveforms, WM8731 much more often fails to pull SDA low in the 9th bit. Seeing that finally convinced me this I2C problem is entirely on the WM8731.

I did put some code into the WM8731 control to automatically detect the problem and retry. It recovers from most of the problem, but with MCLK connected directly there still seems to be some case where the WM8731 gets into a weird mode that doesn't work. I'm not sure if there's anything that can be done about that.

Planning to clean up the auto-retry code and commit it soon. Don't know if there's more I can do here. It's clearly a hardware signal quality problem, where WM8731 appears to be quite sensitive to any ringing or overshoot on the I2S signals. That manifests as the chip not properly responding on I2C, but I'm convinced the real problem is WM8731 isn't internally clocking properly.
 
Ok, thanks for the clarification. That makes more sense. Looking around the web, it seems everyone has issues with I2C with this part using FPGA, or Raspberry PI, or PIC32mx.
 
I've committed code on github to automatically retry when I2C communication fails.

https://github.com/PaulStoffregen/Audio/commit/828ea4a74c3b4db06261a4598fa6bb1bcf4f9787

This will be in the next 1.46 beta, and the upcoming 1.46 release.

I also put in the previously commented reset command. I found that after many times rebooting Teensy (but not power cycling) the WM8731 can sometimes get itself into a non-working state. The WM8731's I2C reset register seems to be able to get it back to a working state.

Here's the code I've been using for testing.

Code:
// https://forum.pjrc.com/threads/55334?p=201494&viewfull=1#post201494

#include <Audio.h>

AudioOutputI2S      out;
AudioSynthWaveform  wave;
AudioConnection     c1(wave, 0, out, 0);
AudioConnection     c2(wave, 0, out, 1);
AudioControlWM8731  codec;

float freq;

void setup() {
  AudioMemory(12);
  while (!Serial) ; // wait
  Serial.println("WM8731 & I2C Test");
  codec.enable();
  codec.volume(0.5);
  wave.begin(WAVEFORM_SAWTOOTH);
  freq = 220.0;
  while (freq < 440.0) {
    wave.frequency(freq);
    freq = freq * 1.08333333;
    wave.amplitude(0.95);
    delay(150);
    wave.amplitude(0);
    delay(20);
  }
  delay(500);
  SCB_AIRCR = 0x05FA0004; // software reset!
}

void loop() {
}

Without the WM8731 reset command in enable(), and with Teensy 3.6 MCLK directly connected (no 220 ohm resistor), I got the WM8731 into the locked up state after about 11 minutes, then no amount of re-running the program would work. Adding the WM8731 command back into the enable() function immediately recovered the locked up WM8731 without needing a power cycle. I'm running this now, so far about 20 minutes and it's still going. Will leave it running for a few hours, before I put all this WM8731 hardware away....

The code retries up to 12 times, which seems to cover nearly all errors. But in this auto-repeating test, I have seen a couple times were even 12 retries was not enough. :(

I'm afraid this is about as good as things are going to get with WM8731.

Anyone using WM8731 in master mode should add a resistor or maybe other circuitry so WM8731 gets a nicely filtered signal more similar to what it expects from the crystal, especially with Teensy 3.6 where the digital signals have far more bandwidth with very fast rising and falling edges. The number of I2C errors is definitely related to the ringing & overshoot received on MCLK at the WM8731 clock input pin.
 
Paul -
I'm delighted to report that your efforts and suggestions have been 100% successful (with the Teensy in Master mode)! After adding the 220R resistor to the MCLK line, and using your updated control_wm8731.cpp, I have run an I2S pass-through test on 3 separate T3.6s and a T3.2, with two different WM8731 chips in every combination of chip and Teensy without a single error :D - except when I attached another I2C device (an Adafruit SI5351 clock generator). In this case I occasionally had up to 5 write attempts before success, but never a complete communication failure. I've had a sketch running for 12+ hours, changing the codec input gain every 500ms, and no reported errors.

Great work and thank you,
Derek
 
Status
Not open for further replies.
Back
Top