I2S input synch error still exists on T4...

Status
Not open for further replies.
D

DerekR

Well-known member
For a couple of years now I've posted about the "bug" where the two I2S channels randomly are out of synch with one another by a single sample. This apparently is a problem on many MCU's, so it's not a specific Teensy phenomenon. It is a huge problem for me however, because a lot of my work involves quadrature signal processing, and the bug induces a frequency dependent phase shift that completely destroys the algorithms involved. You just never know what you are going to get when you turn the power on... So, I was hoping like crazy that the T4 just might solve the issue...

But no such luck :( I finally got around to wiring up the T4 and Audio board :eek: Everything is working (so far) so I connected an external white noise generator to the I2s inputs, and ran the cross-correlation sketch I used to test the same problem on the T3.6. It shows exactly the same random behavior on power-up and program upload.

So sometimes you win, and sometimes you lose. It's my turn to win for a change!
 
DerekR said:
For a couple of years now I've posted about the "bug" where the two I2S channels randomly are out of synch with one another by a single sample. This apparently is a problem on many MCU's, so it's not a specific Teensy phenomenon. It is a huge problem for me however, because a lot of my work involves quadrature signal processing, and the bug induces a frequency dependent phase shift that completely destroys the algorithms involved. You just never know what you are going to get when you turn the power on... So, I was hoping like crazy that the T4 just might solve the issue...

But no such luck :( I finally got around to wiring up the T4 and Audio board :eek: Everything is working (so far) so I connected an external white noise generator to the I2s inputs, and ran the cross-correlation sketch I used to test the same problem on the T3.6. It shows exactly the same random behavior on power-up and program upload.

So sometimes you win, and sometimes you lose. It's my turn to win for a change!
Click to expand...

Is there source code somewhere for the audio drivers ? Point me to it and I'll see if there's an obvious fix. Otherwise I'll be writing my own (which I may do anyway as somebody said it has 2.2mS of latency due to it sampling large blocks rather than using a normal async FIFO).
 
DerekR said:
For a couple of years now I've posted about the "bug" where the two I2S channels randomly are out of synch with one another by a single sample.
Click to expand...

Are we talking here about 1 I2S stereo channel or 2 I2S data channels (two audio boards)?
I assume 1 I2S stereo channel

What do you mean out of sync by one sample?
- are channels switched, or is say right channel missing 1 sample, or is left/right not deterministic?

- how do you use the I2S, with the audio library or by own program?

I connected an external white noise generator to the I2s inputs
Click to expand...
I assume, you mean the two line-in ports of the audio-board and not the two I2S data ports of a Teensy.

The I2S L/R protocol is rigid and it is not clear how data scramble can occur.
Only reason, I can see is that the clocks (MCLK, L/R , Bit) are not clean (interferences (i.e. MCLK coupling into L/R), reflections (long wires)).
If the clocks are not clean, every thing can happen.

But this is not a "bug" with teensy or audioboard, but with the setup.
 
WMXZ said:
Are we talking here about 1 I2S stereo channel or 2 I2S data channels (two audio boards)?
I assume 1 I2S stereo channel

What do you mean out of sync by one sample?
- are channels switched, or is say right channel missing 1 sample, or is left/right not deterministic?

- how do you use the I2S, with the audio library or by own program?


I assume, you mean the two line-in ports of the audio-board and not the two I2S data ports of a Teensy.

The I2S L/R protocol is rigid and it is not clear how data scramble can occur.
Only reason, I can see is that the clocks (MCLK, L/R , Bit) are not clean (interferences (i.e. MCLK coupling into L/R), reflections (long wires)).
If the clocks are not clean, every thing can happen.

But this is not a "bug" with teensy or audioboard, but with the setup.
Click to expand...

Whilst I didn't report the problem, I have seen the same effect if the audio driver uses separate FIFOs for left and right. The sync signal gets delayed slightly between generation and output and the DAC driver takes the left sample from time T and the right sample from T-1 or T+1.
 
PaulStoffregen said:
Just to make sure this earlier question is answered... Yes, here is the source code for the I2S output.

https://github.com/PaulStoffregen/Audio/blob/master/output_i2s.cpp
Click to expand...

Thanks Paul. As I suspected you have used separate left and right buffers. I haven't dug into the code yet to see if that's definitely the problem, but I've always found having a single buffer with left in 0,2,4.... and right in 1,3,5... avoids this problem once and for all. Alternatively if you need separate buffers, for example if performing live reverb so you want to minimize latency, you first write known patterns into both L&R two buffers and allow the output driver to get itself into sync.
 
Thanks for all the answers, guys.
@WMXZ
Are we talking here about 1 I2S stereo channel or 2 I2S data channels (two audio boards)?
I assume 1 I2S stereo channel
Click to expand...
We're talking about two inputs on the line-in of the Audio Board.
What do you mean out of sync by one sample?
- are channels switched, or is say right channel missing 1 sample, or is left/right not deterministic?
Click to expand...
The whole data stream of one channel is slipped by one sample with respect to the other, there are no holes at the beginning of each Audio Block. I have looked at whether LRLR... order in the I2S sequence is flipped (its very hard to tell on a scope or logic analyzer) and I really couldn't tell - but I have looked at whether it is a complete channel swap, and its not.
how do you use the I2S, with the audio library or by own program?
Click to expand...
Audio Library - but also using my own library objects.

But this is not a "bug" with teensy or audioboard, but with the setup.
Click to expand...
I agree. But it is a real bug and regardless of who is responsible it is still a bug.

@defragster
Just checking with WMXZ's clock comment … Did "wiring up the T4 and Audio board" include the 100 Ohm resistor inline with MCLK?
Click to expand...
Absolutely did!

Some general background. This is not just my problem - it has been widely reported by others outside the Teensy community. On this forum there is a discussion here. We originally thought it was a codec initialization problem, and I spent weeks dissecting the SGTL5000 code could not find anything. So in desperation I built a Audio Board pin-compatible WM7831 codec, and after a lot of problems with that chip's I2C hardware (which Paul eventually solved), lo and behold the same old problem was still present. So that eliminated the codec as the issue!

My feeling is that it may be an ARM related issue... It seems that the system randomly gets caught in one of two states on program upload or power-on. The question is whether it is hardware or software?

It would really help if others would demonstrate the problem for themselves using the sketch below. I have used this with the Teensy 3.2, T3.6, and T4.0 all with the same results. The sketch generates a white noise output on the Audio Board line-out channel 0 (left). Hard wire that signal back into BOTH line-inputs chancel 0, and channel 1. The sketch computes the cross-correlation function between the two channels and prints out the results on the serial monitor. If all is well it should print out a lag of zero samples between the channels, but I think you will find that on program load it reports a lag of -1 about 50% of the time, and on power-up about 30% of the time. (I have never seen a lag of +1).

Code: 
//-------------------------------------------------------------------------------------
//                                 *** TestXcorr.ino ***
//
// Sketch to demonstrate Audio board I2S channel synchronization problem.  Computes the
// cross-correlation function between the I2S inputs at lags -2,-1, 0, 1, 2 samples,
// for complete blocks, and prints the lag with maximum correlation, and the values of
// the xcorr estimates for each lag.
//                           --------------------------
//    The cross-correlation function is a statistically optimal measure of
//    the delay between two similar signals f(t) and g(t) that differ only in a delay
//    (tau) between them.   It is a function of the delay, and always has a peak
//    value at tau.   For a discrete time sampled system, the delay is measured 
//    in samples. It has a very simple definition:
//             psi(M) = sum {f(n) * g(n+M)},
//    where the summation is over a finite data block, and where M is the delay.
//    psi(M) will be a maximum at the true delay M between the sample sets. 
//    The value of M

//                           --------------------------
// Input:  Connect both line-inputs (I2S) together, and drive with a common 
//         wide-band input.  
//         1) The sketch generates a "white noise" on I2s line-out channel 0 (left).
//            You can connect the two line-in channels to this output, or 
//         2) You can use an external wide-band source such as a function generator.
// 
// Results: If the input channels are in sync the software will report a lag of 0,
//          however I find that
//          1) on program reload it randomly reports either 0, or -1 with 50% probability
//          2) on power-up it reports 0 with about 70% probability, -1 with 30%.
//          3) I have never seen a report of a lag of 1.
//
// Author:  Derek Rowell
// Updated: July 1, 2017
//  
//-------------------------------------------------------------------------------------
#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SerialFlash.h>
#include "AudioXcorr.h"

const int myInput = AUDIO_INPUT_LINEIN;
// ---
AudioInputI2S        I2S_in;
AudioOutputI2S       I2S_out;
AudioXcorr           xcorr_raw_data;
AudioSynthNoiseWhite noise;
AudioControlSGTL5000 audioShield;
//----
AudioConnection  c1(I2S_in,0, xcorr_raw_data,0);
AudioConnection  c2(I2S_in,1, xcorr_raw_data,1);
AudioConnection  c3(noise,0,  I2S_out,0);
AudioConnection  c4(noise,0,  I2S_out,1);
//----------------------------------------------------------------------------
void setup() {
  Serial.begin(57600);
  delay(2000);
  audioShield.enable();
  AudioMemory(12);
  audioShield.inputSelect(myInput);
  noise.amplitude(.3);
}
//----------------------------------------------------------------------------------
void loop() {}

Code: 
//-------------------------------------------------------------------------------------
//                                 *** AudioLagComp.h ***
//
// Used in debug sketch to evaluate Audio board I2S channel synchronization problem
// 
// Author:  Derek Rowell
// Updated: July 1, 2017
//  
//--------------------------------------------------------------------------
#ifndef AudioLagComp_h_
#define AudioLagComp_h_
#include "AudioStream.h"
#include "Arduino.h"
  class AudioLagComp : public AudioStream {
  public:
    AudioLagComp() : AudioStream(2, inputQueueArray) {}
    virtual void update(void);
    void setLag(int16_t Lag) {
      lag = Lag;
      return;
    }
//
  private:
    audio_block_t *inputQueueArray[2];
    int16_t lag = 1;
    int16_t temp, oldData;
  };
#endif

Code: 
//-------------------------------------------------------------------------------------
//                                 *** AudioXcorr.cpp ***
//
// Used in debug sketch to evaluate Audio board I2S  chaneel synchronization problem
// 
// Author:  Derek Rowell
// Updated: July 1, 2017
//  
//--------------------------------------------------------------------------
#include "AudioXcorr.h"
//
void AudioXcorr::update(void) { 
  audio_block_t *blockI, *blockQ;
  blockI = receiveReadOnly(0);
  blockQ = receiveReadOnly(1);
  if (blockQ && !blockI) {release(blockQ); return;}
  if (blockI && !blockQ) {release(blockI); return;}
  if (!blockI && !blockQ) return;
  //-------------------------------------------
  max = 0.0;
  // Compute the cross-correlation between channels for lags -2, -1, 0, 1, 2
  for (int lag = -2; lag < 3; lag++) {
    // --- Sum of products
    sum = 0.; 
    for (int i = 3; i < 125; i++) { 
      sum += (float)blockI->data[i+lag]/32767. * (float)blockQ->data[i]/32767.;
    }
    xcorr[lag+2] = sum;
    // -- Find the peak cross-correlation
    if (sum > max) {
       max    = sum;
       lagMax = lag;
    }
  }
  // This version simply prints the results 
  Serial.print("Reported lag: "); Serial.print(lagMax); Serial.print(",\t");
  for (int i=0; i<5; i++)  {Serial.print(xcorr[i],4); Serial.print("\t");};
  Serial.println();
  release(blockI);
  release(blockQ);
}
 
@bicycleguy,
Whoops! How stupid of me! You don't need AudioLagComp.h for this demo. That's part of the error compensation package. Here's what you do need:

Code: 
//-------------------------------------------------------------------------------------
//                                 *** AudioXcorr.h ***
//
// Used in debug sketch to evaluate Audio board I2S channel synchronization problem
// 
// Author:  Derek Rowell
// Updated: July 1, 2017
//  
//--------------------------------------------------------------------------
#ifndef AudioXcorr_h_
#define AudioXcorr_h_
#include "AudioStream.h"
#include "Arduino.h"
  class AudioXcorr : public AudioStream {
  public:
    AudioXcorr() : AudioStream(2, inputQueueArray) {}
    virtual void update(void);
    int16_t      XcorrLag(void); 
//
  private:
    audio_block_t *inputQueueArray[2];
    float      sum;
    float      xcorr[5] = {0.,  0.,  0.,  0.,  0.}; 
    float      max;
    int16_t        lagMax;
  };
#endif

It's easy to compensate for the delay, once you know it's present, by using a shift register to delay the leading channel by a single sample. The real problem is detection. Xcorr doesn't work if the signals in the two channels are uncorrelated, as is usually the case.
 
Last edited:
@DerekR

Thanks, that cleared things up. FYI I had to move the #include "Arduino.h" above the #include "AudioStream.h" in AudioXcorr.h for it to compile. This may because I put the AudioXcorr.h & .cpp into tabs in the Arduino environment, don't know, but I've seen it before.

Haven't run the test but will tomorrow and report.
 
@bicycleguy
That's great - I look forward to the outcome!!!

That's funny about the order of the .h files. I've run it as is for a couple of years, but I have always had all component files in the same directory as the .ino. I checked again this morning :confused:

You will probably have to run the code several times using the re-program button, and turning power on and off. Back in 2017 I sat there doing both 150 times each to generate the statistics of 50% error probability on reprogram, and 30% on a power cycle.

The code will report a lag of -1 if the error is present which means
blockI->data[i-1] = blockQ->data.

BTW - Up until the T4, all of my tests have had the Audio Board piggy-backed on top of the Teensy, so it's not a wiring problem.
 
Just saw, that Paul modified the I2S code in audio library (repositioned dma_enable())
Maybe it is worth to check if such mods will change the behavior.
 
I'd be amazed if that change had any effect.

I do have a gut feeling about what might be causing this issue. I've put it on my list of issues to investigate. Can't work on it right now. Getting a software release where the Arduino IDE doesn't crash under the load of Teensy 4.0 sending at max speed is my highest priority at this moment.
 
Just one thing to add: we have exactly the same phenomenon of I2S synch error with the following hardware combinations for our Software Defined Radio Software UHSDR:

STM32F4 & WM8731
STM32F7 & WM8731
STM32H7 & WM8731

So it seems it is an issue that appears with different hardwares.

BTW, as far as I understand the code, in the above setup, we use interleaved buffers, not separate buffers for I&Q (left/right) channels . . .

https://github.com/df8oe/UHSDR/blob/active-devel/mchf-eclipse/drivers/audio/codec/uhsdr_hw_i2s.c
 
Hi,
I am also using an I2S input of the T4 and today I did some tests to check if the I2S channels are sometimes out of sync. My setup is: Laptop -> Minidsp usbStreamer -> (4 stereo i2s channels) Teensy 4.
I decided to do a quick and dirty test by means of the EqualizerApo for Windows 10. The app was already installed and it provides a simple way to make sure that all 8 channels that are transmitted to the Teensy carry the same signal. At the teensy I chose two of the 8 channels and subtracted them. The difference was always exactly zero at all tests.
It seems that all channels are always in sync at my setup. I hope that my test helps at finding the cause for the sync problem. Just let me know if you are interested in some specific details of my setup.
 
I think the problem is in the synchronization of the startup. Basically you should first stop all clocks including that of codec's. Next reset the FIFO's, setup the I2S on the host side with DMA and only then enable the I2S receiver, clocks and codec. If the 'wrong' channel (right) is received first it may still be that the I2S receiver happens to pick it up and put to FIFO but that kind of scenario must be hardware bug. Anyway, one thing which will cause this problem for sure is an incorrect LRCLK phase. Also if the I2S word size 24-32 bit and FIFO is only that wide and for some reason either gets over or underrun it's possible that the channels will be swapped. Anyway, if left channel is input to the first channel and output to the first channel then you can be sure that either the channels are in the same phase or actually two samples apart. If there is just single channel difference then the channels must have been swapped (assuming you send the right channel received back to right output channel and the left to left one).
 
Last edited:
Today I did some further tests. The last time I only checked if the channels are all in sync. This time I tried to test if channels get swapped sometimes. The minidsp usbstreamer that I use must be the i2s master and all clocks are immediatelly active when I connect it to my notebook. Therefore the teensy has no control over the clock signals and I am not able to implement your proposed test procedure. At my tests the usbstreamer was running all the time and I restarted the teensy several times. After a teensy restart the i2s input was starting immediatelly. At the test today it turned out that also the channels are never swapped at my setup.
 
Status
Not open for further replies.
Back
Top