Interference between SDFat/SDIO and I2S on T4.1

[J_Sanders]

Hello,

I'm seeing audio DAC output glitches when reading from Teensy 4.1's onboard microSD card using SDIO.
I'm using Teensyduino 1.54 beta#6.
Here's a sketch that reproduces the issue:

#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include "SdFat.h"
#define FILE_TRANSFER_BUFFER_SIZE 64000
#define N_BUFFERS_IN_FILE 100

AudioPlayQueue           queue1;       
AudioPlayQueue           queue2;      
AudioOutputI2S           i2s1;         
AudioConnection          patchCord1(queue1, 0, i2s1, 0);
AudioConnection          patchCord2(queue2, 0, i2s1, 1);
AudioControlSGTL5000     sgtl5000_1;  

// MicroSD vars
SdFs SDcard;
FsFile Wave0; // File on microSD card
byte fileTransferBuffer[FILE_TRANSFER_BUFFER_SIZE] = {0};
uint32_t playbackFilePos = 0;
bool ready = false; // SD busy flag

int16_t Zeros[256] = {0}; // An empty audio waveform
uint32_t startTime = 0;
uint32_t currentTime = 0;

void setup() {
  AudioMemory(2);
  sgtl5000_1.enable();
  sgtl5000_1.volume(0.5);
  SDcard.begin(SdioConfig(FIFO_SDIO));
  writeTestFile();
  Wave0 = SDcard.open("Wave0.wfm", O_RDWR | O_CREAT);
  Wave0.seek(0);
  startTime = micros();
}

void loop() {
    //------ Fill audio buffers-------
    int16_t *p1 = queue1.getBuffer();
    int16_t *p2 = queue2.getBuffer();
    memcpy(p1, Zeros, 256);
    memcpy(p2, Zeros, 256);
    queue1.playBuffer();
    queue2.playBuffer();

    //------ Read a buffer of data from microSD card every 100ms (disrupts playback!) -----
    currentTime = micros();
    if (currentTime - startTime > 100000) {
      startTime = currentTime;
      Wave0.read(fileTransferBuffer, FILE_TRANSFER_BUFFER_SIZE);
      playbackFilePos += FILE_TRANSFER_BUFFER_SIZE;
      if (playbackFilePos > N_BUFFERS_IN_FILE*FILE_TRANSFER_BUFFER_SIZE) {
        Wave0.seek(0);
      }
    }
}

void writeTestFile() {
  SDcard.remove("Wave0.wfm");
  Wave0 = SDcard.open("Wave0.wfm", O_RDWR | O_CREAT);
  Wave0.seek(0);
  for (uint32_t i = 0; i < N_BUFFERS_IN_FILE; i++) {
    Wave0.write(fileTransferBuffer, FILE_TRANSFER_BUFFER_SIZE);
    while (sdBusy()) {}
  }
  delayMicroseconds(100000);
  Wave0.close();
}

bool sdBusy() {
  return ready ? SDcard.card()->isBusy() : false;
}

The sketch first writes a file to the card. Then, it reads 64kB from that file into a buffer every 100ms. The data read back from the card is not used.
Meanwhile, the audio buffers are filled with 0's from a dedicated array, unrelated to the card readout. The DAC output should remain at its midpoint, but instead it does this:



And zooming into each spike at 100ms:



My best guess is that the SD DMA channel priority is greater than the audio ISR, and samples are being dropped or misread.
I tried setting the audio ISR priority to maximum in output_i2s.cpp by adding a priority argument:

dma.attachInterrupt(isr, 0);

This didn't help.

Is there a way to make microSD reads from T4's onboard card interruptible? Or is something else at fault here?

Thanks!
-J

[WMXZ]

Increase AudioMemory. 2 is too few. try different values.
if thy start with 50 you can use AudioMemoryUsageMax() to learn what is the minimum value you have to set (defined in cores: AudioStream.h)
Also, push at least 2 buffers into play queue to allow DMA to run continuously

[J_Sanders]

Hi WMXZ,

With the sketch I posted, AudioMemoryUsageMax() returns 2.
I set AudioMemory(50) and reduced the SD write buffer from 64k to 1k - same interruption every 100ms (though its duration was shorter, since less data was being read).
Here's the updated sketch:

#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include "SdFat.h"
#define FILE_TRANSFER_BUFFER_SIZE 1000
#define N_BUFFERS_IN_FILE 100

AudioPlayQueue           queue1;       
AudioPlayQueue           queue2;      
AudioOutputI2S           i2s1;         
AudioConnection          patchCord1(queue1, 0, i2s1, 0);
AudioConnection          patchCord2(queue2, 0, i2s1, 1);
AudioControlSGTL5000     sgtl5000_1;  

// MicroSD vars
SdFs SDcard;
FsFile Wave0; // File on microSD card
byte fileTransferBuffer[FILE_TRANSFER_BUFFER_SIZE] = {0};
uint32_t playbackFilePos = 0;
bool ready = false; // SD busy flag

int16_t Zeros[256] = {0}; // An empty audio waveform
uint32_t startTime = 0;
uint32_t currentTime = 0;

void setup() {
  AudioMemory(50);
  sgtl5000_1.enable();
  sgtl5000_1.volume(0.5);
  SDcard.begin(SdioConfig(FIFO_SDIO));
  writeTestFile();
  Wave0 = SDcard.open("Wave0.wfm", O_RDWR | O_CREAT);
  Wave0.seek(0);
  startTime = micros();
}

void loop() {
    //------ Fill audio buffers-------
    int16_t *p1 = queue1.getBuffer();
    int16_t *p2 = queue2.getBuffer();
    memcpy(p1, Zeros, 256);
    memcpy(p2, Zeros, 256);
    queue1.playBuffer();
    queue2.playBuffer();

    //------ Read a buffer of data from microSD card every 100ms (disrupts playback!) -----
    currentTime = micros();
    if (currentTime - startTime > 100000) {
      startTime = currentTime;
      Wave0.read(fileTransferBuffer, FILE_TRANSFER_BUFFER_SIZE);
      playbackFilePos += FILE_TRANSFER_BUFFER_SIZE;
      if (playbackFilePos > N_BUFFERS_IN_FILE*FILE_TRANSFER_BUFFER_SIZE) {
        Wave0.seek(0);
      }
    }
}

void writeTestFile() {
  SDcard.remove("Wave0.wfm");
  Wave0 = SDcard.open("Wave0.wfm", O_RDWR | O_CREAT);
  Wave0.seek(0);
  for (uint32_t i = 0; i < N_BUFFERS_IN_FILE; i++) {
    Wave0.write(fileTransferBuffer, FILE_TRANSFER_BUFFER_SIZE);
    while (sdBusy()) {}
  }
  delayMicroseconds(100000);
  Wave0.close();
}

bool sdBusy() {
  return ready ? SDcard.card()->isBusy() : false;
}

I also tried using AudioSynthWaveformSine with amplitude set to 0 instead of manually filling the audio queue, same result:

#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include "SdFat.h"
#define FILE_TRANSFER_BUFFER_SIZE 1000
#define N_BUFFERS_IN_FILE 100

AudioOutputI2S i2s1; 
AudioSynthWaveformSine   sine1;
AudioConnection          patchCord1(sine1, 0, i2s1, 0);
AudioConnection          patchCord2(sine1, 0, i2s1, 1);
AudioControlSGTL5000     sgtl5000_1;  

// MicroSD vars
SdFs SDcard;
FsFile Wave0; // File on microSD card
byte fileTransferBuffer[FILE_TRANSFER_BUFFER_SIZE] = {0};
uint32_t playbackFilePos = 0;
bool ready = false; // SD busy flag
uint32_t startTime = 0;
uint32_t currentTime = 0;

void setup() {
  AudioMemory(50);
  sine1.amplitude(0);
  sine1.frequency(1000);
  sgtl5000_1.enable();
  sgtl5000_1.volume(0.5);
  SDcard.begin(SdioConfig(FIFO_SDIO));
  writeTestFile();
  Wave0 = SDcard.open("Wave0.wfm", O_RDWR | O_CREAT);
  Wave0.seek(0);
  startTime = micros();
}

void loop() {

    //------ Read a buffer of data from microSD card every 100ms (disrupts playback!) -----
    currentTime = micros();
    if (currentTime - startTime > 100000) {
      startTime = currentTime;
      Wave0.read(fileTransferBuffer, FILE_TRANSFER_BUFFER_SIZE);
      playbackFilePos += FILE_TRANSFER_BUFFER_SIZE;
      if (playbackFilePos > N_BUFFERS_IN_FILE*FILE_TRANSFER_BUFFER_SIZE) {
        Wave0.seek(0);
      }
    }
}

void writeTestFile() {
  SDcard.remove("Wave0.wfm");
  Wave0 = SDcard.open("Wave0.wfm", O_RDWR | O_CREAT);
  Wave0.seek(0);
  for (uint32_t i = 0; i < N_BUFFERS_IN_FILE; i++) {
    Wave0.write(fileTransferBuffer, FILE_TRANSFER_BUFFER_SIZE);
    while (sdBusy()) {}
  }
  delayMicroseconds(100000);
  Wave0.close();
}

bool sdBusy() {
  return ready ? SDcard.card()->isBusy() : false;
}

One detail I didn't specify before: I ran these tests with two microSD cards: Sandisk Ultra (16GB) and Sandisk Extreme Pro (32GB), both formatted as Fat32.
The DAC output level seems to change by a few bits during some (but not all) of these SDIO interruptions - it sometimes oscillates between two values (shown in the first image in my previous post). The buffers only contain 0's, so I'm not sure where the odd values are being read from.

-J

[J_Sanders]

I ran a few more tests to constrain the possible causes of the interference. Here's what I found:

-File reading does not interrupt PWM signals from T4.1 pins
-File reading does not interrupt an IntervalTimer callbacks that toggle a pin (even at default priority)
-The issue exists with the standard Arduino SD library (using Teensyduino 1.53), not just SDFat.

Here's a sketch reproducing the issue with the standard SD library, for compilation with Arduino 1.8.13 + Teensyduino 1.53:

#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SD.h>
#define FILE_TRANSFER_BUFFER_SIZE 1000
#define N_BUFFERS_IN_FILE 100

AudioOutputI2S i2s1; 
AudioSynthWaveformSine   sine1;
AudioConnection          patchCord1(sine1, 0, i2s1, 0);
AudioConnection          patchCord2(sine1, 0, i2s1, 1);
AudioControlSGTL5000     sgtl5000_1;  

// MicroSD vars
File Wave0;
byte fileTransferBuffer[FILE_TRANSFER_BUFFER_SIZE] = {0};
uint32_t playbackFilePos = 0;
bool ready = false; // SD busy flag
uint32_t startTime = 0;
uint32_t currentTime = 0;

void setup() {
  AudioMemory(50);
  sine1.amplitude(0);
  sine1.frequency(1000);
  sgtl5000_1.enable();
  sgtl5000_1.volume(0.5);
  SD.begin(BUILTIN_SDCARD);
  writeTestFile();
  Wave0 = SD.open("Wave0.wfm", FILE_READ);
  Wave0.seek(0);
  startTime = micros();
}

void loop() {
    
    //------ Read a buffer of data from microSD card every 100ms (disrupts playback!) -----
    currentTime = micros();
    if (currentTime - startTime > 100000) {
      startTime = currentTime;
      Wave0.read(fileTransferBuffer, FILE_TRANSFER_BUFFER_SIZE);
      playbackFilePos += FILE_TRANSFER_BUFFER_SIZE;
      if (playbackFilePos > N_BUFFERS_IN_FILE*FILE_TRANSFER_BUFFER_SIZE) {
        Wave0.seek(0);
      }
    }
}

void writeTestFile() {
  
  SD.remove("Wave0.wfm");
  Wave0 = SD.open("Wave0.wfm", FILE_WRITE);
  Wave0.seek(0);
  for (uint32_t i = 0; i < N_BUFFERS_IN_FILE; i++) {
    Wave0.write(fileTransferBuffer, FILE_TRANSFER_BUFFER_SIZE);
  }
  delayMicroseconds(100000);
  Wave0.close();
}

I'm still not sure quite what I'm looking at. Here's a summary of the observations and my best guesses:

-While SD reads are happening, and for their duration, there is noise on audio DAC outputs.
-Noise spike amplitude is about 2-3x the p2p amplitude of the line noise when the DAC output is set to 0
-The DAC's output value changes to a stable non-zero level following some of these events, but returns to 0 after the SD write is complete
-During the SD write events there is no extra jitter in PWM or timer interrupts
-The affected audio devices are the Teensy Audio Board, and the T4.X HiFiBerry breakout board hosting with any of the supported DAC cards (so 3 different audio DACs in total).
-The interference occurs with the Arduino SD library (Teensyduino 1.53) and with SDFat (Teensyduino 1.54 beta 6)
-The interference occurs using two different microSD cards: Sandisk Ultra (16GB) and Sandisk Extreme Pro (32GB), both formatted as Fat32
-The interference occurs in both I2S master and I2S slave mode
-The interference exists with the sketch above and Teensy 3.6 (using Audio board rev B) and Teensy 4.1 with the 3 DACs noted above
-The same interference occurs with sound produced by the Teensy Audio Library (128-sample packets, etc), and also with a sketch with minimal code that initializes the I2S bus +DMA channel and writes audio data via the DMA channel's ISR (so timing in the Audio library's packet pipeline and buffers are not the issue)

My best guesses:
1. An electrical issue, e.g. digital interference on the I2S data or clock lines from high speed switching on the SD bus. If so the issue may be less (or more) present with the I2S2 interface pins, and be less present with lower SD bus speeds (not sure how to set that, or whether SPI speeds <25MHz are valid).
2. A memory or resource issue, e.g. bits from one DMA channel leaking into the second
3. A priority bug, e.g. the audio DMA ISR being lower priority than it's programmed to be, causing buffer underruns

I'll keep testing, and of course any insight into this issue is more than welcome!

-J

[J_Sanders]

I ran one more test, to rule out my acquisition hardware.

Until now, all of the data was captured with an NI USB-6211 DAQ board sampling at 200kHz, but for some reason the distortion was difficult to see on my oscilloscope (high frequency noise drowning the longer disruptions?) so I captured the signal from the sketch in my previous post with a different device.
I used an ASUS Xonar U7 sound card to acquire the output of the Teensy Audio Board. My acquisition sampling rate was 192kHz, 24bit. Here's the resulting signal:



Question to the forum: Is it possible to set up the built-in SD card pins as digital outputs, so I can try high frequency switching on those lines without the SD or SDFat libraries?

[defragster]

...

Question to the forum: Is it possible to set up the built-in SD card pins as digital outputs, so I can try high frequency switching on those lines without the SD or SDFat libraries?

Yes - see : store/teensy41.html#pins

That shows the Arduino pin numbers for the SDIO pins

[J_Sanders]

Thanks, Defragster!

I configured each pin as OUTPUT, and set PWM frequency to 1MHz.
Then, while a zero-amplitude sine wave was playing, I set each builtin-SD pin to 50% duty cycle for a 3ms interval every 100ms.
I captured the Teensy Audio Board output with my NI USB-6211.
Here's the result:



Looks like Pin44 (the clock pin), is the culprit.
Here's the Arduino code I used (manually modifying 'currentPin' for each acquisition session):

#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include "SdFat.h"
#define FILE_TRANSFER_BUFFER_SIZE 64000
#define N_BUFFERS_IN_FILE 100

AudioOutputI2S i2s1; 
AudioSynthWaveformSine   sine1;
AudioConnection          patchCord1(sine1, 0, i2s1, 0);
AudioConnection          patchCord2(sine1, 0, i2s1, 1);
AudioControlSGTL5000     sgtl5000_1;  

uint32_t startTime = 0;
uint32_t currentTime = 0;
byte sdPins[6] = {42, 43, 44, 45, 46, 47};
byte currentPin = 6;

void setup() {
  AudioMemory(50);
  sine1.amplitude(0);
  sine1.frequency(1000);
  sgtl5000_1.enable();
  sgtl5000_1.volume(0.5);
  startTime = micros();
  analogWriteResolution(10);
  for (int i = 0; i<6; i++) {
    pinMode(sdPins[i], OUTPUT);
    analogWriteFrequency(sdPins[i], 1000000);
    analogWrite(sdPins[i], 0);
  }
}

void loop() {
    //------ Toggle sequential built-in microSD pins at 1MHz for 3ms every 100ms -----
    currentTime = micros();
    if (currentTime - startTime > 100000) {
      startTime = currentTime;
      analogWrite(sdPins[currentPin], 512);
      delay(3);
      analogWrite(sdPins[currentPin], 0);
    }
}

If I'm seeing this interference from the built-in SD clock line even as low as 1MHz, a lower SD bus frequency isn't going to help.

It looks like pin 44 is the CS line for the second SPI bus - which should have far fewer transitions in theory. Can I interface with the built-in card via SPI?
More generally, any ideas why this is happening, and whether there's anything I (and others) can do to get around it?

Thanks!
-J

[J_Sanders]

Looks like the same interference from pin 44 affects audio on I2S2:



The noise floor is higher (prototyping wires) but the glitch amplitude is the same (compare to plot from my first post, using the microSD card).

If it were noise coupling onto an I2S bus line, I'd have expected to see at least some difference using a separate bus.

@Paul (and others with access to the T4.1 CAD), any insight and/or suggestions?

Thanks!
-J

[Frank B]

Yes, use more relaible connections - plug the shield directly on the Teensy - , don't use these china wires
As Pin 44 is the SD clock pin, SPI wouldn't help.. needs a clock, too.
Or try the SD on the shield?

We had the contrary effect - communication with the optional flash chips influenced the SD.
Solved that with different PAD Settings..


On the other hand:
The Audio shield is problay the best tested ADC/DAC shield in the Arduino community. There were countless tests , similar to yours.
Nobody has seen this... why?

[J_Sanders]

Hi Frank,

For all tests except for I2S2, I plugged the Teensy 4.1 board directly into the Teensy audio board using pin headers.

From the pinout diagram, it looks like pin 44 is a clock pin for SDIO mode (light blue) and a chip select pin for SPI2 bus (green). The CS line toggles far slower than clock - so if I could initialize the SD library in single bit mode using SPI2, it might reduce the magnitude of interference.

I'm not sure why nobody has seen this, and I'm doing my best to sanity check it. Possibly most users opt for T3.2 and T4.0 with the audio shield, and use the shield's onboard microSD card?

[Frank B]

Hm,
no, for the SD it's always the same pinout (for clock)

SPI-Mode:

Not sure what it could be.. it's not audible, at least for me...

@Pauls (from Netherlands): If you're reading this.. have you tested this?

@J_Sanders, which Teensyduino are you using? The new betas, or 1.53 or older?
Not sure which pads Bill Greiman has changed.. if he changed the settings for CLK, the effect should be much worse with the newer versions (stronger PIN Settings)

[J_Sanders]

OK, maybe it's a typo in the pinout diagram. I was hoping not!

I should add - this glitch is practically inaudible to me with headphones at standard listening levels. The glitches are only 2-3x the noise floor in amplitude. To hear them, you'll need to amplify. I discovered the SD interference initially because it shows up as patterned distortion on spectrograms of the output.

I specified which Teensyduino + Arduino for which tests in the posts above - but in general, 1.53 for SD, newest beta for SDFat.

[J_Sanders]

I found a reason why it may have evaded detection:
At full volume on the SGTL (sgtl5000_1.volume(0.8), the glitch is masked almost entirely by the noise floor:

Attachment 23824

It's more salient at sgtl5000_1.volume(0.7):

Attachment 23825

And at sgtl5000_1.volume(0.5):

Attachment 23826

So the interference really only manifests if you use the SGTL's digital attenuation function.

Unfortunately the same strategy doesn't work when Teensy talks to other audio DACs. Here's a PCM1796 at full scale output:

[Frank B]

Is it the same with the line-out connection?

[J_Sanders]

Oddly enough, no! At first pass, the line out pins don't seem to have the issue. I expected to see it, since I saw them on the headphone channel and two separate PCM series audio DACs. Does the line out have any filtering enabled by default?

I didn't realize that volume() controls a headphone amp on the SGTL - now it makes sense that setting the PCM DACs to 0 digital attenuation did nothing.

[Frank B]

Oddly enough, no! At first pass, the line out pins don't seem to have the issue. I expected to see it, since I saw them on the headphone channel and two separate PCM series audio DACs. Does the line out have any filtering enabled by default?

I didn't realize that volume() controls a headphone amp on the SGTL - now it makes sense that setting the PCM DACs to 0 digital attenuation did nothing.

Hmm... not an expert here... electronics noob, still ... but... this this sounds a problem with the current?
Is it worth to try and add a buffer-capacitor?

[J_Sanders]

I don't suspect a current issue - why would pin44 cause more current to be drawn by an audio board than the other built-in SD pins?
If the line out has digital filtering enabled somewhere by default, it could suppress the spikes...

[PaulStoffregen]

If the line out has digital filtering enabled somewhere by default, it could suppress the spikes...

That's not how unwanted ground loop currents work. They change the observed signal by altering the ground reference. The signal could be absolutely perfect, but if you alter the ground voltage seen by the equipment which receives the signal, to the receiver it looks as if the signal changed by that much.

[J_Sanders]

I had assumed that the spikes were driven by digital noise in the I2S signal - but you do make a good point about my sanity check - the NI USB card and the ASUS Xonar U7 were both grounded to the same USB hub as the Teensy board. It would be really strange if toggling pin 44 but not others caused a ground loop to appear and couple to the DAC output (this is definitely beyond my EE knowledge without seeing the board layout!), but I'll try this with a different power supply and post back.

[J_Sanders]

And I'll be darned. It WAS a ground loop.
The glitches are gone if I record with the NI board on my laptop while the Teensy is powered from a different PC.

Thanks Paul!!