Audio/LED Project Glitching on Teensy 3.6 but Okay on 3.2

Status
Not open for further replies.

SteveManley

New member
May I ask for some advice please? Let me start by briefly summarizing the problem. I’ve designed a control board to drive 350 WS2812B LEDs. The board is designed to accommodate either a Teensy 3.2 or Teensy 3.6 (to provide more I/O Pins). The board also includes a number of other chips, including an audio codec. Everything works great with the Teensy 3.2, but the display starts to flicker and glitch horribly when I swap in a Teensy 3.6, even though nothing else has changed and they are pin compatible (for the pins I’m using).
*
Here's a link to a video showing a crude demo of the display with a Teensy 3.2 followed by a Teensy 3.6 glitching*https://youtu.be/faKecpwiOI0
*
I’ll pose my questions here just in case you loose interest in reading the more in depth summary that follows them:

Q1. Would anyone have an explanation as to why the Teensy 3.2 seemingly works flawlessly, but the 3.6 causes LED display glitches?
*
Q2. Is it possible to implement DMA with this combo of libraries (Teensy Audio, Octo & FastLED)? If so, I seem to be unable to find out how, so please can someone offer some guidance?


OK, here’s the story in a little more depth. I’m an amateur electronics enthusiast with an interest in controlling RGB LEDs using a microcontroller, with Teensy as my main choice in MCU’s. I am able to code to some extent, but by no means am I an expert.
*
I have been developing a 10 by 21 segment Victorian style RGB alpha numeric display (along with some other folk). Each display has 35 LEDs, and the displays are wired in pairs, so I have 5 LED strips each 70 LEDs long, totalling 350 LEDs.
*
The main purpose of the display is to show date and time information, along with text messages and audio visualizations.*As part of this I designed a custom control board to drive the display. This board uses a DS3231S RTC chip as I originally wanted to run the display this from a Teensy LC, so its still included.
*
The control board also currently uses the Teensy Audio & OctoWS2811 libraries, as well as the FastLED library. The board has a 74HCT245 buffer for the standard 8 Teensy Octo outputs, and a SGTL5000 audio codec chip configured in the same way as on the Teensy Audio breakout board minus the SD card reader, audio output, and additional memory capability. I wanted the control board to be able to utilize either a Teensy 3.2 or a 3.6 MCU, so the control board is able to accept either device.
*
Until recently, I was using a Teensy 3.2 to drive the LEDs using the OctoWS2811 and FastLED libraries. So far so good. Using the Octo outputs rather than a single 350-LED string cut the display refresh rate down from about 11ms to a little over 2ms. I’m assuming I’m not benefiting from DMA as my code appears to wait around for the FastLED.show() to complete.
*
I added in the Audio library and managed to get the display to perform a VU meter and an FFT visualization based on the Teensy Audio library examples using the mic or line-in inputs.*
*
Up to this point, everything has been working as I planned, despite the FFT output not appearing very responsive at the higher frequencies (I think this is because my scale is linear at this time and it needs to be exponential).
*
The issue I hit was when I swapped the Teensy 3.2 for a 3.6 and uploaded the same code. The display became incredibly glitchy, whereas it was as steady as can be on the 3.2 (maybe more by luck than judgement).
*
My friend Clive “Max” Maxfield has the same setup, and he was the one who discovered the issue with the Teensy 3.6. His setup also works well with the 3.2. The reason for the 3.6 is that Max has plans to use a lot more I/O than I, else we would stick with the 3.2.
*
After some experimentation on a 3.6, I can significantly reduce but not totally eliminate the glitching by setting the Audio mixer inputs, Audio Mic Gain, and Audio Volume all to zero values. I also applied the AudioNoInterrupts() command, which eliminated the glitching altogether, but that obviously blocks the audio visualizations from working until re-enabled.
*
This indicates to me that the Audio library interrupts are interfering with the FastLED.show() calls, and that I’m definitely not benefitting from DMA for the display updates.

I have uploaded the main INO file which has all the important config. There are several H files with categorised functions, but there are quite a few of them. I can always add them later if need be.

Thanks,
Steve
 

Attachments

  • Foundation_for_Control_Board_V1.01.ino
    20.1 KB · Views: 56
I managed to figure out 2 things myself that have seemingly fixed the issue. Either works alone.

1. Over clocking the Teensy 3.6 at or above 216 MHz.
2. Setting AudioNoInterrupts() for everything except the audio modes. Even the Audio modes stopped glitching after re-enabling the interrupts.
 
Hey Steve, Interesting results. I am seeing something similar going from a Teensy 3.2 to a Teensy 4.1 My Teensy 3.2 setup includes and Octo board + an AdaFruit Airlift board for WiFi and it has worked flawlessly on two 1200 LED projects, as well as a different 600 LED project running very similar code. For my latest LED project, also around 1200 LEDs, I decided to try out a Teensy 4.1 and add an audio shield (along with the Octo and the Airlift). I have noticed if I enable the audio library or the WiFi library, I get occasional glitches on the LED strips (it's much worse with the audio library initialized). I've not dug into the cause yet but I suspect it's some sort of interrupt issue like you suggested. The code running is almost identical to my other projects but there is always the possibility that I introduced a bug somewhere, so it's good in a way to see that others are having similar issues. Cool that you were able to run things using the identical code on different controllers to show this is likely not a user code issue!
 
Hey fussylizard, Yeah interesting that someone else has incurred a similar issue. Wish I new what it was exactly that's causing it. I have experienced the issue after implementing the AudioNoInterrupts() when not using the audio modes. The glitching wasn't as bad, but still happened occasionally. The only thing that reliably stops the glitching is the overclocking. One thing I noticed on my Teensy 3.2, is in the Arduino IDE, the default CPU speed is 96MHz (Overclocked), so I dropped the speed down to 72Mhz and the display didn't just glitch, it flickered really badly. All I can come up with is a compatibility or performance/timing issue with the FastLED lib at lower CPU speeds, or as you suggested, something relating to Interrupts. Overclocking seems to me to be the best way to go. Would love someone more knowledgeable about these things to provide the answers.
 
I suspect you'd be way better off ditching FastLED and using only OctoWS2811: from the documentation, the latter does use DMA (dunno about FastLED).

Bear in mind the Audio library kicks off an interrupt every 2.9ms, and your arrangement takes 2.1ms to update the LED strips (you could improve that to 1.32ms by using 8 strips rather than 5). Your audio CPU load has to take <0.8ms for everything to work stably, even assuming you can sync your LED update to it. You could check the CPU load using AudioProcessorUsageMax(), needs to be <0.8 / 2.9 = 27.5% to stand a chance. Or <54.5% using 8 strips.

There isn't an official way to sync to the Audio engine, but it looks as if you could hack it:
Code:
bool AudioEngineHasRun(void)
{
  bool result = false;
  if (AudioStream::cpu_cycles_total != 0)
  {
    result = true;
    AudioStream::cpu_cycles_total = 0;
  }
  return result;
}
This breaks the processor usage macros, and only tells you the Audio engine has run at some point since you last called AudioEngineHasRun(), not how long ago that was, so you either have to have a really fast loop(), or burn CPU to get excellent sync:
Code:
...
AudioEngineHasRun(); // set cpu_cycles_total to 0
while (!AudioEngineHasRun()) // burn until Audio has run once more: up to 2.9ms wasted CPU
  ;
// start LED update immediately here!
...

Please note - I haven't tested this, beyond checking it compiles!

Cheers

Jonathan
 
Steve, Jonathan, Good to get this discussion going a bit, thanks! I'm using the Octo library, not FastLED, so I don't think the glitching is related to FastLED. The audio sync idea is interesting, I might try something like that soonish, IDK. I'm only using the audio library to do an FFT on incoming audio to display a spectrum analyzer on my LED setup. I'm using all 8 octo channels, with 165 LEDs per channel. So I don't really need the audio to be perfect, rather, I need the LEDs to be stable. I'm also using an AdaFruit AirLift for WiFi control, so that may also be doing stuff with interrupts.

My LEDs definitely glitch with WiFi enabled and audio disabled, and with audio enabled and WiFi disabled. I need to recompile my app with WiFi and audio both disabled and watch it run for a while to be 100% sure it never glitches with both of those disabled. I'm 99% sure that's the case, but I should double check.

I was talking to a coworker about this issue the other day, and he's a super talented firmware developer and one of his gripes with Arduino is that conflicts like this amongst different components and libraries are a huge pain to track down. So...will probably be a ton of fun to get to a root cause haha.

TBH the glitching is not a huge priority for me quite yet. The glitching happens and the display is screwed up for a minute or two, and then it fixes itself and is good for another 10 minutes.
It's just running patterns on a DJ booth, so most people probably just think the glitching is a different kinda of boring pattern haha. But I do want to get this fixed since the spectrum analyzer causes it to fail so much as to be unusable, and I really want to get that into a usable state.
 
A channel of 165 LEDs will take 4.95ms to update, so the Audio engine will run in the middle of that even if you're in sync with it: a bit of double-buffering might be indicated? Definitely worth checking the LEDs alone, though.

Agree with your colleague about the difficulties with conflicts, though I'm sure he's aware such things also happen in the Real World, not just the Arduino / Teensy one (don't get me started on ST's Bluetooth Low Energy API...). Paul's Teensy libraries appear to be pretty well-documented for what's essentially a hobbyist ecosystem, but so many others leave it to the user to read the library source code to figure out what's going on with interrupts, DMA etc.

Cheers

Jonathan
 
True. Maybe I can just have do something like:
loop():
- generate frame for LEDs
- start frame writing
- enable audio engine (or maybe enable audio interrupts)
- let audio engine sample audio for the FFT
- disable audio engine (enable audio interrupts)

Since the audio doesn't have to be continuous I may have some flexibility to work with. There is also the WiFi stack to deal with too, though, IDK how forgiving that will be.

I just ran a test and I do notice some very rare glitching with both the audio library and the WiFi library disabled. If I have the audio library it glitches every minute or so. But with no WiFi and no audio, it will run for 20 minutes at a time with no glitches. I wonder if there is a bug in my code that is causing this, but the glitching persists across pattern changes and the same code works on a Teensy 3.2 with no issues so I'm thinking it's something related to the hardware/integration. But I need to keep an open mind to bugs I've introduced all by myself...
 
Last edited:
Just for reference here is the glitching I'm seeing: https://drive.google.com/file/d/1KG0D1GTM-ny2_ElFLGRBdpaCsAo12uKm/view?usp=sharing

It's interesting since it does those "blocks" of colors. IDK what it's doing, maybe the DMAs are getting out of sync or something. I took a look at the Octo code a bit and it's pretty simple. Since all the transfers happen via DMA, I'd think they just go regardless of there being another interrupt or not, or if interrupts are disabled. If interrupts are disabled (say by the audio library doing something) when the DMA transfer ends, I expect it would just delay the start of the next frame, not cause weird block patterns that magically go away after a few minutes.

Who knows, maybe I'm barking up the wrong tree...?
 
Last edited:
Oh so I just made a discovery that is specific to the Octo library. I was previously looking at the Octo library code for the Teensy 3.x, which does DMA transfers in one go for the entire frame at once.

Teensy 4.x, however, does the DMA in chunks, so each frame will be written out in pieces. I bet if I modify the size of the transfers to be twice as large the "bands" during the glitches will be twice as long...
 
Ah, good observation. And I note that your glitch blocks are about 13 LEDs long, and at line 38 of OctoWS2811_imxrt.cpp we find #define BYTES_PER_DMA 40, which is about (but not exactly) 13 LEDs - not even a multiple of 3 bytes = 1 LED, which would have been vaguely sane... As you say, modifying that would be instructive - I'd pick 48 or 96 to see if you also eliminate the colour shifts, though as it's a bit-based protocol I'd not be surprised one way or the other.

As interrupts of some sort are going off even without Audio and WiFi, I think this approach is definitely what's causing the problems with the Teensy 4.1. (Not so sure about the Teensy 3.2 / 3.6 differences, they seem to use much the same library code.) I don't know how interrupt priorities are set in Teensy, but it would appear that the OctoWS2811 interrupt absolutely needs to be higher than everything else, as servicing it is time critical - it looks as if a delay of 40 byte times = 400us will be enough to cause issues. No idea how long the ISR takes to run, but probably not long.

I kind of take it back about the documentation ... absolutely zero mention of the different interrupt-vulnerable scheme used for the Teensy 4.1 OctoWS2811 library on the relevant page, or the new constructor allowing you to select your pin list. Examples are not documentation.

Cheers

Jonathan
 
Additional ... the FFT may not be massively forgiving of you stopping the Audio engine, but I don't know for sure: you may see huge spikes at multiples of 1000/2.9 = 345Hz because of the missing data. You could try something like:
Code:
uint32_t nextFFT;
enum {waitLEDs, waitFFT} state;

loop()
{
  switch (state)
  {
    case waitLEDs:
      if (!leds.busy())
      {
        AudioInterrupts():
        nextFFT = millis()+9; // allow time for sane FFT
        state = waitFFT;
      }
      break;

    case waitFFT:
      if (millis() > nextFFT)
      {
        AudioNoInterrupts():
        // code involving ...
        leds.setPixel(); // update display from FFT
        // ...ending with
        leds.show();
        state = waitLEDs;
      }
      break;
  }
}
"Obviously" fill wait time with other code to suit, may or may not be state dependent.

Cheers

Jonathan
 
I doubled it to 80 last night and let it run a while. It took a long time to glitch (which makes sense given Paul's note above #define BYTES_PER_DMA 40 about longer values make it less sensitive to interrupt issues). Of course it was 2 a.m. when it happened so I forgot to take a video of it so I'm not sure what affect it had on the segment lengths.

So maybe the quick and dirty fix is to use 128 bytes or something (I'm not sure what the performance implications of this are that Paul references). Since my code (without the FFT) was running comfortably on a Teensy 3.2 I should have plenty of RAM and CPU to "waste" here.

Pity the library code is not in the header file as:
#ifndef BYTES_PER_DMA
#define BYTES_PER_DMA 40
#endif

which would let me change that setting without having to include a copy of the .cpp file in my project, but c'est la vie.

I have no idea how Teensy/Arduino interrupts are selected and how, if at all, interrupt priority is managed via libraries. Something for me to look into I suppose. I am curious what other interrupts are running on the Teensy though that may be interfering with the Octo library.

Oh and thanks for the suggestions on the "starve the FFT" route too. Hopefully I can sort it out interrupts and/or DMA transfer size.

Thanks Jonathan and Steve for the replies!

-- Chris
 
Status
Not open for further replies.
Back
Top