Lockups with FastLED & TeensyThreads

Status
Not open for further replies.
E

Eka

Well-known member
I'm experiencing intermittent lockups on a Teensy 3.6 when using FastLED and TeensyThreads libraries together. Sometimes it will lockup nearly immediately, and other times it can take 15 to 20 minutes. I have two threads that at this point just do some calculations. No IO yet. Eventually one will monitor some sensors, and read the RTC_TSR, RTC_TPR & RTC_TCR registers. The other will block write the data to the SD card. The main loop() does the LED updating, and some serial output for monitoring. It will even lockup without the serial output.

To narrow down where it may have been locking up, I added control of the LED on pin 13 to the threads. At least one of them is still running after the main loop has locked up. I then placed LED on and off only around the FastLED.show() call. The LED was on when it locked up. I need to add more LEDs for a more complete post code output. That will have to wait a bit.

The 2 APA102 144 pixel long strings work fine and are driven with a SN74AHCT125 Quad buffer with the enables fixed. I'm using a buffer for each data and clock for both SPI ports. I was going to use it with chip selects, but FastLED doesn't handle chip selects. I'll be using them in the next HW revision. I should also add pull down resistors to help stop the annoying random changes as the circuit powers up.

I have a serial GPS tied to one of the RS-232 ports, and also with a PPS signal coming in on one pin, but it locks up with it setup and not setup.

I also have a 1F supercapacitor serving as a backup power supply for the RTC. It is charged up to 3V through a diode and resistor connecting it's positive pole to the 3.3VDC rail. I haven't tested how long it will keep the RTC going. With the same circuit I get 4-7 days on a DS3231 with temp corrections applied, and longer without.

I removed lots of code, but some cruft may remain.

With LED control in threads:
Code: 
#include <TeensyThreads.h>

#include <fastspi_nop.h>
#include <cpp_compat.h>
#include <fastled_config.h>
#include <bitswap.h>
#include <fastpin.h>
#include <power_mgt.h>
#include <pixeltypes.h>
#include <noise.h>
#include <fastspi_dma.h>
#include <FastLED.h>
#include <lib8tion.h>
#include <dmx.h>
#include <hsv2rgb.h>
#include <controller.h>
#include <colorutils.h>
#include <fastled_progmem.h>
#include <fastspi.h>
#include <color.h>
#include <platforms.h>
#include <fastled_delay.h>
#include <colorpalettes.h>
#include <chipsets.h>
#include <pixelset.h>
#include <fastspi_types.h>
#include <fastspi_ref.h>
#include <fastspi_bitbang.h>
#include <led_sysdefs.h>

#include <TimeLib.h>

#define numleds 288

#define TDiv_SEC     60.0
#define TDiv_MIN   3600.0
#define TDiv_HOUR 43200.0
#define TI_HOUR   43200

#define DATA_PIN1 11
#define CLOCK_PIN1 14
#define DATA_PIN2 0
#define CLOCK_PIN2 32
#define OEPURPLEPIN 6
#define ENBLUEPIN 15
#define ENWHITEPIN 31

const int offset = -5;   // Central USA

CRGB leds[numleds * 2];
int co = 0;
uint32_t i = 0;
uint32_t pr = 1;
volatile int8_t Sec = 0, Min = 0, Hour = 0;

// file write thread variables
#define logFileBufSize (uint32_t)0x00008000
#define logFileBufWriteSize 4096
volatile uint32_t logFileBufHead = 0;
volatile uint32_t logFileBufTail = 0;
volatile uint32_t lfOK, sdOK;
volatile uint32_t threadFileWriteAct = 0;
volatile uint32_t threadTimeAdjustDataCollectorAct = 0;
volatile uint32_t threadTimeAdjustDataCollectorAct2 = 0;
char logFileBuf[logFileBufSize] = {0};

void setup()
{
  Serial.begin(9600);
  i = 0;
  while ((!Serial) && (i++ < 60)) { // Needed for Leonardo only
    delay(10);
  }
  if(!Serial) pr = 0;
  if(pr) Serial.printf("sr %x\n", RTC_SR);
#define LEDPIN 13
  pinMode(LEDPIN, OUTPUT);
  digitalWrite(LEDPIN, HIGH);
  
  // LED strings
  FastLED.addLeds<APA102, DATA_PIN1, CLOCK_PIN1, RGB, DATA_RATE_MHZ(8)>(leds, 0, numleds/2);
  FastLED.addLeds<APA102, DATA_PIN2, CLOCK_PIN2, RGB, DATA_RATE_MHZ(4)>(leds, numleds/2, numleds/2);

  // start time adjustment data collector thread
  if(!threads.addThread(threadTimeAdjustDataCollector, 1, 8192)) {
    if(pr) Serial.println("failed to start threadTimeAdjustDataCollector.");
  } else {
    if(pr) Serial.println("start threadTimeAdjustDataCollector.");
  }

  // start file write thread
  if(!threads.addThread(threadFileWrite, 1, 8192)) {
    if(pr) Serial.println("failed to start threadFileWrite.");
  } else {
    if(pr) Serial.println("started threadFileWrite.");
  }
}

void loop()                 
{
  static uint32_t lastTSR = 0;
  
  for(i = 0; i < numleds;   i++) {
    leds[i] = CHSV(Sec,Min,Hour);
  }

  // /////////////////////////////////////////////////////////
  FastLED.show();
  // write data out
  if(pr && (lastTSR != RTC_TSR)) {
    Serial.printf("%4d/%02d/%02d %02d:%02d:%02d\n",
                 year(lastTSR), month(lastTSR), day(lastTSR), 
                 hour(lastTSR), minute(lastTSR), second(lastTSR));
    lastTSR = RTC_TSR;
  }
}


void threadTimeAdjustDataCollector(int arg) {
  uint32_t lastTSR;
  uint32_t cTSR, c;
  char strBuf[1024];
  // code to capture data for time adjustment calculations
  digitalWrite(LEDPIN, HIGH);
  lastTSR = RTC_TSR;
  while(1) {
    threadTimeAdjustDataCollectorAct++;

    while(lastTSR == (cTSR = RTC_TSR)) {
      threadTimeAdjustDataCollectorAct2++;
    
      Sec += 1;
      Min += 1;
      Hour += 1;
      digitalWrite(LEDPIN, LOW);
      threads.delay(5);
      digitalWrite(LEDPIN, HIGH);
    }
    lastTSR = RTC_TSR;
    // write data out
    c = sprintf(strBuf, "%4d/%02d/%02d %02d:%02d:%02d\n",
                 year(lastTSR), month(lastTSR), day(lastTSR), 
                 hour(lastTSR), minute(lastTSR), second(lastTSR));
    for(uint32_t x = 0; ((x < 1024) && (x < c)); x++) {
      // check if overrun of buffer
      if(logFileBufHead != ((logFileBufTail - 1) & (logFileBufSize - 1))) {
        logFileBuf[logFileBufHead++] = strBuf[x];
        if(logFileBufHead >= logFileBufSize)
          logFileBufHead = 0;
      } else {
        // buffer overflow error.
        // what to do? put a newline char? Nothing?
      }
    }
    digitalWrite(LEDPIN, LOW);
    threads.yield(); // sleep for now
    digitalWrite(LEDPIN, HIGH);
  }
}

void threadFileWrite(int arg) {
  // code to write buffers as they fill
  digitalWrite(LEDPIN, HIGH);
  while(1) {
    threadFileWriteAct++;
    if((logFileBufHead > logFileBufTail) && ((logFileBufHead - logFileBufTail) > logFileBufWriteSize)) {
      // write out logFileBufWriteSize characters starting at logFileBufTail.
      ///***FIXME***
      logFileBufTail += logFileBufWriteSize;
      if(logFileBufTail >= logFileBufSize)
        logFileBufTail = 0;
    } else if((logFileBufHead < logFileBufTail) && (logFileBufTail < (logFileBufSize - logFileBufWriteSize))) {
      // write out logFileBufWriteSize characters starting at logFileBufTail.
      ///***FIXME***
      logFileBufTail += logFileBufWriteSize;
      if(logFileBufTail >= logFileBufSize)
        logFileBufTail = 0;
    }
    digitalWrite(LEDPIN, LOW);
    threads.yield(); // sleep for now
    digitalWrite(LEDPIN, HIGH);
  }
}
 
OK, I now have 10 additional LEDs hooked up to IO lines. Relax, I used a 74HC04 hex inverter chip to drive 6 of them, with only 4 driven directly by IO pins, and they are driven at under 2mA.

I can confirm all threads I start continue to work after the main loop lockup, and have continued to do so for over 12 hours. It is just the main loop() that halts, and it only halts during the FastLED.show() call. I also had plenty of Serial calls doing serial IO in the main loop. It never stopped in any of them.

I guess this needs to be escalated to a bug report. The code in my previous post consistently reproduces the lockup bug. I figure there maybe is something in the FastLED.show() call that disturbes the TeensyThreads scheduling algorythim and the main loop() process ends or is never scheduled for execution again, or thread execution makes a too long of gap in the FastLED.show() call, and it fails to complete. By putting it in it's own thread with no overhead before it I think I'll be reasonably safe until I can afford some time to debug it.

My workaround because I don't have the time to do the debugging now, I moved the LED calculations, FastLED.show() call, and console serial printing into their own threads like I had intended to do before I ran into the lockups. Nothing remains in the main loop(). The time slice given to the thread with the FastLED.show() needs to be long enough to let the call finish naturally. It hasn't locked up yet with over 36 hours run time with free running FastLED.show() calls only limited in number by the time it takes to calculate a new set of LED values. With my current calculations it reaches well over 300FPS. With the simple calculations in the test code it reaches well over 900FPS and thus has a much higher percentage of time when a task switch can happen in the middle of a FastLED.show(). Before I was usually locking up in under 20 minutes run time with under 3 minutes being most common.

I've also now limited the display to 180FPS. At that rate even my fastest changing sections look smooth. I'm just not happy with FastLED's HSV byte based color routines. They really are HSI not HSV and I use tones and shades in my artwork. HSV uses a mixing of white or black for tone and shade variations of the base hue and saturation where HSI only uses intensity and can't represent the shades and tones I use. I have a float based HSV system running on a RPI3 in python, but it's to slow at 5 to 10 FPS. On the other hand I like it's color rendering much better. We'll see if the T-3.6 is up to it or if I need to parallel a few of them for each light sculpture. The whole thing is written to be wishy washy tolerant for when things happen. The only thing I keep tightly regulated is the time base. I synchronize the hardware RTC to an external PPS input from a GPS, and it is kept to within a few 32768ths of a second. That is so I can synchronize multiple units together.

The LED write thread looks like this. It will write out the buffer to the display each time it gets a time slice and buf0ReadyToShow is true. buf0ReadyToShow is now set true by an timed interrupt, then when threadWriteLEDs next gets CPU time the buffer gets written to the LED string. The time slice for the thread needs to be longer than it tales for FastLED.show() call to complete.
Code: 
void threadWriteLEDs(int arg) {
  while(1) {
    threadWriteLEDsCount++;
    if(buf0ReadyToShow) {
      threadWriteLEDsCountBuf++;
      FastLED.show();
      buf0ReadyToShow = 0;
    }
    threads.yield();
  }
}
 
Lockups of the serial IO become more likely to happen when I use threads.delay() than when just using threads.yield(). I think a long interrupt happening at the right time can do it too, but I haven't tested extending interrupt handler execution times yet. FastLED.show() is stopping during it's execution. I've tested with and without TeensyThreads and it still happens without TeensyThreads being used. It is just that TeensyThreads causes more long interruptions to execution so it happens more often. I've also ran tests with just the first 144 LEDs on the SCI0 bus and it will still halt all serial IO. I'm now running a long test with the second 144 LEDs on the SCI1 bus and so far it hasn't halted in over 12 hours when the same code usually halted in under an hour when using SCI0.

TeensyThreads still thinks the thread with the FastLED.show() call is still running after a serial lockup, but it has ground to a halt. Never returning from the FastLED.show() call. I should see if I can kill that thread, and restart it. Make a watchdog system for the threads I'm running.

Faster data rates on SPI0 make it take on average longer for FastLED.show() to fail to return. With the shorter time period in FastLED.show() they are less likely to get interrupted while it is running.

I thought I may be overloading the 3.3V power rail. So I removed the MicroSD card and moved all non Teensy 3.3V loads onto a separate 3.3V power rail and I still get the same results.
 
Status
Not open for further replies.
Back
Top