Teensy 3.2 with audio, serial, and ili9341 touchscreen hangup problem

[josechow]

Hey All,

System Overview: Teensy 3.2 connected to SGTL5000, ili9341 touch screen display, and RS-485 on TX1/RX1 for audio retrieval and playback

Problem: Hang up of audio library when calling XXX.getBuffer(); from a playqueue object. Processor hangs up approximately 7 to 10 minutes after power up on the .getbuffer() call. Line 255 of the code below.

Elaboration:

Audio packets come into RS-485 and pushed to audio library for playback. Audio packets come from a remote Teensy 3.2.

The code below fails when calling audio_queue.getBuffer(). Checked this is the hangup point with a scope, as I used the RS-485 direction pin as a visual feedback before and after the function call.

Other indications: about 30 seconds before the failure, there is a spike and subsequent return to normal of audio library CPU usage. After the ~30 seconds, memory usage starts ramping up over 5-10 seconds from the nominal 2/3 blocks to the max audiomemory allocated; system hangs up indicated by a buzzing noise since no more audio packets are pushed into playqueue, and the controller resets due to a watchdog time out.

Troubleshooting efforts:

changing audio memory allocation (tried 4, 8, 12, 20) doesn't change outcome.

Removing the LCD touchscreen routines in loop doesn't change outcome.

Adding delay in loop seemed to improve outcome, except a failure was noted after 26 minutes.

The other teensy which sends the audio packets has never failed and has a similar, albeit less strenuous, interrupt routine to check for audio packets and spit them out over RS-485. The other teensy also doesn't connect to a touch screen or any other peripherals besides the recording SGTL5000.



Code:

/*

Special Notes:
  noteFreq library must be modified to reduce calculation time.
  [ Located in analyze_notefreq.h ]
    #define AUDIO_GUITARTUNER_BLOCKS  6
  
  serial buffer size is considerably larger to accomodate audio data transfer. 
  [ Located in Serial1.c ]
    #define TX_BUFFER_SIZE     512 // number of outgoing bytes to buffer
    #define RX_BUFFER_SIZE     512 // number of incoming bytes to buffer
 
To do:
  implement notes about the RGB 565 encoding of the ILI9341
    in the bit compressed format, the Red is 5 MSB's, green is the next 6 bits, and blue is 5 remaining the LSB's
    of a 16 bit string

  setup some bidirectional traffic
    need to figure out if a collision will ever happen when switching UART modes

*/

#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SD.h>
#include <SerialFlash.h>
#include <ILI9341_t3.h>
#include <font_Arial.h> // from ILI9341_t3
#include <XPT2046_Touchscreen.h>
#include "enums.h"

// GUItool: begin automatically generated code
AudioPlayQueue           audio_queue;         //xy=307,242
AudioOutputI2S           i2s1;           //xy=436,173
AudioConnection          patchCord1(audio_queue, 0, i2s1, 0);
AudioConnection          patchCord2(audio_queue, 0, i2s1, 1);
AudioControlSGTL5000     sgtl5000_1; //xy=387,310
// GUItool: end automatically generated code


//Setup the WatchDog Timer
#ifdef __cplusplus
extern "C" {
#endif
  void startup_early_hook() {
    //top value when counting at 200hz rate
    WDOG_TOVALL = 1600;
    //bottom value
    WDOG_TOVALH = 0;
    //start watchdog
    WDOG_STCTRLH = (WDOG_STCTRLH_ALLOWUPDATE | WDOG_STCTRLH_WDOGEN | WDOG_STCTRLH_WAITEN | WDOG_STCTRLH_STOPEN);
  }
#ifdef __cplusplus
}
#endif


// For optimized ILI9341_t3 library
#define CS_PIN  8
#define TFT_DC      20
#define TFT_CS      21
#define TFT_RST    255  // 255 = unused, connect to 3.3V
#define TFT_MOSI     7
#define TFT_SCLK    14
#define TFT_MISO    12
#define LCD_light   A2
ILI9341_t3 tft = ILI9341_t3(TFT_CS, TFT_DC, TFT_RST, TFT_MOSI, TFT_SCLK, TFT_MISO);
XPT2046_Touchscreen ts(CS_PIN);
#define TIRQ_PIN  2
//XPT2046_Touchscreen ts(CS_PIN);  // Param 2 - NULL - No interrupts
//XPT2046_Touchscreen ts(CS_PIN, 255);  // Param 2 - 255 - No interrupts
//XPT2046_Touchscreen ts(CS_PIN, TIRQ_PIN);  // Param 2 - Touch IRQ Pin - interrupt enabled polling
#define x_touch_max   3800
#define x_touch_min   150
#define y_touch_max   4000
#define y_touch_min   130
#define screen_width  320
#define screen_height 240
const double x_scale = (double)screen_width / (double)(x_touch_max - x_touch_min);
const double y_scale = (double)screen_height / (double)(y_touch_max - y_touch_min);


const int standard_color_default[17][4] = {
  // R   G   B   W
  {  0,  0,  0,  0}, //no color
  {255,  0,  0,  0}, //red
  {255,100,  0,  0}, //orange
  {255,255,  0,  0}, //yellow
  {100,255,  0,  0}, //yellow green
  {  0,255,  0,  0}, //green
  {  0,255,100,  0}, //ocean green
  {  0,255,255,  0}, //cyan
  {  0,100,255,  0}, //sky blue
  {  0,  0,255,  0}, //blue
  {100,  0,255,  0}, //purple
  {255,  0,255,  0}, //magenta
  {255,  0,100,  0}, //pink
  {  0,  0,  0,255}, //white
  {255,  0,  0,100}, //light red
  {  0,255,  0,100}, //light green
  {  0,  0,255,100}, //light blue
  };
int button_array[20][5];
int active_preset = 1;
int color_to_correct = magenta;
int vector_origin_color = red;
int vector_one_color = green;
int vector_two_color = blue;
int vector_three_color = white;
int standard_color[17][4];
int previous_menu = title_screen;

uint32_t watchdog_current_time;
uint32_t watchdog_dog_timer = 0;

//serial communication constants for audio transfer stuff
/*

Time(uS)  
Point     P1      p2      p3      p4      p5      p6      p7
          |       |       |       |       |       |       |
          |       |       |       |       |       |       period of wait
          |       |       |       |       |       violin reads
          |       |       |       |       user finishes transmittal
          |       |       |       user starts transmital
          |       |       violin turns off transmitter
          |       violin finishes last byte
          violin starts byte 0 transmission
  
  P1 - Violin Starts transmitting data
  P2 - Calculated or observed Violin Transmit Time finishes
  P3 - Violin turns to transmitter after transmit finished
  P4 - User should transition to transmitter a fixed time after last byte received
  P5 - User Finishes transmission and turns back to receiver
  P6 - Violin Reads the incoming bytes delay
*/
#define check_status_of_trigger_delay         100      //check for arrival of data every 100uS (short routine)
#define transceiver_mode_switch_delay         5        //arbitrary, could be as low as 1uS
#define serial1_speed                         1400000  //slowest baud rate without collisions
#define violin_receive_after_transmission     1950     //violin time
#define user_final_receive_to_transmit_delay  200      //user
#define user_transmit_finished                100      //user
#define violin_read_delay                     300      //violin time
#define audio_packet_length                   256      //audio packets are 16 bit 128 samples wide, thus 256 bytes
#define extra_bytes                           4
#define audio_message_length                  260      //sum of audio_packet_length and extra_bytes

#define transmit_enable_pin         4
byte  receive_buffer[4];
IntervalTimer myTimer;


/////////////////////////////////////////////////////////////////////////////////////////////////////
// Initialization                                                                                  //
/////////////////////////////////////////////////////////////////////////////////////////////////////
void setup(){
  
  //Start up communication with computer
  Serial.begin(250000);
  delay(1);

  //Start up communication with external teensy
  Serial1.begin(serial1_speed);
  pinMode(transmit_enable_pin,OUTPUT);
  digitalWrite(transmit_enable_pin,LOW);
  delayMicroseconds(transceiver_mode_switch_delay);

  //Setup Audio Memory
  //  Inventory of Objects using memory
  //    Output Audio - 2 blocks when writing a packet and another arrives
  //    Buffer for lockup safety - 8 blocks
  AudioMemory(10);

  //SGTL
  //  Turn on the device and set to appropriate volume level.
  sgtl5000_1.enable();
  sgtl5000_1.volume(0.8);
  delay(1);

  //start checking for audio packets
  myTimer.begin(receive_audio, check_status_of_trigger_delay);

  //setup the touchscreen device
  pinMode(LCD_light, OUTPUT);
  digitalWrite(LCD_light, HIGH); //turn on back light
  tft.begin();
  tft.setRotation(1);
  tft.fillScreen(ILI9341_BLACK);
  ts.begin();
  redisplay(title_screen);

  load_standard_colors();
}



/////////////////////////////////////////////////////////////////////////////////////////////////////
// Primary Interrupt Delay Routines                                                                //
/////////////////////////////////////////////////////////////////////////////////////////////////////
void receive_audio(){

  //ensure RS-485 is in receive mode
  digitalWrite(transmit_enable_pin,LOW);

  //check if samples are available
  int bytes_available = Serial1.available();
  if(bytes_available > 0){
    
    myTimer.end();
    uint32_t current_time = micros();
    boolean toss_sample = false;
    int bytes_previously_available = bytes_available;
    
    //wait for packet of fixed length to arrive, time out if middle of packet
    while(bytes_available < audio_message_length){

      //how many bytes do I have now?
      bytes_available = Serial1.available();
      
      //if more bytes have arrived, reset timeout
      if(bytes_available > bytes_previously_available){
        bytes_previously_available = bytes_available;
        current_time = micros();
      }
      
      //check for timeout, if timeout then flush and break
      uint32_t time_out = micros() - current_time;
      if(time_out > 100){
        toss_sample = true;
        Serial1.clear();
        myTimer.begin(receive_audio, check_status_of_trigger_delay);
        break;
      }
      
    }

    //If total amount of samples received in appropriate time
    if(toss_sample == false){

      //kick off transmit timer
      //the code below SHOULD finish before this interrupt gets called 
      myTimer.begin(transmit_data, user_final_receive_to_transmit_delay);
      
      //capture all data in respective buffers
      byte sample_buffer[audio_packet_length];
      for(int i = 0; i < audio_packet_length; i++){
        sample_buffer[i] = Serial1.read();
      }
      for(int i = 0; i < extra_bytes; i++){
        receive_buffer[i] = Serial1.read();
      }
  
      //pass the audio data out to the SGTL Audio Chip
      int16_t *audioBuffer = audio_queue.getBuffer();
      if (audioBuffer != NULL){  
        memcpy(audioBuffer, sample_buffer, audio_packet_length);
        audio_queue.playBuffer();
        //audio_queue.update();
      }

    }
    
  }

  //Sevice the Watch Dog
  watchdog_current_time = millis();
  if((watchdog_current_time - watchdog_dog_timer) > 100){
    noInterrupts();
    WDOG_REFRESH = 0xA602;
    WDOG_REFRESH = 0xB480;
    interrupts();
    watchdog_dog_timer = millis();
  }
  
}



void transmit_data(){

  //dont get interrupted by self
  myTimer.end();
  digitalWrite(transmit_enable_pin,HIGH);
  delayMicroseconds(transceiver_mode_switch_delay);
    
  //write data
  byte send_buffer[4];
  send_buffer[0] = 1;
  send_buffer[1] = 2;
  send_buffer[2] = 3;
  send_buffer[3] = 4;
  Serial1.write(send_buffer,4);
  
  //delay for a little until transmission on its way
  delayMicroseconds(user_transmit_finished);
  digitalWrite(transmit_enable_pin,LOW);
  delayMicroseconds(transceiver_mode_switch_delay);

  //reset timer to check for next serial event
  myTimer.begin(receive_audio, check_status_of_trigger_delay);
  
}




/////////////////////////////////////////////////////////////////////////////////////////////////////
// User Non-time sensitve functions                                                                //
/////////////////////////////////////////////////////////////////////////////////////////////////////
void loop() {
  
  boolean istouched = ts.touched();
  for(int i = 0; i<1000; i++){
    istouched = ts.touched();
    delayMicroseconds(5);
  }
  if (istouched) {

    //delay to allow a proper finger press on the screen
    delay(60); 

    //collect touch point and average over some samples
    int x_raw = 0;
    int y_raw = 0;
    int z_raw = 0;
    int average_samples = 5;
    int press_threshold = 1500; //of 4000
    TS_Point p = ts.getPoint();
    for(int i = 0; i<average_samples; i++){
      x_raw = p.x + x_raw;
      y_raw = p.y + y_raw;
      z_raw = p.z + z_raw;
      p = ts.getPoint();
      delay(10);
    }
    x_raw = x_raw / average_samples;
    y_raw = y_raw / average_samples;
    z_raw = z_raw / average_samples;

    //only permit high pressure press events through
    if(z_raw > press_threshold){

      //convert the raw touch screen input to 320x240 display coordinate
      int x_conv = (int)((x_touch_max - x_raw) * x_scale);
      int y_conv = (int)((y_touch_max - y_raw) * y_scale);

      //print some debug information
      //serial_print_touch_screen_position(x_raw,y_raw,z_raw,x_conv,y_conv);

      //draw the button press location for debug purposes
      draw_button_press_location(x_conv,y_conv);

      //check the position against button matrix
      int action_to_take = check_button_position_array(x_conv,y_conv);
      //Serial.println(action_to_take);

      //take update action
      //value of 100 is take no action 
      int update_screen = take_action_return_new_screen(action_to_take);

      //delay the input and then redraw the display
      delay(50);
      redisplay(update_screen);
      delay(250);
    }
    
  }

  report_audio_stats();

}

Any help is greatly appreciated

[rcarr]

//kick off transmit timer
//the code below SHOULD finish before this interrupt gets called
myTimer.begin(transmit_data, user_final_receive_to_transmit_delay);

Perhaps that is a poor assumption. If there is any delay in completing the function, you will send the 1234 packet and end up with nested versions of receive_audio running. It would be safer to have that code at the end of the function, and have a shorter delay if needed.

[josechow]

Perhaps that is a poor assumption. If there is any delay in completing the function, you will send the 1234 packet and end up with nested versions of receive_audio running. It would be safer to have that code at the end of the function, and have a shorter delay if needed.

Hey rcarr,

thanks for the reply. I did some testing with that assumption, and it holds. I did have the code after the serial reads, but I had to take care of timing correctly and account for micros() wraparound, so I kicked off the interrupt before the serial reads. I do agree that it would be safer.


Further testing.

1.) Possible Hardware issue - nope. I uploaded passthroughstereo, added a simple time print to the serial, and let it run over night. Program never stopped.

2.) Resource management problem - Maybe.

So during eminent audio playback failure, in addition to the cpu usage spike and increase in memory usage, the audio coming out of the speaker starts to get really crackly, as if the data is being corrupted or written while being read. On the oscilloscope you can see the 2.909ms sample get slowly replaced with random noise on one of the ends (couldn't tell if the first or last sample was being "zeroed" to noise).

So I ran with the hunch and ran the program above again with a modification to play_queue.cpp and play_queue.h by changing *queue to 64.

the thought was that the playqueue library has its own resource management to push the samples to the SGTL5000. Im not intimately familiar with how the Audio library works, but I am making a wild guess that playqueue has its own ring buffer which it pushes to the SGTL5000 that is being filled faster than the samples are being pulled off. After a certain amount of time the "head" wraps around and hits the "tail" and the playqueue.getBuffer doesn't know where to go, so it crashes into the sample for playback and the collision results in some errorneous operations and funny resource management results. In a previous test I increased Audiomemory to 60, and the sketch failed at 32 blocks.

I have confirmed one test so far, that increaseing the *queue buffer size (and subsequent head/tail position checks in .getBuffer() and .update() ) delays the collision, as the sketch hasn't failed in over 40 minutes. I will do further testing and report back. So far it seems that head/tail management needs to be performed/coded in play_queue.cpp to prevent these collisions.

[rcarr]

( I sort of expect this will be a rambling useless reply )

I was kind of thinking you tested the delay for normal operation, but might fail when there were some extra processing needed by the audio library or a buffer was not immediately available.

getBuffer();

Returns a pointer to an array of 128 int16. This buffer is within the audio library memory pool, providing the most efficient way to input data to the audio system. The buffer is likely to be populated by previously used data, so the entire 128 words should be written before calling playBuffer(). Only a single buffer should be requested at a time. This function may return NULL if no memory is available.

One would hope that means you are free to request the next buffer after calling playBuffer but before the system is done emptying the buffer that is playing ( meaning at least it is double buffered ). I wonder what happens if you request two buffers, is it like crossing streams in Ghostbusters?

One thing that comes to mind with the description of your system is a classic buffer problem that usually involves re-transmitting serial data streams but I guess it would also apply to audio also. If the sampling system is running just a little bit faster than the player system, data will arrive faster than you can get rid of it. Increasing the buffer size will only delay the inevitable data overrun that will eventually occur. The only solution would be to drop a sample now and then. It the audio arrives slower, you will have holes in the player stream where you do not have a sample to play when one is needed. You did not describe the sending system, so I guess I am assuming it is sampling and not pulling data from a SD card or something like that.

Edit: different sampling rate here would mean play back is at 44.1 khz and sampling is at 44.103 for example. Just a little bit faster.

[josechow]

( I sort of expect this will be a rambling useless reply )

Edit: different sampling rate here would mean play back is at 44.1 khz and sampling is at 44.103 for example. Just a little bit faster.

Yup, I think you are right.

The sending teensy is checking queue1.available() every 100uS. So an audio packet is arriving up to 100uS late sometimes, assuming all other timing operations the same between the two 44khz sampler/printer's, and the 96mhz timings all the same.

I am currently running tests and pulling out the head and tail values from play_queue.cpp. Head is ahead by one or two samples for a while, but tail catches up as expected from subsequent small delays of 100uS, and the difference from head to tail runs 0 for a while, then shows the failure signs as the head finally gets ahead of tail again. (Edit: How does it get ahead ? No idea.)

Pic attached. Gonna put some code in playBuffer(void) to make sure the head is always two samples ahead by skipping (old data will be printed?). And as you said, I will have to define the situation when the head moves faster than tail.





EDIT: From Paul's mouth. This might be one of the caveats of playqueue, haha.

[josechow]

Further troubleshooting, being verbose to help reason through my thought process (may be wrong).

Wrapping the audio_queue.getBuffer() with audio_queue.available(), prevents the sketch from crashing. Still however, at random times, between 7 and 15 minutes, the sketch fills AudioMemory. This is interesting to me as the only difference between getBuffer() and available() is an extra if(userblock) check. Audibly the memory failure causes increased amount of chirping as samples apparently disappear.

I pulled out the interrupt time of play_queue.update() and measured the difference between the update() time and the time when I prospectively want to access a audiomemory Data block using .getBuffer(). Memory problems occur when these two times are very close to each other (<45 uS) and they both occur frequently within the same time window.

I am sure that the audio library is very robust, but when concurrent ISR's try to access the same memory location, I can imagine a collision is bound to happen. This leads me to believe that *queue is the source of the problem, and getBuffer failure is only a symptom.

[josechow]

Alot of nonsense work for a small victory...

Went backthrough and read all the audio stuff and interrupt common notes.

I had IntervalTimer, for my main audio collection ISR to default interrupt priority, which was 128. The Audiostream ISR is 208, so the collisions were happening because of the lengthy code in my routines as compared to the more efficient audio library objects didn't yield to the Audiolibrary. I set it to 220 (lower priority than audio object isr's) and it works fine now. Well... I learned me something today.

Happy to report that one test so far is running for over 1hr 20 minutes with no chirping or hangup.