Implementing Real-Time Audio with Interactive LCD

Willum_

Member
Hello, I'm working on a school project using the Teensy 4.0 and audio board for a real-time audio project that uses encoders and a standard 4x20 LCD using the PCF8574 to control the LCD with the LiquidCrystal_I2C library by YWROBOT for the user interface.

I've been working on creating the GUI for this project and tried using the IntervalTimer library as a way to generate the refresh interrupt for the LCD, but I've quickly ran into the problem of the audio cutting out at whatever interval I set the timer to. I have been trying to mess around with the priority of the interval in reference to the update function for the audio library, but I can't seem to get the audio not to glitch.

The Teensy 4.0 is a very capable device and I've seen projects involving much larger LCDs in conjunction with much higher refresh rates than i'm using so I'm sure my approach to doing this is just incorrect. I am not sure though if it is my choice in LCD library or method of generating the refresh interrupt or something else that is causing me problems. I'm also more generally trying to understand a good approach to designing an interactive system while using the audio library.

Attached below is my simple test that updates a value on the LCD at a given refresh rate while just passing the audio through some buffers. I greatly appreciate any insight!

Code:
#include <Audio.h>
#include <stdio.h>
#include <LiquidCrystal_I2C.h> //YWROBOT
#include <IntervalTimer.h>

#define AUDIO_MASK 0x0000FFFF
#define NUM_BLOCKS 8
#define CIRCULAR_BUFFER_LENGTH AUDIO_BLOCK_SAMPLES * NUM_BLOCKS
#define CIRCULAR_BUFFER_MASK (CIRCULAR_BUFFER_LENGTH - 1)
#define MAX_SAMPLE_BLOCKS NUM_BLOCKS - 1
//Audio library macro, AUDIO_BLOCK_SAMPLES = 128

AudioInputI2S i2s_in;
AudioOutputI2S i2s_out;
AudioRecordQueue Q_in_L;
AudioRecordQueue Q_in_R;
AudioPlayQueue Q_out_L;
AudioPlayQueue Q_out_R;
AudioConnection patchCord1(i2s_in, 0, Q_in_L, 0);
AudioConnection patchCord2(i2s_in, 1, Q_in_R, 0);
AudioConnection patchCord3(Q_out_L, 0, i2s_out, 0);
AudioConnection patchCord4(Q_out_R, 0, i2s_out, 1);
AudioControlSGTL5000 sgtl5000;

LiquidCrystal_I2C lcd(0x27,20,4);  // set the LCD address to 0x27
IntervalTimer lcdUpdate;

const int myInput = AUDIO_INPUT_LINEIN;
unsigned int curSample = 0;
int curBlock = 0;
// Temperary arrays to hold new values of current sample block
double leftIn[CIRCULAR_BUFFER_LENGTH] = {0.0};
double rightIn[CIRCULAR_BUFFER_LENGTH] = {0.0};
double leftOut[CIRCULAR_BUFFER_LENGTH] = {0.0};
double rightOut[CIRCULAR_BUFFER_LENGTH] = {0.0};
volatile int count = 0;
short *bp_L, *bp_R;

void setup(void)
{

  lcd.init();
  // Print a message to the LCD.
  lcd.backlight();
  lcdUpdate.begin(countByOne, 5e5); //lcd refresh rate in usec
  lcdUpdate.priority(255);
  // Audio connections require memory. and the record queue
  // uses this memory to buffer incoming audio.
  AudioMemory(10);
  // Enable the audio shield. select input. and enable output
  sgtl5000.enable();
  sgtl5000.inputSelect(myInput);
  sgtl5000.volume(0.5);
  // Start the record queues
  Q_in_L.begin();
  Q_in_R.begin();
  Serial.begin(9600);
}

void countByOne()
{
  unsigned long curTime = millis();
  count++;
  lcd.clear();
  char str[20];
  sprintf(str, "Count: %d", count);
  lcd.print(str);
  unsigned long time = millis() - curTime;
  Serial.printf("%lu\n", time);
}

void loop(void)
{
  // Wait for left and right input channels to have content
  while (!Q_in_L.available() && !Q_in_R.available());
  
  bp_L = Q_in_L.readBuffer();
  bp_R = Q_in_R.readBuffer();

  int startIndex = curBlock * AUDIO_BLOCK_SAMPLES;
  for (int i = 0; i < AUDIO_BLOCK_SAMPLES; i++) {
    curSample = startIndex + i;
    leftIn[curSample] = (double)bp_L[i];
    rightIn[curSample] = (double)bp_R[i];
  }

  Q_in_L.freeBuffer();
  Q_in_R.freeBuffer();

  // Get pointers to "empty" output buffers
  bp_L = Q_out_L.getBuffer();
  bp_R = Q_out_R.getBuffer();

  // Operate on each value in current block that has been received
  for (int i = 0; i < AUDIO_BLOCK_SAMPLES; i++) {
    curSample = startIndex + i;
  
    leftOut[curSample] = leftIn[curSample];
    rightOut[curSample] = rightIn[curSample];
    
  }

  // copy the processed data block back to the output
  for (int i = 0; i < AUDIO_BLOCK_SAMPLES; i++) {
    curSample = startIndex + i;
    bp_L[i] = (short)leftOut[curSample];
    bp_R[i] = (short)rightOut[curSample];
  }
  

  curBlock++;
  curBlock &= MAX_SAMPLE_BLOCKS;

  // and play them back into the audio queues
  Q_out_L.playBuffer();
  Q_out_R.playBuffer();
}
 
How long does countByOne() take, worst case? I see you're printing that out... It looks as if it could be quite long, especially given the IntervalTimer guideline "keep your function short and avoid calling other functions if possible", which is always sensible for any interrupt-context function. In this instance it needs to be shorter than either the audio processing interval (about 2.9ms) or the acceptable latency for your application.

Let's walk it through one step at a time (ignoring IntervalTimer for now):
  1. at some point the audio engine runs, and makes a packet available
  2. your while() loop spots this, and exits
  3. you process the packet, queue the result and loop() exits
  4. at some point the audio engine runs, absorbs the queued packet and makes a new packet available
  5. back to step 2
  6. and so on...
Now what happens if, 0.1ms before step 1, the IntervalTimer fires, and its service routine takes 3.1ms?
  1. t=-0.1ms: IntervalTimer fires
  2. t= 0.0ms: audio interrupt fires, absorbs queued packet, makes new one available (step 4 above)
  3. t= 0.1ms (say): audio interrupt exits, IntervalTimer service routine resumes
  4. t= 2.9ms: audio interrupt fires: can't absorb non-existent queued packet so inserts silence; makes new one available (now there are two...)
  5. t= 3.1ms: IntervalTimer service routine exits
  6. loop() processes and queues the two available packets
  7. t= 5.8ms: audio interrupt fires, absorbs first queued packet, makes new one available
You have now experienced a glitch (2.9ms of silence), and your processing has got an extra 2.9ms latency. The good news is that because of that latency, there should be no further glitches*.

You may be better off either:
  • making your IntervalTimer run at a 1ms interval, doing your audio processing in it if a queued packet is available (no while loop to wait, obviously...), and...
  • ...updating the LCD in your main loop() - doesn't matter how long it takes, now; or
  • writing an AudioStream object to do your processing
The latter option isn't all that scary - AudioEffectRectifier is about the simplest example I can see.

*EDIT: if you don't have enough AudioMemory you may continue to get glitches because there aren't enough blocks available.
 
Last edited:
Alternative only use IntervalTimer to set a flag
that is then read in the main loop


Code:
volatile uint32_t updateLCD = 0;
void countByOne()
{
    updateLCD = 1;
}
void loop() {
   // your processing code

   if (updateLCD == 1)
   { 
       updateLCD = 0;
       unsigned long curTime = millis();
       count++;
       lcd.clear();
       char str[20];
       sprintf(str, "Count: %d", count);
       lcd.print(str);
       unsigned long time = millis() - curTime;
       Serial.printf("%lu\n", time);
   }
}
 
Hi Willum_
I am doing something similar using TeensyThreads multitasking library. The trick is to use small time slices of only 10 microsecs and to give the important threads 10 slices time.

Code in setup():

Code:
....
  threads.setSliceMicros(10); // 
  threads.addThread(tftThread, 10000); // 1
  threads.addThread(neoThread); // 2
  threads.addThread(footThread); // 3
  int idFile= threads.addThread(fileThread, 40000); //4
  threads.setTimeSlice(idFile, 10); // ten times higher priority
  threads.delay(100);
  queue1.begin();
  int idQueue= threads.addThread(doQueue); // 5
  threads.setTimeSlice(idQueue, 10); // ten times higher priority!
...

https://forum.pjrc.com/threads/6947...ice-Efficiency-Thread-Priority-and-Cycle-Time
Good luck!
Christof
 
This interested me, so I had a bit of a play. I started with this design (you can import it from the code, too, of course):
2022-02-16 16_27_43-Audio System Design Tool for Teensy Audio Library.png

This code has (I think) all the elements you had, but not a real LCD (because I don't have one of those). Instead I just put in a badly-behaved slowFunction() which takes from 1 to 14ms to run, increasing each time it's called. I also decreased the interval from 500ms to 50ms to make the glitches easier to capture. and put the serial print in the loop() where it belongs:
Code:
#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SD.h>
#include <SerialFlash.h>

#include <IntervalTimer.h>

// GUItool: begin automatically generated code
AudioSynthWaveform       waveform2; //xy=152,344
AudioSynthWaveform       waveform1;      //xy=158,285
AudioMixer4              mixer1;         //xy=331,319
AudioRecordQueue         queueIn1;         //xy=474,283
AudioPlayQueue           queueOut1;         //xy=603,283
AudioOutputI2S           i2s1;           //xy=749,308
AudioConnection          patchCord1(waveform2, 0, mixer1, 1);
AudioConnection          patchCord2(waveform1, 0, mixer1, 0);
AudioConnection          patchCord3(mixer1, queueIn1);
AudioConnection          patchCord4(mixer1, 0, i2s1, 1);
AudioConnection          patchCord5(queueOut1, 0, i2s1, 0);
AudioControlSGTL5000     sgtl5000;     //xy=770,352
// GUItool: end automatically generated code

volatile int processCount;
void myProcess(AudioRecordQueue& qIn,AudioPlayQueue& qOut)
{
  int16_t* pIn,*pOut;  

  // get pointers to data in and out:
  pIn  = qIn.readBuffer();
  pOut = qOut.getBuffer();

  // process the data:
  if (NULL != pIn && NULL != pOut)
    memcpy(pOut,pIn,AUDIO_BLOCK_SAMPLES*sizeof *pOut); // simple copy for now

  // send data and free input block:
  qIn.freeBuffer();
  qOut.playBuffer(); 

  // count the number if time we've been called
  processCount++;
}


// Emulate a time-consuming function:
void slowFunction(void)
{
  static int wait = 1;

  // Don't use delay() as that calls yield()
  // which may let other things run!
  uint32_t waitUntil = millis()+wait;
  while (millis() < waitUntil)
    ;

  // Slow it further on every call,
  // within limits
  if (wait < 14)
    wait++;  
}


volatile bool ticked;
volatile int tickedMs,tickedProcessCount;
// Function to run at intervals:
void intvlFn(void)
{
  static int count=0;
  // DO NOT DO THIS! It appears to work, but this function runs in interrupt
  // context so should be short and NOT call long complex functions
  // Serial.printf("Process count %d at %d milliseconds\n",processCount,millis());

  // Do this instead:
  if (count-- <= 0)
  {
    ticked = true;
    tickedMs = millis();
    tickedProcessCount = processCount;
    count = 10;
  }

  // Demonstrate what happens if you call a 
  // long-winded function:
  slowFunction();
}

IntervalTimer intvl;
void setup() {
  // plenty of buffers for the queue:
  AudioMemory(50); 

  // enable audio output:
  sgtl5000.enable();
  sgtl5000.volume(0.05); 

  // start test waveforms:
  waveform1.begin(0.5,500.0,WAVEFORM_SINE);
  waveform2.begin(0.5,550.0,WAVEFORM_SINE);

  // start input queue:
  queueIn1.begin();

  // Set up IntervalTimer
  intvl.begin(intvlFn,50000);  // every 50ms
  intvl.priority(255);         // low priority
}

void loop() {

  // Wait (doing NOTHING) until 
  // audio queue has data
  while (!queueIn1.available())
    ;
  // and process it
  myProcess(queueIn1,queueOut1);

  // This is what you should do to run a long / complex
  // function at regular(ish) intervals.
  if (ticked)
  {
    ticked = false;
    Serial.printf("Process count %d at %d milliseconds\n",tickedProcessCount,tickedMs);
  }
}
Doing this I captured this trace; the source signal is in yellow, the processed one in magenta:
glitch.jpg
This shows 4 glitches, with a zoomed-in view of the second one. You can see the latency is already 5.8ms before the glitch, which inserts 2.9ms of silence, and thus gives a latency of 8.7ms afterwards.

Now here's a New And Improved version. The design intent is the same, but the results are different:
Code:
#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SD.h>
#include <SerialFlash.h>

#include <IntervalTimer.h>

// GUItool: begin automatically generated code
AudioSynthWaveform       waveform2; //xy=152,344
AudioSynthWaveform       waveform1;      //xy=158,285
AudioMixer4              mixer1;         //xy=331,319
AudioRecordQueue         queueIn1;         //xy=474,283
AudioPlayQueue           queueOut1;         //xy=603,283
AudioOutputI2S           i2s1;           //xy=749,308
AudioConnection          patchCord1(waveform2, 0, mixer1, 1);
AudioConnection          patchCord2(waveform1, 0, mixer1, 0);
AudioConnection          patchCord3(mixer1, queueIn1);
AudioConnection          patchCord4(mixer1, 0, i2s1, 1);
AudioConnection          patchCord5(queueOut1, 0, i2s1, 0);
AudioControlSGTL5000     sgtl5000;     //xy=770,352
// GUItool: end automatically generated code

volatile int processCount;
void myProcess(AudioRecordQueue& qIn,AudioPlayQueue& qOut)
{
  int16_t* pIn,*pOut;  

  // get pointers to data in and out:
  pIn  = qIn.readBuffer();
  pOut = qOut.getBuffer();

  // process the data:
  if (NULL != pIn && NULL != pOut)
    memcpy(pOut,pIn,AUDIO_BLOCK_SAMPLES*sizeof *pOut); // simple copy for now

  // send data and free input block:
  qIn.freeBuffer();
  qOut.playBuffer(); 

  // count the number if time we've been called
  processCount++;
}


// Emulate a time-consuming function:
void slowFunction(void)
{
  static int wait = 1;

  // Don't use delay() as that calls yield()
  // which may let other things run!
  uint32_t waitUntil = millis()+wait;
  while (millis() < waitUntil)
    ;

  // Slow it further on every call,
  // within limits
  if (wait < 14)
    wait++;  
}


volatile bool ticked;
volatile int tickedMs,tickedProcessCount;
// Function to run at intervals:
void intvlFn(void)
{
  static int count=0;
  // DO NOT DO THIS! It appears to work, but this function runs in interrupt
  // context so should be short and NOT call long complex functions
  // Serial.printf("Process count %d at %d milliseconds\n",processCount,millis());

  // Do this instead:
  if (count-- <= 0)
  {
    ticked = true;
    tickedMs = millis();
    tickedProcessCount = processCount;
    count = 499;
  }

  // poll to see if queue processing is needed
  while (queueIn1.available())
    myProcess(queueIn1,queueOut1);  
}

IntervalTimer intvl;
void setup() {
  // plenty of buffers for the queue:
  AudioMemory(50); 

  // enable audio output:
  sgtl5000.enable();
  sgtl5000.volume(0.05); 

  // start test waveforms:
  waveform1.begin(0.5,500.0,WAVEFORM_SINE);
  waveform2.begin(0.5,550.0,WAVEFORM_SINE);

  // start input queue:
  queueIn1.begin();

  // Set up IntervalTimer
  intvl.begin(intvlFn,1000);  // every 1ms
  intvl.priority(255);        // low priority
}

void loop() 
{
  // This is what you should do
  if (ticked)
  {
    ticked = false;
    slowFunction();
    Serial.printf("Process count %d at %d milliseconds\n",tickedProcessCount,tickedMs);
  }
}
Full disclosure, slowFunction() is now called every 0.5s, as with your original code! But it appears that it'll run indefinitely with just the inherent 2.9ms delay between source and processed signals. As a bonus, it no longer sits around doing nothing for up to 2.9ms, waiting for the audio engine to queue data for processing, so it can be more responsive. One thing to be aware of is that as written, the processing function needs to execute in less than 1.9ms, because it may take intvlFn() 1ms to notice the audio engine has queued data for it. Obviously you could improve that if necessary.
 
You may be better off either:
  • making your IntervalTimer run at a 1ms interval, doing your audio processing in it if a queued packet is available (no while loop to wait, obviously...), and...
  • ...updating the LCD in your main loop() - doesn't matter how long it takes, now; or
  • writing an AudioStream object to do your

    The latter option isn't all that scary - AudioEffectRectifier is about the simplest example I can see.

    *EDIT: if you don't have enough AudioMemory you may continue to get glitches because there aren't enough blocks available.

As you and manicksan suggested, I put the countByOne code in the main loop below the audio processing and used the interrupt just to update a flag but it produces the same popping result on the test program until I also increased the block memory and not it works!

I understand my mistake of putting the time intensive code in the interrupt, but I'm not entirely understanding the role of allocating more audio memory blocks in this instance. Should I be allocating enough blocks to account for whatever delay I have in updating the screen? i.e. If the code takes 14ms it should allocate ~5 blocks (14/2.9=5) to total memory allocation since it is waiting in the buffer to be read in.
 
Hi Willum_
I am doing something similar using TeensyThreads multitasking library. The trick is to use small time slices of only 10 microsecs and to give the important threads 10 slices time.

Code in setup():

Code:
....
  threads.setSliceMicros(10); // 
  threads.addThread(tftThread, 10000); // 1
  threads.addThread(neoThread); // 2
  threads.addThread(footThread); // 3
  int idFile= threads.addThread(fileThread, 40000); //4
  threads.setTimeSlice(idFile, 10); // ten times higher priority
  threads.delay(100);
  queue1.begin();
  int idQueue= threads.addThread(doQueue); // 5
  threads.setTimeSlice(idQueue, 10); // ten times higher priority!
...

https://forum.pjrc.com/threads/6947...ice-Efficiency-Thread-Priority-and-Cycle-Time
Good luck!
Christof

Thanks for the idea. One of my current concerns is if the simple LCD update function I'm making will scale well if I have a moderately complicated menu system that call audio functions on selection so I'll have to look more into this library.
 
One thing to know with time slices or preemptive multitasking is that you need to take extra care when sharing variables between threads, as the 'task switcher' can interrupt the current running thread at any time.
 
Last edited:
As you and manicksan suggested, I put the countByOne code in the main loop below the audio processing and used the interrupt just to update a flag but it produces the same popping result on the test program until I also increased the block memory and not it works!
Presumably now it works :) - good!
I understand my mistake of putting the time intensive code in the interrupt, but I'm not entirely understanding the role of allocating more audio memory blocks in this instance. Should I be allocating enough blocks to account for whatever delay I have in updating the screen? i.e. If the code takes 14ms it should allocate ~5 blocks (14/2.9=5) to total memory allocation since it is waiting in the buffer to be read in.
Pretty much correct, but bear in mind the number of queues [queue pairs] you're processing. My code had just one, but yours had two, so you'd need at least 10 blocks just for your queues, on top of any requirements from the other parts of your design. That is if you insist on doing your processing within loop(): if you adopt something like my second scheme of running IntervalTimer at a 1ms period and doing the processing there, then you get away with fewer blocks because both IntervalTimer and audio can interrupt your long-winded LCD update.

I suspect you'll find, if you can test it, that you get a few pops early on in the processing until the queues settle down to their correct length. Could be just one pop, if the first update falls just right. Once they have settled you'll have a fixed in-to-out delay - if it's 5 blocks then that'll be 14.5ms. You could pre-load the output queues in setup() by getting, setting to 0, and playing 5 buffers per queue. That should prevent any pops, I think. But it's a lot of effort to go to when better solutions exist...
 
The reason i've been doing the audio processing in the main loop and not utilizing the flow of the audio library is there does not seem to be a clear way to me to dynamically switch between what audio objects are being utilized other than maybe connecting them all to a mixer and setting their gain to 0 when not in use. In the future of this project there will be many features that will dynamically be enabled/disabled in the system.
 
The reason i've been doing the audio processing in the main loop and not utilizing the flow of the audio library is there does not seem to be a clear way to me to dynamically switch between what audio objects are being utilized other than maybe connecting them all to a mixer and setting their gain to 0 when not in use. In the future of this project there will be many features that will dynamically be enabled/disabled in the system.

Well, at the risk of blowing my own trumpet, you might want to take a look at this thread ... https://forum.pjrc.com/threads/66840-Roadmap-quot-Dynamic-Updates-quot-any-effort-going-on. If the existing audio library can implement each of your "many features" one at a time, then this will allow you to create and destroy them on the fly.

Having said that, even with the standard audio library there's still no very good reason to put your audio processing directly in loop(). Adapting my second code example in post#5 to switch between different processing functions is as simple as calling intvl.end(), then intvl.begin() with a pointer to a different processing function, as needed.
 
Back
Top