USB Midi Library blocking sysex messages

[halfordC]

Hey all,
I'm working on a project with the Akai Fire midi controller. It's got a big array of RGB buttons, and takes sysex to set the colors at an index.
Feel free to read about the whole implementation here.
Using a Teensy 4.1, nothing special on the schematic side, just have things wired up through the USB Host port, using the host library. Currently working on Platformio.

I'm trying to build a system where I have an internal array of RGB colors for each button, and send out the changes in that array every 32mS, if there are any changes.
For the sake of brevity, I'll just be including code that is relevant to the issue / question.

What that basically looks like:

Main:
A 1mS loop using interval timer:

/Global Includes
#include <Arduino.h>
#include "USBHost_t36.h"
#include <MIDI.h>

//local Includes
#include "AkaiFireState.h"

// Instatiate Global Objects
USBHost myusb;
MIDIDevice AkaiFire(myusb);
IntervalTimer myTimer;

AkaiFireState *AFState = new AkaiFireState(&AkaiFire, sequencer);

uint8_t msCounter = 0;
uint8_t updateFlip = 0;

//Setup Callbacks

void forgroundLoop()
{

  if(msCounter%16==0)
  {
    //do a screen update
    updateFlip = ~updateFlip;
    if(updateFlip)
    {
      AFState->sendPadLights();
      
    }else
    {
      //do an OLED screen update 
    }
  }

  msCounter++;
}

void setup() {
  myusb.begin();
  delay(4500); //wait for akai fire controller to finish boot sequence. 
  AFState->stateInit();

  myTimer.begin(forgroundLoop, 1000); //running at 1 ms = 1000nS
  Serial.println("Setup Complete");
}

void loop() {
  myusb.Task();
  AkaiFire.read();
}

I've also got a "state" class for the controller. Class with relevant functions:

void AkaiFireState::OnNoteOn(uint8_t channel, uint8_t note, uint8_t velocity)
{
      switch (note)
      {
            case PATTERN_SONG:
                performanceMode = SongMode;
                writePadRow(0,SONG_ROW_COLOR);
                writePadRow(3,PATTERN_ROW_COLOR);
                akaiFire->sendControlChange(PERFORM, 0x02, 0x01);
                akaiFire->sendControlChange(PATTERN_SONG, 0x04, 0x01);
            break;
        }
}
void AkaiFireState::sendPadLights()
{
    //if we have no updates, do not send. 
    if(padLightsToSend.empty())
    {
        return;
    }
    AFControl->sendPadUpdate(padLightsToSend);

    akaiFire->sendSysEx(AFControl->sysexBuffer.size(), AFControl->sysexBuffer.data(), true);

    padLightsToSend.clear(); //after we send, start over. 
}

void AkaiFireState::writePadLight(uint8_t row, uint8_t column, uint32_t color)
{
    
    padLightIndex[row*3][column] = (uint8_t)(color>>16); //red
    padLightIndex[(row*3)+1][column] = (uint8_t)(color>>8); //green
    padLightIndex[(row*3)+2][column] = (uint8_t)(color); //blue

     //if pad is already in vector, edit pad, do not re-write
    for(unsigned int i = 0; i<padLightsToSend.size(); i = i+4)
    {
        if(padLightsToSend[i] == (row*0x10 + column)) //checking to see if the index already exists. 
        {
            padLightsToSend[i+1] = ((color >> 17)& 0x7F);
            padLightsToSend[i+2] = ((color >> 9)& 0x7F);
            padLightsToSend[i+3] = ((color >> 1)& 0x7F);

            return;
        }
    }
    padLightsToSend.push_back(row*0x10 + column); //button index
    padLightsToSend.push_back((color >> 17)& 0x7F); //red
    padLightsToSend.push_back((color >> 9)& 0x7F); //green
    padLightsToSend.push_back((color >> 1)& 0x7F); //blue
}

void AkaiFireState::writePadRow(uint8_t row, uint32_t color)
{
    for(int i = 0; i<16; i++)
    {
        writePadLight(row, i, color);
    }
}

So, the general flow of the issue looks like this:

-> Perform note hit, calling writePadRow twice
-> writePadRow edits each light in the internal array, and pushes changes to vector with writePadLight
-> the 32ms timer hits at some point in time, and sends out the two rows of lights saved in the vector.

After this happens, the teensy locks up, and must be reset.
There seems to be some unsafe memory thing happening, accessing a null part of the vector, or something similar.

Initially, my thoughts are that the sendSysEx function is non-blocking. So, there is some USB midi system sending out this information in the background,
and I end up clearing the vector in the middle of that sending process in the background.

This doesn't happen when I send a single light, or even one row. But something about sending multiple rows causes this to happen.

I also think this might not be the case, and there might be something else happening that I don't understand.

I busted out trusty logic analyzer (since I don't really have a good way to step through teensy code in a proper debugger ATM)
And did some digitalWriteFast's around all of the relevant pieces mentioned here.
I also wrote a version of the sendPadLights function without the clear vector call, from inside the switch in the note_on function.
I seemed to encounter the same issue, in a different way.

Signal Legend:
SendPadRow -> turns high right before AFControl->sendPadUpdate(padLightsToSend);, after the check to see if the vector is not empty.
WritePadRow -> High when entering WritePadRow function, low when exiting.
EnterExitSendPad -> wrapped around AFState->sendPadLights(); in main, high before and low after
WriteScreen/Pad -> high when 32mS has passed and we get the signal to write the screen, low 16ms after that when I plan on writing to the OLED screen
enterSongMode -> the button to start this whole thing off is pressed, and that midi note is sent. Basically, the "PATTERN_SONG" case in the note_on function above.

So, Logic capture with clear vector call:

When this happens, no sysex gets sent, and the teensy crashes / locks up.


Logic capture with no clear vector call:

This time the vector did send the midi message, and the RGB lights did change, but the teensy still crashed / locked up.
You can see the blip on SendPadRow, it would seem that it send the message the first time without issue.
But, since the vector wasn't cleared, when the 30mS interrupt hit, the clearing function happened inside that interrupt call as above.


SO
The two big questions:
- Is the sendSysex call non blocking? Could I be clearing a vector before things get sent out? If so, is there some flag I can check to make sure this doesn't happen?
- Are my assumptions about vectors and C++ interrupts in teensy world just wrong, and my approach is bad?

I do realize that these two things are not mutually exclusive.
Let me know if this should live in Project guidance, I'll delete and post over there.
This just seemed like it might be a documentation gap in the USB Mid Library section, potentially? or not.

Anyway, thanks for reading my book of a question.
-Hal

 

[kd5rxt-mark]

If you suspect a crash/lockup of some sort, add the following at the top of your loop() function (The version proposed by @KurtE can be used to make sure that you have time to read the report before the Teensy reboots):

if (CrashReport) {
    while (!Serial && millis() < 5000) ; // wait awhile to make sure that the Serial port is initialized & available
    Serial.print(CrashReport);
    Serial.println("Press any key to continue");
    while (Serial.read() == -1) ;
    while (Serial.read() != -1);
}

Hope that helps . . .

Mark J Culross
KD5RXT

 

[halfordC]

If you suspect a crash/lockup of some sort, add the following at the top of your loop() function (The version proposed by @KurtE can be used to make sure that you have time to read the report before the Teensy reboots):

if (CrashReport) {
    while (!Serial && millis() < 5000) ; // wait awhile to make sure that the Serial port is initialized & available
    Serial.print(CrashReport);
    Serial.println("Press any key to continue");
    while (Serial.read() == -1) ;
    while (Serial.read() != -1);
}

Hope that helps . . .

Mark J Culross
KD5RXT

This doesn't seem to return anything. Maybe teensy isn't crashing, it's just getting stuck in an infinite loop somewhere? Or, if it's getting stuck in the ISR, It might not ever be getting to the "loop" section of code.

 

[defragster]

Nice to have the logic analyzer! Put a digitalWriteFast() around loop() to tell if it is running.

If SerMon is online and CrashReport isn't stopping to show info then the nature of the problem would seem to be 'stuck somewhere'.

Changing to this might confirm that better when "no CrashReport stored." is shown and that SerMon was indeed online out notice:

while (!Serial && millis() < 5000) ; // wait awhile to make sure that the Serial port is initialized & available
if (CrashReport) {
    Serial.print(CrashReport);
    Serial.println("Press any key to continue");
    while (Serial.read() == -1) ;
    while (Serial.read() != -1);
}
else
    Serial.println("no CrashReport stored.");

Perhaps before exiting setup() put in code for the suspect sendPadLights() a couple of times - whatever can be done from "the general flow" without needing external input to trigger to exercise the code?

 

[halfordC]

So, Now I'm really scratching my head.

I've changed up my approach to the problem. It seemed like the data structure I'm using to represent this stuff is just kindof wrong, and I don't really need to clear out a vector everytime I send something over sysex.
I still want to keep the state of what color the LED matrix is in memory, so now I just have a uint16_t [4] array, and set a bit everytime one of the LEDs get changed. once we write that to the sysex buffer, we clear the bit, but keep the state of the LED in memory. My thought is that getting rid of the clear all together like that plot above would solve this problem, but the exact same issue shows up, even without the clear.

I'll probably be pushing this to source control tomorrow, so yall can better take a look at it. But now I'm thinking that something in the OnNoteOn interrupt is messing with the timing of the interval timer.
Does teensy have an interrupt priority system I can assign these things too? or are the just sort of baked in?

I'll be doing some more work on this tomorrow, and post some results. Maybe some more logic analysis can reveal something I wasn't considering before.