Teensy Crashes on USBHost_t32 MIDI Stop ALL Notes

[browncauk]

Hi,

I have a highly specific crash case that I'm looking for help to resolve.

I have a product that uses a Micromod Teensy 4 and the USBHost_t36 library to communicate with external MIDI devices. I've discovered a crash in my device when connected to a specific MIDI hardware device (Behringer BCF2000) but I have not seen similar crashes in other devices using the same code.

I'm just using Arduino IDE and the test programs below.

The issue lies with the call to: -

hostMIDI.sendControlChange(0x78, 0, channel);

However, it only occurs when this call is made multiple times inside an interval timer.

Here's a worked-through example: -

This code works perfectly fine (when connected to the device)...

#include <USBHost_t36.h>

USBHost usbHost;
MIDIDevice_BigBuffer hostMIDI(usbHost);
unsigned int sw;
int c = 0;

const int NO_OF_CHANNELS = 16;

void setup() {
  Serial.begin(9600);
  usbHost.begin();
}

void loop() {
    Serial.println(c++);
    sw = micros();
  
    for (int channel = 1; channel <= NO_OF_CHANNELS; channel++) {
      // send StopAllNotes msg
      hostMIDI.sendControlChange(0x78, 0, channel);    
    }  

    Serial.print("Time: ");
    Serial.println(micros() - sw);
}

You can see I'm making 16 calls to stop all notes. One call for each MIDI channel. (Yes, I know, but bear with me).

(The stopwatch sw shows around 8000µs).

The following code does the same but from within a fairly slow-running interval timer loop.

#include <USBHost_t36.h>

USBHost usbHost;
MIDIDevice_BigBuffer hostMIDI(usbHost);

IntervalTimer midiLoopTimer;
unsigned int sw;

const int NO_OF_CHANNELS = 2;

void setup() {
  Serial.begin(9600);
  usbHost.begin();

  midiLoopTimer.begin(midiLoop, 1000000);
}

int c = 0;
void midiLoop()
{
    Serial.println(c++);
    sw = micros();
  
    for (int channel = 1; channel <= NO_OF_CHANNELS; channel++) {
      // send StopAllNotes msg
      hostMIDI.sendControlChange(0x78, 0, channel);    
    }  

    Serial.print("Time: ");
    Serial.println(micros() - sw);
}

void loop() {
}

If you set NO_OF_CHANNELS to 2, all works fine, and the stopwatch sw shows only 2µs. However, if you set NO_OF_CHANNELS to 3, the device crashes!

The behaviour is the same if I replace sendControlChange with sendNoteOff. It still crashes when setting the NO_OF_CHANNELS any higher than 2.

I've been through the USBHost_t36 midi.cpp code and nothing obvious jumps out. The USBHOST_PRINT_DEBUG debug flag is only showing the transmitted messages, there's nothing incoming.

If I connect other MIDI devices (e.g. Beatstep Pro) the code works fine, right up to 16 channels.

There's clearly something in the way the BCF device interacts with my device but I'm wondering how to debug it?

It is most confusing and any suggestions on what to try or how to dig deeper would be very much appreciated.

Thanks in advance.

Chris

 

[PaulS]

However, if you set NO_OF_CHANNELS to 3, the device crashes!

Does "crashes" mean that BCF2000 becomes unresponsive or...?

Could it be that the BCF2000 sends a lot data to the host which are not dealt with? I see in all MIDI examples these lines:

void loop() {
  // The handler functions are called when midi1 reads data.  They
  // will not be called automatically.  You must call midi1.read()
  // regularly from loop() for midi1 to actually read incoming
  // data and run the handler functions as messages arrive.
  myusb.Task();
  midi1.read();
}

Perhaps adding these lines to your code help?

Paul

 

[browncauk]

Does "crashes" mean that BCF2000 becomes unresponsive or...?

Could it be that the BCF2000 sends a lot data to the host which are not dealt with? I see in all MIDI examples these lines:

void loop() {
  // The handler functions are called when midi1 reads data.  They
  // will not be called automatically.  You must call midi1.read()
  // regularly from loop() for midi1 to actually read incoming
  // data and run the handler functions as messages arrive.
  myusb.Task();
  midi1.read();
}

Perhaps adding these lines to your code help?

Paul

Cheers for the response.

When I say "crash", I mean my device (teensy) becomes unresponsive. BCF remains responsive, I can still output MIDI data from it using a MIDI Cable as opposed to USB.

It doesn't matter if I include read() code or not, no response messages are received from the BCF. There is no debugging info logged by USBHost_T36, relating to incoming messages, when I send those MIDI messages (in the example where my device does not crash). In the example where my device does crash, I cannot capture any additional debug logs.

 

[PaulS]

To check what the BCF2000 is actually returning when sending your CC message, you could use MIDI-OX. Perhaps you see something that could explain the crashing of ther Teensy.

Connect the BCF2000 to your PC, start MIDI-OX, configure it like below and send out the CC message(s).
[I don't have a BCF2000 but a simple Teensy-based MIDI-controller].



Hope this helps.
Paul

PS: does the BCF2000 need a driver to operate under Windows?

 

[KurtE]

Sorry, I have not done anything with Midi so usually leave that part of the USBHost_t36 stuff to Paul.

Cheers for the response.
...
It doesn't matter if I include read() code or not, no response messages are received from the BCF. There is no debugging info logged by USBHost_T36, relating to incoming messages, when I send those MIDI messages (in the example where my device does not crash). In the example where my device does crash, I cannot capture any additional debug logs.

In USBHost have you turned on debug printing?
Go into USBHost_t36.h and about line 63 uncomment the line:

// Uncomment this line to see lots of debugging info!
#define USBHOST_PRINT_DEBUG

And rebuild your sketch.

If you look into the file midi.cpp you will see lots of lines like:

		println("  Interface is unknown (might be Yahama)");

These will now output, defaulting to Serial.
There are others in this file that are commented out, like:

bool MIDIDeviceBase::claim(Device_t *dev, int type, const uint8_t *descriptors, uint32_t len)
{
	// only claim at interface level
	if (type != 1) return false;
	println("MIDIDevice claim this=", (uint32_t)this, HEX);
	println("len = ", len);

	const uint8_t *p = descriptors;
	const uint8_t *end = p + len;

	if (p[0] != 9 || p[1] != 4) return false; // interface descriptor
	//println("  bInterfaceClass=", p[5]);
	//println("  bInterfaceSubClass=", p[6]);
	bool ismidi = false;

During debugging, you may find you want to uncomment some of them.

 

[browncauk]

Thank you for the suggestions, I really appreciate your help. I'll follow this up in the morning and report back.

 

[browncauk]

To check what the BCF2000 is actually returning when sending your CC message, you could use MIDI-OX. Perhaps you see something that could explain the crashing of ther Teensy.

This was a great suggestion and definitely worth trying.

However, I could detect no response messages from the BCF when manually sending All Sounds Off via MIDI-OX.



I double-checked that all filters were off and that I am receiving CC messages from the BCF.

This is consistent with the behaviour I see in my test program when sending the messages in the normal loop() code, i.e. NOT within an IntervalTimer. There are no response messages from the BCF detected on read(). I wouldn't expect to, given the MIDI standard.

PS: does the BCF2000 need a driver to operate under Windows?

The BCF2000 doesn't need any specific drivers loading, it is detected by Windows 11.

 

[browncauk]

In USBHost have you turned on debug printing?

Yes, I should have posted the log files from my test programs. Please see logs attached for the 3 examples:

View attachment log - Loop example - 16 channels.txt log - Loop example - 16 channels - Working
View attachment log - Timer example - 2 channels.txt log - Timer example - 2 channels - Working
View attachment log - Timer example - 3 channels - Crash.txt log - Timer example - 3 channels - Crashes

On PaulS' suggestion, I added read() logic into my test code and it detects MIDI CC input from the BCF:

MIDIDevice Receive
  MIDI Data: 0B B0 51 12 
avail = 398
queue another receive packet
MIDIDevice Receive
  MIDI Data: 0B B0 51 13 
avail = 397
queue another receive packet
MIDIDevice Receive
  MIDI Data: 0B B0 51 14 
avail = 396
queue another receive packet

So, I know I'm detecting any response from the BCF in the logs.

In the working Timer example (2 channels), there are no incoming messages detected.

In the crashing Timer example (3 channels) there is nothing logged beyond the initial USB device detection because Teensy becomes unresponsive before Serial can output. I assume this happens on the first IntervalTimer loop because the interval timing is 1 second; long enough for serial output logic to be run.

This is a mystery!

Here's a wild theory... I wonder if there is an output buffer (somewhere in the Teensy code) that overflows because the BCF has not returned a valid response or if the BCF has an issue that delays acknowledgement long enough to cause an issue in Teensy. The BCF then recovers but Teensy does not.

I just don't know enough about USB to know what's going on.

I'm stuck for a solution at the moment. While the scope of the problem is currently limited to the BCF, there may be other devices out there that my customers want to use that have the same issue.

I really appreciate your help with this though and I will continue to try to understand more about the USBHost_t36 code.

 

[browncauk]

I just retested the same test program but with a different device (Arturia Beatstep Pro) and using 16 channels. The program works fine and Teensy does not crash

From the logs there is a clear difference and I've highlighted the relevant sections here:-

BCF2000 (sending 2 channels)

    MIDI Endpoint: 2
      tx_size = 4
    MIDI Endpoint: 81
      rx_size = 64
	  
MIDIDevice transmit complete
  MIDI Data: 0B B0 78 00 
MIDIDevice transmit complete
  MIDI Data: 0B B1 78 00

BEATSTEP Pro (sending 16 channels)

    MIDI Endpoint: 2
      tx_size = 64
    MIDI Endpoint: 81
      rx_size = 64
	  
MIDIDevice transmit complete
  MIDI Data: 0B B0 78 00 0B B1 78 00 0B B2 78 00 0B B3 78 00 0B B4 78 00 0B B5 78 00 0B B6 78 00 0B B7 78 00 0B B8 78 00 0B B9 78 00 0B BA 78 00 0B BB 78 00 0B BC 78 00 0B BD 78 00 0B BE 78 00 0B BF 78 00

So, it seems the data is being sent in a single packet (up to 64 bytes) for the Beatstep, but only in 4 byte packets (one for each channel) for the BCF.

Here's the full log for the Beatstep: View attachment log - Timer example - 16 channels - Beatstep - Working.txt

 

[jmarsh]

I think I can guess what's happening:
- IntervalTimer by default sets the priority of the PIT IRQ to 128.
- The USB Host driver doesn't bother setting any priority for the USB HOST IRQ, startup.c initializes all IRQs to 128 so that's what is used.
- Because the BCF2000 only has a tx_size of 4, the outgoing buffer fills up and the code blocks waiting for a USB HOST IRQ to let it know some data has been sent and more can be queued for sending.
- That never happens because it's inside an active PIT IRQ, and only IRQs with higher priority can pre-empt an active IRQ.

 

[browncauk]

I think I can guess what's happening:
- IntervalTimer by default sets the priority of the PIT IRQ to 128.
- The USB Host driver doesn't bother setting any priority for the USB HOST IRQ, startup.c initializes all IRQs to 128 so that's what is used.
- Because the BCF2000 only has a tx_size of 4, the outgoing buffer fills up and the code blocks waiting for a USB HOST IRQ to let it know some data has been sent and more can be queued for sending.
- That never happens because it's inside an active PIT IRQ, and only IRQs with higher priority can pre-empt an active IRQ.

Am I right, then, in thinking that, to resolve this, the USB HOST IRQ needs a higher priority than the PIT IRQ?

 

[jmarsh]

You could try setting a lower priority for the IntervalTimer before starting it:

midiLoopTimer.priority(255); // higher value = lower priority
midiLoopTimer.begin(midiLoop, 1000000);

However this will only work if there are no other IntervalTimers with higher priority since they all share the same IRQ.

Ideally don't do any communication from IRQ handlers, use them to schedule/signal things to the main codepath.

 

[browncauk]

You could try setting a lower priority for the IntervalTimer before starting it:

midiLoopTimer.priority(255); // higher value = lower priority
midiLoopTimer.begin(midiLoop, 1000000);

It works! That's brilliant. I set a lower priority and now the BCF is receiving 16 channels of ALL SOUNDS OFF and I can even lower the Interval timer to run at 30µs and all is well.

However this will only work if there are no other IntervalTimers with higher priority since they all share the same IRQ.

Ideally don't do any communication from IRQ handlers, use them to schedule/signal things to the main codepath.

The reason I have the MIDI comms inside the Timer is that the main loop does the bulk processing to calculate the correct MIDI output and then queues it. The IntervalTimer loop, then, just reads off the queue. This ensures that the calculation code doesn't block the timing of the queued MIDI notes. I originally had it the other way around but really struggled with note timing issues; obviously a big issue for music devices. My device spits out a lot of MIDI data but also has to respond to Incoming MIDI in a timely fashion. It was a post from Paul that suggested the final route I took. It all works and I can output 4 channels of up to 32 notes per tick at 300bpm, which leaves only 28µs to calculate note sequences. This isn't enough, so the MIDI comms loop gets priority.

I don't have any other loops going and the device is solid in every other respect.

I'm not sure if or how lowering the IRQ priority will impact this but I will experiment and report back.

Thank you again for your assistance.

 

[browncauk]

I've added this to my production code and the device is now working perfectly with the BCF2000.

I cannot thank you all enough for your help with this!

jmarsh, KurtE, PaulS, Thank you! You are stars!