USBhost_t36 MIDI losing midi note on / off events (Teensy 3.6) during modwheel change

PaulS · Aug 9, 2023

Nope, unfortunately the updated midi.cpp did not solve the issue...
What I did:
1. Uninstalled the complete Arduino IDE.
2. Deleted the ../Arduino15 directory.
3. Reinstalled Arduino 1.8.19 & Teensyduino 1.58. Updated midi.cpp in C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\USBHost_t36.
4. Recompiled and uploaded the sketch from message #13 to the Teensy 4.1 [at 600 MHz & USB type Serial].
5. Connected the Arturia controller to the USB Host port & opened the serial monitor.
6. Slammed 12 keys at the same time on the Arturia > fail: (did not even manipulate the mod wheel...).

7. Unpowered the Teensy 4.1, disconnected the Arturia and connected the Teensy 4.0-based Arturia emulator [running the code of message #41].
8. Powered the Teensy 4.1 and watched the serial monitor > fail at the red arrow:

Expected Note Off, ch=1, note=60, velocity=63, >>>>>>>> 0.

I will keep debugging.
Paul

rvh · Aug 9, 2023

Thanks Paul, good to have the verification.

PaulStoffregen · Aug 9, 2023

I ordered the Arturia MiniLab MKII.

Can you try the latest code (now available as 0.59.3 in Arduino IDE 2.1.1 Boards Manager) with the Teensy 4.0 test program?

PaulS · Aug 10, 2023

PaulStoffregen said:
I ordered the Arturia MiniLab MKII

Wow, thanks for your great support!
I will try the latest code tomorrow.

Paul

PaulS · Aug 11, 2023

Switching to Arduino 2.1.1 with board package 0.59.3 did not solve the issue.
Steps to reproduce:
1. Updated Arduino 2.1.1 with board package 0.59.3.
2. Recompiled & uploaded both sketches from message #13 and #44 to Teensy 4.1 and Teensy 4.0 respectively.
3. Connected the Arturia and powered the system on.
4. Slammed a whole bunch of keys at the same time on the Arturia [no mod wheel action] > fail:

5. Powered down the system, removed the Arturia, connected the Teensy 4.0 and powered the system up again.
6. Changed one line in the test code: while (ms < 60000) { to while (ms < 600) {, for bursts every 0.6 sec instead of 1 every minute.
7. Let it run for 10 minutes [so ~1000 bursts] > pass.

Paul

PaulS · Aug 11, 2023

All this got me wondering whether the Arturia itsself did not send out correct data when hitting a lot of keys at the same time.
So I checked a whole lot slams with this tool.
Here is a screenshot of 17 keys hit at the same time:

I tried this a lot of times, but always got an even amount of messages.

Paul

rvh · Aug 11, 2023

PaulS said:
All this got me wondering whether the Arturia itsself did not send out correct data when hitting a lot of keys at the same time.
So I checked a whole lot slams with this tool.
Here is a screenshot of 17 keys hit at the same time:

View attachment 31740

I tried this a lot of times, but always got an even amount of messages.

Paul

Also, the Arturia Minilab only causes missing midi events on the Teensy when using the USB MIDI host port.
I have never seen the Teensy lose data when going into the serial or non-host USB MIDI port.

PaulStoffregen · Aug 11, 2023

Can I talk you into trying again with a Teensy 4.0 sending a long-but-finite set of MIDI messages?

My Arturia Minilab hasn't arrived yet. Even when it does, testing by manual input isn't consistent and takes a lot more work. A program on a Teensy 4.0 which sends a large set of MIDI messages allows me to be much more productive when investigating.

I can't use a program that keeps endlessly transmitting, like in msg #41. The USB protocol analyzer I use can only capture so much, and it will even crash on certain type of sustained maximum bandwidth streams. You might imagine something so expensive wouldn't come with so many limitations, but that's just the way it is. I need a test program which send a lot of MIDI messages to reproduce the problem, but then stops so I can look at the USB packet capture in the analyzer software.

PaulS · Aug 11, 2023

PaulStoffregen said:
Can I talk you into trying again with a Teensy 4.0 sending a long-but-finite set of MIDI messages?

Sure. How long is useful for you?

Paul

PaulStoffregen · Aug 11, 2023

Only just long enough to reliably reproduce the problem!

PaulS · Aug 11, 2023

PaulStoffregen said:
Only just long enough to reliably reproduce the problem!

Well, "reliably" turned out to be the keyword for the last few hours... I had to generate a lot of messages in a message burst to consistently get a fail.
[and the 2.1.1 IDE was fighting me with the serial port to fail after a while and requiring a restart, updating stuff in the background and 25% CPU time after closing the application...]

Anyway, here is the test code:

Code:

void setup() {
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, LOW);
  delay(10000);                                       // give Arduino 2.1.1 serial monitor ample time to get ready
}

void loop() {
  for (int runs = 0; runs < 4000; runs++) {
    digitalWriteFast(LED_BUILTIN, HIGH);

    for (int note = 0x3C; note < 0x4C; note++) {      // 16 notes ON
      usbMIDI.sendNoteOn(note, 0x3F, 1);              // incr notes ON, velocity 0x3F, channel 1
    }

    for (int i = 0; i <= 15; i++) {                   // send 16 CC messages
      usbMIDI.sendControlChange(0x01, 0x3F + i, 1);   // send CC modulation wheel, incr mod value, channel 1
    }

    for (int note = 0x3C; note < 0x4C; note++) {      // 16 notes OFF
      usbMIDI.sendNoteOff(note, 0x00, 1);             // incr notes OFF, velocity 0x00, channel 1
    }

    for (int i = 0; i <= 15; i++) {                   // send 16 CC messages
      usbMIDI.sendControlChange(0x01, 0x3F + i, 1);   // send CC modulation wheel, incr mod value, channel 1
    }

    digitalWriteFast(LED_BUILTIN, LOW);
    
    while (usbMIDI.read()) {                          // ignore incoming messages
    }
  }
  while (1) {
  }
}

Paul

rvh · Aug 11, 2023

Thanks Paul. Even with this many messages, the code above only produces 'stuck notes' on my T4.1 synth on something like 60-70% of trials.

rvh · Aug 11, 2023

PaulStoffregen said:
Can I talk you into trying again with a Teensy 4.0 sending a long-but-finite set of MIDI messages?

My Arturia Minilab hasn't arrived yet. Even when it does, testing by manual input isn't consistent and takes a lot more work. A program on a Teensy 4.0 which sends a large set of MIDI messages allows me to be much more productive when investigating.

I can't use a program that keeps endlessly transmitting, like in msg #41. The USB protocol analyzer I use can only capture so much, and it will even crash on certain type of sustained maximum bandwidth streams. You might imagine something so expensive wouldn't come with so many limitations, but that's just the way it is. I need a test program which send a lot of MIDI messages to reproduce the problem, but then stops so I can look at the USB packet capture in the analyzer software.

To localize the problem may require feedback from the receiving teensy back to the host teensy to tell it to stop transmititng when an error occurs (using e.g. known note sequences where the number of notes left on should be zero at the end of the run). I can't see how else you can know whether a captured USB stream is problematic for the host port.

PaulS · Aug 11, 2023

rvh said:
To localize the problem may require feedback from the receiving teensy back to the host teensy to tell it to stop transmititng when an error occurs (using e.g. known note sequences where the number of notes left on should be zero at the end of the run). I can't see how else you can know whether a captured USB stream is problematic for the host port.

Yeah, that came to my mind as well. What I understand is that USB is a polled bus where the host initiates all transactions so perhaps there is a programmatic way to tell the host to no longer request USB packets from the device?

Paul

PaulS · Aug 12, 2023

OK, have been thinking about a better test.
What I came up with is this:
1. have the Arturia emulator generate a specific testpattern with a defined number of notes every 100 msec,
2. let the host check for the correct number of notes at the last MIDI message of each testpattern,
3. if not the correct number of notes, kill the power to the Arturia emulator so no more testpatterns will be outputted.

Here is the Arturia emulator code:

Code:

void setup() {
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, LOW);
  delay(10000);
}

void loop() {
  digitalWriteFast(LED_BUILTIN, HIGH);  // show run activity
  for (int note = 0x00; note <= 0x7F; note++) {   // 128 notes ON
    usbMIDI.sendNoteOn(note, 0x3F, 1);            // incr notes ON, velocity 0x3F, channel 1
  }
  for (int note = 0x00; note <= 0x7F; note++) {   // 128 notes OFF
    usbMIDI.sendNoteOff(note, 0x00, 1);           // decr notes OFF, velocity 0x00, channel 1
  }
  while (usbMIDI.read()) {  // ignore incoming messages
  }
  digitalWriteFast(LED_BUILTIN, LOW);
  delay(100);
}

And here is the host code:

Code:

#include <USBHost_t36.h>

USBHost myusb;
MIDIDevice_BigBuffer midi1(myusb);

void setup() {
  Serial.begin(115200);
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, HIGH);
  delay(1500);
  myusb.begin();

  midi1.setHandleNoteOn(myNoteOn);
  midi1.setHandleNoteOff(myNoteOff);
  midi1.setHandleControlChange(myControlChange);
}

int NumNotes = 0;

void loop() {
  myusb.Task();
  midi1.read();
}

void myNoteOn(byte channel, byte note, byte velocity) {
  Serial.print("Note On, ch = ");
  Serial.print(channel, DEC);
  Serial.print(", note = ");
  Serial.print(note, DEC);
  Serial.print(", velocity = ");
  Serial.print(velocity, DEC);
  Serial.print(", NumNotes = ");
  Serial.println(++NumNotes);
}

void myNoteOff(byte channel, byte note, byte velocity) {
  Serial.print("Note Off, ch = ");
  Serial.print(channel, DEC);
  Serial.print(", note = ");
  Serial.print(note, DEC);
  Serial.print(", velocity = ");
  Serial.print(velocity, DEC);
  Serial.print(", NumNotes = ");
  Serial.println(--NumNotes);
  [COLOR="#FF0000"]if (note == 0x3F && NumNotes != 0) {[/COLOR]
    GPIO8_DR_CLEAR = 1 << 26;  // turn off USB host power
    digitalWrite(LED_BUILTIN, LOW);
  }
}

void myControlChange(byte channel, byte control, byte value) {
}

Now here is the thing: the power to the Arturia is always cut after just 1 test pattern reception.
Am I overlooking something? Is the comparison in red above incorrect?
This is what I see on the serial monitor when the power is cut and the serial monitor no longer updates:

Code:

12:33:45.786 -> Note Off, ch = 1, note = 115, velocity = 0, NumNotes = 12
12:33:45.786 -> Note Off, ch = 1, note = 116, velocity = 0, NumNotes = 11
12:33:45.786 -> Note Off, ch = 1, note = 117, velocity = 0, NumNotes = 10
12:33:45.786 -> Note Off, ch = 1, note = 118, velocity = 0, NumNotes = 9
12:33:45.786 -> Note Off, ch = 1, note = 119, velocity = 0, NumNotes = 8
12:33:45.786 -> Note Off, ch = 1, note = 120, velocity = 0, NumNotes = 7
12:33:45.786 -> Note Off, ch = 1, note = 121, velocity = 0, NumNotes = 6
12:33:45.786 -> Note Off, ch = 1, note = 122, velocity = 0, NumNotes = 5
12:33:45.786 -> Note Off, ch = 1, note = 123, velocity = 0, NumNotes = 4
12:33:45.786 -> Note Off, ch = 1, note = 124, velocity = 0, NumNotes = 3
12:33:45.786 -> Note Off, ch = 1, note = 125, velocity = 0, NumNotes = 2
12:33:45.786 -> Note Off, ch = 1, note = 126, velocity = 0, NumNotes = 1
12:33:45.786 -> Note Off, ch = 1, note = 127, velocity = 0, NumNotes = 0

Any help is appreciated!
Paul

PS: still using Arduino 2.1.1 & Teensyduino 0.59.3

jmarsh · Aug 12, 2023

PaulS said:

Am I overlooking something? Is the comparison in red above incorrect?
This is what I see on the serial monitor when the power is cut and the serial monitor no longer updates:

Code:

12:33:45.786 -> Note Off, ch = 1, note = 115, velocity = 0, NumNotes = 12
12:33:45.786 -> Note Off, ch = 1, note = 116, velocity = 0, NumNotes = 11
12:33:45.786 -> Note Off, ch = 1, note = 117, velocity = 0, NumNotes = 10
12:33:45.786 -> Note Off, ch = 1, note = 118, velocity = 0, NumNotes = 9
12:33:45.786 -> Note Off, ch = 1, note = 119, velocity = 0, NumNotes = 8
12:33:45.786 -> Note Off, ch = 1, note = 120, velocity = 0, NumNotes = 7
12:33:45.786 -> Note Off, ch = 1, note = 121, velocity = 0, NumNotes = 6
12:33:45.786 -> Note Off, ch = 1, note = 122, velocity = 0, NumNotes = 5
12:33:45.786 -> Note Off, ch = 1, note = 123, velocity = 0, NumNotes = 4
12:33:45.786 -> Note Off, ch = 1, note = 124, velocity = 0, NumNotes = 3
12:33:45.786 -> Note Off, ch = 1, note = 125, velocity = 0, NumNotes = 2
12:33:45.786 -> Note Off, ch = 1, note = 126, velocity = 0, NumNotes = 1
12:33:45.786 -> Note Off, ch = 1, note = 127, velocity = 0, NumNotes = 0

Any help is appreciated!

Shouldn't the line in red be comparing note to 0x7F instead of 0x3F? It's probably turning off the power (and the LED) at note 63, with the remaining messages already received but still waiting to be processed.

PaulS · Aug 12, 2023

Aargh...you're absolutely right. Been staring at it the whole time but did not see it...

Thanks!
Paul

PaulS · Aug 12, 2023

OK, thanks to jmarsh and fiddling with the test pattern code, I got it now to fail within a few seconds.

The test pattern code (I removed the delay(100); ):

Code:

void setup() {
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, LOW);
  delay(5000);                                    // give Arduino 2.1.1 serial monitor ample time to get ready
}

void loop() {
  digitalWriteFast(LED_BUILTIN, HIGH);            // show run activity
  for (int note = 0x00; note <= 0x7F; note++) {   // 128 notes ON
    usbMIDI.sendNoteOn(note, 0x3F, 1);            // incr notes ON, velocity 0x3F, channel 1
  }
  for (int note = 0x00; note <= 0x7F; note++) {   // 128 notes OFF
    usbMIDI.sendNoteOff(note, 0x00, 1);           // decr notes OFF, velocity 0x00, channel 1
  }
  while (usbMIDI.read()) {                        // ignore incoming messages
  }
  digitalWriteFast(LED_BUILTIN, LOW);
}

And the corrected host code:

Code:

#include <USBHost_t36.h>

USBHost myusb;
MIDIDevice_BigBuffer midi1(myusb);

void setup() {
  Serial.begin(115200);
  pinMode(LED_BUILTIN, OUTPUT);
  myusb.begin();
  digitalWrite(LED_BUILTIN, HIGH);

  midi1.setHandleNoteOn(myNoteOn);
  midi1.setHandleNoteOff(myNoteOff);
  midi1.setHandleControlChange(myControlChange);
}

int NumNotes = 0;

void loop() {
  myusb.Task();
  midi1.read();
}

void myNoteOn(byte channel, byte note, byte velocity) {
  Serial.print("Note On, ch = ");
  Serial.print(channel, DEC);
  Serial.print(", note = ");
  Serial.print(note, DEC);
  Serial.print(", velocity = ");
  Serial.print(velocity, DEC);
  Serial.print(", NumNotes = ");
  Serial.println(++NumNotes);
}

void myNoteOff(byte channel, byte note, byte velocity) {
  Serial.print("Note Off, ch = ");
  Serial.print(channel, DEC);
  Serial.print(", note = ");
  Serial.print(note, DEC);
  Serial.print(", velocity = ");
  Serial.print(velocity, DEC);
  Serial.print(", NumNotes = ");
  Serial.println(--NumNotes);
  if (note == [COLOR="#FF0000"]0x7F[/COLOR] && NumNotes != 0) {
    GPIO8_DR_CLEAR = 1 << 26;  // turn off USB host power
    digitalWrite(LED_BUILTIN, LOW);
  }
}

void myControlChange(byte channel, byte control, byte value) {
}

The disadvantage of this rude killing of the USB host power is that still another 500-800 MIDI messages will arrive after detection of the fault.
Perhaps there is more elegant way of stopping USB transmissions by the host?

Paul

PS: I noticed that it is not possible to select & copy all the serial monitor info using Arduino 2.1.1. Had to use the 1.8.19 serial monitor to copy all data and look at it using Notepad++.

PaulStoffregen · Aug 12, 2023

PaulS said:
so perhaps there is a programmatic way to tell the host to no longer request USB packets from the device?

Just stop calling midi1.read();

Packets will keep flowing until receive buffers on the host side fill up, so this doesn't stop immediately but it will soon result in communications waiting if messages are still incoming.

PaulS · Aug 12, 2023

Ofcourse...

Here is the updated host code:

Code:

#include <USBHost_t36.h>

USBHost myusb;
MIDIDevice_BigBuffer midi1(myusb);

void setup() {
  Serial.begin(115200);
  pinMode(LED_BUILTIN, OUTPUT);
  myusb.begin();
  digitalWrite(LED_BUILTIN, HIGH);

  midi1.setHandleNoteOn(myNoteOn);
  midi1.setHandleNoteOff(myNoteOff);
}

int NumNotes = 0;
bool error = false;

void loop() {
  myusb.Task();
  if (!error) {
    midi1.read();
  }
}

void myNoteOn(byte channel, byte note, byte velocity) {
  Serial.print("Note On, ch = ");
  Serial.print(channel, DEC);
  Serial.print(", note = ");
  Serial.print(note, DEC);
  Serial.print(", velocity = ");
  Serial.print(velocity, DEC);
  Serial.print(", NumNotes = ");
  Serial.println(++NumNotes);
}

void myNoteOff(byte channel, byte note, byte velocity) {
  Serial.print("Note Off, ch = ");
  Serial.print(channel, DEC);
  Serial.print(", note = ");
  Serial.print(note, DEC);
  Serial.print(", velocity = ");
  Serial.print(velocity, DEC);
  Serial.print(", NumNotes = ");
  Serial.println(--NumNotes);
  if (note == 0x7F && NumNotes != 0) {
    error = true;
    digitalWrite(LED_BUILTIN, LOW);
  }
}

After an error, I now saw 80-200 MIDI messages in the buffer.

Paul

PaulStoffregen · Aug 13, 2023

These test programs really helped.

I found a race condition in MIDI read(). Committed a fix on github.

https://github.com/PaulStoffregen/USBHost_t36/commit/132c968936c8b2f366aa7f0d4209ef9007a415a2

As usual, changed files attached.

PaulS · Aug 13, 2023

Hmm, mixed results...
I copied the changed midi.cpp & USBHost_t36 to both C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\USBHost_t36 and C:\Users\Paul\AppData\Local\Arduino15\packages\teensy\hardware\avr\0.59.3\libraries\USBHost_t36. And cleared C:\Users\Paul\AppData\Local\Temp [just to be sure that the changed files are used at compilation time].
Recompiled and uploaded the host code on to the Teensy 4.1.
Let it run for > 10 minutes with the test pattern code running on the attached Teensy 4.0 > pass!
Then hooked up the Arturia controller, slammed a bunch of keys at the same time > fail.

I will keep looking into it later tonight.

Paul

insolace · Aug 13, 2023

Paul Stoffregen, when you look into these issues are you able to step debug with Teensy?

george.. · Aug 13, 2023

I think there are more bugs not in the MIDI but in the USB routines because non MIDI projects are affected by similar USB faults.

PaulStoffregen · Aug 13, 2023

george.. said:
I think there are more bugs....

You may be right. But hopefully by following this thread you know how this works. A test program which reproduces the bug is needed. Best is when 2 program are posted, one for Teensy 4.1 as USB host and another for a Teensy 4.0 acting at the USB device.

If you believe you're hitting this specific bug, post it here and I'll give it a try. If it's likely an unrelated bug, just start a new thread. And if I miss it, please ping me (but only if you gave a test program.... don't send me any direct reminders to investigate a bug without a test program).

insolace said:
when you look into these issues are you able to step debug with Teensy?

No. Halting debug is pretty much worthless for timing sensitive problems, especially race conditions that only occur with sustained near-maximum speed communication.

PaulS said:
Then hooked up the Arturia controller, slammed a bunch of keys at the same time > fail.

My Arturia Minilab is likely to arrive Monday or Tuesday.

But if you can craft a test program on Teensy 4.0, of course that's much better because it's consistent. I'll investigate either way, but I do much prefer using a test program on Teensy 4.0 as the USB device.

USBhost_t36 MIDI losing midi note on / off events (Teensy 3.6) during modwheel change

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Attachments

Well-known member

Active member

Member

Well-known member