Issue with USB MIDI failing over time when using usbMIDI.sendRealTime

Zenbob

Well-known member
Issue: When connected to Win10 PC USB, sending MIDI clock (usbMIDI.sendRealTime) and MIDI notes, after about an hour the Teensy becomes very sluggish, eventually MIDI notes stop sending but clock still sends, though timing is erratic. This is fairly readily reproducible on our 3 Windows 10 PCs. This issue does NOT happen when connected to an iPad as USB host via camera connection kit.

After spending a few days trying to isolate the issue, we decided to just write some barebones code to do essentially what our own system is doing for clock and MIDI note output. So we whipped this up quick and dirty. The following will compile in Arduino with Teensyduino. There's a blink and some serial output just to see some activity, but you can use MIDI-OX or MIDIClock tool to monitor it. The MIDI sends section takes about 5uS.

Code:
#include <MIDI.h> // make sure to use Teensy optimized driver - 4x USB ports, etc...

unsigned int gMIDI_Count=0;

IntervalTimer T;

void setup() {    
 
    Serial.begin(57600); //'57600
    Serial.println(F("Starting up Test"));
    
    T.priority(96); // try 96 (higher than USB) or 128 (lower than USB)   
    byte responce = T.begin(MIDITicToc, 20833);  // 20833 = 120 BPM

    Serial.println(String("Responce = " ) + responce );

    pinMode(13, OUTPUT);
}

void loop() {
  // put your main code here, to run repeatedly:
}

void MIDITicToc() {

    // SEND MIDI CLOCK OUT TO USB!!!!!!!!!!!!  
    usbMIDI.sendRealTime(usbMIDI.Clock, 0);   //0xF8
    usbMIDI.sendRealTime(usbMIDI.Clock, 1);   //0xF8
    usbMIDI.sendRealTime(usbMIDI.Clock, 2);
    usbMIDI.sendRealTime(usbMIDI.Clock, 3);
    usbMIDI.send_now();

    gMIDI_Count++;

    // double time LED blink
    if (gMIDI_Count%12 == 0) digitalWrite(13, HIGH);
    if ((gMIDI_Count+6)%12 == 0) digitalWrite(13, LOW);
            
    // 
    if (gMIDI_Count%24 == 0) { // 24 typical for MIDI clock to beat 24 pulses per quarter note 
        usbMIDI.sendNoteOn(67, 127, 1, 0);
        usbMIDI.sendNoteOn(67, 127, 1, 1);
        usbMIDI.sendNoteOn(67, 127, 1, 2);
        usbMIDI.sendNoteOn(67, 127, 1, 3); 
        usbMIDI.send_now();
        //Serial.println(String("Play Note... ") + gMIDI_Count);
        Serial.println(gMIDI_Count);
    }


    if ((gMIDI_Count+18)%24 == 0) { // 18 is 75% (of 24) gate time
        usbMIDI.sendNoteOff(67, 0, 1, 0);
        usbMIDI.sendNoteOff(67, 0, 1, 1);
        usbMIDI.sendNoteOff(67, 0, 1, 2);
        usbMIDI.sendNoteOff(67, 0, 1, 3);    
        usbMIDI.send_now();
        //Serial.println(F("Stop Note..."));     
    }

}

Images of clock timing when Teensy goes in the the bad condition, on two different PCs:
midiclocktrack.pngmidiclocktrack_2.png

Teensy 3.2
Teensyduino version: 1.42 (also tested with 1.41)
Arduino version: 1.8.5
Windows audio driver version: 4/11/2018, 10.0.17124.1
"Serial + MIDIx4"
96MHz (overclocked)
Optimize: "Fastest"

Test scenarios:
1) Send notes, but don’t send real time clock (by removing usbMIDI.sendRealTime calls) – fixes the issue
2) Send notes and clock to only one USB port – no change. That is, configured for only one virtual MIDI port rather than four.
3) On our system, we had an interrupt with a higher priority than USB. We changed the interrupt to a lower priority – fixes the issue, but its causing performance problems elsewhere. For instance, SPI display updates that cause a full screen redraw causes the MIDI clock to stutter badly. Hence the effort to fix this.
4) Once in the "condition" if we dynamically stop sending to USB MIDI the system returns to normal performance. We can use an encoder to change from USB to serial MIDI for instance, and performance returns. Then, clock out of HW serial is stable.
5)Doesn't happen with iPad as USB host even with our MIDI loop at a higher interrupt priority than USB.
6)Problem happens whether using a USB hub or directly connected to USB port on motherboard. USB 2 or USB 3 ports.
7)Doesn't seem to matter how fast or slow we are sending MIDI notes.
8)Doesn't seem to matter if software on Windows is consuming the notes or not. We've used MIDI-OX, DEXED and Arturia virtual instruments as MIDI receivers.
9)We recently added "usbMIDI.send_now();" as an experiment but it had no effect.

We could use some recommendations on how to further isolate the issue. It feels like a buffer problem or memory leak, but not sure how to measure it. Maybe some problem with interaction between Win10 USB or MIDI handler and Teensy USB MIDI?
 
What software are you running to get those BPM graphs?

Please keep in mind I'm not a musician, I don't actually use MIDI software, and my primary desktop Linux, not Windows. But I do have a Windows 10 test machine here, and I have a USB protocol analyzer to really look at what's happening with the USB communication... if I can figure out how to set up this test. Any help and specific instructions on what to do with the Windows software setup would really help!
 
Thanks Paul! The two tools we use most are MIDI-OX (http://www.midiox.com/zip/midioxse.exe) for Windows and MIDIClock (http://midiclock.com/wp-content/uploads/2014/04/midiclock4.01.zip) for Windows.

It would be interesting to see if this happens on Linux. Maybe someone can recommend a MIDI tool? I'll take a look too. I tried to test it on an iConnectivity MIDI interface with USB host, but Teensy isn't recognized, it doesn't negotiate for some reason. We'll work on that with iConnectivity later :D

Launch MIDIClock.exe
Set the MIDI Input Port to "Teensy MIDIx4"
MIDICLock.PNG
You can now monitor BPM. To see a graph (Statistics window) over time, click on the little graph icon next to the "Tap Here" button"
MIDICLock graph button.PNG
Uncheck "Chart BPM Out" and check "Raw Clock In", then at the bottom of the window check "Enable Chart Logging" to start logging.
Statistics Window.PNG
 
I should add, that running the above sample code, just plug Teensy into USB on Windows and start the monitoring program. It immediately starts sending clock and notes. Also, I'm going to try this now, but you shouldn't have to use any program to monitor clock if you load a soft-synth and listen to the notes play. They will stop playing eventually.
 
A new and interesting observation

Paul, maybe this will help with diagnosis...

After Teensy goes into the "condition", I opened a serial terminal and Teensy recovered! It started playing notes and the clock stabilized. It ran this way for a while then went back into the condition, at which point the serial monitor could not open the serial port. The content of the serial data had some old values (from when the Teensy was started) then jumped to current values. Like so:
Code:
48
72
96
120
144
168
192
216
240
468504
468528
468552
468576
468600
468624
468648
468672
And from the other system:
Code:
496128
496152
496176
524040
524064
524088
524112
524136
524160
524184
524208
524232
524256
I replicated the same behavior on two systems with two different Teensys. It just did it again but this time I opened the Arduino IDE serial plotter and same result. Clock is stable and its playing notes. ~2500 secs later and the serial interface is gone and no notes or clock being received. LED is still blinking so the loop is running. I left the serial plotter running to see if consuming the serial output makes any difference and it didn't prevent it from failing.
 
I don't see a .read()

You need to read periodically to keep buffer clear...
Code:
void loop() {
...
  // MIDI Controllers should discard incoming MIDI messages.
  while (usbMIDI.read()) {
  }
}
 
Ah yes, good catch. I'll add the following as we have it our system's code and test again.

I don't see a .read()

You need to read periodically to keep buffer clear...
Code:
void loop() {
...
  // MIDI Controllers should discard incoming MIDI messages.
  while (usbMIDI.read()) {
  }
}
 
The 'while' loop always seemed overkill... even calling every main loop on a modern Teensy seems excessive if you're polling anything else on a lesser frequency; but then maybe keep the 'while' loop.
 
Last edited:
In our system's code we call it in the main loop without the While. After seeing your message we decided to try it with the While, and didn't notice any change in general behavior. That is, we do consume incoming MIDI messages, and with the While it was still working.

The 'while' loop always seemed overkill... even calling every main loop on a modern Teensy seems excessive if you're polling anything else on a lesser frequency; but then maybe keep the 'while' loop.
 
Another observation... When we remove all serial initialization and prints, and build with MIDIx4 without serial, we never get into the bad condition. Its run for some hours that way. Next I'll try putting the serial init back in, but not serial prints, add the 'while (usbMIDI.read()) ' and build with MIDIx4 + serial.

Previously, we had tried building without serial, but didn't remove the serial init and prints from the code. It failed in that configuration.
 
Another observation... When we remove all serial initialization and prints, and build with MIDIx4 without serial, we never get into the bad condition. Its run for some hours that way. Next I'll try putting the serial init back in, but not serial prints, add the 'while (usbMIDI.read()) ' and build with MIDIx4 + serial.

Previously, we had tried building without serial, but didn't remove the serial init and prints from the code. It failed in that configuration.

Added back serial.begin, and compiled with MIDIx4 + serial, no serial prints. It ran for 3 hours before getting into the condition. That's much longer than the 45 min to hour it normally takes to fail. Opening a serial terminal did not restore performance, and there was no data in the serial monitor (as expected). And still misspelling "response" :rolleyes:

So, still no real conclusions other than its better when not using the USB serial interface. And best if not building or initializing serial along with MIDI. Sorry I don't know how to pinpoint the issue.

Code:
//Ver 1_4

#include <MIDI.h> // make sure to use Teensy optimized driver - 4x USB ports, etc...

unsigned int gMIDI_Count=0;
unsigned int gFcnTime=0;
unsigned int gTicTemp1=0;

IntervalTimer T;

void setup() {    
 
    Serial.begin(57600); //'57600
    //Serial.println(F("Starting up Test"));
    
    T.priority(96); // try 96 (higher than USB) or 128 (lower than USB)   
    byte responce = T.begin(MIDITicToc, 20833);  // 20833 = 120 BPM

    //Serial.println(String("Responce = " ) + responce );

    pinMode(13, OUTPUT);
}

void loop() {
  // put your main code here, to run repeatedly:
   while (usbMIDI.read()) {
  }
}

void MIDITicToc() {

    //gTicTemp1 = micros(); // measure function time  == 5us
    
    // SEND MIDI CLOCK OUT TO USB!!!!!!!!!!!!  
    usbMIDI.sendRealTime(usbMIDI.Clock, 0);   //0xF8
    usbMIDI.sendRealTime(usbMIDI.Clock, 1);   //0xF8
    usbMIDI.sendRealTime(usbMIDI.Clock, 2);
    usbMIDI.sendRealTime(usbMIDI.Clock, 3);
    usbMIDI.send_now();

    gMIDI_Count++;

    // double time LED blink
    if (gMIDI_Count%12 == 0) digitalWrite(13, HIGH);
    if ((gMIDI_Count+6)%12 == 0) digitalWrite(13, LOW);
            
   
    if (gMIDI_Count%24 == 0) { // 24 typical for MIDI clock to beat 24 pulses per quarter note 
        usbMIDI.sendNoteOn(67, 127, 1, 0);
        usbMIDI.sendNoteOn(68, 127, 1, 1);
        usbMIDI.sendNoteOn(69, 127, 1, 2);
        usbMIDI.sendNoteOn(70, 127, 1, 3); 
        usbMIDI.send_now();
        //Serial.println(String("Play Note... ") + gMIDI_Count);
        //Serial.println(gMIDI_Count);
        //if (gMIDI_Count%240 == 0) Serial.println(String(gFcnTime) + " us"); // print function time
    }


    if ((gMIDI_Count+18)%24 == 0) { // 18 is 75% (of 24) gate time
        usbMIDI.sendNoteOff(67, 0, 1, 0);
        usbMIDI.sendNoteOff(68, 0, 1, 1);
        usbMIDI.sendNoteOff(69, 0, 1, 2);
        usbMIDI.sendNoteOff(70, 0, 1, 3);    
        usbMIDI.send_now();
       // Serial.println(F("Stop Note..."));     
    }

    //gFcnTime = micros()-gTicTemp1; // measure function time
}
 
Just to confirm, I do have this issue on my list to investigate. But at the moment I have another high priority project on my workbench, so I probably can't look at this until at least late into next week. I just one guy, and I'm juggling a lot of priorities. This sort of bug absolutely is on my priority list to investigate & fix, when I can get to it.

In the meantime, if you can find any ways to make the problem happen sooner, that would really help me to fix this problem. I know your focus is on avoiding the problem... but please don't be shy about generating activity on the serial interface. If there's a way to get that 45 minute time down, I can solve this much faster when I do get my workbench cleared off and the Windows machine and USB analyzer set up.
 
Just to confirm, I do have this issue on my list to investigate. But at the moment I have another high priority project on my workbench, so I probably can't look at this until at least late into next week. I just one guy, and I'm juggling a lot of priorities. This sort of bug absolutely is on my priority list to investigate & fix, when I can get to it.

In the meantime, if you can find any ways to make the problem happen sooner, that would really help me to fix this problem. I know your focus is on avoiding the problem... but please don't be shy about generating activity on the serial interface. If there's a way to get that 45 minute time down, I can solve this much faster when I do get my workbench cleared off and the Windows machine and USB analyzer set up.

Thanks again Paul, appreciate your time and effort. I'll try some experiments to see if I can get it to fail faster. I'm thinking that just shoving more data into the serial interface might do it, I'll let you know.
 
Hi Paul, I've tried a bunch of experiments to try to get it to fail faster. Its been a futile exercise, it seems pushing it harder is making it run longer. For instance, cranking up the BPM to 650 and putting a serial print between each MIDI clock tick and it ran for hours. Now I'm just crossing my fingers that you can replicate it.
 
No protocol analyzer yet - just seeing initial testing.

Here's 2 strange things. First, I rebooted (actually several reboots due to a big Windows Update) but left the Teensy running. When I ran the program again, solid 120 bpm.

Then it started doing the problem after only ~8 minutes. I could heat the laptop's fan start to spin up. But Task Manager doesn't show anything else using much CPU. Very mysterious....

capture3.jpg
 
I was able to reproduce the problem here. Takes much longer on my machine, about 75 minutes.

Something that might help, is once it gets into that condition, if you open a serial terminal it will typically recover for a short time (10 minutes maybe?). it might give you a chance to see it fail again without the long wait.
 
It seems you bought an AMD laptop? Does it have AMD video GPU? Something may be firing that up adding HEAT in the box.

If you right click a column heading on that 'Processes' tab you can Add '% GPU' and 'GPU Engine'.

That may show something about where the problem comes from if the PC app is going nuts.

Hopefully 'winver' shows x.x.x.228 - or you don't have yesterday's latest update yet :)
 
Or maybe MS is hiding their background update tasks.

Winvir shows .165 and I don't have any new updates this week showing up in my Updates history, I might force an update and see if it changes anything.

Am I correct in thinking this is the driver to watch? Windows audio driver version: 4/11/2018, 10.0.17124.1
 
Force an update check - not likely related … { but … win update .165 may not be pretty - I've gotten 4 machines in recent days on .165 that were acting slow and ugly for the user. And on the TaskMan/perf page it showed they had not been shut off in days despite powering down. .167 followed in only 6 days and then .191 8 days later. And somehow none of them got the later updates. }

It may be that some background usage isn't shown - but I've not noticed that for CPU. If the App is using GPU and having display/update issues that could add heat fast. The 'GPU Engine' column seems to show when it wires it in.
 
Indeed - if the GPU flies off - supposing it has one and that Paul didn't buy AMD just to get a low end machine - making the fan run then the App may be the issue.

That should show in the USB traffic under analysis if the teensy is sending good - or getting any garbage.
 
Back
Top