Forum Rule: Always post complete source code & details to reproduce any issue!
Page 1 of 3 1 2 3 LastLast
Results 1 to 25 of 53

Thread: Issue with USB MIDI failing over time when using usbMIDI.sendRealTime

  1. #1

    Issue with USB MIDI failing over time when using usbMIDI.sendRealTime

    Issue: When connected to Win10 PC USB, sending MIDI clock (usbMIDI.sendRealTime) and MIDI notes, after about an hour the Teensy becomes very sluggish, eventually MIDI notes stop sending but clock still sends, though timing is erratic. This is fairly readily reproducible on our 3 Windows 10 PCs. This issue does NOT happen when connected to an iPad as USB host via camera connection kit.

    After spending a few days trying to isolate the issue, we decided to just write some barebones code to do essentially what our own system is doing for clock and MIDI note output. So we whipped this up quick and dirty. The following will compile in Arduino with Teensyduino. There's a blink and some serial output just to see some activity, but you can use MIDI-OX or MIDIClock tool to monitor it. The MIDI sends section takes about 5uS.

    Code:
    #include <MIDI.h> // make sure to use Teensy optimized driver - 4x USB ports, etc...
    
    unsigned int gMIDI_Count=0;
    
    IntervalTimer T;
    
    void setup() {    
     
        Serial.begin(57600); //'57600
        Serial.println(F("Starting up Test"));
        
        T.priority(96); // try 96 (higher than USB) or 128 (lower than USB)   
        byte responce = T.begin(MIDITicToc, 20833);  // 20833 = 120 BPM
    
        Serial.println(String("Responce = " ) + responce );
    
        pinMode(13, OUTPUT);
    }
    
    void loop() {
      // put your main code here, to run repeatedly:
    }
    
    void MIDITicToc() {
    
        // SEND MIDI CLOCK OUT TO USB!!!!!!!!!!!!  
        usbMIDI.sendRealTime(usbMIDI.Clock, 0);   //0xF8
        usbMIDI.sendRealTime(usbMIDI.Clock, 1);   //0xF8
        usbMIDI.sendRealTime(usbMIDI.Clock, 2);
        usbMIDI.sendRealTime(usbMIDI.Clock, 3);
        usbMIDI.send_now();
    
        gMIDI_Count++;
    
        // double time LED blink
        if (gMIDI_Count%12 == 0) digitalWrite(13, HIGH);
        if ((gMIDI_Count+6)%12 == 0) digitalWrite(13, LOW);
                
        // 
        if (gMIDI_Count%24 == 0) { // 24 typical for MIDI clock to beat 24 pulses per quarter note 
            usbMIDI.sendNoteOn(67, 127, 1, 0);
            usbMIDI.sendNoteOn(67, 127, 1, 1);
            usbMIDI.sendNoteOn(67, 127, 1, 2);
            usbMIDI.sendNoteOn(67, 127, 1, 3); 
            usbMIDI.send_now();
            //Serial.println(String("Play Note... ") + gMIDI_Count);
            Serial.println(gMIDI_Count);
        }
    
    
        if ((gMIDI_Count+18)%24 == 0) { // 18 is 75% (of 24) gate time
            usbMIDI.sendNoteOff(67, 0, 1, 0);
            usbMIDI.sendNoteOff(67, 0, 1, 1);
            usbMIDI.sendNoteOff(67, 0, 1, 2);
            usbMIDI.sendNoteOff(67, 0, 1, 3);    
            usbMIDI.send_now();
            //Serial.println(F("Stop Note..."));     
        }
    
    }
    Images of clock timing when Teensy goes in the the bad condition, on two different PCs:
    Click image for larger version. 

Name:	midiclocktrack.png 
Views:	17 
Size:	28.0 KB 
ID:	14372Click image for larger version. 

Name:	midiclocktrack_2.png 
Views:	14 
Size:	23.7 KB 
ID:	14373

    Teensy 3.2
    Teensyduino version: 1.42 (also tested with 1.41)
    Arduino version: 1.8.5
    Windows audio driver version: 4/11/2018, 10.0.17124.1
    "Serial + MIDIx4"
    96MHz (overclocked)
    Optimize: "Fastest"

    Test scenarios:
    1) Send notes, but don’t send real time clock (by removing usbMIDI.sendRealTime calls) – fixes the issue
    2) Send notes and clock to only one USB port – no change. That is, configured for only one virtual MIDI port rather than four.
    3) On our system, we had an interrupt with a higher priority than USB. We changed the interrupt to a lower priority – fixes the issue, but its causing performance problems elsewhere. For instance, SPI display updates that cause a full screen redraw causes the MIDI clock to stutter badly. Hence the effort to fix this.
    4) Once in the "condition" if we dynamically stop sending to USB MIDI the system returns to normal performance. We can use an encoder to change from USB to serial MIDI for instance, and performance returns. Then, clock out of HW serial is stable.
    5)Doesn't happen with iPad as USB host even with our MIDI loop at a higher interrupt priority than USB.
    6)Problem happens whether using a USB hub or directly connected to USB port on motherboard. USB 2 or USB 3 ports.
    7)Doesn't seem to matter how fast or slow we are sending MIDI notes.
    8)Doesn't seem to matter if software on Windows is consuming the notes or not. We've used MIDI-OX, DEXED and Arturia virtual instruments as MIDI receivers.
    9)We recently added "usbMIDI.send_now();" as an experiment but it had no effect.

    We could use some recommendations on how to further isolate the issue. It feels like a buffer problem or memory leak, but not sure how to measure it. Maybe some problem with interaction between Win10 USB or MIDI handler and Teensy USB MIDI?

  2. #2
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,576
    What software are you running to get those BPM graphs?

    Please keep in mind I'm not a musician, I don't actually use MIDI software, and my primary desktop Linux, not Windows. But I do have a Windows 10 test machine here, and I have a USB protocol analyzer to really look at what's happening with the USB communication... if I can figure out how to set up this test. Any help and specific instructions on what to do with the Windows software setup would really help!

  3. #3
    Thanks Paul! The two tools we use most are MIDI-OX (http://www.midiox.com/zip/midioxse.exe) for Windows and MIDIClock (http://midiclock.com/wp-content/uplo...iclock4.01.zip) for Windows.

    It would be interesting to see if this happens on Linux. Maybe someone can recommend a MIDI tool? I'll take a look too. I tried to test it on an iConnectivity MIDI interface with USB host, but Teensy isn't recognized, it doesn't negotiate for some reason. We'll work on that with iConnectivity later

    Launch MIDIClock.exe
    Set the MIDI Input Port to "Teensy MIDIx4"
    Click image for larger version. 

Name:	MIDICLock.PNG 
Views:	25 
Size:	15.1 KB 
ID:	14374
    You can now monitor BPM. To see a graph (Statistics window) over time, click on the little graph icon next to the "Tap Here" button"
    Name:  MIDICLock graph button.PNG
Views: 187
Size:  2.1 KB
    Uncheck "Chart BPM Out" and check "Raw Clock In", then at the bottom of the window check "Enable Chart Logging" to start logging.
    Click image for larger version. 

Name:	Statistics Window.PNG 
Views:	27 
Size:	28.1 KB 
ID:	14376

  4. #4
    This MIDI tool for Linux looks useful. You can select what type of MIDI messages to monitor. However, for this exercise its useful to have a graph.
    https://sourceforge.net/projects/kmidimon/

    Discovered at http://tedfelix.com/linux/linux-midi.html

  5. #5
    I should add, that running the above sample code, just plug Teensy into USB on Windows and start the monitoring program. It immediately starts sending clock and notes. Also, I'm going to try this now, but you shouldn't have to use any program to monitor clock if you load a soft-synth and listen to the notes play. They will stop playing eventually.

  6. #6

    A new and interesting observation

    Paul, maybe this will help with diagnosis...

    After Teensy goes into the "condition", I opened a serial terminal and Teensy recovered! It started playing notes and the clock stabilized. It ran this way for a while then went back into the condition, at which point the serial monitor could not open the serial port. The content of the serial data had some old values (from when the Teensy was started) then jumped to current values. Like so:
    Code:
    48
    72
    96
    120
    144
    168
    192
    216
    240
    468504
    468528
    468552
    468576
    468600
    468624
    468648
    468672
    And from the other system:
    Code:
    496128
    496152
    496176
    524040
    524064
    524088
    524112
    524136
    524160
    524184
    524208
    524232
    524256
    I replicated the same behavior on two systems with two different Teensys. It just did it again but this time I opened the Arduino IDE serial plotter and same result. Clock is stable and its playing notes. ~2500 secs later and the serial interface is gone and no notes or clock being received. LED is still blinking so the loop is running. I left the serial plotter running to see if consuming the serial output makes any difference and it didn't prevent it from failing.

  7. #7
    Senior Member oddson's Avatar
    Join Date
    Feb 2013
    Location
    Isle in the Salish Sea
    Posts
    1,165
    I don't see a .read()

    You need to read periodically to keep buffer clear...
    Code:
    void loop() {
    ...
      // MIDI Controllers should discard incoming MIDI messages.
      while (usbMIDI.read()) {
      }
    }

  8. #8
    Ah yes, good catch. I'll add the following as we have it our system's code and test again.

    Quote Originally Posted by oddson View Post
    I don't see a .read()

    You need to read periodically to keep buffer clear...
    Code:
    void loop() {
    ...
      // MIDI Controllers should discard incoming MIDI messages.
      while (usbMIDI.read()) {
      }
    }

  9. #9
    Senior Member oddson's Avatar
    Join Date
    Feb 2013
    Location
    Isle in the Salish Sea
    Posts
    1,165
    The 'while' loop always seemed overkill... even calling every main loop on a modern Teensy seems excessive if you're polling anything else on a lesser frequency; but then maybe keep the 'while' loop.
    Last edited by oddson; 08-08-2018 at 10:17 PM. Reason: Opps

  10. #10
    In our system's code we call it in the main loop without the While. After seeing your message we decided to try it with the While, and didn't notice any change in general behavior. That is, we do consume incoming MIDI messages, and with the While it was still working.

    Quote Originally Posted by oddson View Post
    The 'while' loop always seemed overkill... even calling every main loop on a modern Teensy seems excessive if you're polling anything else on a lesser frequency; but then maybe keep the 'while' loop.

  11. #11
    Another observation... When we remove all serial initialization and prints, and build with MIDIx4 without serial, we never get into the bad condition. Its run for some hours that way. Next I'll try putting the serial init back in, but not serial prints, add the 'while (usbMIDI.read()) ' and build with MIDIx4 + serial.

    Previously, we had tried building without serial, but didn't remove the serial init and prints from the code. It failed in that configuration.

  12. #12
    Quote Originally Posted by Zenbob View Post
    Another observation... When we remove all serial initialization and prints, and build with MIDIx4 without serial, we never get into the bad condition. Its run for some hours that way. Next I'll try putting the serial init back in, but not serial prints, add the 'while (usbMIDI.read()) ' and build with MIDIx4 + serial.

    Previously, we had tried building without serial, but didn't remove the serial init and prints from the code. It failed in that configuration.
    Added back serial.begin, and compiled with MIDIx4 + serial, no serial prints. It ran for 3 hours before getting into the condition. That's much longer than the 45 min to hour it normally takes to fail. Opening a serial terminal did not restore performance, and there was no data in the serial monitor (as expected). And still misspelling "response"

    So, still no real conclusions other than its better when not using the USB serial interface. And best if not building or initializing serial along with MIDI. Sorry I don't know how to pinpoint the issue.

    Code:
    //Ver 1_4
    
    #include <MIDI.h> // make sure to use Teensy optimized driver - 4x USB ports, etc...
    
    unsigned int gMIDI_Count=0;
    unsigned int gFcnTime=0;
    unsigned int gTicTemp1=0;
    
    IntervalTimer T;
    
    void setup() {    
     
        Serial.begin(57600); //'57600
        //Serial.println(F("Starting up Test"));
        
        T.priority(96); // try 96 (higher than USB) or 128 (lower than USB)   
        byte responce = T.begin(MIDITicToc, 20833);  // 20833 = 120 BPM
    
        //Serial.println(String("Responce = " ) + responce );
    
        pinMode(13, OUTPUT);
    }
    
    void loop() {
      // put your main code here, to run repeatedly:
       while (usbMIDI.read()) {
      }
    }
    
    void MIDITicToc() {
    
        //gTicTemp1 = micros(); // measure function time  == 5us
        
        // SEND MIDI CLOCK OUT TO USB!!!!!!!!!!!!  
        usbMIDI.sendRealTime(usbMIDI.Clock, 0);   //0xF8
        usbMIDI.sendRealTime(usbMIDI.Clock, 1);   //0xF8
        usbMIDI.sendRealTime(usbMIDI.Clock, 2);
        usbMIDI.sendRealTime(usbMIDI.Clock, 3);
        usbMIDI.send_now();
    
        gMIDI_Count++;
    
        // double time LED blink
        if (gMIDI_Count%12 == 0) digitalWrite(13, HIGH);
        if ((gMIDI_Count+6)%12 == 0) digitalWrite(13, LOW);
                
       
        if (gMIDI_Count%24 == 0) { // 24 typical for MIDI clock to beat 24 pulses per quarter note 
            usbMIDI.sendNoteOn(67, 127, 1, 0);
            usbMIDI.sendNoteOn(68, 127, 1, 1);
            usbMIDI.sendNoteOn(69, 127, 1, 2);
            usbMIDI.sendNoteOn(70, 127, 1, 3); 
            usbMIDI.send_now();
            //Serial.println(String("Play Note... ") + gMIDI_Count);
            //Serial.println(gMIDI_Count);
            //if (gMIDI_Count%240 == 0) Serial.println(String(gFcnTime) + " us"); // print function time
        }
    
    
        if ((gMIDI_Count+18)%24 == 0) { // 18 is 75% (of 24) gate time
            usbMIDI.sendNoteOff(67, 0, 1, 0);
            usbMIDI.sendNoteOff(68, 0, 1, 1);
            usbMIDI.sendNoteOff(69, 0, 1, 2);
            usbMIDI.sendNoteOff(70, 0, 1, 3);    
            usbMIDI.send_now();
           // Serial.println(F("Stop Note..."));     
        }
    
        //gFcnTime = micros()-gTicTemp1; // measure function time
    }

  13. #13
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,576
    Just to confirm, I do have this issue on my list to investigate. But at the moment I have another high priority project on my workbench, so I probably can't look at this until at least late into next week. I just one guy, and I'm juggling a lot of priorities. This sort of bug absolutely is on my priority list to investigate & fix, when I can get to it.

    In the meantime, if you can find any ways to make the problem happen sooner, that would really help me to fix this problem. I know your focus is on avoiding the problem... but please don't be shy about generating activity on the serial interface. If there's a way to get that 45 minute time down, I can solve this much faster when I do get my workbench cleared off and the Windows machine and USB analyzer set up.

  14. #14
    Quote Originally Posted by PaulStoffregen View Post
    Just to confirm, I do have this issue on my list to investigate. But at the moment I have another high priority project on my workbench, so I probably can't look at this until at least late into next week. I just one guy, and I'm juggling a lot of priorities. This sort of bug absolutely is on my priority list to investigate & fix, when I can get to it.

    In the meantime, if you can find any ways to make the problem happen sooner, that would really help me to fix this problem. I know your focus is on avoiding the problem... but please don't be shy about generating activity on the serial interface. If there's a way to get that 45 minute time down, I can solve this much faster when I do get my workbench cleared off and the Windows machine and USB analyzer set up.
    Thanks again Paul, appreciate your time and effort. I'll try some experiments to see if I can get it to fail faster. I'm thinking that just shoving more data into the serial interface might do it, I'll let you know.

  15. #15
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,576
    Running the test here, first time. Guess I've got 40 minutes to go until the problem....

    Click image for larger version. 

Name:	Capture.jpg 
Views:	16 
Size:	120.7 KB 
ID:	14459

  16. #16
    Hi Paul, I've tried a bunch of experiments to try to get it to fail faster. Its been a futile exercise, it seems pushing it harder is making it run longer. For instance, cranking up the BPM to 650 and putting a serial print between each MIDI clock tick and it ran for hours. Now I'm just crossing my fingers that you can replicate it.

  17. #17
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,576
    I was able to reproduce the problem here. Takes much longer on my machine, about 75 minutes.

    Click image for larger version. 

Name:	capture2.jpg 
Views:	20 
Size:	148.8 KB 
ID:	14463

  18. #18
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    9,709
    At least is wasn't twice as long. Did the USB analyzer catch the traffic usefully or suggestively?

  19. #19
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,576
    No protocol analyzer yet - just seeing initial testing.

    Here's 2 strange things. First, I rebooted (actually several reboots due to a big Windows Update) but left the Teensy running. When I ran the program again, solid 120 bpm.

    Then it started doing the problem after only ~8 minutes. I could heat the laptop's fan start to spin up. But Task Manager doesn't show anything else using much CPU. Very mysterious....

    Click image for larger version. 

Name:	capture3.jpg 
Views:	15 
Size:	130.4 KB 
ID:	14464

  20. #20
    Quote Originally Posted by PaulStoffregen View Post
    I was able to reproduce the problem here. Takes much longer on my machine, about 75 minutes.
    Something that might help, is once it gets into that condition, if you open a serial terminal it will typically recover for a short time (10 minutes maybe?). it might give you a chance to see it fail again without the long wait.

  21. #21
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    9,709
    It seems you bought an AMD laptop? Does it have AMD video GPU? Something may be firing that up adding HEAT in the box.

    If you right click a column heading on that 'Processes' tab you can Add '% GPU' and 'GPU Engine'.

    That may show something about where the problem comes from if the PC app is going nuts.

    Hopefully 'winver' shows x.x.x.228 - or you don't have yesterday's latest update yet

  22. #22
    Or maybe MS is hiding their background update tasks.

    Winvir shows .165 and I don't have any new updates this week showing up in my Updates history, I might force an update and see if it changes anything.

    Am I correct in thinking this is the driver to watch? Windows audio driver version: 4/11/2018, 10.0.17124.1

  23. #23
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    9,709
    Force an update check - not likely related … { but … win update .165 may not be pretty - I've gotten 4 machines in recent days on .165 that were acting slow and ugly for the user. And on the TaskMan/perf page it showed they had not been shut off in days despite powering down. .167 followed in only 6 days and then .191 8 days later. And somehow none of them got the later updates. }

    It may be that some background usage isn't shown - but I've not noticed that for CPU. If the App is using GPU and having display/update issues that could add heat fast. The 'GPU Engine' column seems to show when it wires it in.

  24. #24
    Senior Member oddson's Avatar
    Join Date
    Feb 2013
    Location
    Isle in the Salish Sea
    Posts
    1,165
    Quote Originally Posted by PaulStoffregen View Post
    ...I rebooted (actually several reboots due to a big Windows Update) but left the Teensy running. When I ran the program again, solid 120 bpm.
    It sounds like Windows enters into the 'condition' not Teensy.

  25. #25
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    9,709
    Indeed - if the GPU flies off - supposing it has one and that Paul didn't buy AMD just to get a low end machine - making the fan run then the App may be the issue.

    That should show in the USB traffic under analysis if the teensy is sending good - or getting any garbage.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •