Thanks for your efforts Paul. Are you nevertheless planning to modify usb.c since several users in this thread have reported success with Rolf's suggestions? If so, is there a way to eliminate its current flush command(s) but then still call a flush before returning so you don't need to add it in the user code? I haven't looked at usb.c recently so maybe that wouldn't work. But if it would, maybe it could be done only for sysex transfers to avoid slowing down other midi commands. Or maybe introduce a second USB_send function that users could call for sysex only?
usbMIDI transmission error
- Thread starter Rolfdegen
- Start date
Thanks for your efforts Paul. Are you nevertheless planning to modify usb.c since several users in this thread have reported success with Rolf's suggestions? If so, is there a way to eliminate its current flush command(s) but then still call a flush before returning so you don't need to add it in the user code? I haven't looked at usb.c recently so maybe that wouldn't work. But if it would, maybe it could be done only for sysex transfers to avoid slowing down other midi commands. Or maybe introduce a second USB_send function that users could call for sysex only?
PaulStoffregen
Well-known member
No, not planning to change usb.c. Success of those suggestions comes from adding usb_midi_flush_output() to your program after sending each MIDI message. Removing it from usb.c makes no difference.
I know you are convinced having usb_midi_flush_output() in usb.c or usb_dev.c is somehow causing problems. But all of the testing I've done yesterday and today shows that impression is mistaken, based only on coincidence. When you removed it from usb.c, you added it to your program. Removing it from usb.c did nothing. All the improvement you saw was due to adding it to your code. You can get the same result by just adding it your program while leaving usb.c alone.
But again, the improvement you saw is far from guaranteed to work. Here is a test program. Please, edit your usb.c back to the original code. Then experiment with this program using different message size and either delayMicroseconds_WhileDiscardingInput() or usb_midi_flush_output() after sending each sysex message. Also try SendSX. It is vastly better at receiving than MIDI-OX. But if you send a sustained burst of small sysex messages at maximum speed, neither program on Windows can keep up.
Code:
#define MESSAGESIZE 380
void setup() {
pinMode(2, INPUT_PULLUP);
delay(1); // allow time for pullup to bring pin 2 high
}
void delayMicroseconds_WhileDiscardingInput(unsigned int usec) {
elapsedMicros t = 0;
while (t < usec) {
usbMIDI.read(); // read and ignore incoming USB MIDI
}
}
void loop() {
if (!digitalRead(2)) {
byte patchData[MESSAGESIZE] = {0};
patchData[0] = 0xF0;
patchData[MESSAGESIZE-1] = 0xF7;
for (int P = 0; P < 64; P++) {
for (int i = 0; i < 32; i++) {
patchData[3] = i;
patchData[4] = P;
usbMIDI.sendSysEx(MESSAGESIZE, patchData, true);
//usb_midi_flush_output();
//delayMicroseconds_WhileDiscardingInput(600);
}
//delayMicroseconds_WhileDiscardingInput(50);
}
}
usbMIDI.read();
}I checked your suggestion, but taking out the flush in usb.c really does have an effect in my code. Maybe Rolf can check it in his code as well?
I changed usb.c back to its original form so it retained the usb_midi_flush_output() call in the usb_isr(), but also left the additional flush call in place in my synth code, and recompiled my synth code. Then did a sysex dump from a unit that had 165 patches saved, each requiring 1272 bytes sysex broken up into 6 chunks of 212 bytes with 5ms delay between chunks. I repeated the dump five times and each time there were buffer size errors at random points in the dump (blocks that weren't 212 bytes according to MidiOx's running display and total byte count at the end) .
Then I commented the flush in usb.c out again, recompiled, and did the same test five more times, and each time there were no errors.
In case it's relevant, my low level midi input buffer configuration in MidiOx was set for 32 buffers of 256 bytes each (my intention being that each 212 byte buffer from my code would fit in a MidiOx buffer). And I figured 5ms between chunks should give MidiOx ample time to process the buffer.
p.s. I will try SendSX. Thanks for the suggestion.
Last edited:
PaulStoffregen
Well-known member
The test I mentioned was done from within a large project totally unsuitable for your test purpose. But I'll try to find some time in the next day or two to isolate the section we need, and run the test again. If I get the same results, I'll post that code.
In the meantime, it would be good to hear if anyone else sees the improvement when the flush is commented out in usb.c (whilst retaining an additional flush in your main code after the usb_send command).
I modified your test program from #77 by setting it to my buffer size parameters, and replaced the DelayWhileDiscarding input with a regular delay of 5ms. It behaved as you describe with a noticeable improvement when using a manual flush after the usb_send, regardless of whether the flush occurs in usb.c.
In my full synth code, where I instead see a dependency on whether the flush occurs in usb.c or not, I don't disable interrupts during the sysex dump because I assumed the usb_send needs them enabled. Can you tell me if there's an easy way to disable all usb midi callbacks during the sysex dump to see if that's where the dependency is arising?
In my full synth code, where I instead see a dependency on whether the flush occurs in usb.c or not, I don't disable interrupts during the sysex dump because I assumed the usb_send needs them enabled. Can you tell me if there's an easy way to disable all usb midi callbacks during the sysex dump to see if that's where the dependency is arising?
PaulStoffregen
Well-known member
I've spent 2 days running hundreds of tests using the 2 programs in msg #30 and msg #46 with many variations, such as the code in msg #77. I can't reproduce the problem as you've described. Since you do not have a program I can run to reproduce it, I'm going to consider this matter closed and take this off my workbench. I have a very long list of open issues and pending pull requests and features people want implemented, so I must move on to other things.
If you do come up with a program which reproduces the problem and clearly demonstrates the need to change automatic output flushing, I will put this my onto my list of issues to investigate. But without earlier providing a program to reproduce the problem, you have missed the window of opportunity to have me react quickly. I can not keep this open and the gear set up on my workbench to wait for another program to test. I'm considering this matter closed and the specific claims about usb_midi_flush_output() to be unreproducible and almost certainly a misunderstanding.
If you ask me to look at this again, or anything else, please understand a test program which demonstrates the problem is required. I started working on this 2 days ago only because test programs were provided in msg #30 and msg #46. I've now done everything I can (quite a lot over 2 days) based on those programs and dozens of ways I've tried altering them.
As far as I'm concerned, everything I've found in the last 2 days boils down to much needed updates to the USB MIDI web page documentation, and unfortunately programs which transmit large amounts of data to Windows must do so at a pace where Windows doesn't silently discard data after having received it.
If you do come up with a program which reproduces the problem and clearly demonstrates the need to change automatic output flushing, I will put this my onto my list of issues to investigate. But without earlier providing a program to reproduce the problem, you have missed the window of opportunity to have me react quickly. I can not keep this open and the gear set up on my workbench to wait for another program to test. I'm considering this matter closed and the specific claims about usb_midi_flush_output() to be unreproducible and almost certainly a misunderstanding.
If you ask me to look at this again, or anything else, please understand a test program which demonstrates the problem is required. I started working on this 2 days ago only because test programs were provided in msg #30 and msg #46. I've now done everything I can (quite a lot over 2 days) based on those programs and dozens of ways I've tried altering them.
As far as I'm concerned, everything I've found in the last 2 days boils down to much needed updates to the USB MIDI web page documentation, and unfortunately programs which transmit large amounts of data to Windows must do so at a pace where Windows doesn't silently discard data after having received it.
Thanks for taking the time to look into this. I look forward to the updated USB midi documentation.
However, your comment about slowing the transmission rate with Windows sufficiently may still not not ensure reliable transfers of large sysex files. I ran your #77 code with delays as large as 50 ms between blocks and still saw errors when using only the automated flush in usb.c. They resolved only by calling an additional manual flush after the usb_send, along with a sufficient delay (somewhere between 1 and 5ms in my tests). It might be useful to add a note to that effect in your usb midi documentation.
PaulStoffregen
Well-known member
Post the actual code you ran, and say specifically which Teensy you used and what software you ran on the PC side. Even if the change is as trivial as altering 1 number, post the entire program so I or anyone else can copy it into Arduino and upload to the same Teensy without guesswork. Clearly say which software you used on the PC side, and if any settings were different from the defaults, list those changes so I or anyone else can reproduce the result.
Be specific about that error you saw. If you can't describe precisely with words, give a screenshot.
Again, I want you to understand I will NOT do anything more unless you post a complete program. Even if that complete program is a trivial edit of anything previously posted, you must show the test program if you want me to do anything more than explain yet again that you must do this.
PaulStoffregen
Well-known member
The USB MIDI page has been updated. The "Transmit Issues" section was expanded to mention discarding incoming messages during delay. A new "Transmit Speed and Message Rate Limiting" section was added, which includes a link to this thread.
PaulStoffregen
Well-known member
For the sake of anyone who takes the time to read this long thread, I should probably clear up some misunderstanding from early on, starting back at msg #14.
This is incorrect, in two ways.
1: usb_midi_flush_output() doesn't delete anything.
2: usb_midi_flush_output() is unrelated to usbMIDI.read(). It deals with sending MIDI messages to your PC, not reading messages your PC sent.
The purpose of usb_midi_flush_output() is to cause any MIDI messages you have previously transmitted but are still sitting in a buffer on Teensy to actually send to your computer. Calling this function when any outgoing MIDI messages in sitting in the buffer results in transmission of a less-than-full-size USB packet to your computer.
It doesn't delete any data. It's unrelated to reading incoming messages.
If your program calls usb_midi_flush_output() more than it would have been done automatically based on the elapse of time without writing more MIDI messages into the buffer, it will result is less efficient usage of USB bandwidth, for 2 reasons. First, USB packets have fixed overhead, so smaller data per packet makes less efficient usage of USB bandwidth. Second, the software on your PC (specifically Windows) may be rate limiting the number of packets it tries to read, without consideration for their actual size, so smaller packets can result in much lower overall speed.
This is incorrect, in two ways.
1: usb_midi_flush_output() doesn't delete anything.
2: usb_midi_flush_output() is unrelated to usbMIDI.read(). It deals with sending MIDI messages to your PC, not reading messages your PC sent.
The purpose of usb_midi_flush_output() is to cause any MIDI messages you have previously transmitted but are still sitting in a buffer on Teensy to actually send to your computer. Calling this function when any outgoing MIDI messages in sitting in the buffer results in transmission of a less-than-full-size USB packet to your computer.
It doesn't delete any data. It's unrelated to reading incoming messages.
If your program calls usb_midi_flush_output() more than it would have been done automatically based on the elapse of time without writing more MIDI messages into the buffer, it will result is less efficient usage of USB bandwidth, for 2 reasons. First, USB packets have fixed overhead, so smaller data per packet makes less efficient usage of USB bandwidth. Second, the software on your PC (specifically Windows) may be rate limiting the number of packets it tries to read, without consideration for their actual size, so smaller packets can result in much lower overall speed.
Below is the slightly altered version of your code that I ran on a T4.1 with a fairly large 5ms delay between buffers using the unmodified version of usb.c. Each 'patch' P contains 6 buffers of 212 bytes. and I'm sending 100 of these.
I just tested it again with SXsend. The running midi input display goes by quickly so at the end of the sysex transmission you may need to scroll the display back up to find the errors, which in this case show as incomplete F0..F7 lines, followed (usually?) by an F0 on its own on the next line.
When I comment out the manual flush I usually see one or more errors in a block of 100 patches. With the manual flush left in, I see no errors.
Code:
I just tested it again with SXsend. The running midi input display goes by quickly so at the end of the sysex transmission you may need to scroll the display back up to find the errors, which in this case show as incomplete F0..F7 lines, followed (usually?) by an F0 on its own on the next line.
When I comment out the manual flush I usually see one or more errors in a block of 100 patches. With the manual flush left in, I see no errors.
Code:
Code:
#define MESSAGESIZE 212
void setup() {
pinMode(2, INPUT_PULLUP);
delay(1); // allow time for pullup to bring pin 2 high
}
void delayMicroseconds_WhileDiscardingInput(unsigned int usec) {
elapsedMicros t = 0;
while (t < usec) {
usbMIDI.read(); // read and ignore incoming USB MIDI
}
}
void loop() {
if (!digitalRead(2)) {
byte patchData[MESSAGESIZE] = {0};
patchData[0] = 0xF0;
patchData[MESSAGESIZE-1] = 0xF7;
for (int P = 0; P < 100; P++) {
for (int i = 0; i < 6; i++) {
patchData[3] = i;
patchData[4] = P;
usbMIDI.sendSysEx(MESSAGESIZE, patchData, true);
usb_midi_flush_output();
delayMicroseconds_WhileDiscardingInput(5000);
}
delayMicroseconds_WhileDiscardingInput(5000);
}
}
usbMIDI.read();
}
Last edited by a moderator:
PaulStoffregen
Well-known member
I ran the program in msg #89 on Teensy 4.1 with the usb_midi_flush_output() line commented out. I can't reproduce the problem, after running it many times. I always get complete 212 byte messages received in SendSX. Scrolled up to look all of them. None are truncated.
Didn't look at the USB packets or dig any deeper. I simply can't reproduce the problem with my Windows test machine.
Didn't look at the USB packets or dig any deeper. I simply can't reproduce the problem with my Windows test machine.
PaulStoffregen
Well-known member
If anyone else reading has Windows and a Teensy 4.1 and a few minutes to install SendSX, please give the code in msg #89 a try. Remember to comment out the usb_midi_flush_output(); line before uploading.
Does SendSX sometimes receive incomplete or corrupted messages on your Windows machine? If you horizontal scroll the upper all the way to the right side, it's easy to scroll up and visually see if every line is the correct length and ends with F7.
More results from people with Windows computers would really help to understand the scope of this problem.
While using SendSX, after uploading to Teensy, you'll need to click the MIDI In menu and select Teensy MIDI again. But you don't need to restart SendSX. Just clicking the menu is enough.
Does SendSX sometimes receive incomplete or corrupted messages on your Windows machine? If you horizontal scroll the upper all the way to the right side, it's easy to scroll up and visually see if every line is the correct length and ends with F7.
More results from people with Windows computers would really help to understand the scope of this problem.
While using SendSX, after uploading to Teensy, you'll need to click the MIDI In menu and select Teensy MIDI again. But you don't need to restart SendSX. Just clicking the menu is enough.
Last edited:
Thanks. Unfortunately makes it very hard for you to assess.
PaulStoffregen
Well-known member
Yes it is. I just meant that the fact that our results differ with the same code makes it hard for you to assess the problem I'm describing.
PaulStoffregen
Well-known member
PaulS
Well-known member
Just ran the test 10x [so 6000 messages], saved it to a text file and checked it with Notepad++. No corrupted messages.
Using Arduino 1.8.19/Teensyduino 1.58/Windows 10.
Paul
edit: tested 30K messages now: all good
edit2: tested with MIDI-OX around 6000 messages: all good
Last edited:
Rolfdegen
Well-known member
Hello Paul and rvh
For a neutral test, I reinstalled Arduino 1.8.19 and Teensyduo 1.56 on my Win10 machine and tested Teensy 4.1 sending SysEx data to the PC.
I sent 128 blocks of 401 SysEx bytes each and found no error.
My SysEx function is very simple. I fill a *buffer with 401 byte SysEx data (0xF0 + data + 0xF7) and send it with the command: usbMIDI.sendSysEx(sysexLength, sysexBuffer, true)
After sending a block ( 401 bytes) I always wait 20 ms and draw a progress bar. Then I send the next block. I'm not using usb_midi_flush_output().
Youtube: https://youtu.be/xRoLsR9HgU0
Grüße aus Deutschland,
Rolf
For a neutral test, I reinstalled Arduino 1.8.19 and Teensyduo 1.56 on my Win10 machine and tested Teensy 4.1 sending SysEx data to the PC.
I sent 128 blocks of 401 SysEx bytes each and found no error.
My SysEx function is very simple. I fill a *buffer with 401 byte SysEx data (0xF0 + data + 0xF7) and send it with the command: usbMIDI.sendSysEx(sysexLength, sysexBuffer, true)
After sending a block ( 401 bytes) I always wait 20 ms and draw a progress bar. Then I send the next block. I'm not using usb_midi_flush_output().
Youtube: https://youtu.be/xRoLsR9HgU0
Grüße aus Deutschland,
Rolf
Last edited:
Here is the code from #89 again, but with the flush command already commented out, so you can use it as is.
When I run this on my system (Win10 pro), I get errors in SXsend like this:
Code:
#define MESSAGESIZE 212
void setup() {
pinMode(2, INPUT_PULLUP);
delay(1); // allow time for pullup to bring pin 2 high
}
void delayMicroseconds_WhileDiscardingInput(unsigned int usec) {
elapsedMicros t = 0;
while (t < usec) {
usbMIDI.read(); // read and ignore incoming USB MIDI
}
}
void loop() {
if (!digitalRead(2)) {
byte patchData[MESSAGESIZE] = {0};
patchData[0] = 0xF0;
patchData[MESSAGESIZE-1] = 0xF7;
for (int P = 0; P < 100; P++) {
for (int i = 0; i < 6; i++) {
patchData[3] = i;
patchData[4] = P;
usbMIDI.sendSysEx(MESSAGESIZE, patchData, true);
//usb_midi_flush_output();
delayMicroseconds_WhileDiscardingInput(5000);
}
delayMicroseconds_WhileDiscardingInput(5000);
}
}
usbMIDI.read();
}When I run this on my system (Win10 pro), I get errors in SXsend like this: