USB <-> DIN MIDI loopback and SysEx data

[newbie40]

I'm currently testing a Teensy 4.0 based USB <-> "DIN MIDI" interface. To make sure it passes large amounts of SysEx data correctly I tried a loopback test connecting the DIN MIDI out with the input and then from the computer sending out large amounts of data, making sure I get the same data back in.

Well, it didn't work as expected. Using Bome sendSX on Windows I could observe that SysEx messages longer than about 130 bytes would arrive "shortened". That is, there was data missing in the middle of the transmission. Received data size was usually around 130 to 132 bytes for transmitted SysEx messages of 162 byte length.
So I wrote a Linux app that would send out test patterns and check received data. There I got even less data back (often nothing at all).

Using Teensy as USB->DIN MIDI interface and receiving using some other (off-the-shelf) MIDI interface worked fine. Sending with other MIDI interface and receiving with Teensy also works. So when the Teensy is not being used full-duplex, all is good (which shows that my code doesn't drop/corrupt any data). Full-duplex transmission with other MIDI interfaces I have at hand does work. Interestingly this only effects SysEx data, doing a loopback with note-on/note-off works fine.

Since SysEx first gets "collected" unside the USB MIDI code and then submitted to the callback in chunks of up to 290 bytes (which I then need to send out through slow DIN MIDI) I started to wonder whether the code gets blocked somewhere for too long. And indeed, changing USB_MIDI_SYSEX_MAX in usb_midi.h from 290 to 64 fixes my problem. - Yes I did read the lengthy threads here where people beg for bigger and bigger SysEx buffers so they don't have to re-assemble the data in the callback.

So I guess I have a feature request to make this #define user-adjustable:

#ifndef USB_MIDI_SYSEX_MAX
#   define USB_MIDI_SYSEX_MAX 290
#endif

I guess my real underlying problem is that trying to put large amounts of data into the HardwareSerial output FIFO will block (i.e. I'm submitting more SysEx than the FIFO can hold) which in turn makes the SysEx callback (and thus usbMIDI.read()) execute for so long that I get a buffer overflow on the serial input in the meantime. Lowering the value of USB_MIDI_SYSEX_MAX simply makes sure the callback function returns faster and my code reads MIDI input from serial before an overflow happens. Assuming that HardwareSerial FIFO is 64 bytes, I should probably set USB_MIDI_SYSEX_MAX to 32.

 

[submo]

What Midi (Din) library are you using? The default Midi Library (included with Teensy) also has a SysEx buffer limit which you may need to increase (I think default is 128).

 

[newbie40]

I'm using my own "library". It can use very small SysEx buffers because it was initially developed for PIC16F where you can (almost) count available memory with the fingers of just one hand.

Found another workaround in the meantime: The SysEx callback of USBMIDI submits up to 290 bytes for a single invocation. This is larger than the output FIFO of the serial port (DIN MIDI). So the callback "blocks" for a while until all 290 bytes have been at least written to the FIFO. In all that time my code doesn't read any MIDI input. If I simply send the 290 bytes in chunks of 32 bytes and inbetween call my MIDI reading/parsing code, then it works without having to modify the value of USB_MIDI_SYSEX_MAX.

 

[submo]

In that case it may help to post your MIDI library code, or maybe just have a look how the default Midi library sends SysEx. Sending SysEx (from usbMIDI for example) using the default Teensy (Arduino) Midi library is sent a byte at a time (no output buffer is used).

The Teensy Midi library is however far from perfect when it comes to parsing received SysEx data. I'm not in front of my computer at the moment, but I don't recall having any issues sending and receiving large SysEx data chunks (via loopback) using the default Midi library (albeit with a slightly modified parsing algorithm to better deal with SysEx), but I will check this when I am home and report back.

 

[newbie40]

Had a look at the "official" Teensy MIDI library. The function sendSysEx() does exactly what I do and writes all bytes to the serial output, one byte at a time. That means, in reality it writes it into a FIFO with most likely 64 bytes of size and "in the background" the data from that FIFO gets written to the serial output register whenever the corresponding interrupt fires (at the MIDI baudrate, aka. "slow").

So if somebody would try to make a simple USB-MIDI interface by simply using the offical USBMIDI and the MIDI library, they would (for large SysEx) submit up to 290 bytes in a single call to MIDI::sendSysEx(). Assuming an empty FIFO, the first 64 bytes get written to FIFO "immediately", then the whole thing slows down. The time just to place 290 bytes into the FIFO is (290 - 64) * 320us = 72.32 milliseconds (ignoring the byte or two of extra hardware FIFO in the UART). During that time the MIDI library will not read from the serial input FIFO which overflows after 64 * 320us = 20.48 milliseconds.

Hence my feature request to make USB_MIDI_SYSEX_MAX adjustable by the "user".

 

[submo]

Just gave this a test and I'm seeing the same sort of thing.

I was only able to send/receive SysEx messages (via a physical loopback cable) of about 110 bytes before it started to skip data.

For example when sending a SysEx message of 238 bytes, I only received 108 bytes back via the loopback cable.

....and (as you already pointed out), when using a different Midi Interface (CME U6) to capture the Teensy Midi Output, the Teensy did successfully send all 238 bytes. Also, when sending the SysEx message from the U6 to the Teensy, the Teensy did successfully receive all 238 bytes.

I guess the solution here is to make sure the input buffer is getting read whilst the SysEx message is being sent.

I've been (slowly) working on an alternate Midi Library to the default one (as it has a few issues), which will get rid of any SysEx buffer limit, and will stream SysEx rather than wait for a complete message to be received/sent (which should help with this issue). The plan was then to take a look at usbMIDI and do the same, so thanks for pointing this out.

 

[newbie40]

I guess the solution here is to make sure the input buffer is getting read whilst the SysEx message is being sent.

Exactly that. - That's why I went with interleaving my send code with calls to my input reading/parsing (in addition to a smaller USB_MIDI_SYSEX_MAX value). That also means if your MIDI library provides a function to write a larger buffer of user-provided MIDI data you'd also need to interleave the sending with read calls.

Using small-ish buffers for SysEx and sending it in chunks is the proper way to do it. No matter how large you make this buffer to hold one mans complete SysEx message, there is always another man (or woman) who needs even larger SysEx buffers. I have one synth here that sends 73kByte as a single SysEx message.

Regarding the usbMIDI SysEx handling I have a feeling it doesn't handle "implicitly terminated" SysEx properly. That is SysEx with a missing 0xF7. In DIN MIDI you'd get those if a synth crashes (or gets powered down) while doing a SysEx dump or from unplugged MIDI cables. On the USB side you get those from crashing software that doesn't finish sending the full SysEx data.
At least in 1.59 (haven't checked 1.60 yet) I see invokations for the SysEx callbacks only for the USB packet types 0x04 to 0x07. An implicitly terminated SysEx transmission never sees anything other than 0x04. This causes the remaining bytes in the 290 byte buffer to never get sent to the callback if no 0xF7 is transmitted. MIDI specs say that during a SysEx transmission any other status byte than 0xF7 or System Realtime Messages causes the SysEx transmission to be terminated. I.e. it would need a flag indicating ongoing SysEx and a check for implicit termination which would then invoke the callback if there's bytes in the buffer (or better also if there is no bytes, just to let the user know that there's no more data to come). And all that of course separately for all "cables" (also the 290 bytes SysEx buffer would need to exist for each cable, which is another reason to rather have a smaller buffer).

 

[submo]

Regarding the usbMIDI SysEx handling I have a feeling it doesn't handle "implicitly terminated" SysEx properly. That is SysEx with a missing 0xF7. In DIN MIDI you'd get those if a synth crashes (or gets powered down) while doing a SysEx dump or from unplugged MIDI cables. On the USB side you get those from crashing software that doesn't finish sending the full SysEx data.

This is one of the main problems currently with the Default Teensy (Arduino) Midi Library. If a SysEx message is started with 0xF0, but doesn't receive a subsequent 0xF7 (due to an error of some sort) the Midi Library will get 'stuck' in a SysEx message and never leaves, and will be unresponsive to any further Midi input. Sending any other Midi Status byte after an 0xF0 should also terminate the SysEx message, but the Midi Library also doesn't take this into account.

Midi input becomes unresponsive after "phantom" SysEx start byte. · Issue #367 · FortySevenEffects/arduino_midi_library

Hello, Been having an issue that when hot swapping out TRS Midi input cables it would occasionally cause the MIDI library to freeze or become unresponsive. It seems that hot swapping TRS cables can...

github.com

I put a PR in some time ago which fixes this, but this Library seems somewhat abandoned so I've been working on my own. I was going to look at usbMIDI after and hopefully make a few improvements there (including SysEx for all cables) so your info is helpful.

 

[newbie40]

Looking at usb_midi.[h|cpp] it doesn't seem like that much extra work to add SysEx buffers per cable and add detection for implicit termination. But given the fact that there is even a commercial (with open-source code) Teensy based MIDI interface it seems nobody really cares or uses the code in a situation that would need those fixes.

If you do rewrite this, maybe also consider having callback functions that already get the cable number as first parameter. I find it sort of silly that one currently has to read some globals to get the cable.

Also the function for sending SysEx to USB needs to be changed so that you can submit data in multiple chunks. In my code I'm calling usb_midi_write_packed() directly in order to be able to send chunked SysEx buffers.

 

[PaulStoffregen]

Had a look at the "official" Teensy MIDI library. The function sendSysEx() does exactly what I do and writes all bytes to the serial output, one byte at a time. That means, in reality it writes it into a FIFO with most likely 64 bytes of size and "in the background" the data from that FIFO gets written to the serial output register whenever the corresponding interrupt fires (at the MIDI baudrate, aka. "slow").

For hardware serial, you can use addMemoryForWrite(buffer, size) to increase the serial transmit buffer size large enough to hold your entire outgoing message. Documentation at the hardware serial page.

For USB device MIDI, you are indeed constrained by the fixed size of the transmit buffer. On Teensy 4.x, the default size is 2048 bytes (which is 4 maximum size packets at 480 Mbit/sec speed). Remember USB MIDI packs 3 actual sysex bytes into each 32 bit word, so the 2K buffer is enough for 1536 actual sysex bytes. If you need larger, you'll need to dive into the core library source. Edit usb_midi.c at line 78.

 

[newbie40]

The reason the MIDI specs allow for System Realtime Messages to be inserted into a SysEx transmission is to keep those "realtime" messages, erm, "realtime". I.e. if you send timing clock together with SysEx you want this to be sent at roughly the proper time.

If we have a large SysEx buffer somewhere inside the code that first gets filled we end up with a situation where initially we only send the System Realtime Mesages and no SysEx data (since its still filling the buffer). Once the buffer is full we dump the 290 bytes of SysEx into our enlarged TX FIFO and it takes 290 * 320us to send those bytes (due to the enlarged TX FIFO at least our code doesn't block). In that time no System Realtime Messages will make it through to the DIN MIDI.

Of course you can argue that only very rearely do you need precise timing to continue while sending a firmware update or new sound patches to a synth through SysEx. I'm just trying to point out that the MIDI specs were explicitly designed for SysEx to be broken up into smaller chunks with other bytes being inserted. Using larger and larger buffers and then doing "burst" transfers seems the wrong thing to do. Using large buffers with large chunked SysEx makes the "realtime" timing of the MIDI stream very jittery.

But yes, I get the point that using a larger TX FIFO buffer is good enough for "everybody - 1".

 

[PaulStoffregen]

Please understand the software you're mentioning comes from 2 different authors. USB MIDI for Teensy, and the hardware serial driver, are from PJRC. The USB MIDI code from PJRC comes with all the MIDI parsing stuff, for USB but not for serial. For parsing MIDI in regular serial, the MIDI.h library from FortySevenEffects written by François Best is used.

This forum is PJRC. This is the correct place to discuss any shortcomings in the USB MIDI code for Teensy, and with the hardware serial driver, because all that stuff is published by PJRC. But if your concern is with the way messages are parsed or composed with regular serial MIDI, while the conversation may be interesting and we do allow conversation of almost anything even somewhat related to Teensy, please know in terms of affecting actual change, this forum is the wrong place to bring up improvements you want to see in the FortySevenEffects code.

Regarding USB device MIDI, the scenario you've mentioned with 290 byte sysex will become 388 bytes when converted to the 32 bit frames USB MIDI uses. That will all fit into a single USB packet, which can hold up to 512 bytes. Actual timing of USB communication has complicated overhead, but a good rule of thumb 13 maximum size packets can fit within a 125 us timing frame, according to the USB 2.0 spec Table 5-10 on page 55. Actual communication time is under 10 us. There are many complex factors that affect USB performance, but I'm really not convinced that the "large" 290 byte sysex message size is a meaningful problem for 480 Mbit/sec USB MIDI on Teensy 4.x.

 

[newbie40]

I'll try to ignore DIN MIDI stuff... however:

With the current code (Teensyduino 1.60) in usb_midi.c there is the following problem: If you (by mistake or for bug-finding purposes) send a SysEx without terminating 0xF7 then the data gets added to the internal buffer. If you then send some other properly terminated SysEx the callback gets invoked with the data from both SysEx transmissions together.

How to reproduce: Using Bome SendSX send "F0 00 01 02 03 04" (this is missing the F7). You'll not get an invokation of the callback (which is correct, because the SysEx transmission isn't complete). Now add the missing "F7" and send again. Callback gets invoked with "F0 00 01 02 03 04 F0 00 01 02 03 04 F7" being submitted. That is two separate SysEx transmission in a single invokation.
If you never send a properly terminated SysEx after the broken one, then the callback never gets called (i.e. you never get access to the "incomplete" SysEx data that is still sitting in the buffer).

Attached is fixed code that handles this correctly. The submitted data I get with it looks like this (simply printing the buffers inside the callback, "EOX" indicates that the "complete" flag was set):

This is two invokations now. The line complaining about implicit termination is simply a check (inside the callback) whether the submitted buffer actually really ends with 0xF7. There'll also be an invokation now (after having submitted incomplete SysEx) if instead of another SysEx you send any other message which is not a System Realtime message (like a note-on, for example). That is "implicit termination".

The code also adds the individual SysEx buffers for the cables. It does break backwards compatibility if somebody was accessing the global SysEx buffer directly because that's an array now.

While testing this code I realized that Windows 10 (haven't tested on other versions yet) seems to be quite creative when parsing buffers submitted with midiOutLongMsg(). If there is a System Realtime message inserted in a SysEx message (which is legal), then it doesn't send this message as a separate USB packet (normally of type 0xF) but instead sends it as part of the SysEx data. So you might find System Realtime status bytes (i.e. 0xF8..0xFF) sprinkled into the SysEx buffer you get in the callback. This is also the case with the original, unmodified code.
To "fix" this (if it is considered a bug) there would need to be a check for those bytes in sysex_byte() (which would then require the switch() for the type of System Realtime message to sit in a separate function so that it can be called from multiple places).

EDIT: Test sequence for the above: "F8 F0 00 01 F8 02 03 04 F7". The first F8 triggers the system realtime callback, the one embedded inside the SysEx ends up inside the buffer of SysEx data.
EDIT2: Replaced the ZIP with the code with a version that fixes a problem in my first version when sending SysEx emssages with a total length smaller than 4 bytes (not that this would make any usable message, but anyway). Also added the filtering of System Realtime messages that come embedded inside a SysEx transmission.