Any reason why SysEx length is hardcoded at 60 bytes in usbMIDI usb_api.h?

[PracticalUsage]

The SysEx limit is set to 60 bytes in the usbMIDI file usb_api.h
Is this a USB requirement?
Can I safely change it? If YES, is there an upper limit?

Note that the maximum size of SysEx is set to 255 by default in Arduino Midi Lib 3.2 & up. It can be set to a maximum of 65535.

Thanks.

[PaulStoffregen]

The SysEx limit is set to 60 bytes in the usbMIDI file usb_api.h
Is this a USB requirement?

There no specific reason for 60 bytes. It's just an arbitrary number.

Can I safely change it? If YES, is there an upper limit?

Sure, but increasing it will allocate more memory to hold the message. You'll see the memory estimate increase when you compile. Keep in mind memory is used in 3 ways: static/global, stack, and malloc. The estimate only shows the static/global usage.

Note that the maximum size of SysEx is set to 255 by default in Arduino Midi Lib 3.2 & up. It can be set to a maximum of 65535.

How many applications need larger Sysex messages? I could increase the default if it's commonly needed, but doing so wastes memory for everyone who doesn't edit the code, so it doesn't want to be set needlessly high.

[PracticalUsage]

I'm transferring kilo bytes of SysEx. The receiving Teensy++ 2.0 then pushed the data to EEPROM. I guess I can cut the data in chunks in order not to tax the available memory too much. I can definitely spare 255 bytes. So I'll change it to that. My program is not that big.

SysEx in a Midi environment is mostly used two ways: First, many programs and equipment pieces send short, freely formatted snippets using SysEx. That's where 60 bytes is usually plenty. Most of the time, I see < 20 bytes travelling back and forth. Second, SysEx is also used to transfer large chunks of memory to, and from, Midi devices. For example, my M-Audio Oxygen 49 keyboard dumps 3090 bytes of data when it receives the proper SysEx code. My Behringer FCB1010 pedal board similarly dumps 2352 bytes. My own project receives unknown lengths of SysEx data (up to 4 KB) to tell it what to do with incoming Midi data(configuration).

But I agree that for general use, extra amount of memory reserved for SysEx transfer would be a waste of space.
I'll re-write my code to cut data into 255 byte chunks, and even 60 bytes. I'll then measure performance hits (if any) and post again.

Thanks.

[Nantonos]

How many applications need larger Sysex messages? I could increase the default if it's commonly needed, but doing so wastes memory for everyone who doesn't edit the code, so it doesn't want to be set needlessly high.

Can the memory used be dynamically allocated?

Sysex is commonly used for loading and saving synth patches. Its also used to transfer sample data for hardware samplers and to providefirmware updates. So they can be quite large. As an example, a screenshot from SysEx Librarian shows actual saved sysex ranging from 267 bytes at the low end to 67k at the high end.

[PaulStoffregen]

Can the memory used be dynamically allocated?

That would be quite difficult.

You don't know how long the sysex message will be, so you don't know how much space to allocate. It's possible to keep calling realloc() as more data arrives, but that's a pretty terrible solution. If any other code executes in the meantime and does any allocation (eg, using String), the realloc is forced to allocate a new large block and move all the data, leaving big memory fragmentation holes. That's a far worse outcome than simply allocating a large static buffer.

The main problem is the current API, which is modeled after the MIDI library. That API requires the entire sysex message to be in a single array.

A better solution probably involves an alternate ?more complex) API for sysex that provides the data in chunks. Exactly what that API should be is a good question. I'm open to suggestions?

[el_supremo]

Just off the top of my (balding) head, add a function to the library to permit the user to specify an alternate buffer for SysEx data? By default, you get 60. If you need more it's up to you to provide it. There are only a few places in the library where sysex is handled so it shouldn't be difficult to implement - not that I'm volunteering to do it

Pete

[adrianfreed]

A lot of protocols invoke this problem, e.g. OSC and many other streaming protocols that assume a reasonable amount of memory is available. This is one reason why at the core of the internet is the notion of "best effort" - not a guarantee. Most lost packets on the internet are from someone in the route having inadequate memory.
What we are doing for oscuino now is a small preemptive allocation and realloc: this is relatively efficient. The hardest thing is being careful to test the tricky logic need to abandon the data when you run out of memory, free the memory and report errors.

Note also the weirdness in MIDI that you can send other messages in the middle of a Sysex.

[PaulStoffregen]

Perhaps the API could be extended with an option to change how Sysex is reported.

Right now, you get a single callback or return code for the entire Sysex message. Perhaps setting the alternate API would give you the incoming Sysex in 60 byte chunks? If the message is 6000 bytes long, you'll get 100 calls to your function that handles reception of sysex. But how would you know the message has ended? Maybe a specific Sysex complete return/callback is needed?

[PracticalUsage]

@Paul, that's how I decided to do it: the sending program cuts the SysEx data into chunks and sends the number of chunks along with the first one. The receiving program then knows what to do. I decided to use a MAX_SYS_LENGTH = 255 since my SysEx are rarely less than 100 bytes anyway. I think it's the responsibility of the sending program to know the limits of the receiving library. But I agree with adrianfreed that it's always tricky. In my case, I have to make sure that my users will use the modified library if they want to recompile the code.

The usbMIDI library code should give a warning to users. Right now, I believe that it just truncates the incoming SysEx after 60 bytes. No error and returns 60 as SysEx length. A typical user might not see this. Maybe return a length of 0, or -1, if the incoming length is > than the allocated buffer?

[cmason]

I would love to see a bit better API for sysex. It took a bit of effort to figure out the current one. In particular that the length was in usb_midi_msg_data1 and that this was an 8bit quantity and therefore we couldn't receive messages larger than 254 bytes or so (because of begin and end bytes).

For my pedal, I'm sending JSON data over sysex from Ableton Live. I can only send midi data from Ableton, I can't directly communicate via USB HID RAW or something (Ableton has pure python API with no C module loading). I choose JSON because, a) sysex is already 7 bit, b) i'd have to come up with my own encoding, c) there are stream oriented parsers for JSON, and d) everyone understands JSON! However, JSON is not compact. I'm sending LED information for 64 LEDs and these messages end up being several thousand bytes. I think the teensy and USB are plenty fast enough to receive and parse this.

I've increased the midi buffer size to #define USB_MIDI_SYSEX_MAX 500 and I changed the type of usb_midi_msg_data1 to uint16_t.

The code I'm currently using is something like:

typedef struct {
        jsmntype_t type;
        int start;
        int end;
        int size;
} jsmntok_t;

jsmn_parser parser;
#define NUM_TOKENS 100
jsmntok_t tokens[NUM_TOKENS];

#define BUFF_SIZE 1500
char buffer[BUFF_SIZE];
uint32_t bufcnt = 0;

while(1) {
  bool more = false;
  if (usbMIDI.read()) {
    if (usb_midi_msg_sysex[1] == 0x63) { // start's with 'c', so it has to be my sysexdata, not someone else's :)  
      int i = 2;
      while(bufcnt < BUFF_SIZE && i < usb_midi_msg_data1-1) {
        Debug.print((char)usb_midi_msg_sysex[i]);
        buffer[bufcnt] = (char)usb_midi_msg_sysex[i];
      }
      buffer[bufcnt] = 0;

      int status = jsmn_parse(&parser, buffer, tokens, NUM_TOKENS);
  
      bool reset = false;
      bool more = false;
      if (status == JSMN_SUCCESS) {
        // do something with the JSON data in 'tokens'.
        reset = true;
      } else if (status == JSMN_ERROR_PART) {
        // JSON parser says this is a partial message, keep parsing the next time we get a midi message.
        reset = false;
        more = true;
      }
  
      if (reset) {
        jsmn_init(&parser);
        if (!more) bufcnt = 0;
      }
    }
  }
}

I run into memory problems when trying to increase 'buffer' beyond about 2k. I'm not sure why this is as I think I should have enough ram to receive much more than this. I estimate 'tokens' is about 1.4k, and so I would think I should be able to make buffer at least 5-6k but I can't...

Anyway, i mostly wanted to demonstrate that, a) it is possible to use larger messages, and b) the sysex API could use some cleanup.

[PaulStoffregen]

The existing API was an attempt to closely match MIDI library. Actually, there were 2 libraries, another using C++ object inheritance for callbacks. I added similar callbacks without the object inheritance, and the MIDI library adopted those callbacks in version 3.1.

The sysex API really does need some improvement. I'm open to ideas.

[adrianfreed]

I am going to publish a recommendation soon on how to tunnel OSC with MIDI sysex. I prefer to implement these things well before
making them into a formal specification. The use case I will tune is Teensy 3.0 <-> Max/MSP + PD


It would be nice to have similar efficiencies to Paul's tuned serial library in this case. I plan to make a stab at a sysex API that mirrors what we are doing with the OSC SLIP work, basically extending Stream with calls for the packet model which was designed to look like the ethernet UDP library calls, i.e. available(), endofpacket(), readBytes, read, peek, write, beginPacket(), endPacket().

[jmercer]

Old thread, but I ran into the 60 byte limit as well. It's not important for my application since all messages are likely to be <60, but the convenient testbed I was using sent 2060 bytes. I agree with PracticalUsage, returning -1 or 0 would help to indicate the overflow. Adding this to the documentation would be helpful, too.

[pcbeard]

A streaming API would do nicely. Each time the current SysEx packet is about to fill up, call the SysEx handler with the partial buffer. Something like this:

static int sys_ex_length;
static char sys_ex_buffer[256];

static void onSysEx(uint8_t length, const uint8_t *data, bool complete) {
  if (sys_ex_length + length < 256) {
    memcpy(sys_ex_buffer + sys_ex_length, data, length);
    sys_ex_length += length;
    if (complete) {
      sys_ex_buffer[sys_ex_length - 1] = '\0';
      sys_ex_length = 0;
      Serial.print("data = \""), Serial.print((const char *)(sys_ex_buffer + 1)), Serial.print("\""), Serial.println();
    }
  }
}

void setup() {
   usbMIDI.setHandleSysEx(&OnSysEx);
}

This would let you use whatever size buffer you want. We might also want a way to configure the buffer size dynamically, when we know the max size our SysEx packets will ever be.

[pcbeard]

See my pull request:

https://github.com/PaulStoffregen/cores/pull/17

I only implemented it in teensy3, but it should be easy to backport.

[Garug]

This was long time a go, but have the same problem

trying to do with Teensy 4.1

usbMIDI.sendSysEx(SysExLength, USBhostMIDI.getSysExArray(), true);}

usbMIDI is the computer side of the connection and USBhostMIDI connects the device to USB Host port


The SysExLength is 290. The shorter messages work ok, but is there any examples how to do the longer than 60 messages without changing the buffer.

The sendSysEx has the true/false for the end of the message, so is it just 4 x 60 messages with false and last 50 bytes with true?

How about the getSysExArray I guess that has also the same limitation?