Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 15 of 15

Thread: Any reason why SysEx length is hardcoded at 60 bytes in usbMIDI usb_api.h?

  1. #1
    Junior Member
    Join Date
    Apr 2013
    Location
    Oka, Canada (It's near Montreal)
    Posts
    13

    Any reason why SysEx length is hardcoded at 60 bytes in usbMIDI usb_api.h?

    The SysEx limit is set to 60 bytes in the usbMIDI file usb_api.h
    Is this a USB requirement?
    Can I safely change it? If YES, is there an upper limit?

    Note that the maximum size of SysEx is set to 255 by default in Arduino Midi Lib 3.2 & up. It can be set to a maximum of 65535.

    Thanks.

  2. #2
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,606

    The SysEx limit is set to 60 bytes in the usbMIDI file usb_api.h
    Is this a USB requirement?
    There no specific reason for 60 bytes. It's just an arbitrary number.

    Can I safely change it? If YES, is there an upper limit?
    Sure, but increasing it will allocate more memory to hold the message. You'll see the memory estimate increase when you compile. Keep in mind memory is used in 3 ways: static/global, stack, and malloc. The estimate only shows the static/global usage.

    Note that the maximum size of SysEx is set to 255 by default in Arduino Midi Lib 3.2 & up. It can be set to a maximum of 65535.
    How many applications need larger Sysex messages? I could increase the default if it's commonly needed, but doing so wastes memory for everyone who doesn't edit the code, so it doesn't want to be set needlessly high.

  3. #3
    Junior Member
    Join Date
    Apr 2013
    Location
    Oka, Canada (It's near Montreal)
    Posts
    13
    I'm transferring kilo bytes of SysEx. The receiving Teensy++ 2.0 then pushed the data to EEPROM. I guess I can cut the data in chunks in order not to tax the available memory too much. I can definitely spare 255 bytes. So I'll change it to that. My program is not that big.

    SysEx in a Midi environment is mostly used two ways: First, many programs and equipment pieces send short, freely formatted snippets using SysEx. That's where 60 bytes is usually plenty. Most of the time, I see < 20 bytes travelling back and forth. Second, SysEx is also used to transfer large chunks of memory to, and from, Midi devices. For example, my M-Audio Oxygen 49 keyboard dumps 3090 bytes of data when it receives the proper SysEx code. My Behringer FCB1010 pedal board similarly dumps 2352 bytes. My own project receives unknown lengths of SysEx data (up to 4 KB) to tell it what to do with incoming Midi data(configuration).

    But I agree that for general use, extra amount of memory reserved for SysEx transfer would be a waste of space.
    I'll re-write my code to cut data into 255 byte chunks, and even 60 bytes. I'll then measure performance hits (if any) and post again.

    Thanks.

  4. #4
    Senior Member
    Join Date
    Nov 2012
    Location
    Boston, MA, USA
    Posts
    1,108
    Quote Originally Posted by PaulStoffregen View Post
    How many applications need larger Sysex messages? I could increase the default if it's commonly needed, but doing so wastes memory for everyone who doesn't edit the code, so it doesn't want to be set needlessly high.
    Can the memory used be dynamically allocated?

    Sysex is commonly used for loading and saving synth patches. Its also used to transfer sample data for hardware samplers and to providefirmware updates. So they can be quite large. As an example, a screenshot from SysEx Librarian shows actual saved sysex ranging from 267 bytes at the low end to 67k at the high end.
    Click image for larger version. 

Name:	MainWindowSmall.gif 
Views:	277 
Size:	21.3 KB 
ID:	390

  5. #5
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,606
    Quote Originally Posted by Nantonos View Post
    Can the memory used be dynamically allocated?
    That would be quite difficult.

    You don't know how long the sysex message will be, so you don't know how much space to allocate. It's possible to keep calling realloc() as more data arrives, but that's a pretty terrible solution. If any other code executes in the meantime and does any allocation (eg, using String), the realloc is forced to allocate a new large block and move all the data, leaving big memory fragmentation holes. That's a far worse outcome than simply allocating a large static buffer.

    The main problem is the current API, which is modeled after the MIDI library. That API requires the entire sysex message to be in a single array.

    A better solution probably involves an alternate ?more complex) API for sysex that provides the data in chunks. Exactly what that API should be is a good question. I'm open to suggestions?

  6. #6
    Senior Member
    Join Date
    Nov 2012
    Posts
    1,155
    Just off the top of my (balding) head, add a function to the library to permit the user to specify an alternate buffer for SysEx data? By default, you get 60. If you need more it's up to you to provide it. There are only a few places in the library where sysex is handled so it shouldn't be difficult to implement - not that I'm volunteering to do it

    Pete

  7. #7
    Senior Member
    Join Date
    Mar 2013
    Posts
    126
    A lot of protocols invoke this problem, e.g. OSC and many other streaming protocols that assume a reasonable amount of memory is available. This is one reason why at the core of the internet is the notion of "best effort" - not a guarantee. Most lost packets on the internet are from someone in the route having inadequate memory.
    What we are doing for oscuino now is a small preemptive allocation and realloc: this is relatively efficient. The hardest thing is being careful to test the tricky logic need to abandon the data when you run out of memory, free the memory and report errors.

    Note also the weirdness in MIDI that you can send other messages in the middle of a Sysex.
    Last edited by adrianfreed; 04-15-2013 at 02:38 AM.

  8. #8
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,606
    Perhaps the API could be extended with an option to change how Sysex is reported.

    Right now, you get a single callback or return code for the entire Sysex message. Perhaps setting the alternate API would give you the incoming Sysex in 60 byte chunks? If the message is 6000 bytes long, you'll get 100 calls to your function that handles reception of sysex. But how would you know the message has ended? Maybe a specific Sysex complete return/callback is needed?

  9. #9
    Junior Member
    Join Date
    Apr 2013
    Location
    Oka, Canada (It's near Montreal)
    Posts
    13
    @Paul, that's how I decided to do it: the sending program cuts the SysEx data into chunks and sends the number of chunks along with the first one. The receiving program then knows what to do. I decided to use a MAX_SYS_LENGTH = 255 since my SysEx are rarely less than 100 bytes anyway. I think it's the responsibility of the sending program to know the limits of the receiving library. But I agree with adrianfreed that it's always tricky. In my case, I have to make sure that my users will use the modified library if they want to recompile the code.

    The usbMIDI library code should give a warning to users. Right now, I believe that it just truncates the incoming SysEx after 60 bytes. No error and returns 60 as SysEx length. A typical user might not see this. Maybe return a length of 0, or -1, if the incoming length is > than the allocated buffer?

  10. #10
    I would love to see a bit better API for sysex. It took a bit of effort to figure out the current one. In particular that the length was in usb_midi_msg_data1 and that this was an 8bit quantity and therefore we couldn't receive messages larger than 254 bytes or so (because of begin and end bytes).

    For my pedal, I'm sending JSON data over sysex from Ableton Live. I can only send midi data from Ableton, I can't directly communicate via USB HID RAW or something (Ableton has pure python API with no C module loading). I choose JSON because, a) sysex is already 7 bit, b) i'd have to come up with my own encoding, c) there are stream oriented parsers for JSON, and d) everyone understands JSON! However, JSON is not compact. I'm sending LED information for 64 LEDs and these messages end up being several thousand bytes. I think the teensy and USB are plenty fast enough to receive and parse this.

    I've increased the midi buffer size to #define USB_MIDI_SYSEX_MAX 500 and I changed the type of usb_midi_msg_data1 to uint16_t.

    The code I'm currently using is something like:

    Code:
    typedef struct {
            jsmntype_t type;
            int start;
            int end;
            int size;
    } jsmntok_t;
    
    jsmn_parser parser;
    #define NUM_TOKENS 100
    jsmntok_t tokens[NUM_TOKENS];
    
    #define BUFF_SIZE 1500
    char buffer[BUFF_SIZE];
    uint32_t bufcnt = 0;
    
    while(1) {
      bool more = false;
      if (usbMIDI.read()) {
        if (usb_midi_msg_sysex[1] == 0x63) { // start's with 'c', so it has to be my sysexdata, not someone else's :)  
          int i = 2;
          while(bufcnt < BUFF_SIZE && i < usb_midi_msg_data1-1) {
            Debug.print((char)usb_midi_msg_sysex[i]);
            buffer[bufcnt] = (char)usb_midi_msg_sysex[i];
          }
          buffer[bufcnt] = 0;
    
          int status = jsmn_parse(&parser, buffer, tokens, NUM_TOKENS);
      
          bool reset = false;
          bool more = false;
          if (status == JSMN_SUCCESS) {
            // do something with the JSON data in 'tokens'.
            reset = true;
          } else if (status == JSMN_ERROR_PART) {
            // JSON parser says this is a partial message, keep parsing the next time we get a midi message.
            reset = false;
            more = true;
          }
      
          if (reset) {
            jsmn_init(&parser);
            if (!more) bufcnt = 0;
          }
        }
      }
    }
    I run into memory problems when trying to increase 'buffer' beyond about 2k. I'm not sure why this is as I think I should have enough ram to receive much more than this. I estimate 'tokens' is about 1.4k, and so I would think I should be able to make buffer at least 5-6k but I can't...

    Anyway, i mostly wanted to demonstrate that, a) it is possible to use larger messages, and b) the sysex API could use some cleanup.
    Last edited by cmason; 04-16-2013 at 05:58 AM.

  11. #11
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,606
    The existing API was an attempt to closely match MIDI library. Actually, there were 2 libraries, another using C++ object inheritance for callbacks. I added similar callbacks without the object inheritance, and the MIDI library adopted those callbacks in version 3.1.

    The sysex API really does need some improvement. I'm open to ideas.

  12. #12
    Senior Member
    Join Date
    Mar 2013
    Posts
    126
    I am going to publish a recommendation soon on how to tunnel OSC with MIDI sysex. I prefer to implement these things well before
    making them into a formal specification. The use case I will tune is Teensy 3.0 <-> Max/MSP + PD


    It would be nice to have similar efficiencies to Paul's tuned serial library in this case. I plan to make a stab at a sysex API that mirrors what we are doing with the OSC SLIP work, basically extending Stream with calls for the packet model which was designed to look like the ethernet UDP library calls, i.e. available(), endofpacket(), readBytes, read, peek, write, beginPacket(), endPacket().

  13. #13
    Junior Member
    Join Date
    Mar 2014
    Posts
    19
    Old thread, but I ran into the 60 byte limit as well. It's not important for my application since all messages are likely to be <60, but the convenient testbed I was using sent 2060 bytes. I agree with PracticalUsage, returning -1 or 0 would help to indicate the overflow. Adding this to the documentation would be helpful, too.

  14. #14
    Junior Member
    Join Date
    May 2014
    Posts
    3
    A streaming API would do nicely. Each time the current SysEx packet is about to fill up, call the SysEx handler with the partial buffer. Something like this:
    Code:
    static int sys_ex_length;
    static char sys_ex_buffer[256];
    
    static void onSysEx(uint8_t length, const uint8_t *data, bool complete) {
      if (sys_ex_length + length < 256) {
        memcpy(sys_ex_buffer + sys_ex_length, data, length);
        sys_ex_length += length;
        if (complete) {
          sys_ex_buffer[sys_ex_length - 1] = '\0';
          sys_ex_length = 0;
          Serial.print("data = \""), Serial.print((const char *)(sys_ex_buffer + 1)), Serial.print("\""), Serial.println();
        }
      }
    }
    
    void setup() {
       usbMIDI.setHandleSysEx(&OnSysEx);
    }
    This would let you use whatever size buffer you want. We might also want a way to configure the buffer size dynamically, when we know the max size our SysEx packets will ever be.
    Last edited by pcbeard; 05-20-2014 at 11:51 PM.

  15. #15
    Junior Member
    Join Date
    May 2014
    Posts
    3
    See my pull request:

    https://github.com/PaulStoffregen/cores/pull/17

    I only implemented it in teensy3, but it should be easy to backport.
    Last edited by pcbeard; 05-21-2014 at 03:37 PM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •