Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 11 of 11

Thread: Problem with large volumes of data on usbMidi

  1. #1
    Junior Member
    Join Date
    Feb 2018
    Posts
    6

    Problem with large volumes of data on usbMidi

    Hello everyone

    First of all, thanks to Paul for his work, his support and the dynamism of the Teensy range and sorry for my very rough English and assisted by the translation tools.

    I'm blocking on a problem in Teensy 3.2 + Arduino IDE 1.8.5 and Teensyduino 1.4.1 (and earlier)

    I use the usbMidi to transfer a large volume of SysEx: from 100 to 400 pages of 300 or 400 bytes ...
    I can not post the source code too much because it is very large. I will therefore be content with extracts. I hope they will suffice.

    The blocks are 331 bytes with the headers and I send 100 to 400 blocks by Dump.
    Basically, I have 0.5% of blocks that arrive with a wrong length. In this case, each time 3 bytes are missing. I get a block of 328 bytes instead of 331.
    The first 11 bytes (the header of the SySex block ...) are always good, it is then that a frame of 3 bytes can be loose.

    If I repeat the same transfer, the errors may be in different places. It happens a few times that the transfer is done without error. I tested other USB ports, other cables. I find the errors with both MIDI OX with a personal development fast (C# Api Windows etc ...)

    I must presure that the Teensy is well solicited : Serial1 Serial2 and Serial 3 are open respectively to 210000, 115200, 1152000 ...
    that Serial communications work perfectly and that there is no incoming or outgoing flow during these SysEx transfers.

    I tried MIDI, SERIAL + MIDI, SERIAL + MIDI + AUDIO .... without change and other options like Fast, Faster, Faster With LTO link etc ...

    I do not use hardware interrupts other than normal SPI, I2C, Serial and USB).


    Code:
    static uint8_t dataSys[_SIZEBUFMID];
    static uint16_t maxSys=0;
    ...  
    void dataSysSend()
    {
      switch( sysDest )
      {
      case _USBMIDI:
          usbMIDI.sendSysEx(maxSys, dataSys, true);
          break;
      case _RIVERKEYMIDI:
          MID_IN.sendSysEx(maxSys, dataSys, true);
          break;
      case _BOARDMIDI:
          MID_OUT.sendSysEx(maxSys, dataSys, true);
          break;
      }
      maxSys=0;
      delay(40);
    }
    
    void sysSYS(uint8_t value)
    {
      dataSys[maxSys++] = value;
      if (value == 0xF7 ) dataSysSend();
    }
    
    
    void sysCC(uint8_t value)
    {
      static uint8_t data;
      data = ( value & 0x007F );
      dataSys[maxSys++] = data;
    }
    
    
    void sysNRPN(int16_t value)
    {
      static uint8_t data[2];
      data[0]  = ( ((uint16_t)(value) & 0x3F80) >> 7 ) & 0x007F;
      data[1]  = ((uint16_t)(value) & 0x007F);
      dataSys[maxSys++] = data[0];
      dataSys[maxSys++] = data[1];
    }
    
    
    #define NB_PATCH 200
    
    void SendAllPatch(...)
    {
    	for (int ct = 0; ct < NB_PATCH ; ct++)
    	{
    		sysSYS(0xF0);
    		...
    		sysCC(bytes_1);
    		...
    		sysCC(bytes_300);
    		...
    		sysSYS(0xF7); // With F7 , sysSYS call dataSysSend...
    
    	}
    }


    I tried to slow down the USB, to put delays in usb_midi_write_packed in usb_midi.c,

    to change on
    #define TX_TIMEOUT_MSEC 40 by testing the values ​​from 10 to 200 ...

    to change on
    #define TX_PACKET_LIMIT 6 by testing the values ​​from 1 (to avoid bufferization) to 20 ...

    to change on
    **#define MIDI_INTERFACE 2 // MIDI
    **#define MIDI_NUM_CABLES 1
    **#define MIDI_TX_ENDPOINT 4
    **#define MIDI_TX_SIZE 64
    **#define MIDI_RX_ENDPOINT 5
    **#define MIDI_RX_SIZE 64
    *in usb_desc.h
    *
    to create a function usb_midi_write_packed without queue ...
    *
    all without success.

    That's it, hoping to find a solution other than getting back through a MIDI so slow

    Best regards,

    Laurent
    Last edited by Baloran; 02-14-2018 at 09:20 AM.

  2. #2
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    9,710
    Can you edit post with CODE "#" - hashtag symbol on the edit toolbar to make code more readable?


    Looking at this - assume it is the BULK send - perhaps put a delay(10) [more or less to test] between send requests? not sure if that was tried?
    Code:
    #define NB_PATCH 200
    
    void SendAllPatch(...)
    {
      for (int ct = 0; ct < NB_PATCH ; ct++)
      {
        sysSYS(0xF0);
        ...
        sysCC(bytes_1);
        ...
        sysCC(bytes_300);
        ...
        sysSYS(0xF7);
    
        dataSysSend();
    
        delay(10); // was this tried here?
      }
    }

  3. #3
    Junior Member
    Join Date
    Feb 2018
    Posts
    6
    Quote Originally Posted by defragster View Post
    Can you edit post with CODE "#" - hashtag symbol on the edit toolbar to make code more readable?
    Thanks, CODE insert

    delay is at the end of dataSysSend . I have tried 10, 20 .... 200 ms

  4. #4
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    9,710
    Much better reading : indeed - there is that delay()!

  5. #5
    Senior Member oddson's Avatar
    Join Date
    Feb 2013
    Location
    Isle in the Salish Sea
    Posts
    1,165
    Have you checked the size of value in sysCC by printing each to the serial monitor?

    It would seem to me if they are the correct size here and not after sending then a bug in the library is a possible (perhaps likely) cause.

    If not then the bug is in the code you're not showing us.

  6. #6
    Junior Member
    Join Date
    Feb 2018
    Posts
    6
    Quote Originally Posted by oddson View Post
    Have you checked the size of value in sysCC by printing each to the serial monitor?
    Yes, like this...303 , 83 , 331, 291 and 11 are acceptable. All other are printing. All bloc data pass to usbMIDI.sendSysEx are valid.

    For testing, I also disabled the interrupts of the 3 serials used during transmission. It does not change anything.



    Code:
    void dataSysSend()
    {
    
    #ifdef _DEBUG_SERIAL_
      if ( maxSys != 303 && maxSys != 83 && maxSys != 331 && maxSys != 291 && maxSys != 11 )
      {
          Serial.print(F("dataSysSend"));
          Serial.print(F("-"));
          Serial.println(maxSys);
              
      }
    #endif
      
      
      while( Serial1.available() ) Serial1.read();
      Serial1.flush();
      NVIC_DISABLE_IRQ(IRQ_UART0_STATUS);
    
      while( Serial2.available() ) Serial2.read();
      Serial2.flush();
      NVIC_DISABLE_IRQ(IRQ_UART1_STATUS);
      
      while( Serial3.available() ) Serial3.read();
      Serial3.flush();
      NVIC_DISABLE_IRQ(IRQ_UART2_STATUS); 
         
      switch( sysDest )
      {
      case _USBMIDI:
          usbMIDI.sendSysEx(maxSys, dataSys, true);
          break;
      case _RIVERKEYMIDI:
          MID_IN.sendSysEx(maxSys, dataSys, true);
          break;
      case _BOARDMIDI:
          MID_OUT.sendSysEx(maxSys, dataSys, true);
          break;
      }
      
      maxSys=0;
      delay(40);
    
      NVIC_ENABLE_IRQ(IRQ_UART0_STATUS);
      NVIC_ENABLE_IRQ(IRQ_UART1_STATUS);
      NVIC_ENABLE_IRQ(IRQ_UART2_STATUS);
     
    }
    It is important to note that the same processing with the same data will give different errors. These are not the same blocks that are always in error. So the errors do not seem to be related to the content of the blocks.
    And this random error, but still 3 bytes, whatever the size of the block, crisps me ....

  7. #7
    Senior Member oddson's Avatar
    Join Date
    Feb 2013
    Location
    Isle in the Salish Sea
    Posts
    1,165
    Have you tried increasing the MIDI limit (line 44) in usb_midi.h ?

    If it's a bug it's likely limited to those blocks it has to break up. So you might be able to avoid it by increasing the limit.

    Also, if that works it should go a long way to confirming a bug and pointing to where it might be.

  8. #8
    Junior Member
    Join Date
    Feb 2018
    Posts
    6
    Well, I'm on the track ...

    In usb_dev.c, procedure

    Code:
    void usb_isr (void)
    {
    ....
    if ((status & USB_ISTAT_SOFTOK / * 04 * /)) {
    ....
    #ifdef MIDI_INTERFACE
                           usb_midi_flush_output ();
    #endif
    ....
    This usb_midi_flush_output call can occur during the execution of usb_midi_send_sysex_buffer_has_term
    and puts the chronoligy of tx_buffer in cabbages ...

    If I comment on this line,

    #ifdef MIDI_INTERFACE
    // usb_midi_flush_output ();
    #endif

    All my blocks arrive in perfect condition !!!!

    Now, I would like to understand what this USB_ISTAT_SOFTOK and the repercussion of the disable

    PS: thanks Oddson & defragster for you reply

  9. #9
    Junior Member
    Join Date
    Feb 2018
    Posts
    6
    Finally, the midiUSB flush line in void usb_isr (void) is important because it seems to autoflush unreleased outgoing data.

    So I modified usb_midi.c to add a flag to block flush when transmit SysEx...



    Code:
    ...
    static uint8_t transmit_previous_timeout=0;
    static uint8_t tx_noautoflush=0;
    /* flag anti flush */
    static uint8_t stopFlush = 0;
    
    void usb_midi_send_sysex_buffer_has_term(const uint8_t *data, uint32_t length, uint8_t cable)
    {
    		stopFlush = 1;
    		
    		cable = (cable & 0x0F) << 4;
    ...
    
    		stopFlush = 0;
    
    }
    
    void usb_midi_send_sysex_add_term_bytes(const uint8_t *data, uint32_t length, uint8_t cable)
    {
    	stopFlush = 1;
    	
    	cable = (cable & 0x0F) << 4;
    
    ....
    	stopFlush = 0;
    	
    }
    
    void usb_midi_flush_output(void)
    {
    	if ( stopFlush == 0 ) 
    	{
    		if (tx_noautoflush == 0 && tx_packet && tx_packet->index > 0) {
    			tx_packet->len = tx_packet->index * 4;
    			usb_tx(MIDI_TX_ENDPOINT, tx_packet);
    			tx_packet = usb_malloc();
    		}
    	}
    }

  10. #10
    Senior Member oddson's Avatar
    Join Date
    Feb 2013
    Location
    Isle in the Salish Sea
    Posts
    1,165
    I wonder if calling void usb_midi_flush_output(void) right before sending the block could prevent it from flushing on it's own in the middle of things without changing libraries?

    Also, the three-byte thing seems strange. You'd think it would have a more significant effect on the received block and certainly wouldn't have thought it would always be three bytes... I guess that's how long the interruption stops Teensy from following the incoming stream??

  11. #11
    Junior Member
    Join Date
    Feb 2018
    Posts
    6
    Quote Originally Posted by oddson View Post
    if calling void usb_midi_flush_output(void) right before sending the block could prevent it from flushing
    Tested and no. If block of datas was shorter, possible but yet not tested
    3 bytes are the size of data queuing in a midi over usb message 0x04 or 0x07, this 3 bytes have gived me the direction to find the problem.
    Now, all works fine, 100% of bloc are safe recept

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •