Problem with large volumes of data on usbMidi

Baloran · Feb 14, 2018

Hello everyone

First of all, thanks to Paul for his work, his support and the dynamism of the Teensy range

and sorry for my very rough English and assisted by the translation tools.

I'm blocking on a problem in Teensy 3.2 + Arduino IDE 1.8.5 and Teensyduino 1.4.1 (and earlier)

I use the usbMidi to transfer a large volume of SysEx: from 100 to 400 pages of 300 or 400 bytes ...
I can not post the source code too much because it is very large. I will therefore be content with extracts. I hope they will suffice.

The blocks are 331 bytes with the headers and I send 100 to 400 blocks by Dump.
Basically, I have 0.5% of blocks that arrive with a wrong length. In this case, each time 3 bytes are missing. I get a block of 328 bytes instead of 331.
The first 11 bytes (the header of the SySex block ...) are always good, it is then that a frame of 3 bytes can be loose.

If I repeat the same transfer, the errors may be in different places. It happens a few times that the transfer is done without error. I tested other USB ports, other cables. I find the errors with both MIDI OX with a personal development fast (C# Api Windows etc ...)

I must presure that the Teensy is well solicited : Serial1 Serial2 and Serial 3 are open respectively to 210000, 115200, 1152000 ...
that Serial communications work perfectly and that there is no incoming or outgoing flow during these SysEx transfers.

I tried MIDI, SERIAL + MIDI, SERIAL + MIDI + AUDIO .... without change and other options like Fast, Faster, Faster With LTO link etc ...

I do not use hardware interrupts other than normal SPI, I2C, Serial and USB).

Code:

static uint8_t dataSys[_SIZEBUFMID];
static uint16_t maxSys=0;
...  
void dataSysSend()
{
  switch( sysDest )
  {
  case _USBMIDI:
      usbMIDI.sendSysEx(maxSys, dataSys, true);
      break;
  case _RIVERKEYMIDI:
      MID_IN.sendSysEx(maxSys, dataSys, true);
      break;
  case _BOARDMIDI:
      MID_OUT.sendSysEx(maxSys, dataSys, true);
      break;
  }
  maxSys=0;
  delay(40);
}

void sysSYS(uint8_t value)
{
  dataSys[maxSys++] = value;
  if (value == 0xF7 ) dataSysSend();
}


void sysCC(uint8_t value)
{
  static uint8_t data;
  data = ( value & 0x007F );
  dataSys[maxSys++] = data;
}


void sysNRPN(int16_t value)
{
  static uint8_t data[2];
  data[0]  = ( ((uint16_t)(value) & 0x3F80) >> 7 ) & 0x007F;
  data[1]  = ((uint16_t)(value) & 0x007F);
  dataSys[maxSys++] = data[0];
  dataSys[maxSys++] = data[1];
}


#define NB_PATCH 200

void SendAllPatch(...)
{
	for (int ct = 0; ct < NB_PATCH ; ct++)
	{
		sysSYS(0xF0);
		...
		sysCC(bytes_1);
		...
		sysCC(bytes_300);
		...
		sysSYS(0xF7); // With F7 , sysSYS call dataSysSend...

	}
}

I tried to slow down the USB, to put delays in usb_midi_write_packed in usb_midi.c,

to change on
#define TX_TIMEOUT_MSEC 40 by testing the values from 10 to 200 ...

to change on
#define TX_PACKET_LIMIT 6 by testing the values from 1 (to avoid bufferization) to 20 ...

to change on
**#define MIDI_INTERFACE 2 // MIDI
**#define MIDI_NUM_CABLES 1
**#define MIDI_TX_ENDPOINT 4
**#define MIDI_TX_SIZE 64
**#define MIDI_RX_ENDPOINT 5
**#define MIDI_RX_SIZE 64
*in usb_desc.h
*
to create a function usb_midi_write_packed without queue ...
*
all without success.

That's it, hoping to find a solution other than getting back through a MIDI so slow

Best regards,

Laurent

defragster · Feb 14, 2018

Can you edit post with CODE "#" - hashtag symbol on the edit toolbar to make code more readable?

Looking at this - assume it is the BULK send - perhaps put a delay(10) [more or less to test] between send requests? not sure if that was tried?

Code:

#define NB_PATCH 200

void SendAllPatch(...)
{
  for (int ct = 0; ct < NB_PATCH ; ct++)
  {
    sysSYS(0xF0);
    ...
    sysCC(bytes_1);
    ...
    sysCC(bytes_300);
    ...
    sysSYS(0xF7);

    dataSysSend();

[B]    delay(10); // was this tried here?[/B]
  }
}

Baloran · Feb 14, 2018

defragster said:
Can you edit post with CODE "#" - hashtag symbol on the edit toolbar to make code more readable?

Thanks, CODE insert

delay is at the end of dataSysSend . I have tried 10, 20 .... 200 ms

defragster · Feb 14, 2018

Much better reading

: indeed - there is that delay()!

oddson · Feb 14, 2018

Have you checked the size of value in sysCC by printing each to the serial monitor?

It would seem to me if they are the correct size here and not after sending then a bug in the library is a possible (perhaps likely) cause.

If not then the bug is in the code you're not showing us.

Baloran · Feb 14, 2018

oddson said:
Have you checked the size of value in sysCC by printing each to the serial monitor?

Yes, like this...303 , 83 , 331, 291 and 11 are acceptable. All other are printing. All bloc data pass to usbMIDI.sendSysEx are valid.

For testing, I also disabled the interrupts of the 3 serials used during transmission. It does not change anything.

Code:

void dataSysSend()
{

#ifdef _DEBUG_SERIAL_
  if ( maxSys != 303 && maxSys != 83 && maxSys != 331 && maxSys != 291 && maxSys != 11 )
  {
      Serial.print(F("dataSysSend"));
      Serial.print(F("-"));
      Serial.println(maxSys);
          
  }
#endif
  
  
  while( Serial1.available() ) Serial1.read();
  Serial1.flush();
  NVIC_DISABLE_IRQ(IRQ_UART0_STATUS);

  while( Serial2.available() ) Serial2.read();
  Serial2.flush();
  NVIC_DISABLE_IRQ(IRQ_UART1_STATUS);
  
  while( Serial3.available() ) Serial3.read();
  Serial3.flush();
  NVIC_DISABLE_IRQ(IRQ_UART2_STATUS); 
     
  switch( sysDest )
  {
  case _USBMIDI:
      usbMIDI.sendSysEx(maxSys, dataSys, true);
      break;
  case _RIVERKEYMIDI:
      MID_IN.sendSysEx(maxSys, dataSys, true);
      break;
  case _BOARDMIDI:
      MID_OUT.sendSysEx(maxSys, dataSys, true);
      break;
  }
  
  maxSys=0;
  delay(40);

  NVIC_ENABLE_IRQ(IRQ_UART0_STATUS);
  NVIC_ENABLE_IRQ(IRQ_UART1_STATUS);
  NVIC_ENABLE_IRQ(IRQ_UART2_STATUS);
 
}

It is important to note that the same processing with the same data will give different errors. These are not the same blocks that are always in error. So the errors do not seem to be related to the content of the blocks.
And this random error, but still 3 bytes, whatever the size of the block, crisps me ....

oddson · Feb 14, 2018

Have you tried increasing the MIDI limit (line 44) in usb_midi.h ?

If it's a bug it's likely limited to those blocks it has to break up. So you might be able to avoid it by increasing the limit.

Also, if that works it should go a long way to confirming a bug and pointing to where it might be.

Baloran · Feb 14, 2018

Well, I'm on the track ...

In usb_dev.c, procedure

Code:

void usb_isr (void)
{
....
if ((status & USB_ISTAT_SOFTOK / * 04 * /)) {
....
#ifdef MIDI_INTERFACE
                       usb_midi_flush_output ();
#endif
....

This usb_midi_flush_output call can occur during the execution of usb_midi_send_sysex_buffer_has_term
and puts the chronoligy of tx_buffer in cabbages ...

If I comment on this line,

#ifdef MIDI_INTERFACE
// usb_midi_flush_output ();
#endif

All my blocks arrive in perfect condition !!!!

Now, I would like to understand what this USB_ISTAT_SOFTOK and the repercussion of the disable

PS: thanks Oddson & defragster for you reply

Baloran · Feb 14, 2018

Finally, the midiUSB flush line in void usb_isr (void) is important because it seems to autoflush unreleased outgoing data.

So I modified usb_midi.c to add a flag to block flush when transmit SysEx...

Code:

...
static uint8_t transmit_previous_timeout=0;
static uint8_t tx_noautoflush=0;
/* flag anti flush */
static uint8_t stopFlush = 0;

void usb_midi_send_sysex_buffer_has_term(const uint8_t *data, uint32_t length, uint8_t cable)
{
		stopFlush = 1;
		
		cable = (cable & 0x0F) << 4;
...

		stopFlush = 0;

}

void usb_midi_send_sysex_add_term_bytes(const uint8_t *data, uint32_t length, uint8_t cable)
{
	stopFlush = 1;
	
	cable = (cable & 0x0F) << 4;

....
	stopFlush = 0;
	
}

void usb_midi_flush_output(void)
{
	if ( stopFlush == 0 ) 
	{
		if (tx_noautoflush == 0 && tx_packet && tx_packet->index > 0) {
			tx_packet->len = tx_packet->index * 4;
			usb_tx(MIDI_TX_ENDPOINT, tx_packet);
			tx_packet = usb_malloc();
		}
	}
}

oddson · Feb 14, 2018

I wonder if calling void usb_midi_flush_output(void) right before sending the block could prevent it from flushing on it's own in the middle of things without changing libraries?

Also, the three-byte thing seems strange. You'd think it would have a more significant effect on the received block and certainly wouldn't have thought it would always be three bytes... I guess that's how long the interruption stops Teensy from following the incoming stream??

Baloran · Feb 14, 2018

oddson said:
if calling void usb_midi_flush_output(void) right before sending the block could prevent it from flushing

Tested and no. If block of datas was shorter, possible but yet not tested

3 bytes are the size of data queuing in a midi over usb message 0x04 or 0x07, this 3 bytes have gived me the direction to find the problem.
Now, all works fine, 100% of bloc are safe recept

Problem with large volumes of data on usbMidi

Baloran

Active member

defragster

Senior Member+

Baloran

Active member

defragster

Senior Member+

oddson

Well-known member

Baloran

Active member

oddson

Well-known member

Baloran

Active member

Baloran

Active member

oddson

Well-known member

Baloran

Active member