Teensy 3.2: Drops/corruptions while receiving data via usb_serial under load

Status
Not open for further replies.
S

svobodb

New member
Hello, I'm working on streaming data from PC to WS2811/2/3 strips/strings (Neopixels). Because of strict WS2811-3 timing requirements, Teensy 3.2 was chosen to distribute data from PC to the strips because of the excellent OctoWS2811 board with library and Teensy's USB full speed serial throughput. However I have experienced very often PC->Teensy data corruptions when a larger amount of data is transferred over USB serial.

I have narrowed the problem down to scenario, where PC is sending data to its serial port and Teensy is receiving the data by usb_serial functions and verifying it (so OctoWS2811 does not have anything with this issue). When the Teensy is receiving the data without any processing (without the delay call in the code below), everything works fine. I can see 800 kbps with 512 byte chunks (see the COUNT macro below) or even 1000 kBps with 4096 byte chunks sent to the Teensy. But when I simulate some data processing (e.g. leds.show()) by the delay call in the Teensy code below, Teensy gets corrupted data within a few seconds (i.e. after 30 kB, 60kB or 1MB of data transferred). I can see that the data is corrupted, because when the data verification fails, Teensy is instructed to blink slowly forever.

Note that I'm expecting the Teensy to regulate the incoming serial data throughput from PC via USB Flow Control, which is dedicated to such function. Also, the USB Flow Control already works to some extent because when data verification is disabled and the delay below enabled, I can see stable throughput of 160 kBps. From the big picture, I intend to use the USB Flow Control to automatically maximize the refresh rate of the WS2811-3 strips, i.e. generate and send the data from PC as fast as Teensy is able to receive and process it.

I was searching through this forum whether somebody else have already faced this issue, but have found only problems with Teensy transmit drops. That one could have been fixed by removing timeouting code from usb_serial.c, but that's not the case of Teensy USB serial receive drops.

Teensy code:
Code: 
#define BUSYPIN 13                //This pin will be high when busy

static void blink(int ms)
{
  for(;;){digitalWrite(BUSYPIN, HIGH);delay(ms);digitalWrite(BUSYPIN, LOW);delay(ms);} 
}

static int serial_read(void* dst, int size)
{
  while(usb_serial_available()<size);
  return usb_serial_read(dst, size);
}

void setup()
{
  pinMode(BUSYPIN, OUTPUT);
}

void loop()
{
  uint32_t count;
  serial_read(&count, sizeof(count));
  digitalWrite(BUSYPIN, HIGH);

  for(uint32_t i=0;i<count;i++)
  {
    int a;while((a=usb_serial_getchar())==-1);  //Wait until a byte is ready
    if(a!=(uint8_t)i)blink(400);  //Blink forever when unexpected data received - with the delay below, this happens within a second
  }
  digitalWrite(BUSYPIN, LOW);
  delay(3); //Comment this out to get working setup (seems that Teensy is able to consume the data faster that USB full speed is passing it through)
}

Generator code (VC++):
Code: 
#ifdef _WIN32
#include <io.h>
#else
#include <unistd.h>
#define O_BINARY 0
#endif
#include <fcntl.h>
#include <stdio.h>
#include <time.h>

#define COUNT	512

int main()
{
	unsigned char buf[sizeof(long)+COUNT];
	*(long*)buf=COUNT;
	for(size_t i=0;i<COUNT;i++)buf[sizeof(long)+i]=(unsigned char)i;	//This data will be verified by Teensy

	int fd=open("\\\\.\\COM25", O_RDWR|O_BINARY);
	if(fd==-1){perror("port open failed");return 1;}
	size_t total=0;
	for(time_t lastt=time(NULL), t;;)
	{
		total+=write(fd, buf, sizeof(buf));
		t=time(NULL);
		if(t!=lastt){printf("%d Bps\n", total);total=0;lastt=t;}
	}
	close(fd);
	return 0;
}

I have spent a lot of time investigating this and I don't know what else I can try besides modifying Teensy core library sources. Maybe someone with more background will have the answer right out of the box based on my description.

This whole issue looks like some USB packets are just dropped on seemingly random basis. From my software-experienced developer point of view, this behavior may point to some race conditions when pulling bytes from full USB buffers.

Tested on:
- multiple pieces of Teensy 3.2 at 96, 72, 48 MHz
- Teensyduino 1.44, Arduino 1.8.7
- Windows 7, Visual C 6 (the code works in up to VC 2017 and also under Linux)
 
I was digging a little deeper. After some intermediate tests, I used Serial1 and USB-serial converter for passing debugging output instead of just Teensy's LED blinking, changed the payload length to 64 bytes (COUNT=60 in the generator code above, prepended with 4 byte length) to match the USB full speed packet payload, filled the generated data with the packet order converted to byte (i.e. the first USB packet contains all 0x00s, the second USB packet all 0x01s, etc), verified accordingly on Teensy.

Here's what I got from Teensy on Serial1 output after a few seconds (and hundreds to thousands of correctly received USB packets) of a test run:
Code: 
Started.
Payload offset [0]: expected DD, received DF
- rest of the data: DF DF DF DF DF DF DF DF DF DF DF DF DF DF DF
DF DF DF DF DF DF DF DF DF DF DF DF DF DF DF DF
DF DF DF DF DF DF DF DF DF DF DF DF DF DF DF DF
DF DF DF DF DF DF DF DF DF DF DF DF
- following data:
3C 0 0 0 DE DE DE DE DE DE DE DE DE DE DE DE
DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE
DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE
DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE DE
3C 0 0 0 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4
D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4
D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4
D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4 D4
3C 0 0 0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0
E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0
E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0
E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0 E0
3C 0 0 0 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1
E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1
E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1
E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1 E1
The absolute hex values are different each test run but the relative pattern is repeated every time. I.e. instead of expected packets no. i+0, i+1, i+2, Teensy returns packets no. i+2, i+1, i-9. Range of i-9 to i+2 suspiciously reminds NUM_USB_BUFFERS=12.

I have confirmed that 9 in i-9 is related to NUM_USB_BUFFERS. When I changed it from 12 to 14 in "c:\Program Files (x86)\Arduino\hardware\teensy\avr\cores\teensy3\usb_desc.h", the next test returned data from packets no. i+2, i+1, i-11 instead of the expected packets no. i+0, i+1, i+2.

This means that from time to time, possibly when the teensy's USB serial buffers are full and some race condition occurs, usb_serial functions return data from USB packets no. i+2, i+1, i-(NUM_USB_BUFFERS-3) instead of the expected packets no. i+0, i+1, i+2.

Hope that the description is sufficient. I can share the updated sources for this test if needed (I have chosen not to to prevent TL;DR effect).

Looking forward to hear from anyone with more Teensy-core background than me.

Tested with:
- Arduino/Tools/USB type "Serial" (hence NUM_USB_BUFFERS=12)
 
Status
Not open for further replies.
Back
Top