Losing data with bi-directional USB Serial communication (Teensy 3.1)

Status
Not open for further replies.
C

codingisfun

Member
I have run into the problem that when you use USB Serial as a bi-directional device, Teensy 3.1 tends to lose some data. I was wondering if anybody else has run into this problem and would have some advice on how to solve the problem.

Concretely, I am trying to use a Teensy 3.1 as an interface to my PC, where the Teensy monitors sensors and other microcontrollers, sending a lot of data to the PC to store, visualize and process it there, and receiving a lot of stored data from the PC to feed it to the microcontrollers. The maximum capacity I need is 588 kBytes/sec from the PC to Teensy, and 192 kBytes/sec from Teensy to the PC (in parallel). I can easily achieve either, but when I try to do this in parallel, data is lost going from Teensy to the PC. I have brought it down to a very simple sketch illustrating the issue:

Code: 
const int insize = 3*192;
const int outsize = 4*48;

void setup()
{
  char inbuf[insize];
  char outbuf[outsize];
  int i;
  long sent, received;
  
  Serial.begin(9600);
  while (!Serial) ;
  sent = 0; received = 0;
  for (i = 0; i < 32000; i++) {
    received = Serial.readBytes(inbuf, insize);
    received += rec;
    Serial.write(outbuf, outsize);
    Serial.send_now();
    sent += outsize;
  }
}

void loop()
{
}

And on the PC side, using Processing 2.0:
Code: 
import processing.serial.*;

Serial port;
i
void setup()
{
  int i;
  byte[] inbuf = new byte[4*48];
  byte[] outbuf = new byte[3*192];
  long t0, t1;
  int rec, received, sent;
  
  port = new Serial(this, "COM27", 9600);
  received = 0; sent = 0;
  t0 = millis();
  for (i = 0; i < 32000; i++) {
    port.write(outbuf);
    sent += outbuf.length;
    if (port.available() > 0) {
      received = port.readBytes(inbuf);
    } else {
      println("missed reading data at block "+i);
    }
    if (i % 1000 == 0) {
      t1 = millis();
      println("sent "+sent+", received "+received+" in "+(t1-t0)+" ms");
    }
  }
  t1 = millis();
  println(""+sent+" bytes sent, "+received+" bytes received in "+(t1-t0)+" ms");
}

void draw()
{
}

When I run this, I find that I lose some data going from Teensy to the PC.

Checking out the source code of the Teensy USB modules, I found that Teensy drops data going out, either if there is no space in the buffers anymore, or if the PC does not react for some time. Without having understood all the details of how things work, and understanding the design trade-off that you cannot buffer a lot of data on Teensy (and have to go for either making the application wait or dropping data if the buffers get full), I still would like to get this to work. Note that I have broken down the data into packets that fit the 1kHz rate of USB (i.e. one read and one write per ms), and that they also fit the 64-byte USB-packet size used in the Teensy USB modules, with 192 bytes being 3 such packets).

Any suggestions on what might work? I have been considering the following:
- Go for 64 byte packets (i.e. exactly what the Teensy USB modules use) - just breaking down the current "readBytes" and "write" calls did not make a difference (as expected), but maybe a different ordering of reads and writes might
- Try to increase the number of buffers used in the Teensy USB modules (if I understood it correctly, currently there are 12 buffers of 64 bytes each, which is exactly what I would fill up per ms - I could understand that you can lose data in that case, even though I do not understand how the benchmarks figures of about 1 MByte/sec transfer rates are achieved if you can only store 12*64=768 bytes per ms in the buffers); if possible, I would try to avoid changing anything in the core modules, but if that is the recommended way, I would try that
- Try to implement something using the RAW_HID interface (and thus control exactly what is transferred in every ms) - what keeps me from doing that is the issue of implementing a RAW_HID interface on the host side in Processing.

Any ideas or hints? Thanks a lot for your help!
Cheers,
CodingIsFun
 
if you break the larger data into 588 (reasonably) equal packets and then PC sends 1 byte of data to Teensy and Teensy responds with 192K/588 bytes back and when PC has those it sends the 1 byte 'request' again - PC remains in charge of the conversation at all times, conversation is reciprocal so chances of missing something is very much diminished and I bet it does not take the whole second to transfer this.

If the 588 bytes of data from PC to Teensy is a bundle of messages to go out to the other devices then the 192K of data is probably the responses of those devices - if this was my caper then each of those shorter messages (in the 588 byte bundle) would be sent to Teensy individually as 'requests' with the appropriate response coming back, correctly marked and formatted, reason to send the next request and inappropriate response reason to check status of port, force close it if possible, re-check availability of port and re-open if possible; if open send the first request again...
 
You loop 32000 time on both host and Teensy, the times taken for these loops, have you any idea if they are the same?

Do you know how much data that was actually received by the Teensy, and then how much data the Teensy wrote to its transmissions buffers?

So perhaps what you see is the difference in transmit and receive bandwith when you flood the USB with data in both directions, no lost data just different amount of data

To test your scenario 588kB/s in one direction and 192kB/s in the other you must somehow limit the rates you try to send data.
 
I'll apologise I didn't read your post well enough to go ahead and try a reply; PC side you have far greater speed and very likely much bigger buffers, Teensy side it isn't wise to rely very much on the buffers at all - as part of my apology I wrote you the following for Teensy side; if you make your PC side code do a similar thing (only it needs to send first byte without being prompted by Teensy side) to this code and then add a one second timer to that code and byte counter (just one needed because it is byte for byte) you can find out how much you can transfer 1 for 1 this way.
Code: 
const int insize = 3*192;
const int outsize = 4*48;

volatile int inPtr=0,outPtr=0;
volatile char inbuf[insize],outbuf[outsize];

void setup()
{
  Serial.begin(Serial.baud()); // I can't stand writing values that won't be obeyed.
  while (!Serial) ;
}

void loop()
{
  if(Serial.available()) 
  {
    inbuf[inPtr++]=Serial.read();
    if(inPtr>insize-1) inPtr=0;
    Serial.write(outbuf[outPtr++]);
    if(outPtr>outsize-1) outPtr=0;
  }
}
 
Thanks, and clarifications ....

robsoles said:
I'll apologise I didn't read your post well enough to go ahead and try a reply; PC side you have far greater speed and very likely much bigger buffers, Teensy side it isn't wise to rely very much on the buffers at all - as part of my apology I wrote you the following for Teensy side; if you make your PC side code do a similar thing (only it needs to send first byte without being prompted by Teensy side) to this code and then add a one second timer to that code and byte counter (just one needed because it is byte for byte) you can find out how much you can transfer 1 for 1 this way.
Code: 
const int insize = 3*192;
const int outsize = 4*48;

volatile int inPtr=0,outPtr=0;
volatile char inbuf[insize],outbuf[outsize];

void setup()
{
  Serial.begin(Serial.baud()); // I can't stand writing values that won't be obeyed.
  while (!Serial) ;
}

void loop()
{
  if(Serial.available()) 
  {
    inbuf[inPtr++]=Serial.read();
    if(inPtr>insize-1) inPtr=0;
    Serial.write(outbuf[outPtr++]);
    if(outPtr>outsize-1) outPtr=0;
  }
}
Click to expand...

Thanks for your replies, robsoles and mlu. Let me clarify a couple of things: This is not so much a "request-reply" model, but rather a "streaming in both directions" model: The PC sends (a lot of) data to the Teensy which then distributes it to other microcontrollers (which is the stream PC->Teensy, where I should reach 588 kBytes/sec), and Teensy collects somewhat less data from controllers and sensors, and sends that to the PC (the 192 kBytes/sec stream Teensy->PC). I have condensed down the problem to bare minimum code to spare you with all the details from all the other components in the system.

There are a couple of benchmarks on the maximum speed Teensy 3.1 can do over USB Serial (see https://www.pjrc.com/teensy/benchmark_usb_serial_receive.html) and there is shows a rate of 1'019'395 bytes/sec for "readBytes" - well above what I am trying to transfer. In the context of this system, I have actually sent 384 kBytes/sec Teensy->PC (i.e. with "write") without losing any data, and I have received the 588 kBytes/sec PC->Teensy ("readBytes") as well without any problem. The problems start once I try to use both at once. From looking at the source code for the USB modules on Teensy (concretely, usb_serial.c, lines 145ff and 166ff), I saw some comments that usb_serial drops data going out if it takes too long. The rationale there makes a lot of sense: When the buffers run full, you can either keep the program waiting until the recipient has received the data and the buffers become empty, or you can drop data (which is what Teensy usb_serial does). In usb_desc.h, I also saw that it uses 12 buffers of 64 bytes each, which is just how much I would like to transfer (12*64=768 bytes, and USB transfers data once every 1ms, so you can fill and empty the buffers ideally with 768 kBytes/sec).

I do know that I lose data from the Teensy going to the PC (I have omitted that code). I have not observed losing data in the other direction.

My question is whether just increasing the number of buffers would help, and if it is as simple as changing the constant NUM_USB_BUFFERS in usb_desc.h (my application could afford using a few kBytes of RAM space for this purpose). But that would only be my second best options, as I would try to avoid changing core modules whenever I can. So maybe somebody has ran into the same problem and has found a way to solve it without changing core modules.

I have come up with another idea to solve the problem, with other downsides: Using a second Teensy, and having the first one handle the PC->Teensy stream, and the second the Teensy->PC stream (would be possible from the system setup). Besides having to use two Teensys, one downside would be that I need two USB ports on the PC (I avoid USB hubs for connecting Teensys to the PC - as recommended) and that I cannot be sure that it all works correctly on the PC side (i.e. having multiple serial ports over USB). Does anybody have any experience with multiple Teensys connected to one PC through USB-Serial?

Thanks again for any help and pointers!
CodingIsFun
 
Perhaps you're running into a scenario where the USB stack is running out of packet buffers?

Try editing usb_desc.h, to increase NUM_USB_BUFFERS. There's several copies of NUM_USB_BUFFERS, so make sure you're editing the one corresponding to the Tools > USB Type you're using. When you increase the buffers, you'll see Arduino reports more "global variables" memory used.

Try allocating a LOT of buffers, like 200, just to see if that magically makes the problem go away.... or makes it become something different.
 
PaulStoffregen said:
Perhaps you're running into a scenario where the USB stack is running out of packet buffers?

Try editing usb_desc.h, to increase NUM_USB_BUFFERS. There's several copies of NUM_USB_BUFFERS, so make sure you're editing the one corresponding to the Tools > USB Type you're using. When you increase the buffers, you'll see Arduino reports more "global variables" memory used.

Try allocating a LOT of buffers, like 200, just to see if that magically makes the problem go away.... or makes it become something different.
Click to expand...

Thanks, Paul! I tried with 64 buffers, and the problem is largely solved. What I observe now is the following (not unexpected):
- Some data gets delayed by 1ms or so
- No data is lost, in either direction
- The whole matter seems to be slightly slower (but I need to look into that more).

After making this changes in my software, some other problems popped up that I need to look into, but I wanted to let you know quickly that this helped. Thanks a lot!
 
If you're willing to do some more fiddling, perhaps also try tweaking TX_PACKET_LIMIT in usb_serial.c. Without increasing this, all those new buffers will only get used for receiving. Using more for transmitting might be helpful, if it allows your program to return more quickly to reading data from the many buffers that are holding the data your PC has recently sent.

Maybe the default settings should be reconsidered? Obviously I don't want a default value to allocate a lot more memory than hardly anybody needs. But a little more, or a better balance on those 2 settings might be worth considering for future releases. If you gain any insight by lots of fiddling and testing, please let me know?
 
PaulStoffregen said:
If you're willing to do some more fiddling, perhaps also try tweaking TX_PACKET_LIMIT in usb_serial.c. Without increasing this, all those new buffers will only get used for receiving. Using more for transmitting might be helpful, if it allows your program to return more quickly to reading data from the many buffers that are holding the data your PC has recently sent.

Maybe the default settings should be reconsidered? Obviously I don't want a default value to allocate a lot more memory than hardly anybody needs. But a little more, or a better balance on those 2 settings might be worth considering for future releases. If you gain any insight by lots of fiddling and testing, please let me know?
Click to expand...

Thanks, Paul. I will have a chance to do this over the next weeks, and I will get back with what my experiences are.
 
Status
Not open for further replies.
Back
Top