Teensy 4.0 Serial bandwidth issue

[Sytse Reitsma]

Hi,

I'm having some issues with sending data with 1ms interval over the Serial (USB) port. Every so often the buffer appears to overflow and takes significant time to recover.
The code below reproduces the issue:

uint32_t _prevTimeMs;

void setup() {
  _prevTimeMs = 0;
}

void loop() {
  auto nowMs = millis();
  if (nowMs - _prevTimeMs >= 1)
  {
    _prevTimeMs = nowMs;

    Serial.print(micros());
    Serial.print(',');
    Serial.print(Serial.availableForWrite());
    Serial.print(',');
    Serial.println("Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec.");
    Serial.send_now();
  }
}

It will happily send all the data until all of a sudden Serial.availableForWrite() suddenly plummets and will take a significant time to recover:

Timestamp (us), availableForWrite, Lorem text
6846000,6144,Lorem ....
...
6898000,6144,Lorem ...
6899000,4096,Lorem ...
6900000,2048,Lorem ...
6901000,0,Lorem...
7024000,6144,Lorem...

This pattern repeats:

I have some control over the interval by buffering the lines and batch sending them (the problem remains though), but I do not really understand why this is happening the way it does. You'd expect the availableForWrite to gradually decrease (with 2048 byte decrements when I look at the teensy code), not plummeting to 0 in 3 steps.

Sending less data per ms iteration (e.g. 90 bytes) also works, but I want/need to send more than that.

Is there any way to resolve this, or work around this?

Regards,

Sytse

 

[KurtE]

Maybe it depends on what you are connecting to, and how fast they can process the packets..

Note: USB does not work like Hardware Serial ports. It does not send one byte at a time, but instead sends them as USB packets. Assuming your host can handle USB HS (high speed), the packets are typically 512 bytes and the USB is setup with 4 2048 byte buffers. Each time you do a Serial.write type call it will stuff the new data into the current buffer until full or until timeout interval happens, at which point it will queue the buff er up output.

Your calls to send_now() or flush(), it sort of bypasses all of this and forces the USB to queue up the current 2K buffer to be sent with how many actual bytes are in it, and once all 4 of these groups are queued up, you have to wait until at least one of them has been sent, and the buffer returned back to the free pool.

 

[PaulStoffregen]

Just to mostly echo what Kurt said, we've seen this sort of problem many times. It's almost always caused by software on the PC side running too slowly to keep up with the incoming data. Inefficient Python scripts are the most common in recent years.

 

[Sytse Reitsma]

Thanks. I could have sworn I posted a reply to Kurts message, but apparently I never pressed post.

I gathered as much. For now I was able to work around the problem by compressing the data (reduced the payload size to below 90 bytes). By buffering the data for batch sending I was able to reduce the number of times the problem occurred, but that was not acceptable for my use case. What worries me is that this problem might be worse on other systems, so I'll need to investigate a bit more.

We're running a C# application which monitors incoming data in a separate, high priority thread, can't make that much faster than it presently is (although I'm unsure how much overhead .NET incurs). Could it be windows itself not reading the buffers frequently enough? In any case it is likely beyond what I can fix in within a reasonable effort.

 

[kd5rxt-mark]

We're running a C# application which monitors incoming data in a separate, high priority thread, can't make that much faster than it presently is (although I'm unsure how much overhead .NET incurs). Could it be windows itself not reading the buffers frequently enough? In any case it is likely beyond what I can fix in within a reasonable effort.

Maybe try a simple test as follows:

- have the T4 to send a rolling pattern (example below)
- have the PC to read the incoming data & flag anything that does not match the expected pattern
- keep statistics on good patterns vs bad patterns
- keep the timing data as you have already done
- vary the size of the pattern & see what works & what fails
- at a size where it fails fairly consistently, start adding delays between patterns until it works
- see if the speed (with sizes & with delays) is sufficient to support whatever you need to do

Example rolling pattern:

ABCDEFGHIJKLMNOPQRSTUVWXYZ
BCDEFGHIJKLMNOPQRSTUVWXYZA
CDEFGHIJKLMNOPQRSTUVWXYZAB
DEFGHIJKLMNOPQRSTUVWXYZABC
EFGHIJKLMNOPQRSTUVWXYZABCD
FGHIJKLMNOPQRSTUVWXYZABCDE
GHIJKLMNOPQRSTUVWXYZABCDEF
HIJKLMNOPQRSTUVWXYZABCDEFG
IJKLMNOPQRSTUVWXYZABCDEFGH
JKLMNOPQRSTUVWXYZABCDEFGHI
KLMNOPQRSTUVWXYZABCDEFGHIJ
LMNOPQRSTUVWXYZABCDEFGHIJK
MNOPQRSTUVWXYZABCDEFGHIJKL
NOPQRSTUVWXYZABCDEFGHIJKLM
OPQRSTUVWXYZABCDEFGHIJKLMN
PQRSTUVWXYZABCDEFGHIJKLMNO
QRSTUVWXYZABCDEFGHIJKLMNOP
RSTUVWXYZABCDEFGHIJKLMNOPQ
STUVWXYZABCDEFGHIJKLMNOPQR
TUVWXYZABCDEFGHIJKLMNOPQRS
UVWXYZABCDEFGHIJKLMNOPQRST
VWXYZABCDEFGHIJKLMNOPQRSTU
WXYZABCDEFGHIJKLMNOPQRSTUV
XYZABCDEFGHIJKLMNOPQRSTUVW
YZABCDEFGHIJKLMNOPQRSTUVWX
ZABCDEFGHIJKLMNOPQRSTUVWXY

Mark J Culross
KD5RXT