Teensy 4.1 halts during USB serial transmission

J

jkej

Member
I am running a relatively fast control loop on a Teensy 4.1 which is controlling 2 BLDC motors based on some sensor inputs. The cycle time of the control loop is about 600 microseconds. I have a monitoring mode where the Teensy sends some data to the connected computer over the USB serial every loop cycle (less than 100 bytes). This data is received by a Python program that plots it (similar to the Serial Plotter in the Arduino IDE, but with a little bit more control).

This is mostly working fine, but in certain situations I experience an issue where the plot and the motor control is repeatedly halting for short periods (sub-second duration). This is quite repeatable, but it only happens when in monitoring mode. The only thing the Teensy does differently in the monitoring mode is sending the data over Serial, and there should be plenty of time for this (the Teensy is not particularly busy during most of the loop cycle, it's just waiting for interrupts from sensors). I'm not so surprised that the Python program is struggling with plotting all this data (I have needed to optimize that code a bit get it to work decently), but I wasn't expecting that to spill over to the Teensy control loop. So I'm guessing that because the Python program is busy with graphics, it slips behind on reading the Serial data, and because of this the Teensy somehow gets stuck trying to send data. That's the most reasonable explanation I can think of, but I don't know enough about serial communication to tell if this is actually possible.

So my question is if the explanation I outlined above seems plausible, and if so, what can I do to solve the issue? The monitoring mode is important, but it's acceptable to have these occasional short halts in plotting or missing monitoring data. What I really want to avoid is that it affects the performance of the Teensy's main task (the control loop). If it's something like a buffer that gets full, maybe my Teensy code could just check for that and skip sending data whenever the buffer is full?

I have run the Python program on both Linux and Windows, but only had the problems on Windows. This could be because the Linux computer is more powerful or because I have used it less. The serial data sent consists of 4 long integer values sent as text using a Serial.printf like this:

Code: 
Serial.printf("data %d %d %d %d\n", value1, value2, value3, value4);
 
First guessing: Maybe the USB Serial is having to pause as the PC/Python code is not keeping up with the data being sent and there are no more free USB buffers (or specifically it has allocated the max number of buffers assigned to Serial...).

When the buffers are full. Output to Serial, like your printf will not return until there is sufficient room available to queue the rest of the data.
Which potentially is the pause you are seeing.

You can try to limit this happening, by not writing the data to Serial, if there is not enough room available.
Code: 
if (Serail.availableForWrite() > N) Serial.printf(...)
 
Indeed the slowness of Python has been an issue many times when sending large amounts of data to a PC.

Definitely try availableForWrite() to check whether the USB buffers on the Teensy side of the USB cable have enough room for the data you wish to transmit.
 
Okay, so Serial.availableForWrite sounds promising. Thank you both for the quick and helpful response! I will do some tests with this on Monday. So I checked the code and it seems that a uint8_t is used in availableForWrite to calculate the amount of buffer space available. Does this mean that the total buffer size is 255 bytes or less? If so, maybe requiring 100 bytes to be available before writing might fail quite often? Of course 100 bytes was a very rough estimate for what I'm sending. 52 bytes is probably the real worst case scenario assuming 32 bit signed integers. And I could improve a lot on that by switching to send binary values instead of text. I guess I could also implement my own buffer with a larger size instead of just throwing away data when the buffer is full (assuming the Python program can catch up in the long run).

For some reason I had always thought that if your code tries to send serial data and nothing is there receive it, it just gets lost. Maybe that is true when nothing is connected to the serial port at all? Or is it different for different implementations? Anyway, it's great to learn how these things really work.
 
PaulStoffregen said:
You're looking at the wrong file. Look at the comment on line 31.

Code: 
// This file is used only for Teensy 2.0 and Teensy++ 2.0

This file is the code used on Teensy 4.1.

https://github.com/PaulStoffregen/cores/blob/master/teensy4/usb_serial.c
Click to expand...

Oops! I should have checked more carefully that I was looking at the right file. So the buffer could be much bigger. I found some useful information in this thread, but it's for Teensy 3.2. Looking at this file, it seems NUM_USB_BUFFERS is 12 by default for Teensy 4.1 too, so the total buffer size would be 768 bytes (although it's also shared by read and write). I guess I could also experiment with increasing NUM_USB_BUFFERS. Is 31 still the maximum value for Teensy 4.1?

I read about your very interesting tests of USB speeds that you linked to in the other thread. There seems to be a huge speed loss for sending small packets from a Windows computer. Could it be similar when sending to a Windows computer? Maybe I should have my own large buffer and in each cycle send as much of it as can fit in the result of availableForWrite()?
 
Note: with T4.x typically will run at USB High speed(480m) and the transfer size is 512 bytes versus T3.x which run at full speed(12M) and the transfer size is 64 bytes.

The USB Serial code is setup, to try to pack as much as it can into a buffer and only sends the data when either the buffer is full or there is a timout between characters, something like 1 or 2 ms? or if you do an operation like Serial.flush() which forces it.

As for increasing the numbers buffers there or not. Been awhile since I looked to see what would take. Sometimes that can help or not. That is if you are consistently trying to send more data than the host can process, sooner or later you may run into this issue you are seeing. (Or you could choose to toss the data as an alternative).

Now if it is simply there are times when you are generating the data faster than the host can process, but over time the host will catch up, then adding buffers can help.

For hardware Serial ports (Serial1, Serial2...) we added a method that allows a sketch to add an additional buffer to the default system one to hold the ouput (likewise input).
I don't believe that was ever done for the USB Serial code.

In the past I have setup my own simple external circular buffers or FIFO queue. Where I add all of the generated data to the buffer and each time through the main loop, I would check if y queue had any data in it and output however much I could of this to the Serial.write (or hardware serial back then)...
You can roll your own or others have made libraries like: https://github.com/tonton81/Circular_Buffer
to do stuff like this.

Good luck
 
KurtE said:
Note: with T4.x typically will run at USB High speed(480m) and the transfer size is 512 bytes versus T3.x which run at full speed(12M) and the transfer size is 64 bytes.

The USB Serial code is setup, to try to pack as much as it can into a buffer and only sends the data when either the buffer is full or there is a timout between characters, something like 1 or 2 ms? or if you do an operation like Serial.flush() which forces it.

As for increasing the numbers buffers there or not. Been awhile since I looked to see what would take. Sometimes that can help or not. That is if you are consistently trying to send more data than the host can process, sooner or later you may run into this issue you are seeing. (Or you could choose to toss the data as an alternative).

Now if it is simply there are times when you are generating the data faster than the host can process, but over time the host will catch up, then adding buffers can help.

For hardware Serial ports (Serial1, Serial2...) we added a method that allows a sketch to add an additional buffer to the default system one to hold the ouput (likewise input).
I don't believe that was ever done for the USB Serial code.

In the past I have setup my own simple external circular buffers or FIFO queue. Where I add all of the generated data to the buffer and each time through the main loop, I would check if y queue had any data in it and output however much I could of this to the Serial.write (or hardware serial back then)...
You can roll your own or others have made libraries like: https://github.com/tonton81/Circular_Buffer
to do stuff like this.

Good luck
Click to expand...

Ah okay, so the total buffer size should be 12*512=6144 bytes rather 12*64=768 bytes for Teensy 4.1? That's quite a lot bigger.

Yes, the big question is whether the Python program has trouble keeping up temporarily or in the long run. I will have to do some tests to figure that out. If it's a temporary thing some type of extra buffering could help, but otherwise I will have to skip some data, or find some way to optimize the Python program.
 
BriComp said:
Would it be possible to do away with python? Look at this for some Windows Comms routines/practices.
Click to expand...

Yes, porting the program to C/C++ is something I might do eventually, but I don't really have the time for that now. The serial communication wouldn't take that much time, but the GUI and the plotting would take a bit of time to redo. Other parts of the project have higher priority right now.

I tested the availableForWrite() solution just now, and it seems to work as intended. :D I still lose some data in the plot, but the Teensy doesn't halt anymore. I might experiment with buffers and/or other optimizations to avoid losing data later on, but for now this solution is probably good enough.

Thanks again, everyone!
 
You must log in or register to reply here.
Back
Top