USB transmit buffers

J

jrw

Member
I have numerous applications that store data in files on the SD card. One feature of the program is to print the file to the USB serial interface. "Because of Windows", and in particular "because of usbser.sys", I have to put code on the Teensy that will throttle the output or else Windows cannot keep up. Linux and Mac do not have trouble reading data from Teensy USB serial. To accomplish this I use Serial.availableForWrite() to test and see if space is available in the Teensy USB transmit buffer. This works perfectly for Teensy 4.0 and Teensy 4.1. Unfortunately it does not work on Teensy 3.6. On Teensy 3.6 it returns 64 when it should return 768.

I've been trying to dig through the teensy3 core files for USB to understand how it works. I can't figure it out, as it seems that the registers being used are 8-bit AVR registers, so I'm not sure I'm looking at the right code. For example, cores/usb_serial/usb_api.cpp refers to UENUM and UEBCLX which don't seem to be present on ARM.

Any idea how to proceed? I would offer a pull request to fix this but I have no idea how to begin fixing it.

Below are two different programs to explore this problem. One reports size of the transmit buffer. The other reports the buffer sizes compiled into the code and reports the connection speed.

Thanks,
Jim

C++: 
/*

Teensy 3.6      - Full Speed - 64   - incorrect, should be 768
Teensy 4.1      - Full Speed - 6144 - incorrect, should be 768
Teensy MicroMod - Full Speed - 6144 - incorrect, should be 768

Teensy 4.1      - High Speed - 6144 - correct
Teensy MicroMod - High Speed - 6144 - correct

*/

// wait at most 4 seconds for USB to initialize, in microseconds
#define USBTimeout  4000000UL

void setup(void) {
    unsigned long a;

    pinMode(LED_BUILTIN, OUTPUT);

    // initialize USB serial port
    Serial.begin(115200);
    a = micros();
        while(!Serial) {
        if ((micros() - a) > USBTimeout) {
            break;
        }
    }
    Serial.println();
    Serial.println(";====================================");
}

void loop(void) {
    int avail;

    Serial.flush();
    delay(100UL);
    avail = Serial.availableForWrite();
    Serial.printf("USB transmit buffer has %d bytes available\r\n", avail);
    digitalWrite(LED_BUILTIN, !digitalRead(LED_BUILTIN));
    delay(5000UL);
}


C++: 
// This will be either cores/teensy3/usb_desc.h or cores/teensy4/usb_desc.h
#include <usb_desc.h>

#if defined(__IMXRT1062__)  /* Teensy 4.0 or Teensy 4.1 */
// the variable usb_high_speed is from cores/teensy4/usb.c
extern volatile uint8_t usb_high_speed;
#endif

// wait at most 4 seconds for USB to initialize, in microseconds
#define USBTimeout  4000000UL

void setup(void) {
    unsigned long a;

    pinMode(LED_BUILTIN, OUTPUT);

    // initialize USB serial port
    Serial.begin(115200);
    a = micros();
        while(!Serial) {
        if ((micros() - a) > USBTimeout) {
            break;
        }
    }
    Serial.println();
    Serial.println(";====================================");
}

void loop(void) {
    int bufferSize = -1;

    Serial.println();
#if defined(__MK66FX1M0__)   /* Teensy 3.6 */
    // USB0 in Teensy 3.6 is only capable of Full Speed
    Serial.println("USB connected at Full Speed (12 Mbps)");
    bufferSize = CDC_TX_SIZE;
#elif defined(__IMXRT1062__)    /* Teensy 4.0 or Teensy 4.1 */
    if (usb_high_speed) {
        Serial.println("USB connected at High Speed (480 Mbps)");
        bufferSize = CDC_TX_SIZE_480;
    } else {
        Serial.println("USB connected at Full Speed (12 Mbps)");
        bufferSize = CDC_TX_SIZE_12;
    }
#else
    Serial.println("WARNING: unknown Teensy type");
#endif

    Serial.printf("This Teensy is using %d USB transmit buffers\r\n",
            NUM_USB_BUFFERS);
    Serial.printf("Each transmit buffer is %d bytes\r\n", bufferSize);
    Serial.printf("Total transmit buffer size is %d bytes\r\n",
            NUM_USB_BUFFERS*bufferSize);

    digitalWrite(LED_BUILTIN, !digitalRead(LED_BUILTIN));
    delay(10000UL);
}
 
In the Teensy cores, all of the directories, directly under cores with the name USB_ are for Teensy 2.x (maybe 1.x as well) only
At times I wish we would split this up into a couple of different installs, like 2.x and 3-4x... But...

The USB support for Teensy 3.x is under the directory cores\teensy3. Most of them have the file names starting with usb_.
For USB types that include Serial, a lot of the functionality is in usb_serial.c and usb_serial.h. If you are using the emulated
serial, usb_seremu.*

The device descriptor information is in usb_desc.* The generated compile command lines contains defines -Dxxx which controls which of the USB types will be included in the build.

The main USB code is in usb_dev.*
 
  • Like
Reactions: jrw
This is the code which implements Serial.availableForWrite() on Teensy 3.6.

github.com

cores/teensy3/usb_serial.c at master · PaulStoffregen/cores

Teensy Core Libraries for Arduino. Contribute to PaulStoffregen/cores development by creating an account on GitHub.
github.com github.com

To briefly explain, it first checks whether a packet buffer is allocated, and if not it tries to allocate one. If no more buffers are immediately available, it returns 0. You want that result with availableForWrite, because the transmit code would need to wait for a buffer.

If a buffer is allocated, it just computes the amount of space remaining in the buffer. Simple stuff.

Because Teensy 3.6 has 12 Mbit/sec USB, the USB packet size is 64 bytes. As written, it will never return more than 64 bytes, because it's only looking at the 1 packet buffer the transmit code has allocated. All the rest of the transmit code is designed around the concept of allocating 1 packet buffer and filling it up as the program calls Serial.write() or Serial.print(), which of course calls Serial.write().

With considerable effort, you could try to look into the packet buffer pool to find out how many more packet buffers would be immediately available. But that is risky, which is the reason it was never attempted.

As you can see from the comment in the code, properly implementing availableForWrite() that truly works in practical applications is not easy. When availableForWrite() returns a number, it is essentially making a promise about future performance. With a packet buffer pool shared among all USB interfaces for data flow in both directions, it's easy to over-promise and then in the future not be able to deliver if anything else has allocated some of the packet buffers which were available at the moment availableForWrite() ran.

On Teensy 4, I abandoned this approach of a shared pool of packet buffers, mainly because of this and other thorny problems where excessive memory demands from some interfaces could impact performance of the others. The shared packet buffer pool which makes very efficient use of RAM made much more sense in the days of Teensy 3.0 with 16K RAM and later Teensy LC with only 8K. On Teensy 4 with 1024K, I redesigned the USB stack to allocate buffers within each interface and dedicated to data flow in each direction. It costs more RAM, but delivers consistent high performance... which also matters much more with 480 Mbit USB speed and 512 byte packets.

Whether any of this really helps, I'm not sure. But hopefully it does answer the question of where to find the code and more or less how it works.

You might also consider whether you really need availableForWrite() to return more than 64. With 12 Mbit/sec USB speed on Teensy 3.6, the maximum packet size is 64 bytes. Even if more packet buffers are unused in the USB memory pool, the data you give to Serial.write() is still going to be put into 64 byte packet buffers. That is simply how the USB protocol for bulk endpoints at 12 Mbit works. No amount of creative coding can change that fact of the USB protocol. Maybe something about your code might be able to work better with larger data, but ultimately the USB transmit will always be in chunks of 64 bytes.
 
Thanks Paul, I appreciate the explanation. I understand the evolution of the code following the evolution of the microcontrollers used in the Teensy series. I'm very happy with the teensy4 approach. Now that I understand the philosophy used for the teensy3, I'll figure out a solution that works. I was hoping to have the same code on teensy3 and teensy4 to simplify my code base across numerous payloads using 3.6, 4.0, 4.1 and MicroMod, but it will have to be a bit more complicated. We operate our payloads remotely on the ISS, so dependable data transfer is critical.

My plan was to write data to USB Serial 512 bytes at a time. That is 8 of 12 packets in the transmit buffer for Full Speed, and 1 of 12 packets in the transmit buffer for High Speed. This would keep both systems full of data all the time for maximum utilization, it would not overrun the buffer for either and it would not risk inefficient undersized packets being sent at High Speed. Looks like I'll have to have different code for teensy.

As an aside, with Teensyduino 1.60.0 my code to implement MTP for Teensy3 no longer works. It looks like the MTP code was brought into the core, but only Teensy4 is supported. And using the external library with Teensy3 no longer works due to conflicts with the core. I haven't had time to dig into that to better understand what is going on and what my options are for getting it working again. But that made getting USB Serial communication working reliably an urgent need.
 
I have been trying to find a solution using the new insight about USB serial on teensy3. So far, no success. If I check the value of Serial.availableForWrite() and wait for it to be 64, the program hangs until watchdog reset about 80 seconds later. I'm not sure what is going on for it to report a partial buffer available and continue to report that for many seconds. I changed to test if Serial.availableForWrite() is greater than 0. The program runs. It seems to work OK on Mac and Linux, but I need to write some test code to stress this and make sure I'm not losing any data. However, on Windows I am clearly losing data. The Teensy is not correctly waiting for Windows to read data and be ready for more, so I have USB transmit buffer overruns on the Teensy 3.6.

Below is test code to demonstrate this. It is a minimal extract from a much larger program. You will need an SD card with a large text file. Use the hardcoded filename or change the code to match your filename.

On a new-ish Mac, it runs and reports about 50k to 60k microseconds of delay. On an old-ish Linux box, it runs and reports about 35k microseconds delay. On both I think it is working successfully, but again I need a more rigorous test.

On a new Windows 11 box, it varies between about 150k to 250k microseconds of reported delay, but the actual execution seems a many seconds longer. And it clearly is not working. And my actual USB Host will be a very old single board computer running Windows 7, so even more constrained performance.


A better question is - how can I write large amounts of text to USB serial on teensy3 and teensy4 and not overrun the transmit buffer or otherwise lose data?



Screenshot 2026-06-09 220320.png




Code: 
#include <SD.h>               // https://www.arduino.cc/en/Reference/SD

// wait at most 4 seconds for USB to initialize, in microseconds
#define USBTimeout 4000000UL

bool sd_available = false;

void setup(void) {
    unsigned long a;

    // initialize USB serial port
    Serial.begin(115200);
    a = micros();
        while(!Serial) {
        if ((micros() - a) > USBTimeout) {
            break;
        }
    }
    Serial.println();
    Serial.println(F(";===================================="));

    if (!SD.sdfs.begin(SdioConfig(FIFO_SDIO))) {
        sd_available = false;
        Serial.println("Opening SD card failed, or not present");
    } else {
        sd_available = true;
        Serial.println("System startup");
    }
}

/**************************************************
 * Print contents of a file to USB serial port
 **************************************************/
void printFile(const char* filename) {
    File file;
    char buffer[64];
    int rtrn;
    unsigned long buffWaitCnt;

    if (!sd_available) {
        Serial.println("; ERROR: No SD card detected");
        return;
    }

    file = SD.open(filename, FILE_READ);
    if (!file) {
        Serial.printf("Unable to open file '%s' on SD card\r\n", filename);
        return;
    }

    Serial.println("------------------------------------------");
    Serial.printf("--- Begin %s\r\n", filename);
    Serial.println("------------------------------------------");

    for (unsigned int i = 0; i < sizeof(buffer); i++) { buffer[i] = '\0'; }

    buffWaitCnt = 0;
    while ((rtrn = file.read(buffer, sizeof(buffer) - 1)) > 0) {
        while (Serial.availableForWrite() <= 0) {
            buffWaitCnt++;
            delayMicroseconds(1);
        }
        Serial.print(buffer);

        // overkill, but re-write buffer with all NULL
        for (unsigned int i = 0; i < sizeof(buffer); i++) { buffer[i] = '\0'; }

#ifdef USE_DOG
        petDog();   // don't reboot from watchdog, might take a while!
#endif
    }
    file.close();

    Serial.println("\r\n------------------------------------------");
    Serial.printf("--- End %s\r\n", filename);
    Serial.println("------------------------------------------");
    Serial.printf("--- Delayed %lu microseconds\r\n", buffWaitCnt);
    Serial.println("------------------------------------------");

    if (rtrn < 0) {
        Serial.printf("ERROR: While reading file '%s'", filename);
    }
}

void loop(void) {
    if (Serial.available()) {
        if (Serial.read() == 'p') {
            printFile("20230703.016");
        }
    }
}
 
Windows 7 had buggy usbser.sys, and numerous other USB driver bugs. Microsoft finally got serious about USB driver quality with Windows 10, but they never published updates for pre-10 versions of Windows.

If you really must use Windows, consider updating to Windows 10. Even though it probably means newer SBC is needed, if this needs to be reliable you really should use Windows 10 or later.
 
On the Teensy side, look for TX_TIMEOUT_MSEC inside usb_serial.c. Try increasing it. Or you could try editing usb_serial_write() to not use it and just wait forever, even if the PC appears to have malfunctioned.

Like all editing of core library files, before you invest a lot of time, add a syntax error and click Verify just to make sure you are editing the file Arduino IDE is actually using. There are several files that look similar.

I'm pretty sure the final result will be you're transmitting much faster than software on Windows can process the data. The Windows driver implements a huge buffer, which is not a great design choice on Microsoft's part (in networking problems from too-large buffering called buffer bloat). Sadly, you have no control over Microsoft's driver. The huge buffer masks the real problem for some time, but then you get even worst issues.

But ultimately the bottom line is (probably) transmitting continuously, or in repeating bursts, at a sustained data rate faster than the receiving software can process is just not sustainable. Sooner or later, data loss is inevitable in any system where the data source creates sustained data rate faster than it can be consumed or stored.
 
Thanks, I'll take a look at the USB serial code and see if there is anything I can tweak. But as you say, without any kind of flow control, transmitting faster than the receiver operates will lead to problems.

Agreed on the state of WIndows drivers, and especially usbser.sys, and especially older versions. But even the current Windows 11 driver for USB serial is worse than the Linux driver was 10 years ago. The computer is in space, so we have to live with what we have. I spent years arguing we should continue to use Linux on payloads but got overruled and we have been living with the fallout since then.

I'll keep pounding on things. We have a launch deadline coming up.
 
Lack of flow control is not the problem. USB bulk endpoints / pipes always implement flow control. It's fundamentally part of the USB protocol.

The problem is what to do when flow control effectively stalls everything for too long. You'll see TX_TIMEOUT_MSEC causes Teensy to give up and discard data. If you want Serail.write() to just wait a long time, increase TX_TIMEOUT_MSEC. Or find and delete the code inside usb_serial_write() to make Teensy wait forever.

By checking Serial.availableForWrite(), you should see it returns 0 if Serail.write() would wait. If you only write the amount Serial.availableForWrite() says is safe, you'll shouldn't hit the TX_TIMEOUT_MSEC case. But then your code must decide what to do if Serial.availableForWrite() keeps returning 0 because the PC side is taking too long.
 
Thanks for your continued help and good advice. I try to use the Teensy core without modifications by me. For my use it is better if I put the potential infinite loop into my code rather than in the USB core.

Taking the lessons from above, I think this code is as robust as I can make it. Even with that, it is not 100% reliable. On both Linux and Mac, I get a correct file about 75% of the time. Other times, it has garbage characters inserted. It could be due to my terminal emulator program. I mostly use "screen" although I also have tried with minicom. On my Win11 system, I was originally getting 0% success. Then finally today I realized I have 4 Raspberry Pi Pico boards plugged in for unrelated tests. I unplugged those and now I get maybe 50% success. I am using TeraTerm on Win11. On Win11 I get the "inserted character" problem, and I also have lost data as if packets are missing. For all systems, this is for consecutive runs. I press "p" and it might work. Then press "p" again with no apparent changes, and it doesn't work. I do not have a USB bus analyzer to look deeper. Although I see KurtE has had some success using a Saleae logic analyzer for USB. So that might be an option if I need to go further.

Linux prints in about 5.5 seconds Mac prints in about 7 seconds, Win11 prints in about 27 seconds.

I have never seen either of the ERROR messages due to error returns from availableForWrite() or read().

The code that I have been using for teensy4 works flawlessly. 100% success on any system. This journey started when the same approach did not work on teensy3.

C++: 
#include <SD.h>               // https://www.arduino.cc/en/Reference/SD

// wait at most 4 seconds for USB to initialize, in microseconds
#define USBTimeout 4000000UL

bool sd_available = false;

void setup(void) {
    unsigned long a;

    // initialize USB serial port
    Serial.begin(115200);
    a = micros();
        while(!Serial) {
        if ((micros() - a) > USBTimeout) {
            break;
        }
    }
    Serial.println();
    Serial.println(F(";===================================="));

    if (!SD.sdfs.begin(SdioConfig(FIFO_SDIO))) {
        sd_available = false;
        Serial.println("Opening SD card failed, or not present");
    } else {
        sd_available = true;
        Serial.println("System startup");
    }
}

#if defined(ARDUINO_TEENSY36)
// Teensy 3.6 USB0 hardware can only do Full Speed (12 Mbps).
// Full Speed USB serial packets have up to 64 bytes of data.
#define BUFFSIZE    64
#elif defined(ARDUINO_TEENSY41) || defined(ARDUINO_TEENSY40) || defined(ARDUINO_TEENSY_MICROMOD)
// 512 is only correct if the Teensy4 is connected at High Speed (480 Mbps).
// For our use, assume this is always the case.
#define BUFFSIZE    512
#else
#error "Microcontroller not one of expected Teensy boards"
#endif

/**************************************************
 * Print contents of a file to USB serial port
 **************************************************/
void printFile(const char* filename) {
    File file;
    char buffer[BUFFSIZE];
    int rtrn;
    unsigned long buffWaitCnt;
    unsigned long printCnt;
    unsigned long fileSize;
    bool fileDone;
    bool usbAvail;
    int avail;
    unsigned long startMs, endMs;

    if (!sd_available) {
        Serial.println("; ERROR: No SD card detected");
        return;
    }

    file = SD.open(filename, FILE_READ);
    if (!file) {
        Serial.printf("Unable to open file '%s' on SD card\r\n", filename);
        return;
    }
    fileSize = file.size();

    Serial.println("------------------------------------------");
    Serial.printf("--- Begin %s\r\n", filename);
    Serial.println("------------------------------------------");

    buffWaitCnt = 0UL;
    printCnt = 0UL;
    fileDone = false;
    startMs = millis();
    while (!fileDone) {

        // For teensy3, USB has a shared pool of buffers. So there
        // is a potential race condition where we ask for the available
        // space, but by the time we act on that, it has changed. For
        // our application, when we are printing a file there should
        // be no other USB activity on the Teensy so this should not
        // be a problem.

        // Wait until there is space available for data, and note
        // how much data we can send. This loop could wait forever
        // if the USB Host is having trouble. Watchdog will save us.
        usbAvail = false;
        while (!usbAvail) {
            rtrn = Serial.availableForWrite();
            if (rtrn < 0) {
                break;
            } else if (rtrn == 0) {
                buffWaitCnt++;
                delayMicroseconds(1);
            } else {
                usbAvail = true;
            }
        }
        if (rtrn < 0) {
            Serial.printf("\r\nERROR: availableForWrite() returned %d",
                    rtrn);
            break;
        }

        for (unsigned int i = 0; i < sizeof(buffer); i++) { buffer[i] = '\0'; }
        // Read as much data as possible, but no more than we can fit
        // in the USB packet, and leave room for NULL terminator.
        avail = max(rtrn, BUFFSIZE-1);

        rtrn = file.read(buffer, avail);
        if (rtrn == 0) {
            fileDone = true;
            continue;
        } else if (rtrn < 0) {
            Serial.printf("\r\nERROR: read() returned %d while reading file '%s'",
                    rtrn, filename);
            break;
        }

        Serial.print(buffer);
        // Reliable transmission is far more important than speed.
        // Adding the flush() helps a lot.
        Serial.flush();
        printCnt++;

#ifdef USE_DOG
        petDog();   // don't reboot from watchdog, might take a while!
#endif
    }
    file.close();
    endMs = millis();

    Serial.println("\r\n------------------------------------------");
    Serial.printf("--- End %s\r\n", filename);
    Serial.println("------------------------------------------");
    Serial.printf("--- Elapsed time %0.2f seconds\r\n",
            (float)(endMs-startMs) / 1000.0);
    Serial.printf("--- Cumulative delay %lu microseconds\r\n", buffWaitCnt);
    Serial.printf("--- File is %lu bytes\r\n", fileSize);
    Serial.printf("--- Called Serial.print() %lu times\r\n", printCnt);
    Serial.println("------------------------------------------");
}

void loop(void) {
    if (Serial.available()) {
        if (Serial.read() == 'p') {
            printFile("20230703.016");
        }
    }
}
 
Change
Code: 
    // initialize USB serial port
    Serial.begin(115200);
    a = micros();
        while(!Serial) {
        if ((micros() - a) > USBTimeout) {
            break;
        }
    }
to
Code: 
    // initialize USB serial port
   Serial.begin(115200);
   while (!Serial & millis() < 3000) {} // wait up to 3 seconds for serial
 
You must log in or register to reply here.
Back
Top