Teensy 3.2 Linux Mint USB Serial connection issues with fix

Status
Not open for further replies.
L

levivk

New member
Hello,

I'd like to share an issue that I've been dealing with and what fixed it.

My system:
Asus K501UX
Linux Mint 19.3
Kernel 5.3.0-40
Arduino 1.8.11
Teensyduino 1.50
Teensy 3.2

Example code

Code: 
#define BLINK_PERIOD 500
uint32_t next_blink = 0;
uint32_t counter = 0;

void setup() {
  Serial.begin(9600);
  pinMode(LED_BUILTIN, OUTPUT);
}


void loop() {
  uint32_t now = millis();
  Serial.print(counter++);
  Serial.print("\t");
  Serial.print("this is a ");
  Serial.print("test");
  Serial.println();

  if(now >= next_blink){
    next_blink += (BLINK_PERIOD/2);
    digitalWrite(LED_BUILTIN, !digitalRead(LED_BUILTIN));
  }
  
  delay(5);
}

After uploading with Arduino IDE (or Platformio) and opening the Arduino serial monitor (or Putty), the output would contain a single line with counter between 35 and 40 or be completely blank. Nothing more appeared on the serial monitor, but the Teensy continued to run as indicated by the flashing LED. Sometimes, though rarely, the serial monitor would repeatedly print the full "### this is a test" line with the counter increasing. In this case, I could close and reopen the serial monitor, and it would continue to work. When it did work, however, the counter would start at 4.

I tried programming the Teensy in Linux and then booting into Windows. Using either putty or the Arduino serial monitor, I would continue to receive the expected output. This narrowed it down to a Linux issue...

I tried a different kernel version (4.15.0-88) which did not make a difference.

At this point, after doing some Googling, I was pretty convinced some other program was stealing the device file.
I uninstalled ModemManager, which unfortunately did not make a difference.
I installed the latest udev rules which did not make a difference.

Here, I noticed something interesting. If I increased the delay, or decreased the amount of stuff being printed, the connection would work more often. Still not sure if this is relevant or how this plays into it...

I tried compiling with MIDI USB type instead of serial as suggested by Paul somewhere. This, in fact, worked! I could consistently get a serial connection. I probably would have stopped here, but Putty couldn't open it as it wasn't a tty.

Finally, I figured it out. Adding "while(!Serial)" to the beginning of my setup() worked! I now consistently get a repeating output. Paul suggests this here: https://forum.pjrc.com/threads/56818-Serial-print-lockup in #4. Though he said it's only been reported to solve issues on Raspberry Pis, here's a case where it worked on a different system.

Anyways, I figured I'd share this here in hope that anyone else who deals with this does not have to spend as many hours on it... :)

Any speculation on the factors at play is welcome. I'm interested in learning more about what's going on with USB here.
 
Hopefully this will explain what, why, and how that happens.

First, let's create a simple sketch, a variant of the Blink example, serial-blink.ino:
Code: 
#define  LED  13

void setup() {
  Serial.begin(9600);
  pinMode(LED, OUTPUT);
}

void loop() {
  if (Serial) {
    digitalWriteFast(LED, HIGH);
  } else {
    digitalWriteFast(LED, LOW);
  }
  delay(100);
}
You can compile and build it normally. Instead of just blinking the LED, this one illuminates the LED whenever the Teensyduino Serial device is working, and keeps the LED dark when the Teensyduino Serial device is off/stalled.

If you compile and upload the sketch, you'll see the LED stays dark in normal operation. However, if you open a terminal, and run e.g.
Code: 
bash -c 'exec 3<>/dev/ttyACM0 ; sleep 2'
you'll see that the LED on the Teensy lights up for two seconds, and then goes dark!

The LED on the Teensy also lights up whenever you have the Serial Monitor window in the Arduino environment open!

You see, the Teensyduino Serial object is only operable when an application on the host machine has that USB serial port open.
(Yes, this means you can use that as a detector "switch", having your Teensy with USB Serial support operate in "independent mode" whenever no application has the USB serial port open, but switch to "active serial mode" when an application does have the USB serial port open.)

When (Serial) is false, it can still send data via the USB serial interface. Because there is nothing to receive that data on the other end, and the Linux kernel does not want to drop the data on the floor, it buffers some of it, and then tells Teensy to stop sending more, because the kernel is not willing to spend arbitrary amounts of memory buffering data that some application might or might not be interested in the future.

The "problem" is that the Teensyduino USB serial Serial object does not automagically recover when an application opens the USB serial device after the Teensy has received the stop-sending notification. (I'm not even sure if it even is possible to do it "automagically", because in some cases you want to discard all that old stale data, and in other cases you want to make sure the new application gets it. I guess other operating systems make that choice for you? Well, not Linux: it is a tool for people to use as they wish, not a tool that tells people what to do and how.)

What can we do, then? Well, it really is rather simple. We just monitor the changes in the status of the USB serial object, via (Serial), and do the necessary "recovery" ourselves.

(You could argue that this should really be in a tutorial somewhere, but hey, there is so much knowledge available to us that no single person can encompass all, and Paul Stoffregen and contributors to Teensyduino here and on GitHub are doing a pretty darn good job of getting stuff work on so many operating systems and environments, that I would definitely not blame them for missing this info. I'd love to write this kind of a tutorial myself, but I am damaged myself having issues with both negative feedback and lack of feedback, so although I know this stuff and love to help others with this stuff, I'm not sure I can.)

I'll write a simple example sketch that outputs numbers in increasing order to USB serial, restarting properly whenever the application closes the USB serial device, and a new one opens it. I haven't written it yet, so it'll be in my next post. We'll see how long it takes me! ;)
 
Okay, it was much simpler than I thought. We just use Serial.begin() when we first notice an userspace application has connected (by noticing that (Serial) is now true), and when (Serial) turns false, we stop outputting and do Serial.end(). Very logical!

Here is the example serial-counter.ino I crafted:
Code: 
// LED pin to use for indicating an application connection
#define  LED  13

// Duration between counter increments, in milliseconds
#define  INTERVAL_MS  200

// Initial counter value to use with a new userspace application
#define  COUNTER_START  1

// Flag to indicate whether we believe we are connected to an userspace application
static bool  counting;

// The counter whose value will be provided via the USB serial connection
static uint32_t  counter;


void setup() {
  // We use the LED to indicate when we are connected to an userspace application on the host computer.
  pinMode(LED, OUTPUT);

  // When first powered on, we start disconnected and with a zero counter.
  counting = false;
  counter = 0;

  // but, we also want the USB serial device to be enabled/running, initially.
  Serial.begin(9600);
}


void loop() {

  // Do we have an userspace application connected to us via USB serial?
  if (Serial) {
    // We have a connection to an userspace application.

    if (!counting) {
      // but we must initialize our USB Serial object first.

      Serial.begin(9600);
      counting = true;
      counter = COUNTER_START;

      // Let the LED show we are connected.
      digitalWriteFast(LED, HIGH);
    }

    // Because we do not read any input, we need to clear the input buffer
    // or we will get stuck if the user inputs something.
    Serial.clear();

    // Output the counter and increment it.
    Serial.println(counter);
    counter++;    
    
  } else {
    // No connection to an userspace application.

    if (counting) {
      // Stop counting, and clean up the Serial object.
      Serial.end();
      counting = false;

      // Let the LED show we are connected.
      digitalWriteFast(LED, LOW);
    }
  }

  delay(INTERVAL_MS);
}
You can try this trivially with the Arduino Serial Monitor. Whenever you open the window, it starts counting from 1 upwards, five numbers per second (200ms interval is 1000/200ms = 5 Hz).

Note the Serial.clear(); and the accompanying comment. I did notice that without this, in certain circumstances I could get the Teensy to get stuck (stop outputting, even though it knew it was connected to an userspace application since the LED was lit). So, the read and write sides on the Serial object on the Teensy are not truly independent, and if one gets stopped (because of buffering, and either Teensy or the userspace application is not consuming the data), the other gets stopped too. (Meaning, you cannot just ignore input and do output. You need to either discard the input like I do with Serial.clear() above, or consume it, to be sure you can write too. As I understand it, this is because both sending and receiving via USB uses the same buffer space on the Teensy.)

When using the Linux tty device, we must either discard all input in the Teensy example, or stop the tty device from echoing everything by default. You see, serial ports in POSIXy and UNIXy systems like Linux and Macs, for historical reasons, default to a state where the input is echoed. If we do not change that, then Teensy will receive an echo of everything it outputs! (If you find yourself staring at "infinite output" from Teensy, this is the reason: your serial port is told to echo everything Teensy writes back to Teensy, and if Teensy echoes everything it sees, you get an infinite loop of data between the kernel (more properly, the tty device in the kernel) and the Teensy. Ouch!)

This state is controlled on the host system via an interface called termios; this interface is built in to the C libraries on these systems. There is also a command-line tool, stty, that can be used to change that state.)

So, the proper test to read output from Teensy in Linux and Macs is not cat /dev/ttyACM0; that causes Teensy to get into an echo chamber loop. You need to do something like
Code: 
bash -c 'exec 0<>/dev/ttyACM0 ; stty raw ; cat'
instead, to disable that echo. This runs three commands under the Bash shell. The exec part redirects standard input (as readable and writable) to the USB serial device. The second sets standard input to "termios raw mode"; basically the mode that disables all shenanigans, and lets you send and receive serial data as-is without echo and other magic. The third then outputs everything from input (the USB serial device, since we redirected it) to output. We run it in a Bash subshell, because that way we can return back to the original shell by pressing Ctrl+C, and have control back.

(The reason I didn't do this in the first example, is that I knew there was no data coming out of the Teensy, so infinite echo chamber was not possible. And I wanted to keep everything as concise as I could and omit the tty/termios stuff, because I've very recently been told in another forum that I am basically a troll ("reported to moderators") because I write too long and too complicated posts.)

If you find this interesting, you might consider adding a second counter, to count the number of userspace applications that have connected, and print a line describing that before printing the initial counter value. This is an easy change, just a three lines added at the proper spots, I think; should be interesting!
 
Thanks for the detailed response! That is interesting about the stop-sending notification from the Linux kernel. That explains why a good connection would happen more often as I increased the delay (Less data sent, host buffer fills up slower...). I tried my hand at recovery from rejection by the Linux kernel using Serial.end() followed with Serial.begin(), which did not work. I pasted my code below. The recovery attempt activates after a delay so that I have a chance to trigger rejection. I also tried throwing a Serial.flush() in there and added a delay, but no cigar. I guess I'm not sure what Serial.end() does, if anything, for native USB. Maybe it's required to use lower level functionality? I'm also curious why I couldn't find an example of someone else dealing with this (or such pages avoided my Google-fu...). Do other Linux users not have wait for Serial?

Thanks again for taking the time to write this up! :D I'm looking forward to your sketch!

Code: 
#define BLINK_PERIOD 500
uint32_t next_blink = 0;
uint32_t counter = 0;
bool serial_connected = false;

void setup() {

  Serial.begin(9600);
  pinMode(LED_BUILTIN, OUTPUT);
}


void loop() {
  uint32_t now = millis();
  Serial.print(counter++);
  Serial.print("\t");
  Serial.print("this is a ");
  Serial.print("test");
  Serial.println();

  if(now >= next_blink){
    next_blink += (BLINK_PERIOD/2);
    digitalWrite(LED_BUILTIN, !digitalRead(LED_BUILTIN));
  }

  if(now > 15000){
    if( !serial_connected && Serial){
      serial_connected = true;
      digitalWrite(LED_BUILTIN, HIGH);
      delay(2000);
      Serial.flush();
      digitalWrite(LED_BUILTIN, LOW);
      Serial.end();
      delay(1000);
      Serial.begin(9600);
    }else if(serial_connected && !Serial){
      serial_connected = false;
    }
  }
  
  delay(1);
}
 
Thanks again for all the info! Indeed your code operates as you say. Also interesting facts about the input buffer filling up. I tried sending a lot of data (without Serial.clear()) to the Teensy in my example, and it halts. Very interesting. Thanks again!
 
Note that there is a technical issue to cleanly solve this.

As you noticed, .begin() and .end() don't actually do anything; the reason I used them is more to do with us human programmers than anything technical.

Put simply, to work without issues, you must 1) always consume as much input as you can before attempting to output, and 2) check that there is an user application receiving the data before attempting to write. If that user application hiccups and doesn't consume the Teensy output, the USB serial object will deadlock. If the Teensy side is too slow and doesn't consume the data before trying to write, the USB serial object will deadlock. Without a .reset() interface as mentioned in that technical issue thread, or poking into the Teensyduino USB implementation innards directly, there is no way to resolve either deadlock. Ouch.

I believe Windows doesn't suffer from this, because it is happy to just throw the extra data away, without telling either the device (Teensy) or any application about it.

There are other surprising differences between OSes, too. A comment in cores/usb_serial.h history indicates the RTS information is not reliable on Windows, for example. Note that Teensy cannot know which OS the computer it is connected to is running, so it must deal with ALL quirks of all operating systems. I'm also not an optimal person to suggest fixes here, because I don't use Windows, and can't bring myself to care much about Windows wonkiness, not when it prevents things on other OSes from working in the best possible way... I am tempted to just fork Teensyduino for non-Windows Teensy uses! But I do understand why PJRC and others do care, I'm not a zealot; I'm hoping others would have a solution that works for all.
 
Status
Not open for further replies.
Back
Top