Teensy 4.0 usb serial broken on Ubuntu 20.04

thibault

Member
Hi,

I have spent the last 2 days figuring this one out but have tried all in my capacity.

The Teensy 4.0 usb serial seems to be broken on Ubuntu 20.04. I am trying to setup some T4s as ROS serial nodes in a larger project, on ROS1 so I must use 20.04. I have 00-rules installed and disabled the modemmanager, tried forcing USB2, USB3, with/without a hub, no difference, the usb serial connection is unstable at best and if writing anything from host to T4 the serial socket hangs and I can only quit by physically disconnecting the board. I have tried the while(!Serial) trick, and pretty much tried any smallest suggestion in all posts.

This morning I tried swapping for a Teensy LC and it runs flawlessly on the exact same setup and code.. so there is something to do with the T4 usb serial stack. Steps to reproduce, pretty much any barebone Serial.begin/read/print() and writing/reading from pyserial shows the issue. I tried on 18.04 a barebone read/write loop and it works fine.

Problem is I need the compute power of the T4 so the LC is really not going to cut it.. and I purchased quite a bunch.. Any idea where to look else?

Thibault
 
Also I can confirm that using the hardware serial1 instead of the usb serial via a PL2303 usbserial adapter works flawlessly as well so the code running on the T4 is doing fine. That is a workaround to use the T4 instead of LC but I'd really like to bypass the usbserial adapter if possible.
 
Running fine here on Ubuntu 20.04.3

screenshot.jpg

Can you be more specific about what you're doing? Are you using the Arduino IDE with Serial Monitor as in this screenshot?
 
Sometimes when Linux systems have USB problems, the kernel gives pretty useful messages. How to view those messages varies depending on the distro. On my Ubuntu 20.04.3 system, running "tail -f /var/log/syslog" in a terminal works.
 
Hi Paul,

Thanks a lot for helping out. I've been a long time user of Teensies and love them!

So, basically I'm a bit at a loss of where that could be coming from. I also can reproduce a very simple Teensy -> Serial Monitor, but the problem seems to start when talking from Monitor or Python to the Teensy:

Works fine when just listening:

Screenshot 2022-01-06 at 13.00.51.jpg

But once I start talking back to the Teensy the port hangs, here in this example I should be able to send to the Teensy some chars and see the led blink and get back back "pong", but nothing happens on the Teensy, and after sending a few chars the serial monitor is completely frozen and I must disconnect the teensy to get the IDE back.

Screenshot 2022-01-06 at 13.18.32.jpg

It's the same via Python, when i start the actual ROS node and it starts talking to the Teensy all hell breaks loose, even though I can run a simple serial ping example from Teensy to host, and the node cannot be quit via ctrl-c, i must physically disconnect the board.

Screenshot 2022-01-06 at 13.04.18.jpg
 
Also for debugging in the IDE I use the Serial port, not the Teensy port since I need to use it via Python in the end. Both dont work when trying to send chars, but at least the Teensy Port can be quit, while the Serial port monitor hangs until the board is physically disconnected.

Screenshot 2022-01-06 at 13.31.29.jpg
 
Attachment didn't work on msg #7. Please post again.

Mac and Windows don't allow 2 programs to open the same serial port. Linux does, but 2 program accessing the same port doesn't actually work. Strange problems occur. Often 1 or both can send. Usually 1 or neither can receive anything, or sometimes both receive fragments of incoming data. How exactly it fails is unpredictable.

If you run a python program, or any other program, which opens the /dev/ttyACM0 port while the serial monitor also has it open, expect all sorts of strange problems. The Linux kernel's driver just does not work if more than 1 program opens the device.
 
Here is the attachment again.

Screenshot 2022-01-06 at 13.38.31.jpg

Regarding port access, I might have not been clear but I am absolutely not accessing both ports at the same time. Either only via the Serial Monitor for quick debugging, or only via Python with the serial monitor and Arduino IDE closed, so the issue is not related to concurrent port access.

Currently running 20.04 inside a VM but I have the same usb problem on a RPi4 running 20.04 desktop and server, both showing the same issue as well.
 
Something on your machine tried to run /bin/stty. Looks like it was done by the udev process.

but I am absolutely not accessing both ports at the same time.

You aren't, but sure looks like some software on your machine is. Maybe you've installed other udev rules, or other special software with its own udev rules, which automatically runs something that talks to serial ports? Normally modemmanger is the only widely used program which does this sort of thing, but this looks different. I have no idea what software this is. Can't see enough to guess any more.
 
Yes, but it seems that this command is coming straight from the 00-teensy.rules file no? Here it is:

Code:
KERNEL=="ttyACM*", ATTRS{idVendor}=="16c0", ATTRS{idProduct}=="04*", MODE:="0666", RUN:="/bin/stty -F /dev/%k raw -echo"

From what I grasp, this is executed once I plug in the T4.0, is it meant to continue running? this would make sense that it fails if I unplug the Teensy later on, so is expected?

I have already been reinstalling clean 20.04 machines a few times already to troubleshoot with nothing but the rules and modemmanager disabled, and the issue persists, while it runs without hiccups on an LC.

Are you able to send serial data to a T4.0 from the Serial monitor? Nothing is acknowledged on the Teensy side here.. so I cant even communicate from the host to the teensy.
 
Ah yes, you're right. Looks like I haven't read the udev rules file in a while. I forgot that stty line was added (I believe as a workaround for wrong "unix line discipline" settings when accessed without the serial monitor or any other terminal emulator program which configure "termios" stuff). The udev rules been stable since we switched from 49 to 00.

Here are the syslog messages when I plug that Teensy 4.0 programmed from the screenshot of msg #3 into my Ubuntu 20.04.3 machine:

Code:
Jan  6 09:56:59 preston kernel: [3181584.280971] usb 1-1.4: new high-speed USB device number 109 using ehci-pci
Jan  6 09:56:59 preston kernel: [3181584.394285] usb 1-1.4: New USB device found, idVendor=16c0, idProduct=0483, bcdDevice= 2.79
Jan  6 09:56:59 preston kernel: [3181584.394290] usb 1-1.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jan  6 09:56:59 preston kernel: [3181584.394294] usb 1-1.4: Product: USB Serial
Jan  6 09:56:59 preston kernel: [3181584.394297] usb 1-1.4: Manufacturer: Teensyduino
Jan  6 09:56:59 preston kernel: [3181584.394300] usb 1-1.4: SerialNumber: 7246100
Jan  6 09:56:59 preston kernel: [3181584.406559] cdc_acm 1-1.4:1.0: ttyACM0: USB ACM device
Jan  6 09:56:59 preston snapd[1148]: hotplug.go:199: hotplug device add event ignored, enable experimental.hotplug
 
Yes I get the same:

Code:
[29173.300258] usb 2-2.3: USB disconnect, device number 60
[29181.319892] usb 1-10: new high-speed USB device number 73 using ehci-pci
[29181.476490] usb 1-10: New USB device found, idVendor=16c0, idProduct=0483, bcdDevice= 2.79
[29181.476520] usb 1-10: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[29181.476522] usb 1-10: Product: USB Serial
[29181.476523] usb 1-10: Manufacturer: Teensyduino
[29181.476525] usb 1-10: SerialNumber: 8156850
[29181.483036] cdc_acm 1-10:1.0: ttyACM0: USB ACM device

I meant, are you able to run a sketch that reads from the usb Serial port on the Teensy? I cant seem to see any data written to to the Teensy, only from Teensy to host.

Something like so, and typing manually in the Serial monitor?

Code:
void setup() {
  while(!Serial);
  Serial.begin(9600);
}

void loop() {
  if (Serial.available()>0){
    Serial.println("pong");
  }
  delay(1);
}
 
I'll also mention, since Teensy 4.0 release in August 2019, *many* people have used these boards successfully on Windows, Mac and Linux. Ubuntu is by far the most popular Linux distro.

I know you're feeling frustrated. I know from your point of view it must seem like Teensy doesn't work at all with Linux. But so many people have indeed used it with so many Linux machines since late 2019. Linux can be tricky to support because there are so many different distros, and indeed we've had a variety of Linux problems over the years (most recently, needing to switch to GTK3 and ongoing issues with some distros lacking specific libpng library versions, plus some libudev not parsing the more complex udev rules syntax properly) but the USB serial definitely does work.

This particular case is really a mystery. I really want to understand why this is happening, but I just don't see any info about what might be wrong.

I wish I could help more, but I just can't see what's wrong with your machine. Best I can say it Teensy 4.0 has been well tested with Linux and definitely is know to work. This isn't a situation where a product only got tested on Windows and doesn't work with Mac or Linux. I personally use Ubuntu 20.04 for nearly all development work, so Linux (at least on x86-64) gets really well tested.
 
Oh I want to make it really clear that I am absolutely not pointing the fault at you or Teensy boards, I'm also suspecting rather some weird ubuntu driver behavior in my specific config. I have run right now the same code as you did and the Teensy is not responding to any Serial data, it's just like the ubuntu host is unable to write to the port.

Screenshot 2022-01-06 at 20.57.24.jpg

I had seen this on both my 20.04 VM in Parallels, and on the Pi4 running 20.04 so I assumed this was rather a distro-level issue, but for the sake of it I'll check tomorrow on another 20.04 pc I have around. It works perfectly on my Mac yes, only when I switch to Ubuntu VM or RPi4 is it not working out.. Also a weird thing is on the RPi4 it works perfectly in Raspberry OS but under 20.04 the teensy is not even detected on 2 of the usb ports, on one of the 2 usb3 ports it power cycles and gets a new port number every few seconds, and "works like right now" on the last usb3 port.. which really leads me to believe something is wonky under 20.04..

Do you know of any other debug method I could run to get some logs about to help in debugging this?
 
Works for me on Ubuntu 20.04.

However seeing your picture, with the serial monitor window with an: a
shown in it... Did you ever hit the <cr> or press the send button?

As the monitor does not send data per character bug only by whole lines entered.
Screenshot from 2022-01-06 14-56-56.png
 
yes yes, I did send many chars before, it was just for illustration purpose. I write some chars, send them out and they disappear into the aether, no reply from the board, even though I can listen to the board data back. Will check tomorrow on a different 20.04 system and report back.

thanks a lot for the time.
 
Hi all, so this morning I tried again the same basic read-write loop on different machines, I think the issue is really some conflict with the VM environment:

- on a 20.04 x86_64 pc i didnt test on yet: works perfectly, can read-write serial data without problem
- on a 20.04 x86_64 VM: I can read from the Teensy but not write and the serial connection hangs, impossible to do anything there
- on a 20.04 arm64 Raspi4: this one is a mystery to me as I could not even see the T4 previously on some usb ports but this morning it works perfectly on all ports, I think I might have tried the while(!Serial) trick initially and then fiddled with rules and modemmanager, but didnt add the while(!Serial) at the end again, but it seems that it works perfectly there which is the intended final use.

So I didnt solve the VM problem which prevents me from developing on my laptop, slight annoyance, but seems like I should figure out how to work around that by using that old linux box instead.

So maybe for posterity and search if anyone else hits that: seems like the usb serial port on ubuntu 20.04 and Teensy 4.0 is broken when run in a virtual machine (at least on Parallels, didnt try Virtual Box)
 
Maybe the virtual machine has problems emulating 480 Mbit speed or some aspect of EHCI? (EHCI is a very complicated standard) Maybe it will work if the virtual machine only has to emulate much simpler OHCI for 12 Mbit speed?

I hope everyone can agree it's not Linux's fault, nor Teensy's fault, if the VM software is emulating the host controller incorrectly. (even "direct" access to USB devices from the VM still means the VM software has to emulate the full host controller register set in the virtual machine... the real hardware isn't designed for a VM to also use)

To test this, on one of the real computers, edit {Arduino}/hardware/teensy/avr/cores/teensy4/usb.c.

Uncomment this line and upload your program which responds to incoming characters.

Code:
        //USB1_PORTSC1 |= USB_PORTSC1_PFSC; // force 12 Mbit/sec

You should see the kernel syslog messages say it found a new full speed device, rather than high speed.

Then move it back to the virtual machine. Does it work when the USB runs at 12 Mbit speed?
 
No luck, the full speed device is properly recognized, but again, I receive the data from the Teensy, but writing from the Serial monitor does nothing at all. Works fine on mac. Doesnt seem to be using OHCI though,

Code:
[  113.577970] usb 2-2.4: new full-speed USB device number 6 using uhci_hcd
[  113.683118] usb 2-2.4: not running at top speed; connect to a high speed hub
[  113.692868] usb 2-2.4: New USB device found, idVendor=16c0, idProduct=0483, bcdDevice= 2.79
[  113.692872] usb 2-2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  113.692874] usb 2-2.4: Product: USB Serial
[  113.692875] usb 2-2.4: Manufacturer: Teensyduino
[  113.692876] usb 2-2.4: SerialNumber: 8156850

When commenting again the usb.c line I get that:

Code:
[  265.996947] usb 1-6: new high-speed USB device number 3 using ehci-pci
[  266.152413] usb 1-6: New USB device found, idVendor=16c0, idProduct=0483, bcdDevice= 2.79
[  266.152419] usb 1-6: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  266.152421] usb 1-6: Product: USB Serial
[  266.152422] usb 1-6: Manufacturer: Teensyduino
[  266.152423] usb 1-6: SerialNumber: 8156850
[  266.225147] cdc_acm 1-6:1.0: ttyACM0: USB ACM device
 
Dear all,

-> i had similar problems like you (and others in many other threads) with receiving Serial.print messages over USB from teensy on linux. [on macbookpro my prototype works fine].
my setup:
Raspberry Pi 4B with raspbian 2021-05-07, arduino IDE + Teensyduino + 00-rules installed
teensy 3.6 shows up when i type
Code:
lsusb
and is on /dev/ttyACM0


my issue was:
- i could upload new .ino files to the teensy with arduino IDE and receive them directly after the teensy upload in another software (eg. python or pd)
- but i did NOT receive the serial data after a reboot without disconnecting/connecting usbcable of the teensy or without prior uploading an .ino file. for standalone solutions this does not work.

as teensy showed up with
Code:
lsusb
and i had no output when typing
Code:
lsof | grep /dev/ttyACM0
i thought something might block the device from serial-communication after a reboot so that i could not read data from, it but i can't find the cause.

my workaround is: everytime, prior to starting pd or python code i do an usbreset of the teensy-port with the following command:

Code:
usbreset /dev/bus/usb/001/003

https://wiki.ubuntuusers.de/usbreset/ this article shows how to compile and use usbreset (very simple).

not a very convincing solution but for now it works. better solutions would be appreciated.

hope this helps.
 
Back
Top