Looking for some clues hard Linux crash using Ty Commander

clinker8

Well-known member
Arduino IDE 1.8.19, Teensyduino 1.56, TyTools 0.9.7.
Symptom: Flash of Teensy 4.1 causes hard crash of laptop - screen freeze 10 seconds, unresponsive to keyboard or mouse, then laptop power turns off.
OS: Ubuntu 20.04 LTS & Pop-OS 22.04 fresh install.

Nothing obvious in syslog, nor dmesg. Seems there aren't any breadcrumbs to pick up. :(

On May 26, 2022, Ubuntu update done. It changed some video drivers and a couple of other things. After that, Ty Commander would cause crashes. I purged Nvidia, and reinstalled, but still got crashes using Ty Commander. I spend 5 days messing with this. Then I backed up my disk. Then did a fresh install of Pop-OS 22.04. After a day or two of configuring, same problem.

After playing about a little, if I don't enable TyTools, (after the crash, the integration with Arduino setting was not preserved,) but I just use Teensyduino to program my Teensy 4.1, the download completes and everything seemed fine. Prior to this, every time I used Ty Commander after May 26, 2022 my laptop would crash/turn off.

It would be nice to continue to use Tytools, as it has good support for multiple processors.

Has anyone experienced this? Is it possible to run tytools under valgrind, or some other tool that protects against kernel panics?

This crash occurs when loading my electronic lead screw project onto a Teensy 4.1.

I just now thought of another clue. I am powering the Teensy 4.1, a buffer board, the rotary encoder, and the PJRC ILI9341 board off an active USB3 hub, Atolla - 8 port, with a 4A 5.1V power supply. I noticed that the switched port does not turn on, if the USB cable is connected to the Teensy, but does, if I disconnect the USB cable that powers everything. I will try isolating the active USB3 hub through a passive hub. Maybe there was backfeed into the USB3 port on my laptop, via the active hub?

Can I power the Teensy not through the active hub? What alterations do I need for this?
 
Isolating the active hub from the laptop does not prevent reboot. Total current draw on the USB3 port is 0.23A.
Using valgrind shows memory leaks in tycommander, although they maybe false. I compiled in debug mode, but do not know how to change the optimization setting to O0, via cmake, as recommended.

Finally see something in journalctl.
Code:
Jun 05 13:16:39 pop-os arduino-arduinoide.desktop[14082]: Memory Usage on Teensy 4.1:
Jun 05 13:16:39 pop-os arduino-arduinoide.desktop[14082]:   FLASH: code:73020, data:57240, headers:9000   free for files:7987204
Jun 05 13:16:39 pop-os arduino-arduinoide.desktop[14082]:    RAM1: variables:59520, code:68456, padding:29848   free for local variables:366464
Jun 05 13:16:39 pop-os arduino-arduinoide.desktop[14082]:    RAM2: variables:12384  free for malloc/new:511904
Jun 05 13:16:39 pop-os kernel: usb 1-1.2.4: USB disconnect, device number 20
Jun 05 13:16:39 pop-os arduino-arduinoide.desktop[14082]: An error occurred while uploading the sketch
Jun 05 13:16:40 pop-os kernel: usb 1-1.2.4: new high-speed USB device number 21 using xhci_hcd
Jun 05 13:16:40 pop-os kernel: usb 1-1.2.4: New USB device found, idVendor=16c0, idProduct=0478, bcdDevice= 1.07
Jun 05 13:16:40 pop-os kernel: usb 1-1.2.4: New USB device strings: Mfr=0, Product=0, SerialNumber=1
Jun 05 13:16:40 pop-os kernel: usb 1-1.2.4: SerialNumber: 00123C8D
Jun 05 13:16:40 pop-os kernel: hid-generic 0003:16C0:0478.0008: hidraw4: USB HID v1.11 Device [HID 16c0:0478] on usb-0000:00:14.0-1.2.4/input0
Jun 05 13:16:40 pop-os gnome-shell[2512]: Window manager warning: Buggy client sent a _NET_ACTIVE_WINDOW message with a timestamp of 0 for 0x600003
Jun 05 13:16:42 pop-os kernel: usb 1-1.2.4: USB disconnect, device number 21
Jun 05 13:16:42 pop-os kernel: usb 1-1.2.4: new high-speed USB device number 22 using xhci_hcd
Jun 05 13:16:42 pop-os kernel: usb 1-1.2.4: New USB device found, idVendor=16c0, idProduct=0483, bcdDevice= 2.80
Jun 05 13:16:42 pop-os kernel: usb 1-1.2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jun 05 13:16:42 pop-os kernel: usb 1-1.2.4: Product: USB Serial
Jun 05 13:16:42 pop-os kernel: usb 1-1.2.4: Manufacturer: Teensyduino
Jun 05 13:16:42 pop-os kernel: usb 1-1.2.4: SerialNumber: 11951490
Jun 05 13:16:42 pop-os kernel: cdc_acm 1-1.2.4:1.0: ttyACM0: USB ACM device
and
Code:
Jun 05 13:17:17 pop-os arduino-arduinoide.desktop[14082]: Memory Usage on Teensy 4.1:
Jun 05 13:17:17 pop-os arduino-arduinoide.desktop[14082]:   FLASH: code:73020, data:57240, headers:9000   free for files:7987204
Jun 05 13:17:17 pop-os arduino-arduinoide.desktop[14082]:    RAM1: variables:59520, code:68456, padding:29848   free for local variables:366464
Jun 05 13:17:17 pop-os arduino-arduinoide.desktop[14082]:    RAM2: variables:12384  free for malloc/new:511904
Jun 05 13:17:17 pop-os arduino-arduinoide.desktop[14082]: Opening Teensy Loader...
Jun 05 13:17:17 pop-os arduino-arduinoide.desktop[14082]: An error occurred while uploading the sketch
Jun 05 13:17:17 pop-os kernel: usb 1-1.2.4: USB disconnect, device number 22
Jun 05 13:17:17 pop-os systemd-udevd[19200]: ttyACM0: Process '/bin/stty -F /dev/ttyACM0 raw -echo' failed with exit code 1.
Jun 05 13:17:18 pop-os kernel: usb 1-1.2.4: new high-speed USB device number 23 using xhci_hcd
Jun 05 13:17:18 pop-os kernel: usb 1-1.2.4: New USB device found, idVendor=16c0, idProduct=0478, bcdDevice= 1.07
Jun 05 13:17:18 pop-os kernel: usb 1-1.2.4: New USB device strings: Mfr=0, Product=0, SerialNumber=1
Jun 05 13:17:18 pop-os kernel: usb 1-1.2.4: SerialNumber: 00123C8D
Jun 05 13:17:18 pop-os kernel: hid-generic 0003:16C0:0478.0009: hidraw4: USB HID v1.11 Device [HID 16c0:0478] on usb-0000:00:14.0-1.2.4/input0
Jun 05 13:17:20 pop-os kernel: usb 1-1.2.4: USB disconnect, device number 23
Jun 05 13:17:20 pop-os kernel: usb 1-1.2.4: new high-speed USB device number 24 using xhci_hcd
Jun 05 13:17:20 pop-os kernel: usb 1-1.2.4: New USB device found, idVendor=16c0, idProduct=0483, bcdDevice= 2.80
Jun 05 13:17:20 pop-os kernel: usb 1-1.2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jun 05 13:17:20 pop-os kernel: usb 1-1.2.4: Product: USB Serial
Jun 05 13:17:20 pop-os kernel: usb 1-1.2.4: Manufacturer: Teensyduino
Jun 05 13:17:20 pop-os kernel: usb 1-1.2.4: SerialNumber: 11951490
Jun 05 13:17:20 pop-os kernel: cdc_acm 1-1.2.4:1.0: ttyACM0: USB ACM device
-- Boot d7d6d6b2f91649feb1ac928e8057facc --
At this exact time, my syslog has a pile of \00\00\00... written to it, as the system crashed. My text editor is complaining about this. Then the reboot occurs.
Seems to be some sort of illegal or unkosher use of the USB port. The whole system crashes with no logging at all.

So far it only happens if ty commander is integrated to Arduino. This totally stinks. These crashes are violent. Sometimes my computer does not come up properly after a reboot.
 
So it happened once with just Teensyduino and Arduino. After that, I thought maybe I had a bad install of Arduino. So uninstalled it from the system, and tossed the directory into the trash and deleted it from trash. Re-downloaded and installed it. Reinstalled Teensyduino. Survived one flashing to the board. A minor success - maybe. Not sure I want to install Ty Commander just yet... Think I will hold off a while and see if what I have is stable for a while. I just came to the realization that I can't uninstall tytools easily, because I did a sudo make install. There is no make uninstall in the package. Most unfortunate for me - basically a mess. What a colossal pain this has been.
 
So no comments? Am I the only one with this issue?

What is odd, is that it runs perfectly on RaspiOS64 on an RPI4, but still violently crashes my Pop!OS 22.04 i7 computer (32GB RAM, 1TB SDD). @KurtE, you are a contributor to tytools, no comment? Is there anything I can do to provide more info that could help? I'm not a computer whiz at things like this, so I'm not sure what to do, besides burning the laptop! Of course, I'd rather help fix the problem than do that! Basically, this has stopped all my Teensy development, which is not good. Can't afford for my primary computer to be trashed. Not sure if this is HW, application, or OS related. Need to narrow the possibilities... Helpful suggestions appreciated.
 
@clinker8
Seems most TyCommander users - myself included - are on Windows. @KurtE runs linux and maybe @mjs513 - but not sure they use TyComm there like they do on Windows. I've never seen such an issue posted before by others having trouble.

@koromix is the best bet - mostly on github these days, he picked up a change quickly that KurtE just did for add usb type. So he is still active.

I see the issue #93 made there - be patient (and need I say respectful) - he made an awesome tool that generally works well on MAC/Windows/Linux and supports it in his spare time. His timezone is France and his day job is intense but not as a programmer.

If you can find a way to have the machine be stable without Arduino integration.

Don't have a feel for linux use and cause or solution to such problems.

Would it work to replace the build binaries on your system with those released by @koromix? If your build is broken that might be the cause and if all the released binaries were used instead it might work?
 
@defragster, tycommander was my go to tool, once I learned about it. Extremely useful. I was using it for handling multiple Teensy's, great for that.

My build was from source, which should have gone ok... I did load all the prerequisites, didn't take any shortcuts. I can look into these build binaries. Are they on his main site? or github? Nevermind, I will follow the lead. Thanks.
 
Don't see any binaries on either site. Somehow, I have to undo any tytools integration. Suspect there isn't any (this time, since the laptop crashed on the click to integrate).

Originally, I downloaded the 0.9.7 zip file, is there a more up to date version? Suppose I could clone the git repo and try again, maybe something is off with the zip file?

Been pretty awful to debug this. The crashes occasionally mess with the file system, or the video. Sometimes have to reboot several times for the OS to fully recover.
 
As mentioned, I mainly use TyCommander on Windows. I only use Ubuntu for occasional testing. I have just updated my test machine to 22.04 and trying to install a few other things on it.

At times I do have TyCommander hang/put a real drain on Windows, if I am running something that outputs a ton of data, especially if I forget to output the \n

So it keeps extending out and ...

But it has not crashed the machine, although a few times I had to kill it with the task manager.

Edit: forgot to mention, that I don't see any builds up on his site: https://koromix.dev/files/tytools/
 
Yes, I have forgotten a \n or two in my time :)

Since I am running on PopOS 22.04, which is a modified Ubuntu 22.04, I have some support from System76, who are the originators and maintainers of PopOS. Basically TyCommander stopped working properly on May 26. On that date, there were a lot of software changes pushed out, both on 20.04 and 22.04. One of the changes was libusb. Allegedly it was some kind of bug fix. I downgraded libusb tonight, and was able to get a little further. TyCommander worked exactly once, programming the Teensy. Then maybe 20 seconds later, while I was typing out a message that it was successful, my laptop crashed again. By crash, I mean a screen freeze for 20-30 seconds then power off. So my happy message was premature:p

My laptop works great - except when programming Teensy using TyCommander. (Then the laptop turns off.) It works ok with the Teensy programmer. Apparently there is something subtle going on, that sadly is just beyond my comprehension. Hope to find a satisfactory solution, somehow. Hard crashing your laptop 5 times a day is not good for it. One day the file system is not going to come up.

For my reference, how does the normal Teensy programmer work? Is there any documentation on it that I could read?
 
Oppps - misremembered that the LINUX build was the one with executable provided. ... seems it is MAC and Windows.
 
This source code is the main reference.

https://github.com/PaulStoffregen/teensy_loader_cli/blob/master/teensy_loader_cli.c

While it's long, most of the code is just 4 copies of the USB HID access to hardware, written for 4 different operating systems. The main program from lines 90 to 206 is the part to read.

That was informative. Thank you. On line 215 of your link, there is mention of using libusb 0.1. Checking my computer for libraries, I have libusb-1.0-0, but in arduino-1.8.19/hardware/tools there is a libusb-0.1.so.4. There's probably some difference between 1.0 and 0.1.
 
So far this seems to be an Ubuntu and Ubuntu derivative Linux kernel problem. Regressing libusb did nothing. Reverting the kernel seems to have fixed the crashing problem. There are so few users of TyCommander running Ubuntu, that no one, save for me has reported the problem. I am working with System76 tech support to further isolate the issue. Kernel 5.15.0-37 which predates May 26 is fine, and has no problems. 5.17 (updated on May 26) crashes, and 5.18 (experimental) crashes but takes a little longer.

I just want to be able to use my computer and do development... By reverting the kernel, it was possible to get some work done. But that is a short term fix, so I need to help find the kernel bug, or at least isolate it enough for the kernel team to work on it.
 
Good luck @clinker8. Losing TyComm is a pain - having it take your whole machine offline is disaster.
Wonder if the build or lib tools @koromix uses are causing the trouble after some change. Not sure where he is having time to look at this - if it is something the kernel techs can see maybe they will point it out or fix on their end.
 
Believe me, this has been a royal pain in the neck. When there's a crash, all you get is a frozen screen, then 30 seconds later the laptop turns off. No chance to do anything, or save anything.

For the weekend, I did all my development on my RPI4 because it doesn't crash using TyCommander. With the tech support I got, we could track it down to the kernel. In Linux, nearly all requests to talk to the HW have to go through the kernel. The tech wasn't all that happy it was the kernel, because it is hard to make a minimal example to send off to the kernel developers. They don't want to deal with user applications unless necessary. I even offered to send my tech support a loaner Teensy 4.1 so they could duplicate the problem. I just bought one at Adafruit, who made me go through 2FA just to be able to buy it. Not surprisingly, the tech turned down my loaner offer. This means I have the brunt of the problem of doing the actual testing, with System76 guiding me.

Not what I signed up for, if you know what I mean. For some reason, I was dealt this hand, so I have to work with it.
 
Again sorry, I know I am not much help here, as I normally don't use TyCommander on linux.. 90% of the time I use windows...

I might try setting it up on my Ubuntu machine... Although have to setup again as have RPI4 connected to monitor instead of it...

Have you tried building a debug version of it? Did it make a difference?

Assuming this is an issue you created on Github: https://github.com/Koromix/tytools/issues/93

I might suggest opening another one with a more descriptive title:
Something like: Running TyCommander on Ubuntu 20.04 LTS & Pop-OS 22.04 machine hard crash

Or something like that, saying: there is no method to unistall,
may not have caught the Koromix's eyes.

You might also specify things like:

It crashes when I do???

a) Just receiving data from Teensy? One or more? How much data...
b) When I use it to try to program the Teensy?
c) ???

if it was only b) I would simply recommend don't use Tycommander to program the teensy. A lot of the time I don't... I use Teensy app. Or I use teensy_loader_cli

If a) then it would be good to try to localize what the issue is. But I know this could be hard to do. Areas that I might suspect include:

1) libusb changes, although if Teensy apps still talk to them fine, I would wonder.
2) QT based GUI - Wonder if maybe something changed with what library it may be calling? Wonder if this is static or dynamic linkage?
3) Memory - Memory leak? Or just asking for more memory than your system wants to give? Been awhile since I tracked something like that, but maybe have screen up with htop? maybe will show how much memory is being used on your system... Does it look like it runs out? Do you have swap space...

Again I know not much help. But...
 
Again sorry, I know I am not much help here, as I normally don't use TyCommander on linux.. 90% of the time I use windows...

I might try setting it up on my Ubuntu machine... Although have to setup again as have RPI4 connected to monitor instead of it...

Have you tried building a debug version of it? Did it make a difference?

Assuming this is an issue you created on Github: https://github.com/Koromix/tytools/issues/93

I might suggest opening another one with a more descriptive title:
Something like: Running TyCommander on Ubuntu 20.04 LTS & Pop-OS 22.04 machine hard crash

Or something like that, saying: there is no method to unistall,
may not have caught the Koromix's eyes.

You might also specify things like:

It crashes when I do???

a) Just receiving data from Teensy? One or more? How much data...
b) When I use it to try to program the Teensy?
c) ???

if it was only b) I would simply recommend don't use Tycommander to program the teensy. A lot of the time I don't... I use Teensy app. Or I use teensy_loader_cli

If a) then it would be good to try to localize what the issue is. But I know this could be hard to do. Areas that I might suspect include:

1) libusb changes, although if Teensy apps still talk to them fine, I would wonder.
2) QT based GUI - Wonder if maybe something changed with what library it may be calling? Wonder if this is static or dynamic linkage?
3) Memory - Memory leak? Or just asking for more memory than your system wants to give? Been awhile since I tracked something like that, but maybe have screen up with htop? maybe will show how much memory is being used on your system... Does it look like it runs out? Do you have swap space...

Again I know not much help. But...

Debug build did not help. Same problem. Yes, I should cancel that bug, and write a more descriptive one. At the time I wrote it, I was in a panic, kernel and personal! I have ensured I have backup of everything, and have almost everything now on two different computers. It's a heck of a lot easier to be calm that way.

It seems it only happens sometime after programming Teensy. For the 5.17 kernel (May 26) it happens very shortly after flashing. For the 5.18 kernel, it takes its sweet time, sometimes happening 5 minutes later (when I am doing something else, like looking something up in Firefox). For the 5.15 kernel (pre May 26) it never happens. libusb version doesn't seem to matter which version I used.

My laptop has 32GB RAM, I never see it loaded up. There could be a memory leak in TyCommander, but I don't understand cmake enough to ensure both debug mode and turning all optimization off. As I understand it, to use memory leak tools you need to have -O0 for optimization. I never see any apparent running out of memory, at least using gnome system monitor, maybe 5GB on a 64bit OS. Not sure about swap, since it's been a while. Don't know if modern Linux uses that much any more. Edit: My swapfile is 4GB. So it won't support hibernation, to support that it is recommended to use 1.5 x total RAM, or 48GB. I haven't used hibernation, so I don't know if it is a problem.

When I typed: $ sudo swapon --show, I got size = 4GB, used = 0B, priority -2. I don't know if that is important or not.

TyCommander uses a static version of QT, or at least I think that's what the docs say.

Thanks for your suggestions. This has definitely been a tough one to work through. You know you are in trouble when tech support is telling you to try unreleased kernel builds (for the OS) for fault isolation. I am darn lucky that I bought my laptop from System76, and am using PoP!OS. At least I can get some technical support.

As far as I can tell, the crashes only occur if TyCommander has been used during the session. I have booted into the bad kernel and done other things, for hours at a time, with no problem. Something that TyCommander does is upsetting the kernel. This occurs when TyCommander is integrated with Arduino. I have not tried it without the integration to Arduino.

With kernel 5.15, TyCommander works fine. I can do my work quickly and efficiently on my laptop, it's way faster than a RPI4. I'll keep on plugging at this, because that's my only option.
 
Last edited:
How do I break the integration of TyCommander with Arduino?

My kernel hack no longer is available, unless I run a VM. I have to effectively uninstall tycommander, and break any integration there may have been with Arduino.
Had to do a full-upgrade and my kernel was removed. Now my laptop just shuts down if I program the Teensy.

New kernel is:
$ uname -a
5.18.10-76051810-generic

I programmed my Teensy4.1 at 16:09:35. At 16:10:43 my laptop power went off, due to some unknown reason - but I am blaming the kernel. I cannot function this way anymore.
 
How do I break the integration of TyCommander with Arduino?

My kernel hack no longer is available, unless I run a VM. I have to effectively uninstall tycommander, and break any integration there may have been with Arduino.
Had to do a full-upgrade and my kernel was removed. Now my laptop just shuts down if I program the Teensy.

New kernel is:
$ uname -a
5.18.10-76051810-generic

I programmed my Teensy4.1 at 16:09:35. At 16:10:43 my laptop power went off, due to some unknown reason - but I am blaming the kernel. I cannot function this way anymore.
How do I get the TeensyLoader back? TyCommander is the kiss of death to my laptop :( Do I have to reinstall Teensyduino?
 
Well, I reinstalled Teensyduino. Glad to be rid of the curse of TyCommander. Won't be as nice, but for crying out loud, black screens and shutting down your computer are not worth it.
I did sudo rm tycommander in the /usr/local/bin directory as well. Also removed tycmd and tyuploader.

Tested it out and so far so good. Will keep you posted. If I get brave, maybe I will try this in a VM.

Err, not so good. Can't access the device. What do I use for the USB? Just the default Serial?
The port selection is greyed out. Because of this I cannot get board info. Is this normal?
I cannot get the serial monitor to work. The error is:

Arduino: 1.8.19 (Linux), TD: 1.56, Board: "Teensy 4.1, Serial, 600 MHz, Faster, US English"
Board at /sys/devices/pci0000:00/0000:00:14.0/usb1/1-1/1-1.1/1-1.1.4 is not available

This report would have more information with
"Show verbose output during compilation"
option enabled in File -> Preferences.

udev rules are installed. I get an error uploading the sketch. It instructs me to press the button. How long do I need to press the button. I pressed it for a second or so. The red programming light is sort of blinking, two seconds on and about 1 off, but not the usual thing when programming. I guess this install is borked. Darn, it is hard to recover from this stupid TyCommander. Will start all over again. What a freaking waste of time...
 
Last edited:
You must be bdlabitt@github ... I posted there with reinstall as an option.

... bummer the linux version in use upgrade breaking TyCommander ... it will be missed ...
 
How frustrating... Now I cannot program my Teensy4.1. Which is running right now.

Arduino: 1.8.19 (Linux), TD: 1.57, Board: "Teensy 4.1, Serial, 600 MHz, Faster, US English"

/home/bruce/Apps/arduino-1.8.19/hardware/teensy/../tools/teensy_post_compile -file=ELS001.ino -path=/tmp/arduino_build_606651 -tools=/home/bruce/Apps/arduino-1.8.19/hardware/teensy/../tools -board=TEENSY41 -reboot -port=/sys/devices/pci0000:00/0000:00:14.0/usb1/1-1/1-1.1/1-1.1.4 -portlabel=(null) -portprotocol=(null)
No Teensy boards were found on any USB ports of your computer.
Please press the PROGRAM MODE BUTTON on your Teensy to upload your sketch.
An error occurred while uploading the sketch

I push the button and I get the odd red blinking light by the USB connector. If I power cycle the board, the app I wrote comes up and runs. But I can't program the device. Well, that is no good... :confused:

Just don't know what to do next. Raises hand looking for help.
 
Above issue was a bad cable. Have no idea why it went bad, but it did. It worked for 5 months, most of the time attached. New cable installed.

TyCommander and its friends are removed. 1.8.19 is installed. 1.57 Teensyduino is installed.
My laptop crashed after programming my Teensy with Teensyduino. But it took 10 minutes for it to happen.

I can't even comprehend this. If I don't program a Teensy my laptop is fine - like all day fine. It crashes only when programming Teensy and not right away. The long delay is making this incredibly difficult to solve.
 
Can't prove anything at this point. All I know is I got a crash using a standard Arduino 1.8.19 install and Teensyduino 1.57. Before, I was using Teensyduino 1.56 (and got crashes with TyCommander). I have never had crashes of any sort programming an Arduino, or an M4. (Adafruit M4 Feather Express, 32u4, M0 Feather Express).

Does anyone know what is used to test the usb programming function for the Teensy? Is there a stress test for it? Wondering if there is a smaller code package that could be used than running the whole Arduino package. Honestly don't know if this is a Linux kernel issue, my issue, my laptop hardware, Teensyduino, or something else. Thinking of something closer to a minimal example, rather than some enormous pile of code. Just thinking out loud.
 
With a saved .HEX file - restart the system.

Open Teensy.exe the loader, and use the menu to open the hex file.

Plug in the Teensy and push the Button.

Perhaps open Help / Verbose info to watch Teensy captured traffic.

Wait for crash ... or push button on some interval.

This would be the minimal base upload interaction.
 
Back
Top