Looking for some clues hard Linux crash using Ty Commander

Sorry I am not Paul...

But as for USB Controllers? I am not an expert, but I know on my older machine (now being replaced by my not as older machine), it had a USB 3 controller in it, that was not
100% complete to the released standard. It was to a pre-release standard... And I had some devices/software that never quite worked correctly, so I ended up replacing that board with a new controller and that helped on that machine... (The main thing that did not work well was the Saleae Logic Pro...)
 
Sorry I am not Paul...

But as for USB Controllers? I am not an expert, but I know on my older machine (now being replaced by my not as older machine), it had a USB 3 controller in it, that was not
100% complete to the released standard. It was to a pre-release standard... And I had some devices/software that never quite worked correctly, so I ended up replacing that board with a new controller and that helped on that machine... (The main thing that did not work well was the Saleae Logic Pro...)

Yes, I know he is busy. Unfortunately, I am busy with getting this stuff to work without it dumping its bits all over me!

Wondering if there is a USB controller test of some sort that doesn't require cooperating hardware. Sounds like a hard thing to do actually.

I'm trying to sort out if 1) it is a kernel issue and how can I prove it with a minimal example, or 2) it is a USB hardware controller issue and how I could prove it. Or rule out 1 and 2! I'm lost here at this point.

I might try development via a remote platform. So far the RPI4 seemed to run all the tools without exploding. Have to say, it seems sad there's no apparent way to sort this kind of problem out on my laptop. Definitely a convoluted mess, and far beyond my pay grade.
 
Remote execution of Arduino and TyCommander on an RPI4-64 bit works ok, albeit slowly. Seems ridiculous to me that a sub $100 computer and SSD run the code just fine and a $1500 laptop with much fancier HW can't. Have to say productivity will suffer. The graphics and menu selection are a LOT slower, like 5-10X slower. But working slowly is better than crashing without notice.

One thing I do note, is there are a couple of libGL errors on the RPI4 console screen.
Code:
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
Don't recall ever seeing them before. This is because of running remotely. The fix is apparently to install mesa-utils and optionally libgl1-mesa-glx on the local computer, not the remote RPI4.
 
Last edited:
Seems ridiculous to me that a sub $100 computer and SSD run the code just fine and a $1500 laptop with much fancier HW can't. Have to say productivity will suffer.
...
I am wondering if I just have a bad usb controller, or one that is not fully compliant with the standards - or even a counterfeit device.

Forgive me, as I haven't kept up with every message (been really distracted by supply chain issues), but is this problem confined to just 1 laptop computer?



Regarding Arduino CLI:
Oof, looks like a bit of work to get that going, but it's doable. One quick question, is it effectively a complete new install unto its own? (Independent of any existing installation?)

When you originally worked on this Paul, how did you test for usb compatibility? This is multi-platform, so I am sure you had to go through a lot of work. Is there a usb3 compatibility test anywhere?

Today Teensyduino support for Arduino CLI is compile only. So you could use Arduino CLI instead of a makefile (I believe that was the question I tried to answer with msg #48) and then use teensy_loader_cli to do the upload.

Eventually I will integrate the teensy_loader_cli code into teensy_post_compile which is called by Arduino CLI. It will not get major changes. If teensy_post_compile isn't working on your PC, you can expect the same behavior from future teensy_post_compile used by Arduino CLI.


Regarding USB compatibility testing, I mainly test Teensy with 2 computers, my Linux desktop machine (a generic white box build with Intel Coffee Lake generation processor) which has all USB3 ports, but both USB2 and USB3 hubs (I regularly use both), and with a 2015 Macbook Air which has USB3 ports. I test Windows 10 by rebooting that Macbook Air to a Bootcamp partition. These are the primary "tier 1" machines I use for most testing. Typically Teensy is used directly with a USB3 port on the Macbook Air, or with a hub plugged into a USB3 on the desktop PC.

I have a group of "tier 2" machines, a 2011 Macbook Pro with USB2 ports, a Raspberry Pi 4, Jetson Nano, and "trashcan" 2013 Mac Pro. I really don't pay much attention to which of these has USB3 vs USB2 ports. The 2011 Macbook has a bootcamp partition with saved images for Windows 7 and Windows 10. Those windows partitions also have the USB-IF command compliance test program. I rarely test with these 2nd tier machines, unless someone reports a specific problem. But I do make a point to run the USB-IF compliance test after editing the USB code.

I also have a Beagle 480 USB protocol analyzer. Again, I rarely go to the trouble to use it, unless a specific problem is reported, or when I'm specifically working on the USB code. The protocol analyzer software has extensive decoding for most higher level USB protocols and I do watch for it to flag anything as an error.

Every Teensy model has been tested with all these machines many times over the years. I've dealt with all sorts of USB quirks and issues. But I've never had the sort of crashing problems you're reporting. We've had a pretty large user base for Teensy over many years, large enough to run into all sorts of pretty obscure issues, but this crashing problem really sounds unique. If this is limited to just 1 PC, that's a pretty strong indication something is probably wrong with your PC hardware that needs to be repaired or replaced.
 
[mention]PaulStoffregen[/mention] yes, it is one computer. I only own one "real" computer. It is only 8 months old. i7 32GB RAM, 1 TB NVMe SSD, relatively modern. System76 with PopOS 22.04 installed on it now. Originally it had Ubuntu 20.04, installed by System76. PopOS is an Ubuntu derivative and maintained by System76.

I asked about your testing methodology to get an idea about the extent. It certainly seems you have put in your due diligence. Honestly didn't expect anything different. Do you have an idea of the linux user base size for Teensy? Trying to get an idea of the population size.

I strongly suspect it is hardware, but the sudden onset is quite curious At this point I don't know how to get it repaired as I really don't know how to describe exactly what is wrong. My power supply, the USB controller chip? I don't even know how System76 would test it. I only can cause it by repeatedly programming a Teensy4.1 (that is all I have). The laptop never dies otherwise. I have offered to send them a spare Teensy and have them use it, but so far, they have yet to take me up on it. I don't know who else could repair and diagnose this, do you or anyone else know someone that could take this on?
 
Do you have an idea of the linux user base size for Teensy? Trying to get an idea of the population size.

Best I can offer you the number of times the 7 Teensyduino software files meant for Arduino 1.8.x use have been downloaded from the PJRC website over the last 14 days (our server retains only 2 weeks of log files).

Code:
5140    TeensyduinoInstall.exe  
2618    Teensyduino_MacOS_Catalina.zip
 126    TeensyduinoInstall.dmg
 960    TeensyduinoInstall.linux64
 129    TeensyduinoInstall.linux32
 196    TeensyduinoInstall.linuxarm
 173    TeensyduinoInstall.linuxaarch64
----    --------------
9342    Total downloads

The 4 Linux files add up to about 15.6% of the downloads. However, a quick look at the files shows about 100 of those downloads are with "Wget" user agent, so a chunk of the Linux downloads might be automated scripts rather than real humans.

Edit: caveat, this isn't any sort of good analysis, just the raw number of lines in the web server log for each file name. Sometimes a browser retries. A quick look at the data seems like this happens pretty dramatically for some Windows downloads, sometimes logging 20 to 50 files for the same IP number spanning a few minutes until the user until got the whole thing downloaded. The Windows downloads are probably pretty significantly overestimated by this too-simple count.
 
Last edited:
Was hoping to use a virtual machine to isolate me from system crashes. Programming my Teensy seems to cause system crashes which turn off my laptop power supply. The OS leaves no significant record of what happens during the crash. Just a usb transaction and some random time later the computer turns off. It is unacceptable for this to happen.

I have to abandon something, either Teensy or my laptop, since they just aren't compatible anymore. Seems like a bad choice to have to make. I have spent (wasted) well over 100's of hours on this and am very discouraged. The Teensy is a great platform, but this puts a large damper on my enthusiasm.

@clinker8 / @PaulStoffregen:

Please see <this> thread where I describe successfully programming a Teensy 4.1 from within an Ubuntu VM running under a Windows host. I haven't tried the Ubuntu VM running under a linux host, but I may play with that next week if/when I get some spare time, just for comparison.

Mark J Culross
KD5RXT
 
Last edited:
@clinker8 / @PaulStoffregen:

Please see <this> thread where I describe successfully programming a Teensy 4.1 from within an Ubuntu VM running under a Windows host. I haven't tried the Ubuntu VM running under a linux host, but I may play with that next week if/when I get some spare time, just for comparison.

Mark J Culross
KD5RXT

I used QEMU/KVM aka virt-machine and it was not successful. However, I was not aware of all the USB ID flipping about going on. It seems that as part of normal operation, the USB port is opening and closing a lot and connecting to different addresses. My virt-machine doesn't seem to automatically handle this the same way as the real machine. It seems virtualbox doesn't either and requires assistance.

I don't want to give the impression that the vm and real system always crashes if a connect is made. It only did this once. I don't know if I did something wrong or not. I know the Teensy gets hung up in the bright red light mode, which is the programming mode. For all I know, all I need to do is "attach new HW" at the point. I don't have a map of all the usb states and addresses needed.

On another front, I found that 5.15.0-25 was the release kernel for PoPOS and it will be maintained for the life of the long term release. I will try to revert the kernel to 5.15.0-25 and see if the problem is still there. If this works, I am set for about 4 years. Beats me what will happen after that point.

Can you explain what #7 in your list means? I don't know when this might happen? Does this happen normally? Or are you being thorough?
 
Can you explain what #7 in your list means? I don't know when this might happen? Does this happen normally? Or are you being thorough?

From the other post with the details of successfully programming a Teensy from within an Ubuntu VM running under a Windows Host:
Code:
7) Note that following the successful execution of the 15-second factory restore procedure,
    the T4.1 device will show up in the VBox host menu as Devices / USB / Teensyduino RawHID [0280]
    { NOTE: after executing the claim, when the mouse is hovered over this device,
    it shows Vendor ID: 16C0, Product ID: 0486, Revision 0280, Serial No. 8645610, State: Captured }

With the Teensy 4.x devices, a "recovery" capability was added which allows you to return the T4.x to the "factory configuration" (NOTE: only applies to the "unlocked" T4.x...search the forum for discussion/details on why this is the case, if you are interested). This only happens when you follow the "15-second factory restore" procedure as described in the "Memory Wipe & LED Blink Restore" topic specifically, on <this> PJRC page. There you will find the full details for what this recovery capability does & how it is initiated. This section also contains lots of other detailed info on how programming the T4.x is actually achieved, which may help you to better understand the different states of the red LED that you are observing.

Hope that helps . . .

Mark J Culross
KD5RXT
 
Last edited:
Back
Top