teensy_cli_loader fails midway through .hex file upload

S

snelsond

New member
Hi! I'm working in a university robotics lab right now, and we've set up a Raspberry Pi 4 running headless Ubuntu 22.04 LTS connected by USB to a Teensy 4.1 as the control hardware on an underwater robot we're developing. We've been sshing into the RP4 and using Platform.io and the teensy_loader_cli to do remote development and uploads at times when we can't access the hardware (e.g. sealed up for testing in a lake or pool), and it's worked great.

Lately, however, the teensy_loader_cli has started failing (for the most part, we've had a few successes) .hex file uploads midway through for one particular program (our main one). It compiles fine, and changing the code doesn't appear to help -- it's all just a hex file anyway, right? We've done some digging into the code and messed around with different teensy_write() wait times based on some forums we've read and tested with some smaller .hex files -- they upload fine, it just seems like we're running into problems with this particular program. We identified the error as "-32" (broken pipe) coming from the usb_control_msg() call in teensy_write(), but debugging that is a bit farther into Linux than any of us have gone before.

There a couple things we think might be the problem, but we were hoping to see if we could get some more experienced guidance before diving in:
(1) We have added a substantial amount of code to our project, but the verbose option on the loader says we're at 1.8% capacity -- could that number be wrong and our program is too big?
(2) We're overclocking the teensy right now (using the platformio.ini file) to 800000000 Hz, but it's been working fine like that for the past couple of weeks. Could we have messed up the clock somehow?
(3) We're powering the teensy from the pin on our Raspberry Pi 4, which is also running a wireless access point, which takes some significant amps -- maybe we don't have a good enough power supply to last through a larger upload? We did test uploading the file with the GUI uploader on a desktop linux computer, and it didn't work there as well.
(4) Something else we're missing?

Thanks for your help in reading this post and pointing us in the right direction! I'll attach the .hex file.
 

Attachments

  • firmware.hex.txt
    582.4 KB · Views: 23
snelsond said:
Hi! I'm working in a university robotics lab right now, and we've set up a Raspberry Pi 4 running headless Ubuntu 22.04 LTS connected by USB to a Teensy 4.1 as the control hardware on an underwater robot we're developing. We've been sshing into the RP4 and using Platform.io and the teensy_loader_cli to do remote development and uploads at times when we can't access the hardware (e.g. sealed up for testing in a lake or pool), and it's worked great.

Lately, however, the teensy_loader_cli has started failing (for the most part, we've had a few successes) .hex file uploads midway through for one particular program (our main one). It compiles fine, and changing the code doesn't appear to help -- it's all just a hex file anyway, right? We've done some digging into the code and messed around with different teensy_write() wait times based on some forums we've read and tested with some smaller .hex files -- they upload fine, it just seems like we're running into problems with this particular program. We identified the error as "-32" (broken pipe) coming from the usb_control_msg() call in teensy_write(), but debugging that is a bit farther into Linux than any of us have gone before.

There a couple things we think might be the problem, but we were hoping to see if we could get some more experienced guidance before diving in:
(1) We have added a substantial amount of code to our project, but the verbose option on the loader says we're at 1.8% capacity -- could that number be wrong and our program is too big?
(2) We're overclocking the teensy right now (using the platformio.ini file) to 800000000 Hz, but it's been working fine like that for the past couple of weeks. Could we have messed up the clock somehow?
(3) We're powering the teensy from the pin on our Raspberry Pi 4, which is also running a wireless access point, which takes some significant amps -- maybe we don't have a good enough power supply to last through a larger upload? We did test uploading the file with the GUI uploader on a desktop linux computer, and it didn't work there as well.
(4) Something else we're missing?

Thanks for your help in reading this post and pointing us in the right direction! I'll attach the .hex file.
Click to expand...

1. No, that's probably right. The Teensy MicroMod I'm using has 16MB of flash and I believe the 4.1 does as well. So, with that amount of flash, it's entirely possible that you're using only 1.8% of it.
2. That is entirely possible. I have never overclocked over the 600MHz default. It is entirely possible that overclocking could make the USB flaky. The easiest approach is to try it at 600Mhz and see if stability improves.
3. Power fluctuations and dips could create issues as well. Have you tried using an oscilloscope on the power to see what it looks like?
 
Overclocking, especially without a heatsink, can be pretty stressful on the hardware. We have heard reports on this forum where a substantially overclocked Teensy 4.0 without heatsink eventually died.
 
Is the USB signal passing through multiple cables connected in tandem? If so, are any of them not proper USB cable?
 
Two notes:
  1. T_MM indeed has 16MB Flash - the T_4.1 has only 8MB Flash - which is twice the T_4.0's 4MB Flash.
  2. Overclocking at 816MHz is not marked as cooling required - which assumes good ambient cooling. OC'ing as p#3 notes "can be pretty stressful on the hardware". A new T_4.0 here accidentally swapped in for a prior board with heat sink set at 900+ MHz lasted only some few uploads and a couple of hours before it was acting up and the static "Speed at OC cooling req'd" was noted and the board was lost. Running at 800 MHz and in an "underwater robot" could easily run too hot for long life.
 
I would suspect option 3 - something particular about that HEX causing part of the flash to be erased/written that is causing a brown-out.
 
jmarsh said:
I would suspect option 3 - something particular about that HEX causing part of the flash to be erased/written that is causing a brown-out.
Click to expand...
Why would the content of the hex file matter? What is erased is not a function of the content of the hex file except for the size. As far as I've seen, the hex file is always "contiguous", i.e. it explicitly contains data for every address from start to finish. It doesn't leave blank spots and suddenly jump ahead to another address, or am I wrong about that?
 
joepasquariello said:
Why would the content of the hex file matter? What is erased is not a function of the content of the hex file except for the size. As far as I've seen, the hex file is always "contiguous", i.e. it explicitly contains data for every address from start to finish. It doesn't leave blank spots and suddenly jump ahead to another address, or am I wrong about that?
Click to expand...
Having dealt with a lot of flash devices, programming specific contents to specific locations can cause failures if the power supply isn't quite up to spec.
Programming *can* be skipped for some locations where the erased value already matches the value to be programmed (although I don't know if this is implemented in the Teensy's bootloader), so you can end up with very different patterns of power draw depending on the payload contents.
 
jmarsh said:
... Programming *can* be skipped for some locations where the erased value already matches the value to be programmed (although I don't know if this is implemented in the Teensy's bootloader), ...
Click to expand...
On each upload the T_4.x FLASH is formatted to posted minimum, or beyond as needed for larger HEX.
 
Update: Based on your suggestions, we've decided against and removed the overclocking config params on Platformio for future use, but we're still running into the same upload problem. We're using a single USB cable to connect the Pi to the Teensy and tested uploading with a shorter one today with no success (the shorter .hex files uploaded fine tho). We also tried powering the Teensy independent of the Pi (power cable in the USB snipped) and ran into the same problem. We did do some more extensive testing with other Linux devices and it seems to work great when uploading the same .hex file from other computers.

We're a bit stumped, but we did also recently look into some experimenting with the tytools library and messed around a bit with these dependencies:

sudo apt install build-essential cmake libudev-dev qtbase5-dev

Is it possible some of these libraries overwrote/aren't compatible with teensy_loader_cli in a way that wouldn't affect anything until we uploaded certain (or larger) .hex files?
 
You must log in or register to reply here.
Back
Top