Help with Teensy Loader

[rvh]

I have a large synth project (much too large to post code) that stores many constant arrays containing raw drum samples in FLASH:

FLASH: code:361728, data:7052072, headers:9172 free for files:703492
RAM1: variables:94496, code:358152, padding:2296 free for local variables:69344
RAM2: variables:500224 free for malloc/new:24064

It uses the audio library (with USB and analog I/O) as well as non DMA-based audio processing chain, and includes a touch screen, touch sensors, encoders etc and uses most of RAM2 for a stereo delay line.

I've tried the following with Teensyduino 1.58 and 1.59, and Arduino 1.8.19 and IDE 2.2.1, all on Windows 10 pro (sorry previously said 11).

Teensy loader reports about 91% used. When I try to upload the code to a T4.1 it works fine when the T4.1 had no previous version of this large project in flash. But if I try to do the same when the T4.1 already had a previous version, on the first attempt is shows "erasing" for about 20 sec, and then DOWNLAOD ERROR. But shows "programming" and DOWNLOAD COMPLETE on the second try, every time. The second try can be by either clicking the program button on Teensy loader or via the compiler. Either way the program runs fine after the second try and a reboot.

If I take out the largest drum sample bank to create a smaller version in FLASH, it works first time every time. Except if the larger version was still in the T4.1, then it fails once when erasing the larger old code, but is fine after that, and loads first time every time. So it seems the problem is with erasing the larger version from flash. Is there perhaps some sort of erase time-out in the loader that I'm exceeding?

One thing that puzzles me, is that I wrote a small piece of code that had some vey large dummy arrays, so the loader still reported 95% use, but it didn't have the same problem. It had far fewer arrays - just larger ones. Maybe that matters?

Also, with the problem code, I do need to press the Reboot button on the loader after the second load, which I don't need to do with the smaller version of the code - it boots automatically after the first successful upload. I doubt it makes any difference, but the T4.1 is powered from an external 5V regulator, not USB power. I've tried this both with locked and unlocked T4.1, and get the same result.

If this is something you've seen before, please let me know what the cause was in your case. I know that without code it's just guess work, but I would be grateful for any guesses at this point.

In case it is an erase time-out issue for example, is there a way of changing it's value, or should I be looking for something in my code that could cause this sort of behavior?

Thanks very much.

 

[CollinK]

This may not help but it is seemingly somewhat related:

I took the published teensy_cli_loader and fixed it up to correctly do automatic soft reboot on Windows and Linux (I copied someone else's solution for Windows so I don't want to really take credit for much of anything. I merely slapped patches together and compiled binaries). Anyway, I found that quite often with my Teensy MicroMod it would timeout when trying to write to the board. This turned out to be a timeout value which was too low for my needs. If I drastically increased the timeout then it would work. If I did not increase the timeout then it would fail the first time but work the second time. This sounds similar to what you are experiencing. So, yes, it is potentially a fix to increase the timeout. However, if you are using the GUI program then we don't have the source code for that. If you're using the cli program then the timeout can be changed and this might help you.

 

[rvh]

This may not help but it is seemingly somewhat related:

I took the published teensy_cli_loader and fixed it up to correctly do automatic soft reboot on Windows and Linux (I copied someone else's solution for Windows so I don't want to really take credit for much of anything. I merely slapped patches together and compiled binaries). Anyway, I found that quite often with my Teensy MicroMod it would timeout when trying to write to the board. This turned out to be a timeout value which was too low for my needs. If I drastically increased the timeout then it would work. If I did not increase the timeout then it would fail the first time but work the second time. This sounds similar to what you are experiencing. So, yes, it is potentially a fix to increase the timeout. However, if you are using the GUI program then we don't have the source code for that. If you're using the cli program then the timeout can be changed and this might help you.

Thanks very much - definitely sounds like it's worth a try. Is the Windows version available for download? Another plus might be that it could be an alternative for my end users having to use both Teensy Loader and TyCommander - the latter to force the bootloader to start when using ehex firmware updates, as that doesn't seem to work with the GUI Teensy loader. Although, some of them will be OS users. Do you have any plans to do the same for Mac?

 

[rvh]

Also, I'm not sure who authored the GUI version of the loader (presumably Paul?), but it would be ideal if the timeout value for that could be changed.

 

[PaulStoffregen]

Can you share the large .hex file? Really helps if I have a way to reproduce the problem. Also if you've only tested on 1 system, please tell me which one.

 

[rvh]

Thanks for the response Paul. Unfortunately I can't share the problem hex file as it stands, as it's commercially sensitive. But maybe I can modify it so it isn't functional but still produces the loader problem. If so, should I send you a dropbox link (as it will be very large)?
The system I've tested it on is Windows 10 pro (sorry not Windows 11)

 

[defragster]

Seeing this I went to the Code4Code example done to test on Beta Locking Teensy4 with TD 1.59.

Memory Usage on Teensy 4.1:
  FLASH: code:2959140, data:808368, headers:9000   free for files:4349956
   RAM1: variables:179680, code:45864, padding:19672   free for local variables:279072
   RAM2: variables:12416  free for malloc/new:511872

That was crafted to do code and data from FLASH - and fit on a T_4.0 as well.
Just bumped it up from 4,000 called func()'s to 10,000 and it got a bit bigger - but when running it ends up out of stack space. According to CrashReport - so after collecting working code from USB_2 - rebuilt without USB_2 and then it runs - still crashes.
> (3 DWORD params and 2 local DWORDS )* 10,000 call()'s added to stack eats 270KB it seems

Building it trashed IDE 1.8.19 - but that was up for some time and had many other open sketches.
Seemed to load and build okay in IDE 2 after update done to 2.3.1 - takes 5 minutes to build and upload.

The IDE 2 popped up 'Done uploading' while the Teensy Erase was still in progress - and before 'Programming' {46% Used} begins. That takes some time, does Reboot and the IDE GUI doesn't offer any feedback after that prior 'Done' message.

So, with this there is no 'Download Error'

Dropped to 8,000 Made code calls from 10K - still runs out of stack:

Memory Usage on Teensy 4.1:
  FLASH: code:2380092, data:651784, headers:8376   free for files:5086212
   RAM1: variables:151456, code:48936, padding:16600   free for local variables:307296
   RAM2: variables:24736  free for malloc/new:499552

But again on upload IDE 2 says 'Done' during initial erase and then says no more as the programming takes time needed to program 3MB.

 

[rvh]

Thanks Defragster. Maybe I misunderstand your example, but the problem I'm seeing relates to how much FLASH is taken up (mostly) by my large constant data arrays stored there, rather than the RAM2 use for the delay line. And it also needs to be very high usage for me to see the issue.

For example deleting my largest bank of samples from progmem still leaves pretty high usage (5.6M of samples in flash):

FLASH: code:361664, data:5589676, headers:8336 free for files:2166788
RAM1: variables:94432, code:358088, padding:2360 free for local variables:69408
RAM2: variables:506720 free for malloc/new:17568

But I have no probelms with this hex file (note RAM2 is pretty much fully used too here).

Also, it seems there's more to it than just the size because this totally different test program that I wrote to try to let otheres reproduce the problem, has a an even larger % usage (7.7M progmem data!) than the problem program, yet causes no trouble at all for teensy loader:

FLASH: code:14000, data:7715292, headers:9072 free for files:388100
RAM1: variables:5152, code:11304, padding:21464 free for local variables:486368
RAM2: variables:18176 free for malloc/new:506112

 

[PaulStoffregen]

Also, it seems there's more to it than just the size because this totally different test program that I wrote to try to let otheres reproduce the problem, has a an even larger % usage (7.7M progmem data!) than the problem program, yet causes no trouble at all for teensy loader:

Indeed I've previously tested with similar const-data hex files. This is why I asked for the .hex file to reproduce the problem.

 

[rvh]

I opened two new Teensy 4.1 boards and tested those today, as well as retesting two of the boards that I've been developing with, which both showed the problem I've mentioned in this thread. For the new boards, I cut the VCC/USB trace and soldered pins onto one so it could go into my custom PCB for testing under the same sonditions as the older boards, and left the other one unodified for USB powered uploading tests.

Upload tests were with stand-alone Teensy Loader (1.58) using previously compiled ehex files (Arduino not running). I also used TyCommander to use its 'Bootloader" button which seems required when using ehex files in TeensyLoader. I need this because my end users won't have access to the program button, but it would be better to have this functionality built into the Teensy Loader if possible.

The intersting result was that neither of the two new boards had any issues, no matter which program I uploaded. Wheras both older ones were as described in this thread. They've seen far more uploads and pogram variations, and possibly use of differrent EEPROM locations in case it's relevant. But I'm inclined to think it's more coincidental and my sample size is just too small.

If it's an easy thing to implement in a test version of Teensy loader, I'd be very curious to see whether simply extending the time-out makes a diffence. Maybe the current value happens to be marginal for my test case and by chance two of my four T4.1s were OK with it, and two weren't?

 

[rvh]

Indeed I've previously tested with similar const-data hex files. This is why I asked for the .hex file to reproduce the problem.

Sure, understood. I'll see what I can do, but so far the modifications I've made to allow me to share the hex have also alleviated the problem.

 

[PaulStoffregen]

If it's an easy thing to implement in a test version of Teensy loader, I'd be very curious to see whether simply extending the time-out makes a diffence.

I'm curious too. The situation with timeouts isn't just 1 simple number, but here's a copy of Teensy Loader with the longest timeout meant to cover the initial erase increased from 30 to 45 seconds.

https://www.pjrc.com/teensy/beta/teensy_loader_forum74506.zip

There is also a minor complication. PJRC's code signing certificate for Windows expired a few days ago. It appears the company we previously used (3 years ago) has either gone out of business or just isn't answering emails. Even worse, looks like middle of last year the rules for issuing code signing certs changed. But very few of the sites selling them have updated their info. How to get the new cert (which apparently now requires a hardware dongle) and how to integrate that into my cross-compile build process inside a virtual machine is still a big unknown. It's yet another issue on my long to-do list, but not really urgent as we just released version 1.59 and I wasn't planning to package up new beta releases for at least a month. But then this thread came along...

Just know this executable isn't signed like I normally do. If you get angry message from you browser or Windows Defender or other lame anti-virus software, you'll have to deal with it somehow.

 

[rvh]

Excellent, thanks very much Paul!

45 seconds seems to do the trick for me. No errors so far with any of the T4.1 boards I've been testing, and the program auto-boots after upload every time. Do you happen to also have a MAC/OS version that I could try sometime? Will you make this publicly available once you've sorted the certification issue?

Also, I know your to-do list is perpetually overloaded, but any chance of integrating something like TyCommander's Bootloader function into Teensy Loader, rather than having to use TyCommander for ehex files?

 

[defragster]

integrating something like TyCommander's Bootloader function into Teensy Loader, rather than having to use TyCommander for ehex files?

Not sure what this means? TyCommander cannot upload .eHex - it has a Bootloader GUI button that will invoke the Teensy.exe loader program without a physical Button press?

Perhaps that is what is desired? A way to have Teensy.exe when not in Auto Mode enable the 'Manual Gui' button for "Program" and "Reboot" without a physical Button push? The unique thing about TyCommander in that case it is allows 'selection of unique USB port' to perform the 'Bootloader' GUI button on. Teensy.exe only 'knows' the desired Teensy when it is already waiting with a Button holding it in Bootloader {or when TyCommander does that} - or on command from the IDE telling it what specific device.

Thanks Defragster. Maybe I misunderstand your example, but the problem I'm seeing relates to how much FLASH is taken up (mostly) by my large constant data arrays stored there,

That example is specifically about FLASH based code
FLASH: code:2,959,140 and FLASH: code:2,380,092
It was contrived to test data and code running from FLASH on a LOCKED Teensy during the Beta test of those units.
Those examples have 2MB to ~3MB maintained in FLASH during execution, so not as much as your case as the Stack and a T_4.0 can't handle that much FLASH storage. But it did show that .eHex encryption could reliably run and decode during execution and only some small percentage of overhead could be observed versus execution from TCM RAM.
The Code4Code.ino 'writes code to USB2' output that works to create 4,000 calls calling calls when copied back into the build. But going beyond 4,000 the DEPTH of calling that many calls just happens to blow away the stack space.

 

[rvh]

Not sure what this means? TyCommander cannot upload .eHex - it has a Bootloader GUI button that will invoke the Teensy.exe loader program without a physical Button press?

Perhaps that is what is desired? A way to have Teensy.exe when not in Auto Mode enable the 'Manual Gui' button for "Program" and "Reboot" without a physical Button push? The unique thing about TyCommander in that case it is allows 'selection of unique USB port' to perform the 'Bootloader' GUI button on. Teensy.exe only 'knows' the desired Teensy when it is already waiting with a Button holding it in Bootloader {or when TyCommander does that} - or on command from the IDE telling it what specific device.

Yes, after I load an ehex file into Teensy loader, it doesn't automatically start the upload to the Teensy. It shows the notice to press the physical button on the Teensy, and disables both Program and Reboot options (regardless of whether Auto is on or off). But my end users won't have access to the physical button. I can force the Teensy Loader to start the uplaod without a physical button press at that point by starting TyCommander and pressing its GUI Bootloader button. But this means my end users need to download and run two different programs to update their firmware, whch is not ideal. Hence my request to integrate a similar function into Teensy Loader.

In a previous version of my application I didn't use a lockable Teensy so end users could just use TyUploader for the whole process, but it doesn't support ehex.

 

[rvh]

p.s. note that I'm referring here to stand-alone Teensy Loader use, without Arduino running. When called from within Arduino (after compiling), Teensy Loader starts the uplaod automatically, and with the extended 45 sec time-out I get no errors that way now too.

 

[defragster]

Yes, after I load an ehex file into Teensy loader

Okay - that makes sense. It seems that @luni (?) has made a version that can push .eHex? Someone did (??) - so that could be incorporated into TyComm if so - not sure of @Koromix having bandwidth to do that ... @KurtE has made some builds/edits to TyComm - not sure if it might be done as a pull request?

I see this that may be a path to try a custom app?: https://github.com/luni64/TeensySharp
ReadME notes out of date - not sure if it defers to Teensy.exe for upload?

 

[luni]

It seems that @luni (?) has made a version that can push .eHex?

I have successfully worked on a proof of concept to upload ehex files with TeensySharp. However, I haven't integrated it into the library yet. Anyway, the purpose of TeensySharp is to allow users to integrate Teensy detection and firmware upload from within a dotNet PC application. Looks like this is not the use case @rvh had in mind.

 

[defragster]

I have successfully worked on a proof of concept to upload ehex files with TeensySharp

Seemed I had seen it - thanks for confirming details! And good to know the .eHex process is otherwise doable.
It just might be that @rvh could make use of a custom app if a single program allowed his process to complete.

 

[rvh]

Yes, what I'm after is a simple standalone GUI program that can upload ehex files to T4.1 without pressing the button on the board, and can run on Windows or MacOS. I sent a message to Koromix last year asking whether there were plans to incorporate ehex abilities into TyUploader, but didn't hear back. Hence my request to modify Teensy Loader instead. I'm a bit surprised others haven't found the need for such a tool yet, as the locked Teensy has been available for a while now.

 

[KurtE]

Okay - that makes sense. It seems that @luni (?) has made a version that can push .eHex? Someone did (??) - so that could be incorporated into TyComm if so - not sure of @Koromix having bandwidth to do that ... @KurtE has made some builds/edits to TyComm - not sure if it might be done as a pull request?

I see this that may be a path to try a custom app?: https://github.com/luni64/TeensySharp
ReadME notes out of date - not sure if it defers to Teensy.exe for upload?

Not something I want to touch.

if i were easy to do then maybe Paul would integrate it into teensyloadercli and then others could then migrate it into different apps like TyCommander