'Over the Air' firmware updates, changes for flashing Teensy 3.5 & 3.6

Thanks for the clarification and all your work on this, this should work perfectly for my needs I just need to send the file over ethernet. Assuming I verify the program with either a CRC or sending back each line, is this sufficient to ensure the program is correct? Is there a way to verify the program after it has been flashed?

With Ethernet you have a CRC on every packet, so if you verify that packets received = packets sent, that's pretty good. A cumulative CRC would be better. Regarding verification after Flash, the first step is does it run. After that, you can compute the CRC of what's in Flash. You need logic to figure out where the end of the firmware is, or you can compute the CRC of a block that you're sure goes beyond the end of the firmware and then do the same thing in whatever client is sending the firmware to Teensy. I think I read somewhere that the program size is embedded in the code, but not sure if that's true for all Teensy.
 
With Ethernet you have a CRC on every packet, so if you verify that packets received = packets sent, that's pretty good. A cumulative CRC would be better. Regarding verification after Flash, the first step is does it run. After that, you can compute the CRC of what's in Flash. You need logic to figure out where the end of the firmware is, or you can compute the CRC of a block that you're sure goes beyond the end of the firmware and then do the same thing in whatever client is sending the firmware to Teensy. I think I read somewhere that the program size is embedded in the code, but not sure if that's true for all Teensy.

Great! After playing around with it I am able to read the program on the flash simply by reading a variable pointed to the flash address, I thought I would need a special function to read from the flash.. does the cpu abstract reading the flash in the background or cache it or something? Is this the correct way to read from the flash?
 
Great! After playing around with it I am able to read the program on the flash simply by reading a variable pointed to the flash address, I thought I would need a special function to read from the flash.. does the cpu abstract reading the flash in the background or cache it or something? Is this the correct way to read from the flash?

Yes, as far as I know it's okay to do that. I'm not sure of the terminology with respect to read, but my understanding is the processor "translates" accesses to QSPI flash or RAM into the proper QSPI transactions and provides the results as if they were in-line accesses. When you put a PSRAM chip on the bottom of your Teensy 4.1, you get a large RAM, and you can read/write in software, but it's actually doing all of the read/write via the QSPI interface. It's much slower than internal RAM, but fast enough for many applications. The processor can also execute code directly from the QSPI flash, which is called Execute-In-Place, or "XIP" for short. This is my super-simple understanding of how it works. There is a lot more to it, I'm sure. With the cache, and depending on what your code is doing, it can be reasonably fast. It's a pretty cool architecture.
 
Recently I started to have a weird issue with the FlasherX code. I've been using it to read firmware from the sdcard and flash it to a Teensy MicroMod. This has been working but recently it stopped working properly.

I've noticed that the code defaults to writing the new program to the top of FLASH (it figures out how large the current firmware is and writes the new copy past that point) then it stops everything and writes the program from upper flash to lower flash. This always used to work. Now as soon as the first line from the hex file that contains firmware is read the whole program dies. I've traced this seemingly to a call to flash_wait which bricks the processor now and causes an automatic reset. I've tried reverting my Teensyduino version but it doesn't help. I've tried 1.56 and 1.57b1 and 1.57b3 all work the same. Everything else in the program works fine. As soon as it tries to write to upper flash it dies a horrible death.

I then tried to buffer the program in RAM. This works because my program is about 326kb and I do have enough RAM free on the TMM to allow this amount to be buffered. It successfully buffers then dies in flash_move so once again as soon as flash memory is written to it dies. It should be noted that I can freely program this chip over USB so the chip isn't bad or anything. It works fine, it flashes fine so long as I use the PC based tools. As soon as it tries to write to itself it dies now and it never used to do that with, so far as I can tell, all the same code I was using before. I'm just not sure what else I could try....
 
Testing with T4.1 and T3.2 and it seems good. Can you revert to a version of your code where you know for sure it worked?

I grabbed a version I had compiled and that worked fine. That version can update even to the newest version when I place said file on the sdcard. I traced my git commits to the exact commit where it stops working. Oddly enough, the code changed between a version that always works and the next commit that never works is... well, nothing special at all. It has nothing to do with firmware flashing or flash memory at all. The only thing I can seem to correlate between these versions is that it got slightly larger as versions went on and at a certain size perhaps it started to act funny.

Here's a strange thing that happens as soon as I get to the commit where the problems start to happen:
1. I flash the sketch and it reboots.
2. I then power off and insert the sdcard
3. I reboot. It stops responding while saying it is waiting for hex lines (which it is getting from the sdcard)
4. Since it is stopped I power cycle it.
5. This time it loads the entire image then says "abort - new code missing string fw_teensyMM" but I can confirm the image does contain that string
6. Even more oddly, hereafter, every time I reboot this line "buffer = 7264K FLASH (600E4000 - 607FC000)" will show the buffer area shrinking by the exact size of my firmware image. It seems it is loading the firmware image to the flash buffer, figuring out that the ID doesn't match (though it does!) and then stopping but the buffer was still filled and does not get erased. So, every reboot the flash gets more and more filled but the move never happens.

Now, that behavior is repeatable with my program every time. If I try a given version of my code it either always works or never works. The code didn't change in ways that make any sense. I even tried commenting out anything that changed between the good and bad version of my code that I thought might even have the slightest chance of interfering. Now I'm left just with misc code that expands the size a little but doesn't do anything fancy or noteworthy at all.

With the newest version of my program the available buffer only drops by 4k each reboot. But, it might be a similar problem. This newest version of my code also automatically reboots itself so the fault is somewhat different.

I'm about to give up on it for now I think. I've sunk a lot of hours into trying to find the problem only to find that it sure used to work but magically doesn't anymore and for no good reason at all. I guess I'd just be curious if these sort of oddities happen to anyone else. I'm sure eventually I'll find the cause but I can upload the sketch the normal way perfectly fine so it's not worth the time any longer to keep digging. The good news is that the code found in my GEVCU7 repo still should work perfectly fine for anyone else who wants to do firmware updates from the SDcard. I didn't change anything for months and my older versions of the code can update perfectly fine from sd. So, odds are decent that it normally works and I've just hit some obscure corner case somehow.
 
Sounds sorta like a sneaky bug in the unrelated code. Writing to an array index that is negative or out of bounds or similar. Can cause all sorts of impossible to diagnose behaviors. I would go over everything between versions with a fine toothed comb and comment them out one at a time until it works again if there's nothing obvious. Couldn't tell you the number of times I've had to find my own mistakes this way.
 
I grabbed a version I had compiled and that worked fine. That version can update even to the newest version when I place said file on the sdcard. I traced my git commits to the exact commit where it stops working. Oddly enough, the code changed between a version that always works and the next commit that never works is... well, nothing special at all. It has nothing to do with firmware flashing or flash memory at all. The only thing I can seem to correlate between these versions is that it got slightly larger as versions went on and at a certain size perhaps it started to act funny.

I had no internet the last two days, so just able to respond now. Are you saying that the same FlasherX code was used before and after the changes to your own code? I think that's what you're saying, so I have to agree with @ipaq3115 as to how to trouble-shoot. The other thing that came to mind is there have been lots of changes to the SD library in the recent TeensyDuino releases, but you don't seem to be saying there was any correlation with updates to Arduino/TeensyDuino.
 
I finally found the problem. Two rules of thumb apply here:

1. 99.99999% of the time, when you think it isn't your fault... It's your fault
2. When you use pointers you should check to make sure they're not null first

Yep, the dreaded null pointer dereference. Took me forever to figure out how and why that happened.

Here's a line of code that caused the trouble:
Code:
if (sysConfig->logLevel > Debug)

That was found within the code that handles logging to the serial port. The code can obviously check whether the log level is proper and then makes decisions on whether to output a line of text or not based on that. So far, so good. It's worked for many years. However, this new code actually uses a pointer, as above. This is new. That pointer is defined pretty early in code. The I2C is initialized and then an EEPROM is read to fill the now defined class for system configuration. That all works fine. The problem here is that I had placed the code to do firmware updates even earlier because it's best if you don't have a lot going on when you decide to overwrite all of flash. So, the best place was early in start up just after hardware configuration of the sdcard. So, sysConfig was explicitly set as nullptr when the updater code was running. The updater code did NOT call the logging routines because I at least had the presence of mind not to use them yet in those early routines. However, I did accidentally place a line just before calling the routines that looked something like Logger::debug("Found teensy firmware! Flashing it!"); Now, that was a bad idea. A very bad idea. The frustrating part is that following the null pointer did not immediately break the code. It continued executing instructions but now things had gone sideways. It would still sometimes read the firmware but the firmware it read would have corruptions. Basically, the stack had been ruined and random things would happen based on exactly how the code was laid out. I could change behavior by compiling with different optimizations, etc. This may have been easier to find had I had any sort of hardware debugger but of course we don't have that on the Teensy 4 based designs. I really wish I could have used JTAG to step through. It may have helped. But, ultimately what happened is that I was blind to my own error and it cost me many, many hours of looking for a very simple bug.

Anyway, the solution was just to check pointers before using them. I had been lazy and just assumed things were OK and that the object exists and is allocated. Don't do that. Especially not with a 600MHz processor. It probably has the cycles to spare to check pointers before clobbering everything.
 
No, it was basically just a struct that gets filled out with configuration as soon as EEPROM comes up. It's just used by logger to check whether the user wants a given log to be displayed or not. The levels are off, errors only, warnings, info, debug. So, you can turn off stuff you don't want to see. I probably shouldn't have used a pointer for that. Had the struct just been allocated by default it would have saved me a lot of trouble. There were reasons it ended up as a pointer but it did mess me up.
 
I finally found the problem. Two rules of thumb apply here:

1. 99.99999% of the time, when you think it isn't your fault... It's your fault
2. When you use pointers you should check to make sure they're not null first

Man, if only I had a dollar for every time I've said pretty much the same thing... This would make a great poster! lol

I've learned not to blame anybody until I'm really sure it's not my fault. I find lots of bugs that way!

Also awesome that you posted the whole breakdown. Somebody will read this one day and learn something.
 
Hi,

there's a way to flash teensy using binary elf instead ?
The hex update is 400k the binary is half size, 200k.
I've a low bandwidth (I updated it using a modem) and I want to speed up it.

Any hint ?

Thank you,
Lude
 
there's a way to flash teensy using binary elf instead ? The hex update is 400k the binary is half size, 200k.
I've a low bandwidth (I updated it using a modem) and I want to speed up it.

I can think of a few ways of making the transfer faster. If you are using UART, then a higher baud rate is faster, but I’m sure you know that. There are many different file formats for firmware files, such as bin, s3, etc. Binary formats are more efficient in terms of total bytes. It also helps a lot to have more data bytes per record. You could search for a program that reads a hex file and creates a new one with the same data but with longer records. That would be more efficient and also have the advantage of not requiring changes on the Teensy side.
 
Using the ELF file would probably be a pain because that's a full-scale binary suitable for use on UNIX systems. It has all sorts of additional data and fields you don't really need or want to parse. In reality, you're basically going to be putting the binary data right at the start of the flash space every time so you know for sure what the starting address is. You don't really need to store that, I don't think. So, a raw binary would seem to be suitable. It doesn't seem like the Teensy build chain really creates a raw binary but I'm sure you could make one either from the HEX or ELF file. This would be the smallest thing you could send to the processor unless you compressed the binary first then decompressed it on the fly. That's probably too complicated to be worthwhile. Ultimately this over the air flashing code parses the hex file into binary so skipping that step and directly having the binary should work fine and be easy enough to code up. I didn't bother because I'm loading from an sdcard and the read speed of an SDCard is sufficiently fast that I don't care that hex files are extra big.
 
It doesn't seem like the Teensy build chain really creates a raw binary but I'm sure you could make one either from the HEX or ELF file. This would be the smallest thing you could send to the processor.

@CollinK, this is exactly what I do for my applications. My Windows host program reads the HEX file, converts to a BLK (binary) format and sends that to Teensy. It's about as efficient as it can be without using compression, and I agree that's not called for.
 
@CollinK, this is exactly what I do for my applications. My Windows host program reads the HEX file, converts to a BLK (binary) format and sends that to Teensy. It's about as efficient as it can be without using compression, and I agree that's not called for.

Thank you both, it was very easy.
I need to add some checks (size and crc32) before flash but it works!
Just to clarification, to create the raw binary I use the teensyduino toolchain:

I get the folder where elf and hex are created from arduino ide then I run:

Code:
arm-none-eabi-objcopy -O binary -S -R .eeprom "foo.ino.elf" "foo.ino.bin"

Thank you again,

Lude
 
Just to clarification, to create the raw binary I use the teensyduino toolchain. I get the folder where elf and hex are created from arduino ide then I run:

Code:
arm-none-eabi-objcopy -O binary -S -R .eeprom "foo.ino.elf" "foo.ino.bin"

Excellent. Thanks for documenting how to create a "bin" file.
 
Hi all,

Has anyone gotten FlasherX to update a Teensy 4.x from the SD card?
Is there a size limit to the size of the HEX file? Currently my program uses roughly 33% of Flash memory on a Teensy Micromod.

I've been searching and searching but can't find one place with all the needed info. One post asking about a similar approach was referred to a test sketch but no link was provided.

If anyone can provide insight it would be highly appreciated
 
Has anyone gotten FlasherX to update a Teensy 4.x from the SD card? Is there a size limit to the size of the HEX file? Currently my program uses roughly 33% of Flash memory on a Teensy Micromod

Hi Rezo. I seem to remember people saying they have done it, but I don't think anyone has posted a working sketch. I have tested FlasherX on Teensy Micromod, and I have tested with very large hex files sent via USB, so I'm sure it will work. The size limit will be slightly less than 50% of the flash size. My suggestion is to start from the USB serial example and replace the code that receives and parses hex records via USB with similar code that reads and parses hex records from an SD file. The processing of each record will be identical. All of the buffering and everything else will be the same. When the end of file is reached, use the same strategy of writing the number of records to the monitor and asking for confirmation, just to be sure you don't write anything you know is obviously wrong.
 
FlasherX v2.3 with support for update from hex file on SD card

  • run-time option for update via USB serial or hex file on SD card
  • separate streams for hex file input (serial or SD) and user interaction
  • hex file functions moved from ".ino" file to new files FXUtil.cpp and FXUtil.h

https://github.com/joepasquariello/FlasherX

FlasherX is still just a single sketch, but with more options available, my plan for the next release is to publish it as a library with examples of the various transfer method.

SD file update has so far been tested only on T3.5 and T4.1 so far, with chip select BUILTIN_SDCARD.
 
  • run-time option for update via USB serial or hex file on SD card
  • separate streams for hex file input (serial or SD) and user interaction
  • hex file functions moved from ".ino" file to new files FXUtil.cpp and FXUtil.h

https://github.com/joepasquariello/FlasherX

FlasherX is still just a single sketch, but with more options available, my plan for the next release is to publish it as a library with examples of the various transfer method.

SD file update has so far been tested only on T3.5 and T4.1 so far, with chip select BUILTIN_SDCARD.

I've tried using this for T4.1, Serial, for now just over USB Serial, but I'm stumbling over some issues.
Issue materialises already in the example .ino. Using a bare Teensy41.
Using TeraTeam for the hex file uploads.
Issue I think is with the Intel hex file lines ':02' record type that set base address. Those result(?) in a premature reboot. Once rebooted, it restarts upload over serial because it sees '1' characters in the iHex file that it gets bombarded with.

First time I tried it all worked ok.
But subsequent attempts fail. I suspect that is because the first time it uses a RAM buffer, while subsequent attempts decide to use the FLASH buffer. I'm unclear on what that flash buffer actually is. Like is it the external flash chip or not?

Output looks like this (nb i added some debug printfs just to see what some variables were at, and I gave it extra time before REBOOT so that I could see the abort messages in TerTerm) :

Code:
FlasherX v2.3 - Dec  9 2022 15:32:37
WARNING: this can ruin your device!
target = fw_teensy41 (8192K flash in 4K sectors)
created buffer = 0K FLASH (607FC000 - 607FC000)
enter 1 for hex file via serial, 2 for hex file via SD
reading hex lines...
:0200000460009A
:100000004643464200000156000000000103030081
abort - max address 60000010 too large
abort - FLASH_BASE_ADDR= 60000000 abort - buffer_size = 00000000
erase FLASH buffer / free RAM buffer...
rebooting in a fes seconds now..
FlasherX v2.3 - Dec  9 2022 15:32:37
WARNING: this can ruin your device!
target = fw_teensy41 (8192K flash in 4K sectors)
created buffer = 0K FLASH (607FC000 - 607FC000)
enter 1 for hex file via serial, 2 for hex file via SD
enter 1 for hex file via serial, 2 for hex file via SD
enter 1 for hex file via serial, 2 for hex file via SD
enter 1 for hex file via serial, 2 for hex file via SD
enter 1 for hex file via serial, 2 for hex file via SD
enter 1 for hex file via serial, 2 for hex file via SD
enter 1 for hex file via serial, 2 for hex file via SD


The issue seems to be that 'buffer_size' is zero. I think that's because I'm using a Teensy4.1 and the buffer is (has become?) Flash, not RAM. But I might be wrong here. Maybe buffer_size should have been set to FLASH_SIZE??
But if i force buffer_size to FLASH_SIZE, then the reading of the iHEX file goes fine, but I get a reboot while the Teensy is busy searching for a ID string in (check_flash_id( buffer_addr, hex.max - hex.min ). With no code changes apparently because it reboots as before into the same old program.

I'm lost now. Were we not already writing into the Teensy external flash chip? So why still bother about that check? And why ask user for a confirmation before moving from buffer into flash, if the buffer was that flash already?
 
I've tried using this for T4.1, Serial, for now just over USB Serial, but I'm stumbling over some issues. Issue materialises already in the example .ino. Using a bare Teensy41. Using TeraTeam for the hex file uploads. Issue I think is with the Intel hex file lines ':02' record type that set base address. Those result(?) in a premature reboot. Once rebooted, it restarts upload over serial because it sees '1' characters in the iHex file that it gets bombarded with.

First, what is your platform, IDE, versions, etc? FlasherX works for all types of ihex records, pretty well tested.

First time I tried it all worked ok. But subsequent attempts fail. I suspect that is because the first time it uses a RAM buffer, while subsequent attempts decide to use the FLASH buffer. I'm unclear on what that flash buffer actually is. Like is it the external flash chip or not?

If you want your buffer to be in RAM, just set the RAM buffer size to non-zero value, say 256*1024. That works.

Code:
created buffer = 0K FLASH (607FC000 - 607FC000)

The issue seems to be that 'buffer_size' is zero. I think that's because I'm using a Teensy4.1 and the buffer is Flash, not RAM. But I might be wrong here. Maybe buffer_size should have been set to FLASH_SIZE?? But if i force buffer_size to FLASH_SIZE, then the reading of the iHEX file goes fine, but I get a reboot while the Teensy is busy searching for a ID string in (check_flash_id( buffer_addr, hex.max - hex.min ). With no code changes apparently because it reboots as before into the same old program.

I'm lost now. Were we not already writing into the Teensy external flash chip? So why still bother about that check? And why ask user for a confirmation before moving from buffer into flash, if the buffer was that flash already?


Default buffer location is flash, and the FlasherX sketch determines how much flash is available at run-time. You could probably modify the code to use a fixed FLASH buffer size and location, but the buffer certainly cannot be of size FLASH_SIZE, as that would imply no existing code, which is not possible. The confirmation is requested just to get the user to confirm that the transfer seemed to go okay, and because there is no confirmation of any overall checksum or other error test for the file transfer. My recommendation is to go back to the original sketch and it should work every time. From there you can explain how you want/need to adapt for your application. Quite a few people are using it successfully, including on T4.1. If you have trouble, please share the smallest program you can devise that shows the issue so that we can trouble-shoot.
 
First, what is your platform, IDE, versions, etc?

Arduino IDE 2.0.3, Teensyduino 1.58.0-beta2.

If I go for RAM buffer of 256k instead of 0k as in the .h in the library then it works ok.Is it expected that T41 users patch this .h file to make it work for them?

If I want to use the code as-is, so with 0k buffer, then do i need to manually compute file size first, then look at existing code size in bottom of flash rom, then compute where to place a temporary buffer in flash so that it lands above existing code? Why not always push the temporary buffer to the flash chip ceiling and have the code at the floor?
 
Arduino IDE 2.0.3, Teensyduino 1.58.0-beta2.

If I go for RAM buffer of 256k instead of 0k as in the .h in the library then it works ok.Is it expected that T41 users patch this .h file to make it work for them

No, it is not necessary. T4.1 can work with buffer in RAM or in FLASH (default).

If I want to use the code as-is, so with 0k buffer, then do i need to manually compute file size first, then look at existing code size in bottom of flash rom, then compute where to place a temporary buffer in flash so that it lands above existing code?

No, you don't have to do that. The program does it for you.

Why not always push the temporary buffer to the flash chip ceiling and have the code at the floor?

That's exactly what it does.

If you build the FlasherX sketch for T4.1, with no modifications, it should report that it has allocated a flash buffer of a very large size, i.e. from the top of the application to the top of flash. Could you build for T4.1 with no changes and then post the serial output?
 
Back
Top