OTA through Ethernet with Teensy 4.1

[joepasquariello]

"FlasherX buffers the new code in flash by default, and automatically defines the largest possible flash buffer, from the top of existing code to the bottom of FLASH_RESERVE (settable in FlashTxx.h). "

I think what you're seeing is a consequence of changes that have been made to the Teensy bootloader. FlasherX determines the largest possible buffer by starting at the bottom of FLASH_RESERVE and then working downward until non-erased flash is found. In older versions of the bootloader, every firmware update caused the entire flash to be erased (except for space reserved for LITTLEFS and EEPROM). I can't remember why the change was made, but the latest bootloader erases less of the flash.

You could verify this is what is happening by doing the 15-second hold of the program button, which causes the entire flash to be erased and the built-in blink program to be loaded. After that, reload your application and see if it reports a larger available buffer.

Paul just wrote a post a few days ago about the logic for how much flash is erased on a given upload. I'm going to have to think about how FlasherX could be modified to accommodate the new logic. One thing to know, though, is that if start with a completely erased T4.1, install your sketch with Teensy bootloader, and thereafter do updates only via FlasherX, the buffering will work correctly because FlasherX erases the buffer after it does the update.

Another thing you could try is setting FLASH_RESERVE to a larger value, such as (256*FLASH_SECTOR_SIZE). You might find that there is enough erased flash below that address to do the update.

@Paul, if you can point me to your recent post on the topic of erasing flash, I would appreciate it.

 

[joepasquariello]

I found Paul's recent post (https://forum.pjrc.com/threads/7376...memory-storage?p=332976&viewfull=1#post332976)

The bootloader erases either the amount of memory needed (based on
the size of your program), or a minimum amount if the prior program was small.
In normal mode, the minimum is 512K. In secure mode (Lockable Teensy only)
the minimum is 1024K.

 

[joepasquariello]

I found Paul's recent post (https://forum.pjrc.com/threads/7376...memory-storage?p=332976&viewfull=1#post332976)

The bootloader erases either the amount of memory needed (based on the size of your program), or a minimum amount if the prior program was small. In normal mode, the minimum is 512K. In secure mode (Lockable Teensy only) the minimum is 1024K.

@Razon, do you know why you have unerased flash in those relatively high addresses?

 

[jmarsh]

@Razon, do you know why you have unerased flash in those relatively high addresses?

LittleFS_Program will use the highest region of flash possible (counting down from the reserved area) so anyone who has ever used it will probably have the same issue.

 

[joepasquariello]

LittleFS_Program will use the highest region of flash possible (counting down from the reserved area) so anyone who has ever used it will probably have the same issue.

Yes, if anyone uses EEPROM and/or LittleFS, they must set the value of FLASH_RESERVE to a value at least as large as the total reserved for those two uses. That has been explained a few times, but it's not obvious. I don't know how much space that can be, but I should probably set the default FLASH_RESERVE to allow for that, since loading sketches that require all of Flash is probably less common than using EEPROM/LittleFS.

 

[AndyA]

My application ended up being a large flash image (it includes an FPGA image that is downloaded to the hardware over SPI on startup). It has external SPI RAM fitted.
Using the default flash buffering worked (after adjusting the FLASH_RESERVE value), the image just fit in the spare space, but the process was slow and didn't leave much headroom if things got larger.
I instead added a special upgrade mode in my system, when an upgrade starts the code that uses the external ram as a buffer gets disabled. I then use the external ram as the buffer location.
This eliminated the risk of running out of flash space and reduced the upgrade time by almost 2/3. I would highly recommend this approach if you have external ram available.

 

[Razon]

I only have external flash (32Mb), no external ram.
LittleFS is enabled only for the external QSPI flash so I would expect the normal 8Mb of flash to not be affected and FlashX should be able to allocate more memory to the buffer.

However, Increasing the FLASH_RESERVE worked!!! ...I'll keep looking into this though.



#define USE_SD 0 // SDFAT based SDIO and SPI
#define USE_LFS_RAM 0 // T4.1 PSRAM (or RAM)
#define USE_LFS_QSPI 1 // T4.1 QSPI
#define USE_LFS_PROGM 0 // T4.1 Progam Flash
#define USE_LFS_SPI 0 // SPI Flash

Serial Monitor output:

Checking for new Firmware...
New Firmware Available!
Starting OTA!
FlasherX v2.3 - Oct 26 2023 10:05:39
WARNING: this can ruin your device!
target = fw_teensy41 (8192K flash in 4K sectors)
created buffer = 6128K FLASH (60104000 - 60700000)
Verifying file. Please wait...
reading hex lines...

hex file: 16586 lines 265216 bytes (60000000 - 60040C00)
new code contains correct target ID fw_teensy41
enter 16586 to flash or 0 to abort
calling flash_move() to load new firmware...

 

[joepasquariello]

I only have external flash (32Mb), no external ram.
LittleFS is enabled only for the external QSPI flash so I would expect the normal 8Mb of flash to not be affected and FlashX should be able to allocate more memory to the buffer.

However, Increasing the FLASH_RESERVE worked!!! ...I'll keep looking into this though.

Okay, thanks for letting us know. That means there is some unerased main Flash near the top. Do you also use EEPROM, or do you think you may have at one point used EEPROM or LittleFS on the main Flash of that particular T4.1? If so, but you are not using it now, I would try doing the 15-second button hold, which will erase the entire main Flash for sure, and then see if you can use the smaller FLASH_RESERVE.

 

[jmarsh]

Yes, if anyone uses EEPROM and/or LittleFS, they must set the value of FLASH_RESERVE to a value at least as large as the total reserved for those two uses. That has been explained a few times, but it's not obvious. I don't know how much space that can be, but I should probably set the default FLASH_RESERVE to allow for that, since loading sketches that require all of Flash is probably less common than using EEPROM/LittleFS.

Would it take too long to scan from the top of the current program image up to FLASH_RESERVE to look for the largest erased region?

 

[joepasquariello]

Would it take too long to scan from the top of the current program image up to FLASH_RESERVE to look for the largest erased region?

The existing code searches downward from FLASH_RESERVE, because we know where that is, but we don't know the size (top) of the currrent image. As far as I know, the size of the image is not stored within the image, so we need to determine it via search.

 

[jmarsh]

The existing code searches downward from FLASH_RESERVE, because we know where that is, but we don't know the size (top) of the currrent image. As far as I know, the size of the image is not stored within the image, so we need to determine it via search.

The linker script stores it in an unsigned long named _flashimagelen at compile time. This is how LittleFS_Program is able to verify there is enough space left on the flash for the requested filesystem size: https://github.com/PaulStoffregen/LittleFS/blob/main/src/LittleFS.cpp#L959

 

[Razon]

Joe, you are right again! I am using EEPROM for storing some flags and variables.
I have about 200bytes starting at address "0", but since that's emulated EEPROM I don't know where that falls in the flash. FlasherX readme mentions it's supposed to be at "the top of the flash".

Also, to me it's counter intuitive that increasing the FLASH_RESERVE would result in a bigger buffer.
|<------------------------------ FLASH_SIZE ------------------------------------>|
^FLASH_BASE_ADDR
|<------- code ------->|<--------- buffer ---------->|<-- FLASH_RESERVE -->|

 

[joepasquariello]

Joe, you are right again! I am using EEPROM for storing some flags and variables. I have about 200bytes starting at address "0", but since that's emulated EEPROM I don't know where that falls in the flash. FlasherX readme mentions it's supposed to be at "the top of the flash".

The very top 4K of main flash contains the recovery Blink program. Below that would be EEPROM, and I'm not sure how much space it takes up. I have to understand that so that FlasherX can be easier to use for those using EEPROM. If LittleFS was used in main flash, then I think it would be below EEPROM.

Also, to me it's counter intuitive that increasing the FLASH_RESERVE would result in a bigger buffer.
|<------------------------------ FLASH_SIZE ------------------------------------>|
^FLASH_BASE_ADDR
|<------- code ------->|<--------- buffer ---------->|<-- FLASH_RESERVE -->|

What was happening was you were searching for empty space in the area that included EEPROM. Once you increased FLASH_RESERVE, you started the search below the beginning of EEPROM, so a larger buffer was possible.

 

[joepasquariello]

The linker script stores it in an unsigned long named _flashimagelen at compile time. This is how LittleFS_Program is able to verify there is enough space left on the flash for the requested filesystem size: https://github.com/PaulStoffregen/LittleFS/blob/main/src/LittleFS.cpp#L959

Thanks. I'll have to think about how to use this. The existing code uses the same logic for T3.x and T4.x, which keeps things simple. I think it should be possible to test for the existence of EEPROM symbols and automatically begin the buffer search below them, rather than the user having to manually modify the FLASH_RESERVE macro. With T4.1, it's possible to buffer the new firmware not only in the main flash, but also in the optional SDIO flash/PSRAM. I haven't added explicit support for those, but I have tested them.

 

[Razon]

Thanks for the explanation. That makes sense.

To make it more general we would have to search for an empty memory location that can fit the new firmware file.

So, when looking for buffer space, we can start from FLASH_RESERVE as we do now, but if we run into data before buffer_size>FW_file_size, we reset buffer_size and update FLASH_RESERVE to the first new address that doesn't contain data; then we keep searching until we find enough memory or we reach FLASH_SIZE

Or we could search the entire memory regardless, remembering the start/length addresses for the largest block then use that for the buffer. no need for file size.

 

[joepasquariello]

To make it more general we would have to search for an empty memory location that can fit the new firmware file.

So, when looking for buffer space, we can start from FLASH_RESERVE as we do now, but if we run into data before buffer_size>FW_file_size, we reset buffer_size and update FLASH_RESERVE to the first new address that doesn't contain data; then we keep searching until we find enough memory or we reach FLASH_SIZE

Or we could search the entire memory regardless, remembering the start/length addresses for the largest block then use that for the buffer. no need for file size.

The only thing that was getting in your way was that you did not take into account the space used for EEPROM. I am working on an application that uses EEPROM, so I can experiment there, and then move on to LittleFS.

 

[Razon]

yeah, I did some EEPROM testing and so far I didn't find a relation between the amount of data I write to eeprom and how it affects the "FlashReserve" address. That's why my suggestion.

Another thing I noticed is it doesn't work with .ehex files (Code Security enabled). I didn't have time to think what the best workaround would be, but it can't find the target ID which is understandable since the file is encoded.

 

[joepasquariello]

yeah, I did some EEPROM testing and so far I didn't find a relation between the amount of data I write to eeprom and how it affects the "FlashReserve" address. That's why my suggestion.

For T4.X, T4.1, and TMM, the top sector (4K) is the recovery Blink program. Below, that EEPROM reserves 15 sectors (60K) on T4.0 and 63 sectors (252K) on T4.1 and TMM. It doesn't matter how much EEPROM space you use. All of this will be "used" be EEPROM, which explains your 96K buffer. If you set FLASH_RESERVE to (64*FLASH_SECTOR_SIZE), that lets your buffer begin just below EEPROM and should allow FlasherX to define the largest possible buffer.

Another thing I noticed is it doesn't work with .ehex files (Code Security enabled). I didn't have time to think what the best workaround would be, but it can't find the target ID which is understandable since the file is encoded.

I'm not sure that's correct. I think I remember that someone tried FlasherX with EHEX files and said that it worked.

 

[Razon]

Pre-allocating memory and marking it for EEPROM usage as you explained is consistent with what I tested.

As far as code security, from what I see the check_flash_id routine compares the FLASH_ID against raw data in flash.
ehex data is encoded and therefore the data in the buffer is encoded.

But... I guess it depends HOW we store the ID.
I'm simply using: #define FLASH_ID "fw_teensy41" . This doesn't work since it gets encoded with the rest of the program and we'll never find the decoded version of "fw_teensy41" in the buffer.

I don't have to deal with multiple platforms, so I'll take the easy way out and skip the ID check. I tested and can confirm it works.

 

[joepasquariello]

Pre-allocating memory and marking it for EEPROM usage as you explained is consistent with what I tested.

I don't have to deal with multiple platforms, so I'll take the easy way out and skip the ID check. I tested and can confirm it works.

Thanks. I wasn't sure if you had tested with (64*FLASH_SECTOR_SIZE), and thanks for the confirmation that EHEX is okay (except for ID check).

 

[jmarsh]

If the payload is an EHEX, are you doing any extra sanity check(s) to verify the integrity? It seems a bit dangerous to write the payload to flash without any sort of verification first...

 

[joepasquariello]

If the payload is an EHEX, are you doing any extra sanity check(s) to verify the integrity? It seems a bit dangerous to write the payload to flash without any sort of verification first...

The ID we're talking about is just to make sure the build was for the correct model of Teensy. That's important for T3.x, where on-board Flash contains some model-specific critical values, and less so for T4.x. The FlasherX demo application transfers the firmware by sending the hex file. In my own applications, I convert to binary on the host side, compute the CRC of the image, and the Teensy verifies that CRC after receiving the complete image.

 

[jmarsh]

Right, but I meant if you're using this on a "production" level device (which is really the only reason a secure teensy would be used) you'd want to add some stronger checks (better than just checking the target id, even if that worked) to ensure the payload is legitimate before flashing it otherwise the device could easily be rendered inoperable.

 

[Frankthetech]

I know this is an older thread but I'm reading this to figure out how to do these updates to the 4.1. been working with the older 3.1. Anyway I noticed the issue about the flash ID and found it's necessary to use the ID in your code somewhere, like Serial.println("fw_teensy41");
Did not work for me as a define only.

 

[AndyA]

I know this is an older thread but I'm reading this to figure out how to do these updates to the 4.1. been working with the older 3.1. Anyway I noticed the issue about the flash ID and found it's necessary to use the ID in your code somewhere, like Serial.println("fw_teensy41");
Did not work for me as a define only.

The FlasherX code defines a macro FLASH_ID which contains a text string of the board type. It then uses a function check_flash_id() to verify that the new firmware contains the expected string and rejects the image as invalid if it doesn't.

Macros aren't included in the code unless they are used, having the string purely as a #define isn't going to add anything to the compiled code. You then need to use that macro somewhere in code that doesn't get optimised out.

Since the check function does use that macro any application that is built including the ability to upgrade the code using the flasherX library should already contain that string somewhere. So for most applications this requirement is automatically met. However if you want to use it to flash some content that doesn't contain that library you will need to manually include the string in your code as you indicated. Or remove that check and implement your own sanity checking to ensure that the uploaded file is valid.

Personally I've moved to checking that a couple of magic numbers that should be in the flash image at fixed locations are correct. It's teensy 4.x specific rather than generic for all versions but is a little safer since it verifies the firmware at least looks bootable rather than checking it happens to have some arbitrary text string somewhere in the code.