Teensy 4: What triggers Halfkay to do a full flash erase

roberthh

Active member
I am extending the Micropython implementation on a Teensy 4 and similar. The set-up uses all flash beyond 1MByte for a file system. It happens more or less regularly, that the halfkay loader performs a full flash erase on firmware update, which wipes also the file system. Which is inconvenient. But that happens only when the file system used more than just a few blocks of the flash. I already tried to keep the upper 64kByte of the flash untouched, but that did not help.
So my question is: what triggers the halfkay loader to do a full flash erase, and not only erasing the sectors it need for the new firmware. At the moment, the firmware has a size of ~140k.

Note: With a MIMXRT1020-EVK board (with a different loader) this does not happen. It's a halfkay phenomenon. When it runs, I can tell by the duration of the short stop after a few actions what happens.

roberthh
 
As far as I know halfkay always performs a full flash erase (everything but the eeprom area). There are plans to add more of such areas, but I think that can take a a long time.. if it ever comes.
 
So my question is: what triggers the halfkay loader to do a full flash erase, and not only erasing the sectors it need for the new firmware

The entire flash is always erased at the beginning of each upload, except for the small area meant to be used for EEPROM emulation.

Historically this full erase has been meant for people who edit FSEC to turn on code read security Teensy 3.x or who use Teensy 2.0 where security is always on. If the upload only erases the flash as needed for a new program, an attacker who wishes to capture someone's firmware from a chip with security turned on could upload a tiny program which prints all the rest of the memory to a serial port.

On Teensy 4.0, your best option is to use the 60K space meant for EEPROM. It's address range 601F0000 to 601FF000. The very top 4K holds the known-good LED blink program which gets copied when you use the 15 second button press. That top 4K is read-only. So if you want to fully erase the other 60K of the top 64K, you need to use 15 4K erase operations, or 1 32K and 7 4K. Just know than a single 64K block erase at the very top of the flash won't work because the top 4K is read-only memory which can never be erased or modified.

Indeed, I am planning to make a future bootloader smarter about which portion of the flash to erase. Support for encrypted firmware is also planned. But there is no time frame for these, and the reality of the pandemic is PJRC has been running short-staffed since March 2020, which has really limited how much development can be done. Until we can rehire, odds are slim these sorts of major software developments will happen anytime soon.
 
The entire flash is always erased at the beginning of each upload, except for the small area meant to be used for EEPROM emulation.
Hello Paul, thank you for your swift response. According to my test, the flash is not always fully erased. Some data is left untouched, other parts are overwritten. That's why I was asking.
 
Hello Paul, thank you for your swift response. According to my test, the flash is not always fully erased. Some data is left untouched, other parts are overwritten. That's why I was asking.

Was there a test sketch that found the areas of the flash not erased?
 
Was there a test sketch that found the areas of the flash not erased?

The firmware create a littlefs type file system, starting at offset 0x103000 of the flash. So I wrote a few short files to that area, and then re-flashed the firmware. The firmware checks on boot whether the file system exists and creates a new one, if not. That file system will be empty then. But is was not. It still had the names of the file in it, meaning that the directory was not erased. And since there was a fill reboot with power cycling, there is not data retained in cache. Starting a new file system, the super block and the directory are at the lower block numbers. Data will be spread over the whole available range, and indeed the data in the files was most of the times changed. But not for all files.
 
The firmware create a littlefs type file system, starting at offset 0x103000 of the flash. So I wrote a few short files to that area, and then re-flashed the firmware. The firmware checks on boot whether the file system exists and creates a new one, if not. That file system will be empty then. But is was not. It still had the names of the file in it, meaning that the directory was not erased. And since there was a fill reboot with power cycling, there is not data retained in cache. Starting a new file system, the super block and the directory are at the lower block numbers. Data will be spread over the whole available range, and indeed the data in the files was most of the times changed. But not for all files.

That hasn't shown itself here. Most testing - and just attempted repro - done with LFSIntegrity.INO.

Is the current TD 1.54 Beta 7 or Beta 8 in use?

It always starts like this after Code Upload:
Code:
printDirectory PRO_DISK
--------------

 0 dirs with 0 files of Size 0 Bytes
 Total 0 files of Size 0 Bytes
Bytes Used: 8192, Bytes Total:7995392

That is using a disk 8,000,000 bytes big - about all the space left after code storage : FLASH: code:75968, data:9712, headers:7212 free for files:8033572

Having written data before and soiling the media - DIR on startup shows nothing after TeensyLoader displays 'Formatting'.

And in that sketch doing 'y' has it walk the media searching for any non-blank areas and finding none.

Also doing "F" 3 in a row for complete low level Format takes the same short time each time, only wiping out the new ROOT dir data and then recreating it.


If there is data left after the Upload Full Format - then something is wrong and repro case in use would be needed to see it.

One way to confirm this showing SerMon output would be to go to file : ...\hardware\teensy\avr\libraries\LittleFS\src\LittleFS.cpp

In :: bool LittleFS_Program::begin(uint32_t size)

And uncomment all the :: //Serial.printf
Those will indicate the state of the Program Flash memory, and the steps taken to mount or prepare the media.

Doing that here adds the BOLD lines below as expected:
Code:
T:\arduino_1.8.14\hardware\teensy\avr\libraries\LittleFS\examples\LFSintegrity\LFSintegrity.ino May 17 2021 02:11:20
LittleFS Test : File Integrity
[B]Program flash begin
available_space = 8029252
baseaddr = 60020000
attempting to mount existing media
couldn't mount media, attemping to format
attempting to mount freshly formatted media
[/B]printDirectory PRO_DISK
--------------

Repeating that there should always begin that way after a Fresh Upload where the format completed.
 
Thanks for the reply. I do not use Teensyduino at all for the device. As said, this is a MicroPython based firmware, with it's own implementation of the LFS, using the low level NXP/Freescale drivers. So what I have learned so far: Halfkay attempts to erase all of the flash but the top 4k. Whether it does a full erase or some blocks are skipped does not matter for my actual topic. It would only be interesting if there ways a way to prevent erasing areas which will not be filled with new code. Now my load script has to force a FS re-creation after every firmware upload to ensure a clean startup state.

Besides that it is obviously of interest, if the halfkay loader should erase everything, but it does not.
 
So I did another test. After reflashing the firmware with what seemed to include a flash erase (there is a longish stop during upload) I ran a test reading all blocks of the file system area. That starts at block 259 of the 512 4k blocks of the Teensy 4.0 flash. It reads a block and checks the content to be all 0xff. It reports the following blocks as non-empty:
Code:
Number of blocks 252
Blocks not empty:
  0,   1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12, 237, 238, 239, 240, 241, 242, 243,
244, 245, 246, 247, 248, 249, 250, 251,
And indeed, reading them shows non 0xff content.
Block 0 and 1 may be fine, since these are the superblocks of the FS, which will be recreated on device boot. All other blocks are not empty. If I use my own script to erase the blocks, all blocks are reported as empty, unless I boot. Then block 0 and 1 are flagged. For me it looks like an inconsistency in halfkay. I do not mind too much, since the file system anyhow erase block before use.
 
So I did another test. After reflashing the firmware with what seemed to include a flash erase (there is a longish stop during upload) I ran a test reading all blocks of the file system area. That starts at block 259 of the 512 4k blocks of the Teensy 4.0 flash. It reads a block and checks the content to be all 0xff. It reports the following blocks as non-empty:
Code:
Number of blocks 252
Blocks not empty:
  0,   1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12, 237, 238, 239, 240, 241, 242, 243,
244, 245, 246, 247, 248, 249, 250, 251,
And indeed, reading them shows non 0xff content.
Block 0 and 1 may be fine, since these are the superblocks of the FS, which will be recreated on device boot. All other blocks are not empty. If I use my own script to erase the blocks, all blocks are reported as empty, unless I boot. Then block 0 and 1 are flagged. For me it looks like an inconsistency in halfkay. I do not mind too much, since the file system anyhow erase block before use.

Interesting information, but I don't really know what that means. As mentioned it could be a bug that Paul would probably want to fix, especially with the possibility later of encrypted...

But what I wonder about, is, is it that the sections are not erased, or is it that that those sectors have stuff written into them by the halfkey?

For example if you have code that runs in the ITCM (lower memory) which is by default, there is copy of it in the flash that is copied down into that region. Likewise for initialized variables in the DTCM, there is data stored in the flash that us copied into the DTCM...

Again might help to look at the programs map and see if those blocks by chance correspond with some of the blocks you show.
 
For me it looks like an inconsistency in halfkay. I do not mind too much, since the file system anyhow erase block before use.

I've added this to my list of issues to investigate. But just to be realistic, it may be quite a long time until I do anything. Without code to reproduce this issue, I'll have to write something from scratch to try to cause it to happen here. I don't have a great history of managing to reproduce problems that way...
 
Thanks for the reply. I do not use Teensyduino at all for the device. As said, this is a MicroPython based firmware, with it's own implementation of the LFS, using the low level NXP/Freescale drivers. So what I have learned so far: Halfkay attempts to erase all of the flash but the top 4k. Whether it does a full erase or some blocks are skipped does not matter for my actual topic. It would only be interesting if there ways a way to prevent erasing areas which will not be filled with new code. Now my load script has to force a FS re-creation after every firmware upload to ensure a clean startup state.

Besides that it is obviously of interest, if the halfkay loader should erase everything, but it does not.

Not an expert here especially with MicroPython, but the question would also be what is MicroPython doing on booth. Does it write anything to flash or .. I have seen implementations for CircuitPython but not for MicroPython for the T4. Could also be a issue with NXP/Freescale LFS implementation if you are running it automatically from start? Hard to tell - might be halfkey or might be an issue with MicroPython/LFS driver?

Just as a note I have seen that before with LFS leaving non-empty blocks (not 0xFF) while we were developing LFS for the Teensy but that was an issue with implementation of LFS erase block function until we fixed it. Again hard to say exactly since their implementation might be different than using the non-micropython version.
 
Thanks for all. I have written the LFS block level driver myself, and it definitely does not write stuff to flash unless told so. Micropython itself is not hardware aware and has not driver itself for writing to flash. As long as I do not run Halfkay, I can start Micropython as often as I like, and it definitely does not write to flash, unless file operations start to do so.
I can write a script using the LFS driver to erase the flash and that works. Flash is empty.
If I start LFS on that empty flash, only blocks 0 and 1 get written as result of creating the file system. The check shows only block 0 and 1 as non-empty.
If I run a series of file creation operations, more blocks are changed. The list below show the changed blocks after 100 files being create and deleted.
Code:
Number of blocks 252
Blocks not empty:
   0,    1,  160,  161,  162,  163,  164,  165,  166,  167,  168,  169,  170,  171,  172,  173,  174,  175,  176,  
177,  178,  179,  180,  181,  182,  183,  184,  185,  186,  187,  188,  189,  190,  191,  192,  193,  194,  
195,  196,  197,  198,  199,  200,  201,  202,  203,
If I the do a firmware upload and boot again, I would expect a changed block 0 and 1. The FS is recreated if corrupt. An empty flash would be considered as corrupt. What I get is this list:
Code:
Number of blocks 252
Blocks not empty:
   0,    1,  160,  161,  162,  163,  164,  165,  166,  167,  168,  169,  170,  171,  172,
Below if a section from block 166:
Code:
bytearray(b'\x01\x00\x00\x00\xff\xef\xff\xfafile1 \x00\x01\x05\x01\x02\x03\x04\x05\x06
\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e
\x1f !"#$%&\'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefg
hijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f
\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5
\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb
\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2
\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8
\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff\x00\x10
\x00\x01\x08\x00\x00CR\xcfL#\r0\x00\x0c\rfile2 \x00\x01\x05\x02\x03\x04\x05\x06\x07\x08\t\n
\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'
()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrst
uvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8
It is the pattern I had written as test, which should have been erased. But it is not.
Again: for my purpose I do not need an answer. Assuming that firmware upload is meant to clear all flash, I recreate the file system as part of the firmware build & upload script. It may be a problem of my specific board.
 
Just an update to that thread: I got a Teensy 4.1, and on that board the bootloader does indeed erase all flash. So maybe the Teensy 4.0 and 4.1 behave different, or it was just my device, which is a pretty early model, behaved different. Even if I do not like that full erase behavior, it is at least consistent, and my code does nor have to take additional actions in clearing the flash.
 
Oming back to this topic. @PaulStoffregen
I've added this to my list of issues to investigate. But just to be realistic, it may be quite a long time until I do anything. Without code to reproduce this issue, I'll have to write something from scratch to try to cause it to happen here. I don't have a great history of managing to reproduce problems that way...
So it seems, that with the teensy loader 1.07 the memory content beyond some limit is preserved (see https://www.pjrc.com/teensy/td_code_security.html, paragraph "Minimum Erase Size".) Where in the file is the information taken from, how much memory is used for code and how much shall be preserved? I looked into a file created by a sample Teensyduino code to the Image vector table and the Boot data structure. Both revealed not much of a difference to the NXP RM, besides the version type in the IVT header, which seems to be 0x43 instead of 0x40 or 0x41, as stated by NXP.
 
Indeed, once a 1062 MCU's bootloader chip is updated to v1.07, as indicated on that page, only a required portion of the FLASH is erased. At least the indicated minimum, and if used space is beyond that then it extends at least 64K beyond the last used portion of FLASH needed

Flash used for all of "FLASH: code:30656, data:7496, headers:8948" likely included in the total to determine if beyond the 'minimum erase size'

With bootloader version 1.07, only the first 512K in normal mode or first 1M in secure mode is unconditionally erased, if the flash memory contains a previously written program with valid IVT+BootData fields.

Board Maximum LittleFS Size Retained Across Code Update
bootloader < 1.07 bootloader >= 1.07
Any Mode Normal Secure
Teensy 4.0 0 1472K 960K
Teensy 4.1 0 7424K 6912K
MicroMod Teensy 0 15616K 15104K
While uploading with bootloader version 1.07, flash memory beyond the minimum erase size is erased in 64K blocks on an as-needed basis, depending on the program size. For example, loading a 513K program in normal mode results in 576K flash erased.
 
the memory content beyond some limit is preserved (see https://www.pjrc.com/teensy/td_code_security.html, paragraph "Minimum Erase Size".) Where in the file is the information taken from, how much memory is used for code and how much shall be preserved?

The amount of memory erased versus preserved is explained on that page under Minimum Erase Size.

But to repeat it again, when you upload a new program a minimum size (either 512K or 1M) is always erased, then additional flash beyond that minimum is erased in 64K chucks as needed by your program. So if you upload a 100K program to an unlocked Teensy 4.0, the first 512K is erased. If you upload a 515K program, 576K erased is erase, first the 512K minimum, then the last 3K of your program causes another 64K block to be erased before it is written to flash.

To answer your question about how much is preserved, simply subtract the amount erased from the total size. See the table under Minimum Erase Size for the amount of flash preserved for programs smaller than the minimum erase size.

The minimum erase size is fixed. You can not change or reconfigure it.
 
Indeed, once a 1062 MCU's bootloader chip is updated to v1.07, as indicated on that page, only a required portion of the FLASH is erased. At least the indicated minimum, and if used space is beyond that then it extends at least 64K beyond the last used portion of FLASH needed

Flash used for all of "FLASH: code:30656, data:7496, headers:8948" likely included in the total to determine if beyond the 'minimum erase size'

Thank you for the swift reply. The text you cited lead to the the assumption, that the data section of the flash is preserved. I do not use code created by Teensyduino, I use MicroPyhton firmware Images, like this one: https://micropython.org/resources/firmware/TEENSY40-20220425-unstable-v1.18-377-geb9674822.hex
The total code size is about 314k. The LFS file system it creates starts at 1MB. On an attempt to upload, it fails on the first attempt after printing three dots, at the second attempt it passes, but the flash is erased.
In the hex file boot header at address 60001024, the firmware size is set to 2MB. Setting it to 1 MB does not change the behavior. So the magic happens somewhere else. And that's what I am asking for.
 
The amount of memory erased versus preserved is explained on that page under Minimum Erase Size.

But to repeat it again, when you upload a new program a minimum size (either 512K or 1M) is always erased, then additional flash beyond that minimum is erased in 64K chucks as needed by your program. So if you upload a 100K program to an unlocked Teensy 4.0, the first 512K is erased. If you upload a 515K program, 576K erased.

To answer your question about how much is preserved, simply subtract the amount erased from the total size. See the table under Minimum Erase Size for the amount of flash preserved for programs smaller than the minimum erase size.

The minimum erase size is fixed. You can not change or reconfigure it.

Thanks Paul for the fast reply. I was happy to see, that you implemented a feature which you were skeptic about when to impoement.
The way it is described is perfect for this application. But it does not work for me. And so I wonder why. The space I need is beyond the 1M limit.
 
Setting it to 1 MB does not change the behavior.

When you try setting to 1MB size, are you configuring the bootdata length field to 0x100000, or to 0x0FD000 ?

Remember, you're using 0x002000 as the bootdata start address. So a length of 0x100000 means your program actual flash usage is 1M+8K, because you also used the first 8K of the flash for metadata. A 1MB program image located after the metadata would conflict with the first 8K of the flash you're intending to use as a filesystem which begins as a fixed 1MB offset from the start of the flash.
 
When you try setting to 1MB size, are you configuring the bootdata length field to 0x100000, or to 0x0FD000 ?

Remember, you're using 0x002000 as the bootdata start address. So a length of 0x100000 means your program actual flash usage is 1M+8K, because you also used the first 8K of the flash for metadata. A 1MB program image located after the metadata would conflict with the first 8K of the flash you're intending to use as a filesystem which begins as a fixed 1MB offset from the start of the flash.

The actual code size is like 300k. Let's assume it is 512k. So which value has to be placed into the header?
 
In Teensyduino 1.56, the linker script defines a "_flashimagelen" symbol which is the actual image length.

Thank's. I will check. But I am not sure that this is the proepr path. Below is copy of the attempt to flash the image. What causes the upload top fail at the first attempt, and to pass at the second?
Code:
robert@hh3:~/Downloads/MicroPython/Daily Builds$ teensy_loader_cli --mcu=TEENSY40 -v -w TEENSY40-20220425-unstable-v1.18-377-geb9674822.hex
Teensy Loader, Command Line, Version 2.2
Read "TEENSY40-20220425-unstable-v1.18-377-geb9674822.hex": 314564 bytes, 15.5% usage
Found HalfKay Bootloader
Programming...error writing to Teensy
 
@paulstoffregen

Defining that symbol properly did not change anything. The loader still errors. It seems that happens at the point where it attempts to do a partial flash erase. What could be the reason for the fail? Does the loader parse the data and check the value at the boot header. Or does it just refer to the highest load address in the hex file? Or is the amount of data in the hex file compared to the value in the boot header. The file I try to load has gaps.
 
Last edited:
My main suggestion is to use Arduino+Teensyduino and look at the HEX files it creates. The complete source code and linker script used to create those HEX files is available to you.
 
Back
Top