Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 14 of 14

Thread: Teensy 4: What triggers Halfkay to do a full flash erase

  1. #1
    Member
    Join Date
    May 2015
    Location
    Germany
    Posts
    20

    Teensy 4: What triggers Halfkay to do a full flash erase

    I am extending the Micropython implementation on a Teensy 4 and similar. The set-up uses all flash beyond 1MByte for a file system. It happens more or less regularly, that the halfkay loader performs a full flash erase on firmware update, which wipes also the file system. Which is inconvenient. But that happens only when the file system used more than just a few blocks of the flash. I already tried to keep the upper 64kByte of the flash untouched, but that did not help.
    So my question is: what triggers the halfkay loader to do a full flash erase, and not only erasing the sectors it need for the new firmware. At the moment, the firmware has a size of ~140k.

    Note: With a MIMXRT1020-EVK board (with a different loader) this does not happen. It's a halfkay phenomenon. When it runs, I can tell by the duration of the short stop after a few actions what happens.

    roberthh

  2. #2
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    8,552
    As far as I know halfkay always performs a full flash erase (everything but the eeprom area). There are plans to add more of such areas, but I think that can take a a long time.. if it ever comes.

  3. #3
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    24,451
    Quote Originally Posted by roberthh View Post
    So my question is: what triggers the halfkay loader to do a full flash erase, and not only erasing the sectors it need for the new firmware
    The entire flash is always erased at the beginning of each upload, except for the small area meant to be used for EEPROM emulation.

    Historically this full erase has been meant for people who edit FSEC to turn on code read security Teensy 3.x or who use Teensy 2.0 where security is always on. If the upload only erases the flash as needed for a new program, an attacker who wishes to capture someone's firmware from a chip with security turned on could upload a tiny program which prints all the rest of the memory to a serial port.

    On Teensy 4.0, your best option is to use the 60K space meant for EEPROM. It's address range 601F0000 to 601FF000. The very top 4K holds the known-good LED blink program which gets copied when you use the 15 second button press. That top 4K is read-only. So if you want to fully erase the other 60K of the top 64K, you need to use 15 4K erase operations, or 1 32K and 7 4K. Just know than a single 64K block erase at the very top of the flash won't work because the top 4K is read-only memory which can never be erased or modified.

    Indeed, I am planning to make a future bootloader smarter about which portion of the flash to erase. Support for encrypted firmware is also planned. But there is no time frame for these, and the reality of the pandemic is PJRC has been running short-staffed since March 2020, which has really limited how much development can be done. Until we can rehire, odds are slim these sorts of major software developments will happen anytime soon.

  4. #4
    Member
    Join Date
    May 2015
    Location
    Germany
    Posts
    20
    The entire flash is always erased at the beginning of each upload, except for the small area meant to be used for EEPROM emulation.
    Hello Paul, thank you for your swift response. According to my test, the flash is not always fully erased. Some data is left untouched, other parts are overwritten. That's why I was asking.

  5. #5
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,419
    Quote Originally Posted by roberthh View Post
    Hello Paul, thank you for your swift response. According to my test, the flash is not always fully erased. Some data is left untouched, other parts are overwritten. That's why I was asking.
    Was there a test sketch that found the areas of the flash not erased?

  6. #6
    Member
    Join Date
    May 2015
    Location
    Germany
    Posts
    20
    Quote Originally Posted by defragster View Post
    Was there a test sketch that found the areas of the flash not erased?
    The firmware create a littlefs type file system, starting at offset 0x103000 of the flash. So I wrote a few short files to that area, and then re-flashed the firmware. The firmware checks on boot whether the file system exists and creates a new one, if not. That file system will be empty then. But is was not. It still had the names of the file in it, meaning that the directory was not erased. And since there was a fill reboot with power cycling, there is not data retained in cache. Starting a new file system, the super block and the directory are at the lower block numbers. Data will be spread over the whole available range, and indeed the data in the files was most of the times changed. But not for all files.

  7. #7
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,419
    Quote Originally Posted by roberthh View Post
    The firmware create a littlefs type file system, starting at offset 0x103000 of the flash. So I wrote a few short files to that area, and then re-flashed the firmware. The firmware checks on boot whether the file system exists and creates a new one, if not. That file system will be empty then. But is was not. It still had the names of the file in it, meaning that the directory was not erased. And since there was a fill reboot with power cycling, there is not data retained in cache. Starting a new file system, the super block and the directory are at the lower block numbers. Data will be spread over the whole available range, and indeed the data in the files was most of the times changed. But not for all files.
    That hasn't shown itself here. Most testing - and just attempted repro - done with LFSIntegrity.INO.

    Is the current TD 1.54 Beta 7 or Beta 8 in use?

    It always starts like this after Code Upload:
    Code:
    printDirectory PRO_DISK
    --------------
    
     0 dirs with 0 files of Size 0 Bytes
     Total 0 files of Size 0 Bytes
    Bytes Used: 8192, Bytes Total:7995392
    That is using a disk 8,000,000 bytes big - about all the space left after code storage : FLASH: code:75968, data:9712, headers:7212 free for files:8033572

    Having written data before and soiling the media - DIR on startup shows nothing after TeensyLoader displays 'Formatting'.

    And in that sketch doing 'y' has it walk the media searching for any non-blank areas and finding none.

    Also doing "F" 3 in a row for complete low level Format takes the same short time each time, only wiping out the new ROOT dir data and then recreating it.


    If there is data left after the Upload Full Format - then something is wrong and repro case in use would be needed to see it.

    One way to confirm this showing SerMon output would be to go to file : ...\hardware\teensy\avr\libraries\LittleFS\src\Lit tleFS.cpp

    In :: bool LittleFS_Program::begin(uint32_t size)

    And uncomment all the :: //Serial.printf
    Those will indicate the state of the Program Flash memory, and the steps taken to mount or prepare the media.

    Doing that here adds the BOLD lines below as expected:
    Code:
    T:\arduino_1.8.14\hardware\teensy\avr\libraries\LittleFS\examples\LFSintegrity\LFSintegrity.ino May 17 2021 02:11:20
    LittleFS Test : File Integrity
    Program flash begin
    available_space = 8029252
    baseaddr = 60020000
    attempting to mount existing media
    couldn't mount media, attemping to format
    attempting to mount freshly formatted media
    printDirectory PRO_DISK
    --------------
    Repeating that there should always begin that way after a Fresh Upload where the format completed.

  8. #8
    Member
    Join Date
    May 2015
    Location
    Germany
    Posts
    20
    Thanks for the reply. I do not use Teensyduino at all for the device. As said, this is a MicroPython based firmware, with it's own implementation of the LFS, using the low level NXP/Freescale drivers. So what I have learned so far: Halfkay attempts to erase all of the flash but the top 4k. Whether it does a full erase or some blocks are skipped does not matter for my actual topic. It would only be interesting if there ways a way to prevent erasing areas which will not be filled with new code. Now my load script has to force a FS re-creation after every firmware upload to ensure a clean startup state.

    Besides that it is obviously of interest, if the halfkay loader should erase everything, but it does not.

  9. #9
    Member
    Join Date
    May 2015
    Location
    Germany
    Posts
    20
    So I did another test. After reflashing the firmware with what seemed to include a flash erase (there is a longish stop during upload) I ran a test reading all blocks of the file system area. That starts at block 259 of the 512 4k blocks of the Teensy 4.0 flash. It reads a block and checks the content to be all 0xff. It reports the following blocks as non-empty:
    Code:
    Number of blocks 252
    Blocks not empty:
      0,   1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12, 237, 238, 239, 240, 241, 242, 243,
    244, 245, 246, 247, 248, 249, 250, 251,
    And indeed, reading them shows non 0xff content.
    Block 0 and 1 may be fine, since these are the superblocks of the FS, which will be recreated on device boot. All other blocks are not empty. If I use my own script to erase the blocks, all blocks are reported as empty, unless I boot. Then block 0 and 1 are flagged. For me it looks like an inconsistency in halfkay. I do not mind too much, since the file system anyhow erase block before use.

  10. #10
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,206
    Quote Originally Posted by roberthh View Post
    So I did another test. After reflashing the firmware with what seemed to include a flash erase (there is a longish stop during upload) I ran a test reading all blocks of the file system area. That starts at block 259 of the 512 4k blocks of the Teensy 4.0 flash. It reads a block and checks the content to be all 0xff. It reports the following blocks as non-empty:
    Code:
    Number of blocks 252
    Blocks not empty:
      0,   1,   2,   3,   4,   5,   6,   7,   8,   9,  10,  11,  12, 237, 238, 239, 240, 241, 242, 243,
    244, 245, 246, 247, 248, 249, 250, 251,
    And indeed, reading them shows non 0xff content.
    Block 0 and 1 may be fine, since these are the superblocks of the FS, which will be recreated on device boot. All other blocks are not empty. If I use my own script to erase the blocks, all blocks are reported as empty, unless I boot. Then block 0 and 1 are flagged. For me it looks like an inconsistency in halfkay. I do not mind too much, since the file system anyhow erase block before use.
    Interesting information, but I don't really know what that means. As mentioned it could be a bug that Paul would probably want to fix, especially with the possibility later of encrypted...

    But what I wonder about, is, is it that the sections are not erased, or is it that that those sectors have stuff written into them by the halfkey?

    For example if you have code that runs in the ITCM (lower memory) which is by default, there is copy of it in the flash that is copied down into that region. Likewise for initialized variables in the DTCM, there is data stored in the flash that us copied into the DTCM...

    Again might help to look at the programs map and see if those blocks by chance correspond with some of the blocks you show.

  11. #11
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    24,451
    Quote Originally Posted by roberthh View Post
    For me it looks like an inconsistency in halfkay. I do not mind too much, since the file system anyhow erase block before use.
    I've added this to my list of issues to investigate. But just to be realistic, it may be quite a long time until I do anything. Without code to reproduce this issue, I'll have to write something from scratch to try to cause it to happen here. I don't have a great history of managing to reproduce problems that way...

  12. #12
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    7,190
    Quote Originally Posted by roberthh View Post
    Thanks for the reply. I do not use Teensyduino at all for the device. As said, this is a MicroPython based firmware, with it's own implementation of the LFS, using the low level NXP/Freescale drivers. So what I have learned so far: Halfkay attempts to erase all of the flash but the top 4k. Whether it does a full erase or some blocks are skipped does not matter for my actual topic. It would only be interesting if there ways a way to prevent erasing areas which will not be filled with new code. Now my load script has to force a FS re-creation after every firmware upload to ensure a clean startup state.

    Besides that it is obviously of interest, if the halfkay loader should erase everything, but it does not.
    Not an expert here especially with MicroPython, but the question would also be what is MicroPython doing on booth. Does it write anything to flash or .. I have seen implementations for CircuitPython but not for MicroPython for the T4. Could also be a issue with NXP/Freescale LFS implementation if you are running it automatically from start? Hard to tell - might be halfkey or might be an issue with MicroPython/LFS driver?

    Just as a note I have seen that before with LFS leaving non-empty blocks (not 0xFF) while we were developing LFS for the Teensy but that was an issue with implementation of LFS erase block function until we fixed it. Again hard to say exactly since their implementation might be different than using the non-micropython version.

  13. #13
    Member
    Join Date
    May 2015
    Location
    Germany
    Posts
    20
    Thanks for all. I have written the LFS block level driver myself, and it definitely does not write stuff to flash unless told so. Micropython itself is not hardware aware and has not driver itself for writing to flash. As long as I do not run Halfkay, I can start Micropython as often as I like, and it definitely does not write to flash, unless file operations start to do so.
    I can write a script using the LFS driver to erase the flash and that works. Flash is empty.
    If I start LFS on that empty flash, only blocks 0 and 1 get written as result of creating the file system. The check shows only block 0 and 1 as non-empty.
    If I run a series of file creation operations, more blocks are changed. The list below show the changed blocks after 100 files being create and deleted.
    Code:
    Number of blocks 252
    Blocks not empty:
       0,    1,  160,  161,  162,  163,  164,  165,  166,  167,  168,  169,  170,  171,  172,  173,  174,  175,  176,  
    177,  178,  179,  180,  181,  182,  183,  184,  185,  186,  187,  188,  189,  190,  191,  192,  193,  194,  
    195,  196,  197,  198,  199,  200,  201,  202,  203,
    If I the do a firmware upload and boot again, I would expect a changed block 0 and 1. The FS is recreated if corrupt. An empty flash would be considered as corrupt. What I get is this list:
    Code:
    Number of blocks 252
    Blocks not empty:
       0,    1,  160,  161,  162,  163,  164,  165,  166,  167,  168,  169,  170,  171,  172,
    Below if a section from block 166:
    Code:
    bytearray(b'\x01\x00\x00\x00\xff\xef\xff\xfafile1 \x00\x01\x05\x01\x02\x03\x04\x05\x06
    \x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e
    \x1f !"#$%&\'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefg
    hijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f
    \x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5
    \xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb
    \xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2
    \xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8
    \xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff\x00\x10
    \x00\x01\x08\x00\x00CR\xcfL#\r0\x00\x0c\rfile2 \x00\x01\x05\x02\x03\x04\x05\x06\x07\x08\t\n
    \x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&\'
    ()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrst
    uvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8
    It is the pattern I had written as test, which should have been erased. But it is not.
    Again: for my purpose I do not need an answer. Assuming that firmware upload is meant to clear all flash, I recreate the file system as part of the firmware build & upload script. It may be a problem of my specific board.

  14. #14
    Member
    Join Date
    May 2015
    Location
    Germany
    Posts
    20
    Just an update to that thread: I got a Teensy 4.1, and on that board the bootloader does indeed erase all flash. So maybe the Teensy 4.0 and 4.1 behave different, or it was just my device, which is a pretty early model, behaved different. Even if I do not like that full erase behavior, it is at least consistent, and my code does nor have to take additional actions in clearing the flash.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •