FlasherX buffer size is 4k issue

Frankthetech

Frankthetech

Well-known member
So i am running on a T4.1 the example FlasherX code and getting odd results. on 1 computer it works with 7844 k flash (60017000 - 607C0000) and all is good. the code I have running is about 900k hex file so 60170000 minus 6000 0000 = 17 0000 to top of code space, seems workable.
So I expect top of code to where the reserve starts is usable for buffer space,
On the other computer it's created buffer = 4K FLASH (607BF000 - 607C0000)
607B F000 minus 6000 0000 = 7B F000, no way my code should take up that much space, so I think the base is wrong somewhere.
Now on both I did set the these #define FLASH_RESERVE (64*FLASH_SECTOR_SIZE) // reserve top of flash
Could it be the board definitions?
Any ideas to get me looking in the right place for the issue.
Frank
 
Last edited:
Take my note with a grain of salt because I don't fully understand the problem, but might it be related to the fact that FlasherX stores the HEX file first? That's at least twice the size of the binary data.

To address this, I rolled my own updater that only stores the binary data in flash, not the full ASCII HEX file.
 
FlasherX can work in different ways. The demo program receives Hex records via USB or UART serial, parses each one as it arrives, and writes only the binary data to the buffer (in Flash or RAM). I think there is also a demo program where the Hex file can be on the SD, and once again it is parsed one record at a time and only the binary data is written to the buffer.

Frank, how are you using FlasherX?
 
There is a known issue with FlasherX and using the EEPROM library. Even if the current code doesn't use it if it's been used on that unit in the past it could cause issues.

FlasherX starts near the end of the flash and works backwards from there to find the first non-blank part of flash. It then defines the usable buffer as being from that point onwards. The EEPROM library uses a section of flash towards the end, flasherX takes that as the start of your program and so creates a tiny buffer.

Quick solution if you aren't using EEPROM in this project: Factory reset the teensy, that will wipe all the flash.
Better long term solution: Change where flasherX puts its buffer so that it skips the area used for EEPROM. To do this increase FLASH_RESERVE (how much space at the end flasherX avoids using) to (64*FLASH_SECTOR_SIZE)
 
Not sure why the _flashimagelen global variable still isn't being used to know where the end of the program is rather than guessing.
 
jmarsh said:
Not sure why the _flashimagelen global variable still isn't being used to know where the end of the program is rather than guessing.
Click to expand...
What if the user was using some other part of the flash as a filesystem or some other use?

I have one use case where the teensy has 3 different programs in flash, each sitting at a different start address in flash. A basic bootstrap/bootloader, a failsafe ethernet OTA application and a main application. For two of those 3 the _flashimagelen would imply it was ok to overwrite the main app, for the main app it would need to add the adjusted start address to get the correct first available location.
I use the external psram for the upload temporary buffer so wouldn't actually be an issue in this specific case but I could see it causing issues in some situations.

Checking the memory and only using blank sections is safer. Using _flashimagelen as a starting point for the search would however probably be a good idea.
 
AndyA said:
What if the user was using some other part of the flash as a filesystem or some other use?
Click to expand...
Then they're probably going to be boned anyway because LittleFS places the filesystem as high as possible.
AndyA said:
I have one use case where the teensy has 3 different programs in flash, each sitting at a different start address in flash. A basic bootstrap/bootloader, a failsafe ethernet OTA application and a main application. For two of those 3 the _flashimagelen would imply it was ok to overwrite the main app, for the main app it would need to add the adjusted start address to get the correct first available location.
I use the external psram for the upload temporary buffer so wouldn't actually be an issue in this specific case but I could see it causing issues in some situations.
Click to expand...
Which one of these actually performs the updating? Just put it above the others...
AndyA said:
Using _flashimagelen as a starting point for the search would however probably be a good idea.
Click to expand...
Not suggesting it should be the starting point for a search (teensy loader does not erase the majority of pages that aren't rewritten) but rather there should be no search at all - the flasher should have a fixed area that it can use that gets defined at compile time, either by default (_flashimagelen up to the eeprom reserved space) or a const array that reserves needed space.
 
jmarsh said:
Not suggesting it should be the starting point for a search (teensy loader does not erase the majority of pages that aren't rewritten) but rather there should be no search at all
Click to expand...
I realized that. I don't think that is a good idea.
Blindly erasing and overwriting parts of the flash that may contain data is not a good idea in a generic library. If you were writing code for your own use then that approach is fine. But in a library for general use you need to err on the side of caution and not assume it's safe to erase user data on the grounds they probably don't need it any more.
 
Then it's just massively flawed - any Teensy that has used the majority of its flash at any point will have unerased pages due to Teensy loader not erasing unused space.
 
OK, so the question I have is why 1 computer complies the flasherX code and only shows a 4k buffer, on my other computer it's fine and compiles with a 7000+ buffer. What I need to know is how/where does flasherX get the base address for the T4.1?
Joe, the FlasherX works just fine in my code too, just not when I compile on my main computer (4k buffer issue). I'm putting the hex file on the sd card and then run the flasherX to update.
 
jmarsh said:
Then it's just massively flawed - any Teensy that has used the majority of its flash at any point will have unerased pages due to Teensy loader not erasing unused space.
Click to expand...

@jmarsh, when I was working on FlasherX, the bootloader always erased the entire Flash except for EEPROM or LittleFS, if they were in use. FlasherX works well if you start with a full erase, install your sketch via the bootloader, and thereafter do your updates via FlasherX, which is how I use it. For someone who is developing their application, and sometimes updating via bootloader and sometimes via FlasherX, they are likely to run into problems, as you say. Likewise if they are using flash for custom storage, they must modify the source to take that into account, and I could do a better job of documenting that.

I was not aware of _flashimagelen, and I'll incorporate that when I do an update, though it only exists for T4, and not for T3/LC.

I can see that an improvement would be to search for the largest contiguous area of blank flash, as opposed to starting from the point where the user indicates that flash is available (top of flash - FLASH_RESERVE) and assuming that the all flash below the first unerased byte is in use.
 
Frankthetech said:
OK, so the question I have is why 1 computer complies the flasherX code and only shows a 4k buffer, on my other computer it's fine and compiles with a 7000+ buffer. What I need to know is how/where does flasherX get the base address for the T4.1?
Joe, the FlasherX works just fine in my code too, just not when I compile on my main computer (4k buffer issue). I'm putting the hex file on the sd card and then run the flasherX to update.
Click to expand...
Frank, if you're working on multiple computers, check to make sure that you have the FLASH_RESERVE macro set to 64 sectors (or larger if you are using LittleFS) on all of them.
 
Joe, those settings are done as you suggest. It just looks like the base address is set too high, and flasherx sees very little room left.
Strange as it did work on both computers, Do you know where the base address is set, which file its in?
 
Frankthetech said:
It just looks like the base address is set too high, and flasherx sees very little room left.
Strange as it did work on both computers, Do you know where the base address is set, which file its in?
Click to expand...
The base address is not set. What FlasherX does is start at the TOP (highest address) of flash and work downward. First, it skips over the FLASH_RESERVE. You said you had that set to 64 sectors, and if so, that's enough to get past the EEPROM area. Then it walks downward, checking to make sure that all words are 0xFFFFFFFF (erased). When it finds a word that is not erased, it stops, and reports that the buffer size is the number of bytes from the first non-erased word to the bottom of FLASH_RESERVE.

Please do the following as @jmarsh suggested.
  • do a 15-second reset on your T4.1 (will erase all flash)
  • install your sketch using Arduino
  • trigger a FlasherX update and see what it reports for buffer size
Also, are you using EEPROM in your application? Have you modified the EEPROM file to create a larger space?
 
I was away for a week, so now I'm thinking the t4.1 is not getting erased with the 15 second button press, leaving the flash with random data somewhere causing flasherX to find un-erased data. Question, should the blink code that comes on a new t4.1 also start running after the 15 sec reset ? OK, yes it does. I was not waiting long enough after releasing the button, so from the blink program I flash my code and that works, and now flasherX is working. Not sure why, but it's working again so the full reset is the ticket. Thanks for the help.
 
Last edited:
Frank, that’s exactly what FlasherX does. After your 15 sec erase, install app, and update with FlasherX, whst was the reported buffer size? It should have been close to 8 MB less FLASH_RESERVE less the size of your sketch.
 
Joe, yep this is the same board that was reporting a 4k buffer and now after the full erase the buffer is now where I'd expect, about 7+MB
I ran FlasherX 3 times and it worked each time reporting a buffer big enough to do the update.
I was trying the 15 sec erase but I did not wait long enough for the full erase to finish, takes a while, have to wait till the default blink code runs.
 
Just a thought, if the 15 sec reset erases flash, (and since it takes a while it must be writing/clearing flash in 4k sectors)
Now I'm thinking its not working, so I just yank out the power mid flashing. Could this leave some part of flash in an undefined state?
Then I run FlasherX and it sees this undefined part in flash thinking this to be the end of the code? then the 4k buffer issue I had.
A lot of maybe's.
 
Frankthetech said:
Just a thought, if the 15 sec reset erases flash, (and since it takes a while it must be writing/clearing flash in 4k sectors)
Click to expand...
Yes, it takes a while.
Frankthetech said:
I just yank out the power mid flashing. Could this leave some part of flash in an undefined state?
Click to expand...
Yes, it could.
Frankthetech said:
Then I run FlasherX and it sees this undefined part in flash thinking this to be the end of the code?
Click to expand...
Yes, it would.

If you use more than the default 64 sectors of Flash for EEPROM, or if you use LitteFS, or if you install a very large program at some point, or you do something custom to write to high flash, any of those could leave unerased flash in the area that FlasherX expects to be empty if available for use as a buffer.
 
I'm glad to have a handle on this now. I was thinking my computer was the problem, maybe I should reinstall things, but now things make sense. Thank you for your help.
 
You must log in or register to reply here.
Back
Top