Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 12 of 12

Thread: Flash - Storing preferences

  1. #1
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    7,642

    Flash - Storing preferences

    @Paul:

    Hello Paul,
    on my short trip to the ESP32 world i learned to enjoy the "preferences" library.
    It is very handy because it allows to store key/value pairs. This could be a PIN assignment, volume or filter settings, for example. These are retained even when reflashing.
    It is similar to a config file, but binary.
    So that different programs can use them, there is a parent namspace. For example, the same key can be stored for "Sketch1" and "Sketch2" (and have different values)

    "preferences" is the name of the Arduino interface, in the IDF it is called Non-volatile storage library. (link to docu)

    My question now: Teensy 4.x has some flash free at the top end.
    Can this be used partially? I assume you need less than the 64k.
    Is the area partially writable?
    Will it be erased when flashing (I hope not) ?

    For example, if you could use 48K of it, that would be a practical thing.

    If that doesn't work - is there any other way to reserve some memory that is not erased?

    Something like EEPROM emulation?

  2. #2
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    7,642
    LOL, forget some of my questions. It's already used for EEPROM emulation, right?
    Ok, then I must be a bit bold and ask if you would spend another 64k for such a feature

  3. #3
    Senior Member
    Join Date
    May 2015
    Location
    USA
    Posts
    832
    Seems like such a library should be built on top of teensy's EEPROM code. For many uses, far less than 64K is adequate.

  4. #4
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    8,197
    I could be wrong, but I believe it was in the LittleFS conversation(s), Probably this one: https://forum.pjrc.com/threads/58033...PIFlash/page12
    About the ability to create an FS in Program space:

    Code:
    #if defined(__IMXRT1062__)
    class LittleFS_Program : public LittleFS
    {
    public:
    	LittleFS_Program() { }
    In that discussion, if I remember correctly Paul said something along the line that it will take a change to the bootloader chip to preserve the LittleFS FS during reprogramming.

    Then again maybe I just dreamed it.... But my guess is that this level of change may be similar.

  5. #5
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    7,642
    Storing it in the EEPROm makes it useless, as the first write of a program not aware of it (=> all) will kill the data.
    But that's the trick: Unlike EEPROM, it can store settings which stay available, even if you run other sketches that save their own settings and are not aware of foreign data.

  6. #6
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    7,642
    Kurt.. maybe yes.
    Years ago, we all said ESP is crap and nothing works. Now, the software and api-side are better than the "old-style" arduino stuff.. so, times are changing...and we should see that.
    Yes, that means one should accept that and think about changing some old things.
    The questions is, why the bootloader works this way. It would be enough to erase the space needed by the actual program*.

    An other nice thing is the esp exception decoder. It is very very helpful when debugging and displays where your program crashed. the source-file, even line-number. incredible useful. it makes debugging fun. I dare not even ask about it..for Teensy.....


    *or make that configurable in a way that Arduino does not forget that flag.
    Last edited by Frank B; 11-23-2020 at 03:23 PM.

  7. #7
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,456
    Quote Originally Posted by Frank B View Post
    Ok, then I must be a bit bold and ask if you would spend another 64k for such a feature
    Probably not another fixed 64K, but....


    Quote Originally Posted by KurtE View Post
    In that discussion, if I remember correctly Paul said something along the line that it will take a change to the bootloader chip to preserve the LittleFS FS during reprogramming.
    Yes, this, more or less.

    I'm still not 100% sure of all the details, but my general plan is to make future bootloader versions scan the flash for any LittleFS filesystem and avoid erasing it. The scan will probably start at 512K or maybe 1M and be done at 64K block boundaries. That's why I made a commit to LittleFS_Program to align the filesystem to 64K.

    Maybe if we modify LittleFS_Program to also avoid using the last 64K, then that area could become de-facto reserved for a preferences library.

    Or maybe the preferences could just be stored as 1 or more files on the LittleFS volume?


    Quote Originally Posted by Frank B View Post
    Storing it in the EEPROm makes it useless, as the first write of a program not aware of it (=> all) will kill the data.
    We could also explore the possibility of some emulated EEPROM addresses being "reserved". The EEPROM emulation would preserve them, and allow them to be read but not written (at least not without a special API used only by the parameter stuff).

    But this wouldn't give much storage, maybe a few hundred bytes. This parameter stuff probably wants a model of infrequent writes (low endurance) in exchange for larger storage, right?

  8. #8
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    7,642
    Oh, I even could imagine a "hidden" file in LittleFS that survives formatting and stores the "preferences".

    Depends. It can save "blobs", too. I think the max size of a blob is around 4k (have to read about it..) But yes, it's intended for infrequent writes.

  9. #9
    Senior Member
    Join Date
    May 2015
    Location
    USA
    Posts
    832
    Besides the clear need for more structure on top of teensy non-volatile storage, there is the barrier-to-change business issue. It should be easy to port software from ESP32 to teensy. The reverse direction depends on your point of view :-).

  10. #10
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    7,642
    I have thought about it.
    I now think it would be far over-engeneered to put it in LittlFS. More disadvantages than advantages.
    Placing it in LittleFS would mean that there is an unwanted connection between them. Deleting LittleFS would mean losing all configurations. LittleFS "hacking" to not touch the configuration is more (unnecessary) work and complicates things unnecessarily.
    So, please not. Keep both unconnected.

    My Question remains. Why gets the flash deleted?
    I understand it makes sense on Teensy 3.x. It maybe was a kind of protection against "stealing" code by executing a small stub of code that "prints out" the flash.
    For Teensy 4, I have no Idea why it works this way and still deletes all the Flash?. It's dead-easy to just desolder the flash -chip and read it out. Easier than messing with code, jtag etc.
    Later, with flash encryption, there are better techniques than deleting everything (I hope



    Quote Originally Posted by jonr View Post
    Besides the clear need for more structure on top of teensy non-volatile storage, there is the barrier-to-change business issue. It should be easy to port software from ESP32 to teensy. The reverse direction depends on your point of view :-).
    Jonr, you're right :-)

  11. #11
    Senior Member
    Join Date
    May 2015
    Location
    USA
    Posts
    832
    Any easy, standards based way for a PC to access non-volatile data (eeprom or other) in a teensy would be useful. Perhaps virtual files accessed via MTP?

    For example, I run audio convolution with large impulse responses. Currently I have to recompile and flash a new program binary just to change the impulse response data. Fine for me, very hard to explain to a non-technical user.

    With a backup on PC, I have little concern about unaware programs deleting teensy non-volatile data. Happens on many devices - load incompatible firmware and you lose config data.

  12. #12
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    7,642
    For ESP, there is a tool that can do this at least for SPIFFS- without reflashing the program.
    As it is as python-tool, i think we can use (parts of) it (and I did not look at the lincense, so far)
    Search for "ESP Sketch Data Upload".
    Last edited by Frank B; 11-23-2020 at 10:59 PM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •