Teensy 4.1 PROGMEM remap?

P

Peter_R

Member
My project uses a large array of complex structures which I would like to retain in non-volatile memory.
Rather than setting up a LittleFS Filesystem to load and save the data, is there a way in which the PROGMEM section could be remapped to use the LitttleFS area in main Flash, or even the additional QSPI Flash?
 
the PROGMEM section could be remapped to use the LitttleFS area in main Flash
Click to expand...

Yes, this is exactly what LittleFS_Program does. See the readme file and examples for usage info.

When you call it's begin() function, you need to give the size (in bytes) of the portion of leftover program flash you want it to use. If using Teensy 4.0 which has 2MB flash, and your code is currently using 600K, you might choose to allocate 900K for LittleFS_Program, which would still work even future versions of your program grown by another 500K.

You should also be aware uploading new programs always wipes the first 512K of flash for standard Teensy or first 1024K for Lockable Teensy, even if your program is smaller. If you want your filesystem to survive across uploading new code, make sure you don't allocate so much of the program memory that LittleFS_Program uses that portion of the flash.


or even the additional QSPI Flash?
Click to expand...

For this, use LittleFS_QSPIFlash or LittleFS_QPINAND or LittleFS_QPI (which automatically detects if you have a normal vs NAND type flash chip).
 
Thanks for your reply.
I was hoping that there would be something like a definition of PROGMEM somewhere within the core files which could be changed to allocate more flash but maybe this is not considered a wise move.

I had looked at LittleFS_QSPIFlash but decided that creating a filesystem was not really what I wanted since I am only creating an array of variables (structures) which I need to have random access to.
That said, you did make me think of an alternative if redefining PROGMEM is more involved than my simple hope.
Since only one array element can be edited at a time, the array could be a filesystem and the active element loaded int RAM for editing and saved on completion.
This would have the advantages of saving on RAM usage and giving me a far large space for array expansion if desired.
 
Peter_R said:
Thanks for your reply.
I was hoping that there would be something like a definition of PROGMEM somewhere within the core files which could be changed to allocate more flash but maybe this is not considered a wise move.
Click to expand...
Sorry, I am sort of confused by your question. That is, I believe today your sketch can use all of the flash, except the EEPROM section.
That is, if you are looking at the memory section of the T4.1 product page:
1734793925273.png

The sections shown for LittleFS filesystem and unused flash, depend on the sizes of the other sections. That is PROGMEM
variables can grow up until the unused space that can be used for LittleFS is more or less 0. Or put another way, I don't
believe that there is any space reserved for LittleFS, but instead if your sketch requests to have a little FS Progmem File system of some
size, it will look at how space that is left from where the EEPROM emulation starts and sees if there is enough room from their down by the
size of your requested file system, that it does not run into the end of the initialized variables.
 
Thank you.
That would explain why I could not find anything defining the size of PROGMEM.
I am using PlatformIO in VScode so the term 'sketch' does not apply, not that that has any effect.
I shall recheck the error message I got on compile and also recalculate my expected flash usage, I may well have miscalculated and actually used it all.
 
Is your PlatformIO setup configured to run teensy_size after building the .elf file? It prints a memory usage summary specific to Teensy 4.x.

Arduino IDE always does this, and it really should be done with PlatformIO too, but sometimes people customize PlatformIO in ways we can't anticipate when answering questions.
 
Is your PlatformIO setup configured to run teensy_size after building the .elf file? It prints a memory usage summary specific to Teensy 4.x.
Click to expand...
Yes.
This is the current working version of the project with the array of structures in RAM.
teensy_size: Memory Usage on Teensy 4.1:
teensy_size: FLASH: code:97532, data:298632, headers:8312 free for files:7721988
teensy_size: RAM1: variables:381120, code:91304, padding:7000 free for local variables:44864
teensy_size: RAM2: variables:12416 free for malloc/new:511872

I have done some checks and am sure it is not a size problem but something odd is happening in PROGMEM when I move the structures.
There is a large array already in PROGMEM and when I move the array of structures in there is a clash reported ...
Moving the large array into EXTMEM stops this clash and the project compiles.
Sadly, this is not a solution since the running programme crashes when attempts are made to write values into the structures in PROGMEM, so another problem to solve.
Given the rethink I had after your initial reply, rearranging the code to make use of LittleFS would be a far more beneficial use of my efforts.
 
PROGMEM is mapped as read-only, it's intended only for const data. If you want to write to flash, you have no choice but to use something like LittleFS where you have to manually perform write operations (due to flash needed to be erased/programmed in pages).

In theory, if you learned how the MPU works you could set up a one-page "window" region and use it to do on-the-fly writing as needed but in practice the MPU doesn't really like having its regions adjusted once they're enabled.
 
Ha-ha-ha ... perhaps I should have checked that first ...
Well, I haven't lost much time and I've learnt something ...
 
If you want to write to PROGMEM arrays, first you need to define them with align attribute so they are placed within 4K sectors or 32K or 64K blocks of the flash memory. Or you can access the flash by simply computing addresses, but you need to be careful not to conflict with other usage of the flash. Erasing can only be done in 4K, 32K or 64K sizes. Erasing turns all the bits to 1s. Programming turns some 1 bits to 0s, but never a 0 back to a 1 bit. Programming can be done within 256 byte pages, where you can program between 1 to 256 bytes at a time. This is simply how the flash memory chip works, so your software must be designed to work within these hardware constraints.

Actual writing is done by sending flash memory commands into the FlexSPI hardware. It's "IPCMD" interface is used. You can find documentation in the reference manual chapter 27 starting on page 1601. Writes are never done into the actual address range of the flash memory, so this doesn't run afoul of the MPU's read-only restriction. Interrupts are disabled while the flash memory is busy. After erase or write is completed, the CPU cache is flushed. If you want to do this, you don't need to reinvent this (rather difficult) wheel. It already exists in the eeprom emulation code inside the core library. You might also look at LittleFS_Program, which simply calls the eeprom emulation functions to do the heavy lifting, but does it from C++ code. It's probably the closest to what you would want if you just wish to get write access to the raw flash memory.

There are a few possible "gotchas" to know. The "restore program" part of the flash, which is the last 4K sector, is permanently hardware write protected. Any attempt to erase or write that memory will fail. But you can overwrite anything else, so be careful not to destroy your program. If using Lockable Teensy in secure mode, you should also be aware the program image is encrypted and signed. Altering it will cause the signature to become invalid and the chip won't boot (until you use Teensy Loader to reprogram the entire program image). Altering program images on normal Teensy (or Lockable Teensy not yet locked into secure mode) will attempt to boot and run, but unless you're very crafty with self-modifying code the likely outcome is a crash which the MPU may or may not detect.
 
Last edited:
Thanks Paul.
I should have realised that there is a lot going on for storing in flash, I did it many years ago in a works project.

Since this project is an upgrade to a previous version and a step towards a far more complex version, implementing LittleFS can be done now rather than later.
I did look at the Reference Manual earlier and decided it was a lot of work for what is just an intermediate step.
 
One more "gotcha" I should mention, in case anyone else later finds this thread and tries to craft their own code or copy parts of LittleFS_Program, is compiler optimization. Even after the flash has been writtena and the CPU cache emptied of any remants of the prior data, you still need to be careful that your code actually reads the flash. Normally the compiler "knows" that memory doesn't magically change. Usually you would use a volatile pointer. But if you care about performance, volatile causes much slower code if do more than just read it once (not only does it force memory access, but CPU registers get allocated to do so, reducing the number of local variables held in the other registers which can have a substantial performance impact for complex code). There is an asm memory barrier which might let ordinary access work without the overhead of volatile. Maybe? The compiler does a lot of optimizations which normally give your code a huge performance boost, but this is one place they can really bite.
 
I have taken another look at implementing LittleFS and it seems rather involved for what I want to do so I started to consider other possible? solutions.
Battery backup for the QSPI_PSRAM and use EXTMEM for non-volatile storage ... Not a foolproof solution since the battery could discharge and lose the data.
Replace the QSPI_PSRAM with QSPI_FRAM ... Looks possible if you don't want very large devices (I've found 1 to 4 MBIT devices at the time of writing this).
Anybody looked into this as I've just done a superficial hardware compatibility check at the moment?
 
I did play with using EXTMEM as an array in a very simple way
Code: 
EXTMEM unsigned int extSong[2097151];     //8 megabytes

I'm sure you want to do something more interesting, but it is possible to use the memory without a file system.
 
Check the clock speed specs before you buy a FRAM chip. The reason we don't support FRAM on QSPI in LittleFS is because the FlexSPI clock is difficult to configure at slow speeds (like below ~50 MHz) and most of those FRAM chips don't support fast clock speed.

FRAM works great with regular SPI.
 
I realised, after posting, that I had gone somewhat off topic from my original enquiry so I did a search on PSRAM alternatives and found a 3 year old thread about using MRAM : by Michael MacDonald The MRAM he uses is of similar speed to the PSRAM (108 MHz clock).
So I decided to give it a try.

The work I have done so far revolves around the RAM based array of structures which would have to be undone and rewritten to use a filesystem.
Because of the regular use of the array, the read and write activity could be very high if just one array element was loaded for editing before moving to the next.
Also, hardware is my speciality so getting to grips with even a simple filesystem is an effort which, given the usage (to simulate non-volatile RAM) is a little bit daunting. If the project works well enough to warrant further expansion I may reconsider.
 
You should probably also anticipate needing to configure the FlexSPI hardware for whatever type of chip you connect. LittleFS does this for the chips it supports, and the core library startup code does it for the supported PSRAM.

If you connect a different chip, don't expect it to automatically show up as addressable memory at 0x70000000. You'll need to configure FlexSPI to use whatever chip you solder. At least you will have the existing startup code and LittleFS for examples. But FlexSPI is complicated, as you can see from the reference manual.
 
Hi

Is there an easy way to increase the eeprom allocated size?
For example using the 4.1 amount on a 4.0?

Or adding a second "eeprom" type of memory on other flash locations and maybe with a word size of 4 bytes instead of 1.
 
PaulStoffregen said:
You should probably also anticipate needing to configure the FlexSPI hardware for whatever type of chip you connect. LittleFS does this for the chips it supports, and the core library startup code does it for the supported PSRAM.

If you connect a different chip, don't expect it to automatically show up as addressable memory at 0x70000000. You'll need to configure FlexSPI to use whatever chip you solder. At least you will have the existing startup code and LittleFS for examples. But FlexSPI is complicated, as you can see from the reference manual.
Click to expand...
The device I have ordered is a 3V one AS3004204

Yes, I have looked at the existing core code and see that there are many issues but Michael MacDonald has seemingly addressed them in his (untested?) GitHub project.
 
You must log in or register to reply here.
Back
Top