memory, config storage, sd cards

Status
Not open for further replies.

snowsh

Well-known member
Sorry for the vague title.

I am having some trouble working out how to store some large arrays...

as an example I have a library array:

Code:
uint8_t array[128][16][32].

I am intending to keep these libs on the SD card. In use I only need to call one of the 128 instances to use in SRAM

I have looked at arduinoJSON, none of the examples there seem to do what I am after.

I have looked at the memory chips to expland my 4.1

I have tried FLASHMEM. PROGMEM....

Im just a bit lost.

I want to be able to keep my library files - there are many - off the main SRAM as I am rapidly running out of memory. anytime I want to update the SRAM array from the library, I could edit values. these need to be stored back to the library so PROGMEM is out. FLASHMEM too I guess?

I am having a hard time finding any examples to work from..... Could anyone point me in the right direction.

Am I correct that the memory upgrade chips will give me more SRAM so I can continue working on a growing project? Will I have to do any messing about to move data around?
 
For the T_4.x this is a good set of memory details :: pjrc.com/store/teensy41.html#memory

512KB RAM1 : holds code not kept on flash with "FLASHMEM" and all runtime variables and stack. It runs at CPU speed and so does not use any of the 32KB data or code caches.

512KB RAM2 : is DMAMEM has some allocated for USB and 'other' by the 'system' - but near 500KB left that can be used for that 65KB 'array':
Code:
DMAMEM uint8_t array[128][16][32];

It runs more slowly (1/4 of CPU speed) when the 32KB data cache isn't able to cover the accessed area - but the access is normal and fully functional in the ARM Flat memory space.
Usage of a 'variable' there will not be initialized { that is: int foo=42; // won't work }- but will be allocated with normally usable pointer or other access methods.
This should work for the purpose at hand to free up RAM1 but allow reading to that RAM2 memory from SD, using/modifying, writing back to SD.

this build console output shows the memory regions in use:
Code:
Memory Usage on Teensy 4.1:
FLASH: code:55884, data:5504, headers:7212   free for files:8057864
RAM1: variables:17088, code:53048, padding:12488   free for local variables:441664
RAM2: variables:12384  free for malloc/new:511904
 
And as the link defragster mentioned (https://www.pjrc.com/store/teensy41.html#memory), on the Teensy 4.1, you can add two more memory modules via soldering the chips to the Teensy 4.1. You can add either 2 PSRAM chips (8 megabytes each) or 1 PSRAM chip and 1 FLASH memory chip. The PSRAM chips are reset when the teensy is power cycled. The FLASH chips keep their state. Generally, the FLASH memory chip is used to hold a Littlefs file system. These memory chips are slower than the main SRAM, except when items are in the cache.

If your soldering skills are not up to SMT (surface mount soldering), the company protosupplies.com offers a Teensy 4.1 with a PSRAM and FLASH chip soldered on to the chip.
 
@defragster well DMAMEM has solved it for now..... I will order some of the 8meg ram chips and see what I can do with them. Thanks to all for the advice!
 
@defragster well DMAMEM has solved it for now..... I will order some of the 8meg ram chips and see what I can do with them. Thanks to all for the advice!

Great that was the answer!

pjrc.com/store/psram.html PSRAM's are fun :) They are volatile when power is lost - and they are not FULL speed - but the 1062 has hardware QSPI direct I/O access so use is just like DMAMEM or other things:
Code:
EXTMEM uint8_t array[128][16][32];
They are covered by data cache so I/O within that are full speed.

The chips have legs (not like some WSON FLASH that work okay too) so when properly oriented and the first leg tacked down square the others can go easy ( as long as there are no pins in the way ).
Done more than a few of them using flux paste gets good smooth liquid solder connection - then an alcohol brushing to make sure they are clean as needed for their high speed access.
 
Great that was the answer!

pjrc.com/store/psram.html PSRAM's are fun :) They are volatile when power is lost - and they are not FULL speed - but the 1062 has hardware QSPI direct I/O access so use is just like DMAMEM or other things:
Code:
EXTMEM uint8_t array[128][16][32];
They are covered by data cache so I/O within that are full speed.

The chips have legs (not like some WSON FLASH that work okay too) so when properly oriented and the first leg tacked down square the others can go easy ( as long as there are no pins in the way ).
Done more than a few of them using flux paste gets good smooth liquid solder connection - then an alcohol brushing to make sure they are clean as needed for their high speed access.

@defragster so its SMD hot air soldering only? I have a steady hand and a sharp soldering bit.......

If I decide to go with the flash memory too, how do you reference that?
 
@defragster so its SMD hot air soldering only? I have a steady hand and a sharp soldering bit.......

If I decide to go with the flash memory too, how do you reference that?

I used a decent Sparkfun iron for all done here - works fine. Got finer gauge solder that worked well, had some solder paste but it is getting old.

For reference of PSRAM it is EXTMEM, Flash takes 'helper functions' as implemented in LITTLEFS ( at least for any writes )::
> https://forum.pjrc.com/threads/58033-LittleFS-port-to-Teensy-SPIFlash

> just edited that post to show the supported FLASH chips known in LittleFS
 
@defragster so its SMD hot air soldering only? I have a steady hand and a sharp soldering bit.......

If I decide to go with the flash memory too, how do you reference that?

I've been able to solder the PSRAM chips on using my normal soldering iron and thinner solder.
 
So I now have some 8meg PSRAM modules. I have soldered one to a new 4.1, all good. memory test succesful and I seem to be able to store to it in the following way:

Code:
const EXTMEM char keyboardInputMap[8][10] = {
  {32, 49, 97, 98, 99, 65, 66, 67},                                             // SPACE a,b,c,A,B,C,1
  {50, 100, 101, 102, 68, 69, 70},                                              // d,e,f,D,E,F,2
  {51, 103, 104, 105, 71, 72, 73},                                              // g,h,i,G,H,I,3
  {52, 106, 107, 108, 74, 75, 76},                                              // j,k,l,J,K,L,4
  {53, 109, 110, 111, 77, 78, 79},                                              // m,n,o,M,N,O,5
  {54, 112, 113, 114, 115, 80, 81, 82, 83},                                     // p,q,r,s,P,Q,R,S,6
  {55, 116, 117, 118, 56, 84, 85, 86},                                          // t,u,v,T,U,V,7,8
  {57, 48, 119, 120, 121, 122, 87, 88, 89, 90}                                  // w,x,y,z,W,X,Y,Z,9,0
};

compiles fine.

When I check, any use of this memory shows garbage text. What am I missing? Do I have to do somthing to pull the memory back from EXTMEM?
 
So I now have some 8meg PSRAM modules. I have soldered one to a new 4.1, all good. memory test succesful and I seem to be able to store to it in the following way:

Code:
const EXTMEM char keyboardInputMap[8][10] = {
  {32, 49, 97, 98, 99, 65, 66, 67},                                             // SPACE a,b,c,A,B,C,1
...
};

compiles fine.

When I check, any use of this memory shows garbage text. What am I missing? Do I have to do somthing to pull the memory back from EXTMEM?

Yes, something must be done ...
See this link in post #2 : pjrc.com/store/teensy41.html#memory

As noted in that post #2 EXTMEM :: Usage of a 'variable' there will not be initialized { that is: int foo=42; // won't work }- but will be allocated with normally usable pointer or other access methods.

EXTMEM - Variables defined with EXTMEM are placed in the optional PSRAM memory chip soldered to the QSPI memory expansion area on bottom side of Teensy 4.1. These variables can not be initialized, your program must write their initial values, if needed.

In setup() perhaps that initialized data would be copied from PROGMEM to EXTMEM:
Code:
PROGMEM char keyboardInputMap_init[8][10] = {
  {32, 49, 97, 98, 99, 65, 66, 67},                                             // SPACE a,b,c,A,B,C,1
...
  {57, 48, 119, 120, 121, 122, 87, 88, 89, 90}                                  // w,x,y,z,W,X,Y,Z,9,0
};
EXTMEM char keyboardInputMap[8][10];
...
setup() {
   // not sure if this is sufficient or correct for sizeof()
   memcpy( keyboardInputMap, keyboardInputMap_init, sizeof( keyboardInputMap_init ) );
};
 
I've been able to solder the PSRAM chips on using my normal soldering iron and thinner solder.

Ditto that. Putting a tiny bit of solder on kitty corner pads (and then taking most of it off with solder wick) seems to help me tack the chip down and then go to work on the remaining pins. Stone age, I know, but it works for me.
 
Yes, something must be done ...
See this link in post #2 : pjrc.com/store/teensy41.html#memory

As noted in that post #2 EXTMEM :: Usage of a 'variable' there will not be initialized { that is: int foo=42; // won't work }- but will be allocated with normally usable pointer or other access methods.



In setup() perhaps that initialized data would be copied from PROGMEM to EXTMEM:
Code:
PROGMEM char keyboardInputMap_init[8][10] = {
  {32, 49, 97, 98, 99, 65, 66, 67},                                             // SPACE a,b,c,A,B,C,1
...
  {57, 48, 119, 120, 121, 122, 87, 88, 89, 90}                                  // w,x,y,z,W,X,Y,Z,9,0
};
EXTMEM char keyboardInputMap[8][10];
...
setup() {
   // not sure if this is sufficient or correct for sizeof()
   memcpy( keyboardInputMap, keyboardInputMap_init, sizeof( keyboardInputMap_init ) );
};

oh.... wel that scuppers my plan.... ok. So I can allocate the memory in the PSRAM but i need to then initialise the data during runtime, setup for example.....

OK. my issue is im running out of memory. So, I guess I need to move to the SD card for my library. the example array above is not really what I wanted the extra memory for. I am building a music sequencer which is very memory heavy.

Any suggestions on how to store multidimensional arrays in SD and pull them into main memory when needed, I can already edit them. I need to be able to store them back the the SD....

ideas?
 
oh.... wel that scuppers my plan.... ok. So I can allocate the memory in the PSRAM but i need to then initialise the data during runtime, setup for example.....

OK. my issue is im running out of memory. So, I guess I need to move to the SD card for my library. the example array above is not really what I wanted the extra memory for. I am building a music sequencer which is very memory heavy.

Any suggestions on how to store multidimensional arrays in SD and pull them into main memory when needed, I can already edit them. I need to be able to store them back the the SD....

ideas?

Are you sure? The T_4.1 has an 8MB Flash so a decent amount of MB free after code. That could be pulled into PSRAM - the copy doesn't take any memory - just FLASH storage. If the data doesn't need to be changed it could be read from PROGMEM:FLASH.

Certainly the SD is there and provides much more space. Anything stored in a known format on SD could be read back using available file I/O.

Using TeensyDuino 1.54 Beta 9 or later has the SD.H library for access and that now includes the faster more complete SdFat library with alternate access methods. There are examples included that should allow enough info to craft a write and read process for accessing the needed data.
 
Are you sure? The T_4.1 has an 8MB Flash so a decent amount of MB free after code. That could be pulled into PSRAM - the copy doesn't take any memory - just FLASH storage. If the data doesn't need to be changed it could be read from PROGMEM:FLASH.

Certainly the SD is there and provides much more space. Anything stored in a known format on SD could be read back using available file I/O.

Using TeensyDuino 1.54 Beta 9 or later has the SD.H library for access and that now includes the faster more complete SdFat library with alternate access methods. There are examples included that should allow enough info to craft a write and read process for accessing the needed data.

no the data needs to be editable. But PROGMEM:FLASH is a new option for me, I will look into it for placing some predefined libraries... how would I use that with a struct? I keep getting compile errors wjen trying to use EXTMEM etc....

Code:
struct drummerConfig {
  uint8_t screenMode;
  uint8_t midiOutChannel;
  uint8_t resolution;
  uint8_t stepLength;
  uint8_t kitId;
  char kitName[CHANNEL_NAME_LENGTH];
  int kitColors[4];
  char instrumentNames[NUMBER_OF_DRUMMER_INSTRUMENTS][CHANNEL_NAME_LENGTH];
  uint8_t instrumentNoteNumbersAr[NUMBER_OF_DRUMMER_INSTRUMENTS];
  uint8_t instrumentMidiOutChannel[NUMBER_OF_DRUMMER_INSTRUMENTS];
  uint8_t instrumentVelocity[NUMBER_OF_DRUMMER_INSTRUMENTS];        // set low to allow for accents
  uint8_t instrumentStepLength[NUMBER_OF_DRUMMER_INSTRUMENTS];
  uint8_t instrumentResolution[NUMBER_OF_DRUMMER_INSTRUMENTS];
  bool instrumentDirection[NUMBER_OF_DRUMMER_INSTRUMENTS];        // 0 forward 1 reverse
} drummer;

void initDrummer()
{
  drummer.screenMode = 0;
  drummer.midiOutChannel = 1;
  drummer.resolution = 6;
  drummer.stepLength = 32;
  drummer.kitId = 0;

  drummer.kitColors[0] = MONOLITH_808_RED;
  drummer.kitColors[1] = MONOLITH_ORANGE;
  drummer.kitColors[2] = MONOLITH_808_YELLOW;
  drummer.kitColors[3] = MONOLITH_808_WHITE;

  for (uint8_t i = 0; i < NUMBER_OF_DRUMMER_INSTRUMENTS; i++)
  {
//    drummer.instrumentNames = {};
//    drummer.instrumentNoteNumbersAr = {};
    drummer.instrumentMidiOutChannel[i] = {1};
    drummer.instrumentVelocity[i] = {60};        // set low to allow for accents
    drummer.instrumentStepLength[i] = {32};
    drummer.instrumentResolution[i] = {6};
    drummer.instrumentDirection[i] = {0};        // 0 forward 1 reverse
  }
}
 
no the data needs to be editable. But PROGMEM:FLASH is a new option for me, I will look into it for placing some predefined libraries... how would I use that with a struct? I keep getting compile errors wjen trying to use EXTMEM etc....
...

Post #10 showed PROGMEM for the constant image that stays in FLASH.

The code shown isn't a complete program - so I can't play with it directly and compile ...

But it shows something that wouldn't work to have a PROGMEM or const drummerConfig and then modify it. It shows all the initialization done the runtime in initDrummer().

Given that is not provided any compile time initial values - it could be just as well defined for EXTMEM .... perhaps:
Code:
struct drummerConfig {
  uint8_t screenMode;
  uint8_t midiOutChannel;
  uint8_t resolution;
  uint8_t stepLength;
  uint8_t kitId;
  char kitName[CHANNEL_NAME_LENGTH];
  int kitColors[4];
  char instrumentNames[NUMBER_OF_DRUMMER_INSTRUMENTS][CHANNEL_NAME_LENGTH];
  uint8_t instrumentNoteNumbersAr[NUMBER_OF_DRUMMER_INSTRUMENTS];
  uint8_t instrumentMidiOutChannel[NUMBER_OF_DRUMMER_INSTRUMENTS];
  uint8_t instrumentVelocity[NUMBER_OF_DRUMMER_INSTRUMENTS];        // set low to allow for accents
  uint8_t instrumentStepLength[NUMBER_OF_DRUMMER_INSTRUMENTS];
  uint8_t instrumentResolution[NUMBER_OF_DRUMMER_INSTRUMENTS];
  bool instrumentDirection[NUMBER_OF_DRUMMER_INSTRUMENTS];        // 0 forward 1 reverse
};

EXTMEM drummerConfig  drummer;
 
Status
Not open for further replies.
Back
Top