Best way to organize eeprom locations

U

Userli

Member
Hi,

I'm using a PID controller and not to loose the tuning values on re programming, those are, amongst others, stored in the simulated EEPROM.
I was wondering, what would be the best way to define the locations in the EEPROM. Ideally you don't want to care about the variable size, so sizeof should be used. This, however, refers to the variable before, which makes the code break easily on reordering of the EEPROM content. This also means that additions can easily be messed up, when referring to the wrong variables sizes. The best I could come up with by now is the following:

Code: 
#define EEPROMLocation(x, y) const uint16_t x = (eepromLocationCount += sizeof(y)) - sizeof(y)

unsigned int eepromLocationCount = 0;
float pidP = 0.005, pidI = 0.0001, pidD = 0.01;
EEPROMLocation(pidPeEPromLocation, pidP);
EEPROMLocation(pidIeEPromLocation, pidI);
EEPROMLocation(pidDeEPromLocation, pidD);

How do you define your EEPROM layout?
 
If I'm being lazy and know I won't be near the EEPROM size, I define a struct that gets written to and read from EEPROM. As an example:

Code: 
struct Config {
  float p;
  float i;
  fload d;
};

uint8_t buf[sizeof(Config)];

/* Write to EEPROM */
void Write(uint8_t *buffer, std::size_t len) {
  if (buffer) {
    for (std::size_t i = 0; i < len; i++) {
      EEPROM.write(i, buffer[i]);
    }
  }
}
/* Read from EEPROM */
void Read(uint8_t *buffer, std::size_t len) {
  if (buffer) {
    for (std::size_t i = 0; i < len; i++) {
      buffer[i] = EEPROM.read(i);
    }
  }
}

void Load(Config * const cfg) {
  if (!cfg) {return;}
  Read(buf, sizeof(buf));
  memcpy(cfg, buf, sizeof(cfg));
}

void Save(const Config &cfg) {
  memcpy(buf, cfg, sizeof(buf));
  Write(buf, sizeof(buf));
}

The Write and Read are helper functions used to write and read buffers to EEPROM. The Load and Save are the functions actually used to load the struct from EEPROM and to save the struct to EEPROM. I also typically define a two byte header and a two byte checksum. I use the two byte header to determine if I've written the configuration to EEPROM, if I haven't, I can load and save a default configuration. The two byte checksum (i.e. a Fletcher 16) is used to make sure the EEPROM didn't get corrupted.

I mean that this is ok if you're lazy because there could be padding bytes added in the struct, so we're not necessarily making the most efficient use of the EEPROM memory.

I use this a ton, I should probably just write a templated class, so I never have to think about it again.
 
It also may depend on what processor you are doing this on? Some may have built in EEPROM support. Others like Teensy 4.x may emulate it.
The 4.x emulation tries to spread your writes across the different sectors that are reserved for the EEPROM (15 on T4, 63 on T4.1 and Micromod).
They are split up again to go across the different sectors in groups of 4 bytes.
That is address 0-3 go to first sector, 4-7 go to sector 1, 8-11 sector 2... And then when you get up to 4*number_sectors, it wraps back to sector 0...

Each new write, into sector goes to the first free byte in that sector, and the read code knows the last value wins. (I don't remember if it checks first to see if wrote the same value if skips it or not)... When a sector fills up, it then compresses out all of the duplicates, it then erases that sector and writes it out with only the unique values...

Sorry, not sure if that level of detail helps you out or not.
 
brtaylor said:
If I'm being lazy and know I won't be near the EEPROM size, I define a struct that gets written to and read from EEPROM. As an example:

Code: 
struct Config {
  float p;
  float i;
  fload d;
};

uint8_t buf[sizeof(Config)];

/* Write to EEPROM */
void Write(uint8_t *buffer, std::size_t len) {
  if (buffer) {
    for (std::size_t i = 0; i < len; i++) {
      EEPROM.write(i, buffer[i]);
    }
  }
}
/* Read from EEPROM */
void Read(uint8_t *buffer, std::size_t len) {
  if (buffer) {
    for (std::size_t i = 0; i < len; i++) {
      buffer[i] = EEPROM.read(i);
    }
  }
}

void Load(Config * const cfg) {
  if (!cfg) {return;}
  Read(buf, sizeof(buf));
  memcpy(cfg, buf, sizeof(cfg));
}

void Save(const Config &cfg) {
  memcpy(buf, cfg, sizeof(buf));
  Write(buf, sizeof(buf));
}

The Write and Read are helper functions used to write and read buffers to EEPROM. The Load and Save are the functions actually used to load the struct from EEPROM and to save the struct to EEPROM. I also typically define a two byte header and a two byte checksum. I use the two byte header to determine if I've written the configuration to EEPROM, if I haven't, I can load and save a default configuration. The two byte checksum (i.e. a Fletcher 16) is used to make sure the EEPROM didn't get corrupted.

I mean that this is ok if you're lazy because there could be padding bytes added in the struct, so we're not necessarily making the most efficient use of the EEPROM memory.

I use this a ton, I should probably just write a templated class, so I never have to think about it again.
Click to expand...

I agree with using a struct however, as a c++ newby, I do not understand your code. It also does NOT compile.

I also tend to use #pragma pack(push,1)....#pragma pack(pop) to compact the data. I use this mainly when talking to two different MCU types over radio or serial. It also helps to compact the data for EEPROM storage.

Below is my suggestion for what it's worth:
#include <Arduino.h>
#include <EEPROM.h>

#pragma pack(push,1)

struct configType {
uint16_t dataId;
float p;
float i;
float d;
};

configType cfg;

#pragma pack(pop)

uint32_t cfgSaveAddr;
const uint16_t signature = 0xDFEA;

bool loadData(uint32_t addr) {
EEPROM.get(addr,cfg);
return (cfg.dataId == signature);
}

void saveData( uint32_t addr) {
cfg.dataId = signature;
EEPROM.put(addr, cfg);
}

void setup() {
Serial.begin(9600);
cfgSaveAddr = 0;
if (!loadData(cfgSaveAddr)) {
saveData(cfgSaveAddr); //...with default data
}
}

void loop() {}
Click to expand...
 
brtaylor said:
The Write and Read are helper functions used to write and read buffers to EEPROM. The Load and Save are the functions actually used to load the struct from EEPROM and to save the struct to EEPROM. I also typically define a two byte header and a two byte checksum. I use the two byte header to determine if I've written the configuration to EEPROM, if I haven't, I can load and save a default configuration. The two byte checksum (i.e. a Fletcher 16) is used to make sure the EEPROM didn't get corrupted.

I mean that this is ok if you're lazy because there could be padding bytes added in the struct, so we're not necessarily making the most efficient use of the EEPROM memory.
I use this a ton, I should probably just write a templated class, so I never have to think about it again.
Click to expand...

If memory serves, the EEPROM code was changed some time ago so that it used templates for the get and put functions. Thus you can write/read the whole structure all at one go without having to convert it to bytes and write each individual byte. I would have to imagine this is more efficient, since it does one write/read instead of multiples writes/reads for each byte.
 
Not sure about template support ... but the post #4 shows use of .get() and .put() that will transfer a structure/Byte group of known size: docs.arduino.cc/learn/built-in-libraries/eeprom#get

Looking at T_4 sources recently the EEPROM lib for writes it seems to read verify each byte and only write on changed value with p#3 noted process.

Code: 
put()   Description   Write any data type or object to the EEPROM.
EEPROM.put(address, data)
Parameters
  address: the location to write to, starting from 0 (int)
  data: the data to write, can be a primitive type (eg. float) or a custom struct
 
Thanks for all the suggestions. A struct is the way to go. It's also good to know, that the processor distributes the writes to use the maximum write cycle lifetime optimally.
I'm not sure about the packing yet. Whilst it saves space in the eeprom, it apparently reduces the memory access efficiency due to the on aligned data.
Does somebody know the real performance effect in numbers?
Modifying the put and get to do the (un)packing, would seem to me an elegant solution, if needed.
 
Userli said:
...
Modifying the put and get to do the (un)packing, would seem to me an elegant solution, if needed.
Click to expand...

Not seeing value in modifying? If unpacked the unused bytes won't be inclined to change - they'll take space, but not cause wear on subsequent updates - hopefully that won't be a problem. If packed the sizeof() the struct will be smaller and only those bytes written.
 
Not seeing value in modifying
Click to expand...
If space is not an issue, I would not pack. If space is an issue and the performance impact is negligible I would pack. If space is an issue and the performance impact is non negligible I would read/write individual variables and not a struct.
By modifying the get/put I thought you could optimize performance and space usage and keep a simple interface. I missed that the struct structure is not known to get/put and thus this will not work.
 
I missed that the struct structure is not known to get/put and thus this will not work.
Click to expand...
Struct does work with EEPROM.Get() and EEPROM.put.

put()
Description
Write any data type or object to the EEPROM.

Syntax
COPY
EEPROM.put(address, data)
Parameters
address: the location to write to, starting from 0 (int)

data: the data to write, can be a primitive type (eg. float) or a custom struct

Returns
A reference to the data passed in

Note: This function uses EEPROM.update() to perform the write, so does not rewrites the value if it didn't change.

Example
COPY
#include <EEPROM.h>

struct MyObject {
float field1;
byte field2;
char name[10];
};

void setup() {

Serial.begin(9600);
while (!Serial) {
; // wait for serial port to connect. Needed for native USB port only
}

float f = 123.456f; //Variable to store in EEPROM.
int eeAddress = 0; //Location we want the data to be put.


//One simple call, with the address first and the object second.
EEPROM.put(eeAddress, f);

Serial.println("Written float data type!");

/** Put is designed for use with custom structures also. **/

//Data to store.
MyObject customVar = {
3.14f,
65,
"Working!"
};

eeAddress += sizeof(float); //Move address to the next byte after float 'f'.

EEPROM.put(eeAddress, customVar);
Serial.print("Written custom data type! \n\nView the example sketch eeprom_get to see how you can retrieve the values!");
}

void loop() { /* Empty loop */ }
Click to expand...
 
Struct does work with EEPROM.Get() and EEPROM.put.
Click to expand...
Yes, but the struct is treated as a binary array. To optimize, get/put would need to know the contained types and locations.
 
Userli said:
Yes, but the struct is treated as a binary array. To optimize, get/put would need to know the contained types and locations.
Click to expand...
True, hence the use of pack.

I mainly use pack where I am communicating between different MCU types over radio or serial. One does not know that ESP32 will pack the same as Teensy for example.
 
Userli said:
Yes, but the struct is treated as a binary array. To optimize, get/put would need to know the contained types and locations.
Click to expand...

As mentioned you could use the #pragma packed... stuff or alternatively use the __attribute__ stuff like
typedef struct __attribute__((packed, aligned(4))) {
...

Note: this could also be aligned(1) or skip the aligned part...

Or when possible, simply align the items in your struct in a naturally packed order. For example put all of the 4 byte items first, 2 bytes next, 1 byte at end... Or carefully count how many bytes are used and make sure each field is naturally packed.

As for being optimized? Again, for me hard to say what is meant by optimized...
 
If you care about optimizing runtime performance, create global or static variables to hold a copy of your tuning parameters. Read the (emulated) EEPROM at startup to initialize those variables. If you change the tuning, obviously you would write to both.

Ordinary variables are allocated in DTCM, by far the fastest memory. It clocks at the same speed as the CPU (600 MHz by default) and is connected with a 64 bit bus which allows the CPU to fetch 2 words within the same cycle, if they're aligned properly.

Flash memory is much slower, using a 4 bit bus with considerable overhead, clocked at appox 100 MHz. But it is cached by the 32K data cache inside the Cortex-M7 processor, so you only suffer the slow access for cache misses. A separate 32K cache is used for instructions (but most code runs from ITCM which doesn't use the cache) so code executing directly from flash doesn't rob the data cache. The data cache is 4 way associative, so some use of RAM2 (which runs at 1/4 the CPU speed and is cached) may or may not result in more cache misses. Actually reading from the flash involves a linear search within a 4K flash sector, so if the sector has been nearly filled with prior values written, your chances of a cache miss are greater.

Writing to EEPROM typically takes only dozens of microseconds, but can potentially be a lengthy operation if a 4K sector needs to be erased.

So again, if you want your PID algorithm to always run fast, create global or static variables to hold a copy of the tuning parameters. The DTCM memory is always fast and never suffers cache misses, so you get consistent best performance.
 
You must log in or register to reply here.
Back
Top