TeensyLC EEPROM emulation

Status
Not open for further replies.

darrenji

Member
Hi,

I'm seeing an issue where every 3-5 times I re-flash the Teensy, the data in the emulated EEPROM appears to be getting corrupted.
I'm saving a 1 byte version number at address zero, and an array of packed structs at address one.
EEPROM.put(PRESET_DATA_ADDR, channels);
Total data is 109 bytes.

This works in general, but fairly often after updating the firmware, the value at address zero reads back a 0 instead of the version number I wrote there.

Curious if this mechanism has been reliable for other folks.
 
Can you craft a simple sketch that demonstrates the corruption?

How much beyond the affected one byte is corrupted?

not sure if writing a whole 4 bytes not be more reliable - not sure of the micro details

Also how often has this same address written to?

Is that write at a safe and rare place or is it done so often that it could be interrupted by an upload?

There is some multiple of bytes away that will use an alternate subset of flash pages for emulation
 
Can you craft a simple sketch that demonstrates the corruption?
How much beyond the affected one byte is corrupted?

I'll work on a simplified sketch and provide more details on the extent of the corruption this afternoon.

How much beyond the affected one byte is corrupted?

not sure if writing a whole 4 bytes not be more reliable - not sure of the micro details

I tried using 4 bytes for the version_id but same problem persisted

Also how often has this same address written to?

Address zero where the version_id is stored is only written to once when the firmware first runs.
The configuration data writes to address 1-108 happen only after the user interacts with the encoders on the device.

Is that write at a safe and rare place or is it done so often that it could be interrupted by an upload?

The write is never done without a user interaction, so couldn't be interrupted by an upload except by deliberate action. The firmware prints a message when it stores data to EEPROM so I can see that it's not happening more frequently than expected.

Thanks for clarifying questions... I'll post again when I have more info.
 
I'll work on a simplified sketch and provide more details on the extent of the corruption this afternoon.

Please try to focus on the simplified program and clear steps to follow to reproduce the problem. The extent of corrupted data isn't really important. Even 1 byte not stored as it should be is plenty to get me to investigate. No need to waste a lot of time documenting how much the problem happens. Even pretty obscure issues to get investigated when there's a way to reproduce them.

But if I can't reproduce the problem from the code and steps given, you can be sure nothing will be done to fix it. Clearly written instructions about how to use the code can make all the difference. I will try to use whatever you provide. Please understand we get all sorts of reports of problems on this forum. Some really are bugs, and those usually do get fixed, but the first essential step is a way to reproduce the problem.
 
Ok cool, thanks Paul. It was happening to me all afternoon yesterday, and today I'm having a hard time reproducing reliably. Will keep at it.
 
Is it theoretically possible that a trivial interrupt (like a button press or encoder turn that just increments a variable and returns) could bork the EERPOM.put()?
Should I be disabling interrupts during the EEPROM write?
 
The eeprom emulation code disables interrupts in the places it needs exclusive access. So no, interrupts aren't supposed to mess up writing to the emulated eeprom space.

At least that's the way it's supposed to work. I'm denying the possibility of a bug. It's unlikely for such mature code, but not something I ever want to completely rule out.

Interrupts in general are tricky to use properly. All sorts of difficult to diagnose problems tend to happen if you're not extremely careful.
 
Understood.
I've written a lot of interrupt code over the years for my day jobs, so I feel relatively confident that I'm not doing anything obviously stupid/crazy, but there's still room for accidentally stupid/crazy.
This problem has turned out to be infuriatingly difficult to reproduce. So far it literally only happens when I'm not trying to make it happen, yay.

It's looking more likely that it's a corrupt-on-write issue. Previously I'd only seen it after re-flashing, but once yesterday I saw the problem after a simple power cycle.
After 50 power cycles this morning, it hasn't happened again. So I'm going to focus on writes now and see if I can find the timing or order of events that triggers it.

The one hint I've gotten is that it's not just the byte at address zero that's changing to a 0, it's every other byte for the first 16 or so bytes. I don't have the exact number yet because I forgot that reflashing erases the output in the serial monitor window and I lost the data when I forgot to copy it.

Hopefully I can narrow it down today and either find the problem or have a reproducible test case that I can post.
 
OK I finally have narrowed it down to a trivial example that demonstrates the problem.

My test program writes to emulated EEPROM a preset format version number (3) at address zero. This only happens at power on if corruption is detected (or if the firmware has never been run), i.e. something other than 3 was read from address zero.

The program then writes 104 bytes (all 1) starting at address four.

What I've discovered is that periodically, the first write of 104 data bytes after power on corrupts address zero. This ONLY ever occurs on the first write after a power cycle.

You can see this in the output as follows, a 3 is written and a 1 is read back:
Code:
Serial Initialized
preset version
3
preset data size: 104
Data to write:
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1  
saved preset
Data read back:
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1  
 
read back preset version: 1
EEPROM DATA CORRUPTION DETECTED after write.
Halting test

After writing a 3 to address zero, and 104 1s to address 4, it reads back a 1 from address zero. Somehow writing the 104 bytes at address 4 has changed the contents at address 0.

Hopefully this is enough for y'all to either tell me I did something really dumb in the code, or provide a path forward to debug at a lower level.

Thanks!

Darren
 

Attachments

  • TLC_EEPROM_TEST.ino
    3.3 KB · Views: 72
Here's an even simpler version. It just uses a byte array instead of the more elaborate array of structs.

Code:
#include <EEPROM.h>


#define PRESET_VERSION          3

#define PRESET_VERSION_ADDR     0
#define PRESET_DATA_ADDR        4

static uint8_t preset_data[104];
static uint8_t preset_data_copy[104];

void PresetInit(void)
{
    memset((void *)preset_data, 1, sizeof(preset_data));  
}


//================================================================================
void setup(void)
{ 
    Serial.begin(9600);

    delay(500);

    for (int i = 0; i < 20; i++)
    {
        Serial.println();
    }

    Serial.println("Serial Initialized");

    // Only read preset from EEPROM if we know what version it is
    uint8_t byte = EEPROM.read(PRESET_VERSION_ADDR);
    Serial.println("preset version");
    Serial.println(byte);
    Serial.print("preset data size: ");
    Serial.println(sizeof(preset_data));

    // Compare stored preset version without currently expected version.
    // If they are the same, read the data into memory.
    // If they are not the same, re-write EEPROM with current version and data format
    if (byte == PRESET_VERSION)
    {
        EEPROM.get(PRESET_DATA_ADDR, preset_data);
    }
    else
    {
        Serial.println("EEPROM DATA CORRUPTION DETECTED at power on, reformatting.");
        EEPROM.get(PRESET_DATA_ADDR, preset_data);

        uint8_t *data_ptr = (uint8_t *)preset_data;
        for (uint8_t i = 0; i < sizeof(preset_data); i++)
        {
            Serial.print(*data_ptr); Serial.print(" ");
            data_ptr++;
        }
        Serial.println(" ");
        Serial.println(" ");
        Serial.println(" ");
        Serial.println(" ");

        Serial.println("Reformatting preset in EEPROM...");
        
        PresetInit();

        EEPROM.write(PRESET_VERSION_ADDR, PRESET_VERSION);
        EEPROM.put(PRESET_DATA_ADDR, preset_data);

        delay(5000);
    }
}

void loop()
{
    static bool data_was_saved = false;

    // Write data to EEPROM and do read-back/compare peridically to test.
    if (!data_was_saved)
    {
        Serial.println("Data to write:");
        uint8_t *data_ptr = (uint8_t *)preset_data;
        for (uint8_t i = 0; i < sizeof(preset_data); i++)
        {
            Serial.print(*data_ptr); Serial.print(" ");
            data_ptr++;
        }
        Serial.println(" ");

        EEPROM.put(PRESET_DATA_ADDR, preset_data);

        Serial.println("saved preset");

        EEPROM.get(PRESET_DATA_ADDR, preset_data_copy);
        Serial.println("Data read back:");
        data_ptr = (uint8_t *)&preset_data_copy;
        for (uint8_t i = 0; i < sizeof(preset_data_copy); i++)
        {
            Serial.print(*data_ptr); Serial.print(" ");
            data_ptr++;
        }
        Serial.println(" ");
        Serial.println(" ");

        uint8_t byte = EEPROM.read(PRESET_VERSION_ADDR);
        Serial.print("read back preset version: ");
        Serial.println(byte);

        if (byte != PRESET_VERSION)
        {
            Serial.println("EEPROM DATA CORRUPTION DETECTED after write.");
        }

        data_was_saved = true;
        Serial.println("Halting test");
    }
}
 
Are there any restrictions on how to use the EEPROM library? I've looked at the examples and the reference docs. Can I treat the storage as random byte addressable (i.e. I can write to any location without having to think about other memory locations)? Is there any need for me to do a read/erase/write cycle?
 
If I use EEPROM.update() in a loop to write one byte at a time instead of EEPROM.put() to write the whole data structure, (so far, after 50 attempts) I don't see the corruption.
 
If I use EEPROM.update() in a loop to write one byte at a time instead of EEPROM.put() to write the whole data structure, (so far, after 50 attempts) I don't see the corruption.

That's good info for @Paul - the .get .put will be a separate code path added for Arduino added support.
 
I think I may be encountering this bug also (or misunderstanding something about EEPROM! :) )

Here's some sample code where I write random bytes and check them a few seconds later:

Code:
#include <EEPROM.h>

unsigned attempts = 0;
byte testData[128];

void setup() {

  Serial.begin(38400);
  
  for(int i = 0; i < 128; i++) {
    testData[i] = EEPROM.read(i);
  }
}

void loop() {
  
  Serial.print("Attempt ");
  Serial.println(attempts);
  
  for(int i = 0; i < 128; i++) {

    if(EEPROM.read(i) != testData[i]) {
      Serial.print("Data error ");
      Serial.println(i);
    }
    
    testData[i] = random(100);
    EEPROM.update(i, testData[i]);
  }
  Serial.println("----");

  attempts++;
  
  delay(3000);
}

I've tried running this on two Teensy LCs (one brand new) and writes usually fail partially on the first several attempts after power on. Here's a typical log:

Code:
Write attempt 1
Data error 0
Data error 1
Data error 2
Data error 3
Data error 4
Data error 5
Data error 6
Data error 7
Data error 8
Data error 9
Data error 10
Data error 11
Data error 12
Data error 13
Data error 14
Data error 15
Data error 16
Data error 17
Data error 18
Data error 19
Data error 20
Data error 21
Data error 22
Data error 23
Data error 24
Data error 25
Data error 26
Data error 27
Data error 28
Data error 29
Data error 30
Data error 31
Data error 32
Data error 33
Data error 34
Data error 35
Data error 36
Data error 37
Data error 38
Data error 39
Data error 40
Data error 41
Data error 42
Data error 43
Data error 44
Data error 45
Data error 46
Data error 47
Data error 48
Data error 49
Data error 50
Data error 52
Data error 53
Data error 54
Data error 55
Data error 56
Data error 57
Data error 58
Data error 59
Data error 60
Data error 61
Data error 62
Data error 63
Data error 64
Data error 65
Data error 66
Data error 67
Data error 68
Data error 69
Data error 70
Data error 71
Data error 72
Data error 73
Data error 74
Data error 75
Data error 76
Data error 77
Data error 78
Data error 79
Data error 80
Data error 81
Data error 82
Data error 83
Data error 84
Data error 85
Data error 86
Data error 87
Data error 88
Data error 89
Data error 90
Data error 91
Data error 92
Data error 93
Data error 94
Data error 95
Data error 96
Data error 97
Data error 98
Data error 99
Data error 100
Data error 101
Data error 102
Data error 103
Data error 104
Data error 105
Data error 106
Data error 107
Data error 108
Data error 109
Data error 110
Data error 111
Data error 112
Data error 113
Data error 114
Data error 115
Data error 116
Data error 117
Data error 118
Data error 119
Data error 120
Data error 121
Data error 122
Data error 123
Data error 124
Data error 125
Data error 126
Data error 127
----
Write attempt 2
Data error 0
Data error 1
Data error 2
Data error 3
Data error 4
Data error 5
Data error 6
Data error 7
Data error 8
Data error 9
Data error 10
Data error 11
Data error 12
Data error 13
Data error 14
Data error 15
Data error 16
Data error 17
Data error 18
Data error 19
Data error 20
Data error 21
Data error 23
Data error 24
Data error 25
Data error 26
Data error 27
Data error 28
Data error 29
Data error 30
Data error 31
Data error 32
Data error 33
Data error 34
Data error 35
Data error 36
Data error 37
Data error 38
Data error 39
Data error 40
Data error 41
Data error 42
Data error 43
Data error 44
Data error 45
Data error 46
Data error 47
Data error 48
Data error 49
Data error 50
Data error 51
Data error 52
Data error 53
Data error 54
Data error 55
Data error 56
Data error 57
Data error 58
Data error 59
Data error 60
Data error 61
Data error 62
Data error 63
Data error 64
Data error 65
Data error 66
Data error 67
Data error 68
Data error 69
Data error 70
Data error 71
Data error 72
Data error 73
Data error 74
Data error 75
Data error 76
Data error 77
Data error 78
Data error 79
Data error 80
Data error 81
Data error 82
Data error 83
Data error 84
Data error 85
Data error 86
Data error 87
Data error 88
Data error 89
Data error 90
Data error 91
Data error 92
Data error 93
Data error 94
Data error 95
Data error 96
Data error 97
Data error 98
Data error 99
Data error 100
Data error 101
Data error 102
Data error 103
Data error 104
Data error 105
Data error 106
Data error 107
Data error 109
Data error 110
Data error 111
Data error 112
Data error 113
Data error 114
Data error 115
Data error 116
Data error 117
Data error 118
Data error 119
Data error 120
Data error 121
Data error 122
Data error 123
Data error 124
Data error 125
Data error 126
Data error 127
----
Write attempt 3
Data error 0
Data error 1
Data error 2
Data error 4
Data error 5
Data error 6
Data error 7
Data error 8
Data error 9
Data error 10
Data error 11
Data error 12
Data error 13
Data error 14
Data error 15
Data error 16
Data error 17
Data error 18
Data error 19
Data error 20
Data error 21
Data error 22
Data error 23
Data error 24
Data error 25
Data error 26
Data error 27
Data error 28
Data error 29
Data error 30
Data error 31
Data error 32
Data error 33
Data error 34
Data error 35
Data error 36
Data error 37
Data error 38
Data error 39
Data error 40
Data error 41
Data error 42
Data error 43
Data error 44
Data error 45
Data error 46
Data error 47
Data error 48
Data error 49
Data error 50
Data error 51
Data error 52
Data error 53
Data error 54
Data error 55
Data error 56
Data error 57
Data error 58
Data error 59
Data error 60
Data error 61
Data error 62
Data error 63
Data error 64
Data error 65
Data error 66
Data error 67
Data error 68
Data error 69
Data error 70
Data error 72
Data error 73
Data error 74
Data error 75
Data error 76
Data error 77
Data error 79
Data error 80
Data error 81
Data error 82
Data error 83
Data error 84
Data error 85
Data error 86
Data error 87
Data error 88
Data error 89
Data error 90
Data error 91
Data error 92
Data error 93
Data error 94
Data error 95
Data error 96
Data error 97
Data error 98
Data error 99
Data error 100
Data error 101
Data error 102
Data error 103
Data error 104
Data error 105
Data error 106
Data error 107
Data error 108
Data error 109
Data error 110
Data error 111
Data error 112
Data error 113
Data error 114
Data error 115
Data error 116
Data error 117
Data error 118
Data error 119
Data error 120
Data error 121
Data error 122
Data error 123
Data error 124
Data error 125
Data error 126
Data error 127
----
Write attempt 4
Data error 0
Data error 1
Data error 2
Data error 3
Data error 4
Data error 5
----
Write attempt 5
----
Write attempt 6
----
Write attempt 7
----
Write attempt 8
----

You can see that it comes good after a few attempts. This is fairly consistent in my testing – after flashing or power cycling it only occasionally works first time and usually takes a few attempts before it will write consistently.

Any ideas? Am I missing something obvious?
Thanks!
 
I'm seeing this issue too. I can't believe that no serious action has taken place yet to resolve it yet. I have some time next week, so I'll be investigating.

Is the issue that the Teensy LC's emulated firmware just isn't ready right at startup?
 
For me the issue is that after programing the device, sometimes reading a value from EEPROM is being returned as zero, even though a previously non zero value was stored. I need to do some more tests to get a better idea of what's going on.
 
From the testing I've done this evening, I'm confident that it you are in the process of saving when the device looses power, parts of the memory get reset to zero.
 
Excellent news. I would like to incorporate the fix into my current installation. Is there a particular file I should be looking at?
 
Status
Not open for further replies.
Back
Top