What happens when you don't close a SD card file?

Status
Not open for further replies.
N

neltnerb

Well-known member
I'm writing a logging system using the Teensy 3.6 and onboard SD card with the sdfat library and the RTC.

In my application I log the time that the device turns on, and then on the next line I immediately store a time indicating the time it turned off. Every 10 seconds I delete the last line and replace it with an updated time so that when the device turns off it has a stored log showing the duration accurate to ~10 seconds. The only way the device stops logging is if it is powered off.

In my code I'm opening the file and closing the file each time like this (well, this is the initial write, but it's all done with the same structure):

Code: 
if (!file.open("TrialLog.txt", O_RDWR | O_CREAT | O_AT_END)) {
    errorHalt("Open File Failed.");
}
file.seek(file.size());
file.println(getTime() + " - Started.   T" + String(now()));
file.println(getTime() + " - Ended.     T" + String(now()));
file.close();

The idea here is that I don't want it to run at all if the logging isn't working (they need to bring it in for repair if logging is broken so I'd rather it just not work at all). But the log file is extremely valuable at the end, so I need to ensure that data is actually written to the SD card.

What happens if I fail to close a file before it is powered off? I have other threads running so I'd much rather avoid the time involved in opening and closing the file every time (might cause glitches in other behavior), but it is critical that I avoid corrupting the file or losing the data when the system is powered off without warning.

I'm worried that the file class may only store changes in RAM and only flush to the SD card when you run close. Does anyone know for sure what the promised behavior is if the system is powered off after a write but before close? Or during a write, since theoretically they could power it off precisely at the wrong time?
 
Probably nothing good. If the last data isn't in the Teensy RAM - it may be in the SD card RAM - not likely written. Perhaps file problems with size not updated.

There is a small piece of RAM associated with the RTC that survives restart - given RTC battery. I don't find the now older post on that IIRC by 'Frank B' that showed how to use that.

Pushing to EEPROM would be nice - as long as the writes were spread out perhaps. Writing the same area of SD requires rewrite and update of the data on the card
 
Hmm, I suppose I'm only updating once every ten seconds anyway due to worries about glitches. Still, even that slowly I'd reach 100,000 in just over 10 days. Might be okay here though since it's only active for an hour or so a day, so that'd take it out to a year of usage. At least since I'd probably be able to spread out the writes to different addresses. Good idea.

EEPROM might not be too bad, but the 3.3ms vaguely referenced write times won't do. I'm trying to simultaneously play audio with the DAC so that'd definitely cause a glitch in playback. Well, I'm assuming it's not buffered, maybe it is and is handled without blocking code execution.

I suppose with that I could store one int with the pointer to the last entry and store log entries as a struct with the type of log event and the timestamp. Wouldn't be too bad. Using the SD card does seem like a better choice given the write endurance.

So I guess it's an answerable question of whether the EEPROM writes cause a glitch. I can just give that a try and see, the documentation isn't super obvious here.

But it would be cool if the SD card writes happened in the background in a way that's guaranteed safe if the system gets powered off! I'd definitely prefer that, although for the super slow logging rates here it might be alright. I'd love if I could just open the file and then write to it whenever is needed with a lazy backend that updates the actual SD card occasionally. If my log file is off by 10 seconds it's no big deal, I just can't end up with a corrupted text file at the end.
 
Easy to try - may audio get glitches - T_3.6 over 120 MHz has to play clock and power games with HSRUN mode to get power boost for cpu speed that doesn't fly with EEPROM writes. It doesn't seem to stop interrupts in kinetis_hsrun_disable or enable.

If you could locate in on the WORD(s) of RTC backed memory and add a battery that would work … it seems I didn't save that - not sure if it got linked in 'Wiki Coming'? It seemed cool but I may not have snagged it as it went by as I didn't find it. It will be in the MCU PDF manual in some fashion ...
 
Oh, yes, I already have a battery for the RTC in the design. I'm not seeing any reference to it online, but maybe if it's big enough. I guess the idea is that the memory never has to be flushed to be safe? I'll have to check the datasheet, it might be more trouble than it's worth if I'm having to manually write to registers.
 
Sure it will!

#1> When you open do the open SD record and ZERO a 32 bit counter in VBAT register file. Increment it each 10 seconds - 6 times per minute.

#2> On a normal shutdown or end of log period with the start time at hand write the end time as you choose and Zero VBAT reg.

#0> On startup BEFORE doing the above step #1 - check VBAT Reg for ZERO, if not ZERO - write out what you need for records to track that log period, then proceed to #1.


** Modify as needed to suit your purposes

** One year would only be 3,153,600 updates - uint32_t there would be good for about 1,360 years?


<< EDIT >> :: updated detail note here > System-VBAT-register-files
 
Last edited:
Ah, sorry, I didn't explain this part but it's storing each start and end time separately so that we know the schedule it was used on rather than just total on-time.

Perhaps writing just the most recent data to the VBAT Reg and then writing that to the SD card on the next boot would work though. That way it's only the at-risk entry that is saved there since the rest can be written during setup() when latency doesn't matter.
 
neltnerb said:
Ah, sorry, I didn't explain this part but it's storing each start and end time separately so that we know the schedule it was used on rather than just total on-time.

Perhaps writing just the most recent data to the VBAT Reg and then writing that to the SD card on the next boot would work though. That way it's only the at-risk entry that is saved there since the rest can be written during setup() when latency doesn't matter.
Click to expand...

Indeed I was keying off this text - even though I didn't see it in the posted code:

neltnerb said:
Every 10 seconds I delete the last line and replace it with an updated time so that when the device turns off it has a stored log showing the duration accurate to ~10 seconds.
Click to expand...

So post #8 would allow that to be accomplished just knowing when it started and then tracking the count of 10 sec intervals.
 
Yup! What I meant to describe was that each time it's started it creates a "started at" and "aborted at" line, rather than a single cumulative duration. But I think the hybrid approach of storing the data being updated in RTC memory and that it simply writes this to the SD card on the next startup is entirely reasonable. That should only take a single 32-bit register to hold the last t_time seen while the system was powered, and perhaps another slot for whether it was aborted or completed successfully.

So when it boots it would know whether the last session succeeded or failed, and would write as much to the log file. Then it would indicate that it started a session at the RTC time when powered on but not bother writing updates to the SD card at all. It would just get picked up the next time it's turned on instead. I love how clever it is, thanks very much.

Of course, I'd still like to know if not running close corrupts the file or if it risks data loss, if anyone reading this later ends up finding it. I know you can do datalogging at very high speeds on the onboard SD card so I know writing is fast, I'm sure I'll do another project in the future where this comes up -- how can I guarantee that all the data collected (or at least all but the last few entries) will actually be written to the SD card if it gets powered off without warning? I could do some trickery with updating both the RTC memory and the SD card at the same time so that I can compare the last entries for data integrity checking. But this all seems like a lot of effort for something that is designed to be used this way, it seems like they probably have thought about how to do this well.
 
There are other threads and samples for LowLatencyLogger. To use FAT file system it involves pre-allocating a blank file of known length. Then tracking progress into the file where real data is written - . I haven't gotten one of those sketches working yet - barely opened one to try and first attempts are hitting an error just starting the sample so far on my setup not like the sample was tested on it seems.

Doing the preallocation avoids file growth on the fly - with FAT updates to track/chain new clusters and updating the directory entry for new length that can take a couple hundred milliseconds every now and again.

If the file is not 'closed' you can't expect the data to be there. You can easily try that - perhaps a looping process creating some 20 files in turn and writing some amount of data to them - and then in between turn off or reset the Teensy { use a byte of EEPROM for the test to start at 0 and run to 20 creating a unique file each time } and do the next file without closing them. The data integrity and maybe the state of the file system will be suspect.
 
Status
Not open for further replies.
Back
Top