SD Datalogging - Best Practice in 2019

P

peteculmer

New member
Hi,

I'm looking for some guidance on current 'best practice' in logging data to the (internal) SD card on a Teensy 3.6.

As background, the project involves gathering data from an array of I2C sensors, storing either ~400 or ~800 bytes per 'read' (total data from the array of sensors) @ 100Hz to the internal SD card of the Teensy. This would be for maximum 5 minutes - so not a huge file size or data throughput.
I've read around the forums and looked at various examples - all of which have been very helpful - but left me slightly uncertain about which direction to go with the current state of libraries available for this kind of operation. The closest and most recent relevant example seems to be this:

https://forum.pjrc.com/threads/43834-Real-low-latency-logging-for-Teensy-3-5-3-6-SDIO-SD/page2

And so I'm looking for advice on:

a) Library: should I use the default SD library (comes with Arduino, with optimisation for Teensy?), SDfat (as in example above) or the new SDFS (does this supercede SDfat?)
b) Filesystem: FAT32 on the SD card seems ok for this application?
c) Preallocated data file - is this preferable to maintain consistent (and higher) data rates?
d) Any 'fancy' buffering or padding of data before using file write (to make into 512 /1024 byte chunks? I'd prefer to keep the code 'simple/readable as possible'!

I'm a reasonably competent programmer - happy to spend time developing - but before committing time want to make sure I choose a sensible route
Any comments, advice etc very much appreciated,
cheers
Pete
 
peteculmer said:
I'm looking for some guidance on current 'best practice' in logging data to the (internal) SD card on a Teensy 3.6.
Click to expand...
For me there is only one rule:
Any data acquisition should run at interrupt level (i.e. elevated priority) and disk access should be done in loop() (i.e. in the background)

This implies: data acquisition delivers data ASAP into a large buffer and data storage runs async from acquisition whenever disk is ready to receive data.

The size of the buffer should be large enough to keep say 150 - 200 ms worth of data

concerning filing SW I guess stock SD (from teensyduino) is a good starting point. There are examples that can be build on.
Question weather SDFAT of SDFS (both from Bill Greiman, AFAIK, SD is also derived from earlier SDFAT version) is better, I cannot answer. For me SdFS is better as it supports exFAT but it seems Bill maintenance goes more into SDFAT (Arduino community?) so it is more looked after.
 
Thanks WMXZ, really useful reply.

The example I posted contains a good link to interrupt driven logging using SDfat so I'll have a look at that, particularly if SDFat is more actively maintained.

I'd still be interested to hear if anyone knows of the relative merits between the simpler looking SD library and SDFat.

I'll report back on my own findings too!
 
Datalogging is a problem I've been trying to brute-force for half a year now.

My solution (that I hope to release in a couple more months of testing) runs on top of the SdFs library for the purpose of low-latency high-frequency binary logging. I'm currently able to (somewhat reliably) maintain a logging speed of 925bytes at 500Hz (452KB/s). So in short, it is possible to have a good logging system, although it's highly dependent on how you interact with your data.

My current best practises:

1. Initialize the SdFs on SdioConfig(FIFO_SDIO) mode.
2. Preallocate the maximum length of your log file using the .preAllocate(size_t size) method.
3. Important: Pre-erase the entire file with zeroes.
4. Before you start logging (to ensure the file is committed fully to the card), call sync().
5. Write in chunks of a multiple of 512bytes only. (512, 1024, 2048 byte chucks per write() command)
6. To minimize blocking, try to write only if .card()->isBusy() returns false. This seems to be related to the internal card data management (I'm guessing things like TRIM and what not). Step 3 helps ensure that the card spends as little time as possible in this busy time.
7. Use a SLC-type SD card. I cannot stress enough the difference this makes. Before buying this card, I had write() operations take up to 10ms on a SanDisk card, but with this expensive card they 99.9% of the time stayed below 500us.
8. Call truncate() and then sync() after you've finished writing your entire file. The first function removes any extra pre-allocation and the second makes sure that it's a proper FAT32 file. Note that if you're using an exFAT partition your file will not be saved until you call sync() or close().
7. On that note, FAT32 is always faster than exFAT, but you do have less capacity.

I'm still learning more about best practises for logging. My current hang-ups are (I think) related to heap-overflow because of the many sensors I have attached interrupting one another during a write() operation. I'm a very fresh programmer so it's taking me a while but I hope the tips above are helpful enough for you to get started on your 100Hz project.

Good luck! Let me know if you find any new practises that I can also test.
 
XFer said:
Hello Pete,
I did a few latency tests with T3.6 and both SD and SDFat (using onboard uSD slot).
SDFat came well ahead, and pre-allocation and pre-zeroing helped too.

Quick link to my test:
https://forum.pjrc.com/threads/54241-Simple-SdFat-write-benchmark-for-T3-5-3-6

For my video recording application, I did more or less what Linearism suggests.
Obtained good results (800x600 @ 15 fps on average as MJPEG, about 5 mbit/s).
Click to expand...

Not OP, but thanks for the code and link, I'm tackling a data logging project currently. FWIW, my test results for a SanDisk Extreme Plus card are:

Code: 
SdFatSdioEX write benchmark
SdFatSdioEx uses extended multi-block transfers without DMA

Preallocating file (takes awhile)...

Starting write benchmark

Transfer size	Write speed	Avg write time	Max write time
bytes		KB/s		us		us

512		20462.2		25		9319
1024		20709.8		49		9077
2048		20853.8		98		13465
4096		21033.0		194		9425
8192		21003.5		390		8840
16384		21096.7		776		9876
32768		21045.2		1557		10440
65536		21116.3		3103		12357
131072		21030.4		6232		20540

 SD clock speed (kHzSdClk): 48000 KHz

Done
 
Nice, I have more or less same results with a Sandisk Extreme Plus 32 GB.
Pretty fast, the little bugger. :)
 
@linarism,

Thanks for the tips. I'm having severe write issues with my data logging system. I'm writing around 126 characters every second to an 8GB sandisk card. I'm using good old fashion SD.h libs. My issues is sometimes write fails, sometimes it completely corrupts the file etc.

My process is

if elapsed time > 1 sec then
1. MeasureData();
2. DisplayResultsToScreen()
3. WriteDataToSD();
4. SendDataToWirelessUnit()


in my WriteDataToSD function, I'm
Opening the file
writing data
closing file // because I have no warning when user will shut off data logger and i' scared an open file will can't be opened later
//again this process happens every second (yes open/close every second)


Thoughts on my workflow, any room for improvement?
1. my choice in library
2. my open / close every second
3. implementing your tips
4. others???


Thanks in advance.
 
Hello, I tried most of these steps. My program preallocate space on the file fine but later on in the program when I try and append to the file it starts writing after the preallocated space (at the very bottom of the file). What can I do to stop this from happening? Ive tryed to write in blocks of 512 bytes.

My goal for the program is to open the file in setup(), write one line then open, write/append and close the file repeatedly in loop().

Here is my code:

Code: 
//Teensy SD library

#include <SD.h>

String fileName = "data4.txt";
uint32_t id = 0;
uint32_t micOld = 0;

void setup() {
  Serial.begin(115200);
  while (!Serial);

  Serial.println("Initializing SD card");
  while(!SD.sdfs.begin(SdioConfig(FIFO_SDIO))) {
    Serial.println("Initialization failed");
    delay(1000);
  }
  Serial.println("Initialization done");
  
  FsFile myfile = SD.sdfs.open(fileName, O_WRONLY | O_CREAT);
  
  unsigned int len = myfile.fileSize();
  if(myfile.fileSize() > 0) {
    myfile.truncate();
  }
  
  if(!myfile.preAllocate(40*1024*1024)) {
    Serial.println("Unable to preallocate this file");
  }
  Serial.println("Allocate 1 megabyte for " + fileName);
  
  if(!myfile.sync()) {
    Serial.println("File not fully committed");
  }
  Serial.println("File fully committed");

  if(!myfile.isBusy()) {
    myfile.println("Time, ax, ay, az, gx, gy, gz, temperature");
    //myfile.write(0);
  }
  myfile.close();
}

void loop() { 
  uint32_t mic = micros();
  uint32_t dif = mic - micOld;
  FsFile myfile = SD.sdfs.open(fileName, O_WRONLY | O_AT_END | O_CREAT);
  if(!myfile.isBusy()) {
    String len = String(id)+","+String(dif)+",";
    int extra = 511 - len.length();
    myfile.print(id);
    myfile.print(",");
    myfile.print(dif);
    myfile.print(",");
    for(int i=0;i<extra;i++) {
      myfile.print("1");
    }
    myfile.println(",");
    //myfile.write(0);
  }
  myfile.close();
  id ++;
  micOld = mic;
  //delay(100);
}
 
I prefer to leave the file open, rather than open, write, close at each storage interval. I do call sync() about once per second to make sure that the directory is updated. That way, if something goes wrong, the file state is up to date.

If you close, then reopen a FAT32 file each time you write, the open process can get time consuming in a few weeks as the file system may have to traverse the FAT table to find the next write location.

If I'm going to log for a long time, I usually close a file just after midnight and start a new file for the next day. Some of my oceanographic loggers collected at about 2KB/second for six to 12 months. The design specs were for 6 months, but they didn't get back to pick up the logger until a cruise six months later than planned. The logger eventually drained its batteries---but the data was good until the end.
 
Thank you, these changes made a big difference. I have some questions in relation to Linarism's comment further up. I don't understand what it means to "Pre-erase the entire file with zeros" and what is the difference between write() and print(). Also is there an advantage to making the file .bin instead of .txt, is this what binary logging is or are there other steps needed for this?
 
To quickly answer just 1 of these questions..

Ash228 said:
what is the difference between write() and print().
Click to expand...

write() stores the raw data you give it. Usually print() does some conversion to human readable text, and then internally it calls write(). If you call print(123456), it will convert the number into 6 characters. But some things like strings don't require any conversion, so using print() or write() ends up doing the same thing in those cases.
 
Lately I've been working on a data logging project using the latest SD and Sdfat from TD 1.58b3. In post #4 of this thread, item 3 in the list of best practices is "Pre-erase" the file. Does anyone know what this means?

I've experimented with calling contiguousRange() to get the first/last sectors of a preAllocated file, then calling erase() on various sector ranges. In my tests, erase() succeeds as long as the file size is 3MB or less, and the number of sectors erased in a single call can be anything from 1 to the total number of sectors in the file. For files > 3MB, erase() always fails, whether for 1 sector, all sectors, or any number in between, such as 64, 128, etc.

According to a post by Bill Greiman on his github, erase() is a "trick" that doesn't really do anything to the underlying medium. Does anyone have a comment on whether erase() is helpful? I also wondered whether the term "erase" in the original list meant "write all 1's" or "write all 0's", as opposed to calling the erase() function.

1. Initialize the SdFs on SdioConfig(FIFO_SDIO) mode.
2. Preallocate the maximum length of your log file using the .preAllocate(size_t size) method.
3. Important: Pre-erase the entire file with zeroes.
Click to expand...
 
KrisKasprzak said:
@linarism,

Thanks for the tips. I'm having severe write issues with my data logging system. I'm writing around 126 characters every second to an 8GB sandisk card. I'm using good old fashion SD.h libs. My issues is sometimes write fails, sometimes it completely corrupts the file etc.

My process is

if elapsed time > 1 sec then
1. MeasureData();
2. DisplayResultsToScreen()
3. WriteDataToSD();
4. SendDataToWirelessUnit()


in my WriteDataToSD function, I'm
Opening the file
writing data
closing file // because I have no warning when user will shut off data logger and i' scared an open file will can't be opened later
//again this process happens every second (yes open/close every second)


Thoughts on my workflow, any room for improvement?
1. my choice in library
2. my open / close every second
3. implementing your tips
4. others???


Thanks in advance.
Click to expand...

I have found that opening the file once at the beginning and calling sync() at regular intervals is sufficient to make files readable up to the last sync() position in case of a power failure. Sync() updates the directory information and end-of-file position. Since it has to write the directory block, it takes some extra time. However, I think it is faster than an open/close cycle.

I think A card formatted as EXFat and pre-allocated results in much faster random access, since the file system can calculate the read or write block address without having to traverse the FAT cluster chain.

As noted in other posts, using interrupts to collect data into large buffers, then writing buffers when filled from loop() seems the best way to handle high rate logging that you want to happen at fixed intervals. Even if you preallocate and erase, you can't really know when the internal MPU in the card will decide it is time to swap out or erase a 128KB block for wear-leveling reasons. That's when your write time can go up into the 100mSec range.

I should point out that the SDFat formatting example erases all the blocks by default. I assume that is done for a reason.
 
mborgerson said:
I should point out that the SDFat formatting example erases all the blocks by default. I assume that is done for a reason.
Click to expand...

That makes sense. If you know your disk is formatted, then everything has been erased, and there is no reason or benefit to use the "erase" function on a pre-allocated file.
 
You must log in or register to reply here.
Back
Top