Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 8 of 8

Thread: SD Datalogging - Best Practice in 2019

  1. #1
    Junior Member
    Join Date
    Feb 2019
    Posts
    2

    SD Datalogging - Best Practice in 2019

    Hi,

    I'm looking for some guidance on current 'best practice' in logging data to the (internal) SD card on a Teensy 3.6.

    As background, the project involves gathering data from an array of I2C sensors, storing either ~400 or ~800 bytes per 'read' (total data from the array of sensors) @ 100Hz to the internal SD card of the Teensy. This would be for maximum 5 minutes - so not a huge file size or data throughput.
    I've read around the forums and looked at various examples - all of which have been very helpful - but left me slightly uncertain about which direction to go with the current state of libraries available for this kind of operation. The closest and most recent relevant example seems to be this:

    https://forum.pjrc.com/threads/43834...-SDIO-SD/page2

    And so I'm looking for advice on:

    a) Library: should I use the default SD library (comes with Arduino, with optimisation for Teensy?), SDfat (as in example above) or the new SDFS (does this supercede SDfat?)
    b) Filesystem: FAT32 on the SD card seems ok for this application?
    c) Preallocated data file - is this preferable to maintain consistent (and higher) data rates?
    d) Any 'fancy' buffering or padding of data before using file write (to make into 512 /1024 byte chunks? I'd prefer to keep the code 'simple/readable as possible'!

    I'm a reasonably competent programmer - happy to spend time developing - but before committing time want to make sure I choose a sensible route
    Any comments, advice etc very much appreciated,
    cheers
    Pete

  2. #2
    Senior Member
    Join Date
    Jul 2014
    Posts
    2,055
    Quote Originally Posted by peteculmer View Post
    I'm looking for some guidance on current 'best practice' in logging data to the (internal) SD card on a Teensy 3.6.
    For me there is only one rule:
    Any data acquisition should run at interrupt level (i.e. elevated priority) and disk access should be done in loop() (i.e. in the background)

    This implies: data acquisition delivers data ASAP into a large buffer and data storage runs async from acquisition whenever disk is ready to receive data.

    The size of the buffer should be large enough to keep say 150 - 200 ms worth of data

    concerning filing SW I guess stock SD (from teensyduino) is a good starting point. There are examples that can be build on.
    Question weather SDFAT of SDFS (both from Bill Greiman, AFAIK, SD is also derived from earlier SDFAT version) is better, I cannot answer. For me SdFS is better as it supports exFAT but it seems Bill maintenance goes more into SDFAT (Arduino community?) so it is more looked after.

  3. #3
    Junior Member
    Join Date
    Feb 2019
    Posts
    2
    Thanks WMXZ, really useful reply.

    The example I posted contains a good link to interrupt driven logging using SDfat so I'll have a look at that, particularly if SDFat is more actively maintained.

    I'd still be interested to hear if anyone knows of the relative merits between the simpler looking SD library and SDFat.

    I'll report back on my own findings too!

  4. #4
    Junior Member
    Join Date
    Oct 2016
    Posts
    6
    Datalogging is a problem I've been trying to brute-force for half a year now.

    My solution (that I hope to release in a couple more months of testing) runs on top of the SdFs library for the purpose of low-latency high-frequency binary logging. I'm currently able to (somewhat reliably) maintain a logging speed of 925bytes at 500Hz (452KB/s). So in short, it is possible to have a good logging system, although it's highly dependent on how you interact with your data.

    My current best practises:

    1. Initialize the SdFs on SdioConfig(FIFO_SDIO) mode.
    2. Preallocate the maximum length of your log file using the .preAllocate(size_t size) method.
    3. Important: Pre-erase the entire file with zeroes.
    4. Before you start logging (to ensure the file is committed fully to the card), call sync().
    5. Write in chunks of a multiple of 512bytes only. (512, 1024, 2048 byte chucks per write() command)
    6. To minimize blocking, try to write only if .card()->isBusy() returns false. This seems to be related to the internal card data management (I'm guessing things like TRIM and what not). Step 3 helps ensure that the card spends as little time as possible in this busy time.
    7. Use a SLC-type SD card. I cannot stress enough the difference this makes. Before buying this card, I had write() operations take up to 10ms on a SanDisk card, but with this expensive card they 99.9% of the time stayed below 500us.
    8. Call truncate() and then sync() after you've finished writing your entire file. The first function removes any extra pre-allocation and the second makes sure that it's a proper FAT32 file. Note that if you're using an exFAT partition your file will not be saved until you call sync() or close().
    7. On that note, FAT32 is always faster than exFAT, but you do have less capacity.

    I'm still learning more about best practises for logging. My current hang-ups are (I think) related to heap-overflow because of the many sensors I have attached interrupting one another during a write() operation. I'm a very fresh programmer so it's taking me a while but I hope the tips above are helpful enough for you to get started on your 100Hz project.

    Good luck! Let me know if you find any new practises that I can also test.

  5. #5
    Senior Member
    Join Date
    Oct 2015
    Location
    Roma (IT, EU)
    Posts
    117
    Hello Pete,
    I did a few latency tests with T3.6 and both SD and SDFat (using onboard uSD slot).
    SDFat came well ahead, and pre-allocation and pre-zeroing helped too.

    Quick link to my test:
    https://forum.pjrc.com/threads/54241...k-for-T3-5-3-6

    For my video recording application, I did more or less what Linearism suggests.
    Obtained good results (800x600 @ 15 fps on average as MJPEG, about 5 mbit/s).

  6. #6
    Senior Member brtaylor's Avatar
    Join Date
    Mar 2016
    Location
    Portland, OR
    Posts
    494
    Quote Originally Posted by XFer View Post
    Hello Pete,
    I did a few latency tests with T3.6 and both SD and SDFat (using onboard uSD slot).
    SDFat came well ahead, and pre-allocation and pre-zeroing helped too.

    Quick link to my test:
    https://forum.pjrc.com/threads/54241...k-for-T3-5-3-6

    For my video recording application, I did more or less what Linearism suggests.
    Obtained good results (800x600 @ 15 fps on average as MJPEG, about 5 mbit/s).
    Not OP, but thanks for the code and link, I'm tackling a data logging project currently. FWIW, my test results for a SanDisk Extreme Plus card are:

    Code:
    SdFatSdioEX write benchmark
    SdFatSdioEx uses extended multi-block transfers without DMA
    
    Preallocating file (takes awhile)...
    
    Starting write benchmark
    
    Transfer size	Write speed	Avg write time	Max write time
    bytes		KB/s		us		us
    
    512		20462.2		25		9319
    1024		20709.8		49		9077
    2048		20853.8		98		13465
    4096		21033.0		194		9425
    8192		21003.5		390		8840
    16384		21096.7		776		9876
    32768		21045.2		1557		10440
    65536		21116.3		3103		12357
    131072		21030.4		6232		20540
    
     SD clock speed (kHzSdClk): 48000 KHz
    
    Done

  7. #7
    Senior Member
    Join Date
    Oct 2015
    Location
    Roma (IT, EU)
    Posts
    117
    Nice, I have more or less same results with a Sandisk Extreme Plus 32 GB.
    Pretty fast, the little bugger.

  8. #8
    Senior Member
    Join Date
    Apr 2017
    Posts
    155
    @linarism,

    Thanks for the tips. I'm having severe write issues with my data logging system. I'm writing around 126 characters every second to an 8GB sandisk card. I'm using good old fashion SD.h libs. My issues is sometimes write fails, sometimes it completely corrupts the file etc.

    My process is

    if elapsed time > 1 sec then
    1. MeasureData();
    2. DisplayResultsToScreen()
    3. WriteDataToSD();
    4. SendDataToWirelessUnit()


    in my WriteDataToSD function, I'm
    Opening the file
    writing data
    closing file // because I have no warning when user will shut off data logger and i' scared an open file will can't be opened later
    //again this process happens every second (yes open/close every second)


    Thoughts on my workflow, any room for improvement?
    1. my choice in library
    2. my open / close every second
    3. implementing your tips
    4. others???


    Thanks in advance.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •