Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 11 of 11

Thread: SD library deemed harmful due to GPL license

  1. #1
    Member
    Join Date
    Apr 2016
    Location
    Honolulu, Hawaii
    Posts
    37

    SD library deemed harmful due to GPL license

    The default SD library installed by TeensyDuino claims to be a wrapper for sdfatlib, and it carries a GPL license "because sdfatlib is licensed with this." Unfortunately, devices subject to certain types of regulatory oversight cannot reasonably comply with GPL requirements. For this reason, any GPL code is a poison pill for such projects.

    Looking at the actual code, it appears that SD is now actually a wrapper for a 2009 version of SdFat (from Bill Greiman) that also used the GPL.

    About a year ago Bill Greiman solved this problem for us all by making recent versions of his excellent SdFat library available under the MIT license, which makes it acceptable for regulated devices. Other updates such as high speed optimizations and long filenames are merely bonuses. Ongoing development against modern SD cards, a large number of solid examples, good utilities, and *very* good documentation are other good reasons to use current releases of the SdFat library.

    I use the current version of SdFat on Teensy 3.6 with no problems.

    However, a number of other TeensyDuino -installed libraries (most notably the audio library) take a dependency on the supplied SD wrapper, and I have not found it straightforward to modify that dependency to use SdFat while continuing to benefit from library updates.

    I've been unable to completely untangle the licensing trail, but it appears to me that if SD could wrap a current version of SdFat, then the GPL license could be replaced with an MIT license. I'm not sure whether SparkFun would need to modify their wrapper license label or whether their disclaimer would cover this, or whether the wrapper would need to be rewritten, but all of those are possible now that the underlying codebase has a better license.

    I seem to remember a thread suggesting that perhaps SdFat could be made "SD-Compatible" or that SD could use the current version of SdFat by adding a few macros, but I haven't been able to find that thread again so perhaps I imagined it.

    Bottom line: I propose that the GPL license should be removed from the TeensyDuino-supported version of SD by any means necessary.

    In the meantime, if anybody is aware of a existing work towards a wrapper or set of configuration settings that can allow me to substitute a recent version of SdFat to appear as SD (e.g., to make it work with the unmodified audio library) I would be very interested.

  2. #2
    Senior Member brtaylor's Avatar
    Join Date
    Mar 2016
    Location
    Portland, OR
    Posts
    493
    Hi Craig,

    Interesting that recent versions are now MIT license; I've been avoiding all SD related code and relying on using other means to log data (i.e. flash memory, transferring to a BeagleBone Black, etc) in order to avoid GPL licenses. Can you point to the MIT license versions of his library?

  3. #3
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    7,795
    Selecting 'Bill Greiman' and four clicks and EDGE/BING pointed me here ::

    The Arduino SdFat library provides read/write access to FAT16/FAT32 file systems on SD/SDHC flash cards.
    SdFat requires Arduino 1.6x or greater.
    Key changes:
    The SPI divisor has been replaced by SPISettings in the begin() call.
    bool begin(uint8_t csPin = SS, SPISettings spiSettings = SPI_FULL_SPEED);
    Several macros have been defined for backward compatibility.
    #define SD_SCK_MHZ(maxMhz) SPISettings(1000000UL*maxMhz, MSBFIRST, SPI_MODE0)
    // SPI divisor constants
    /** Set SCK to max possible rate. */
    #define SPI_FULL_SPEED SD_SCK_MHZ(50)
    /** Set SCK rate to F_CPU/3 for Due */
    #define SPI_DIV3_SPEED SD_SCK_HZ(F_CPU/3)
    /** Set SCK rate to F_CPU/4. */
    #define SPI_HALF_SPEED SD_SCK_HZ(F_CPU/4)
    // ...
    There are two new classes, SdFatEX and SdFatSoftSpiEX.
    Teensy 3.5/3.6 SDIO support has been added. Try the TeensySdioDemo example. Many other example will work with Teensy SDIO if you use the SdFatSdio classes and call begin with no parameters.

  4. #4
    Senior Member brtaylor's Avatar
    Join Date
    Mar 2016
    Location
    Portland, OR
    Posts
    493
    Cool, thanks! Wanted to make sure I wasn't grabbing the wrong one.

  5. #5
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    7,795
    Pulled it down and the example TeensySdioDemo works.

    Sketch uses 26220 bytes (2%) of program storage space. Maximum is 1048576 bytes.
    Global variables use 39360 bytes (15%) of dynamic memory, leaving 222784 bytes for local variables. Maximum is 262144 bytes.
    Demo samples both :: Type '1' for SdFatSdioEX or '2' for SdFatSdio

    Worked on the aData card just opened - the SanDisk failed a pass with Read Error … SdFormatter Quick and the SanDisk passed
    Last edited by defragster; 08-19-2018 at 11:19 PM.

  6. #6
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    7,795
    On another thread where GPS w.SD was logging times would average 5-10 ms for a good group, and some 1% were over 50 ms and then there would be an abominable 200-400 ms write of 1,000 bytes.

    I just linked it with SdFat with Minor changes looking at the sdioDemo noted above. With this the SD.xxx file operations were all supported and functional:
    Code:
    #include "SdFat.h"
    //#include <SD.h>
    //#include <SPI.h>
    
    SdFatSdio SD;
    
    setup() {
    // …
    //  while (!SD.begin(BUILTIN_SDCARD)) {
        while (!SD.begin()) {
    
    }
    I am seeing worst case - after only 420 writes as 15.4 ms and typical average at or under 4.4 ms. Just stopped it at 420+ writes and the data reads well. Will restart and let it get to 5-10K writes to see if it keeps going that well. Also this is with the self same Teensy 3.6 internally generating\emulating GPS like test data not reading from Serial.

    Might be a good change - not just for licensing - but it seems faster as well so far. Below is Flash/RAM use for this minimal example with each library.

    Using SDFat.h:
    Code:
    Sketch uses 25116 bytes (2%) of program storage space. Maximum is 1048576 bytes.
    Global variables use 8676 bytes (3%) of dynamic memory, leaving 253468 bytes for local variables.
    Using prior SD.h:
    Code:
    Sketch uses 26948 bytes (2%) of program storage space. Maximum is 1048576 bytes.
    Global variables use 8160 bytes (3%) of dynamic memory, leaving 253984 bytes for local variables.
    <EDIT>
    Even this library quickly devolved to about the same state after 4360 writes of 1K the max write time is 310ms and 40 were over 50 ms … about the same 1%.
    4.4MB of data looks good.
    Did fresh (barely out of box) format of aData brand card - it hit 161 ms write in 60 writes at 5 per second. Hit 166 ms in the next 100 writes. Showing 2% over 50 ms where avg in 20 fast ones is under 12 ms
    Last edited by defragster; 08-20-2018 at 05:29 AM.

  7. #7
    Senior Member
    Join Date
    Oct 2012
    Location
    Portland OR
    Posts
    588
    I assume all this write-time variability comes from some low-level physics in the individual storage cells (randomly some sticky / pinned states needing extra time to inject enough charge). If so, curious how the larger world of video cameras etc. that rely on high sustained data transfer rates to flash memory, ever manage to work reliably. They just use enormous RAM buffers to handle the few % of memory blocks that write very slowly, I guess?

  8. #8
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    7,795
    Quote Originally Posted by JBeale View Post
    I assume all this write-time variability comes from some low-level physics in the individual storage cells (randomly some sticky / pinned states needing extra time to inject enough charge). If so, curious how the larger world of video cameras etc. that rely on high sustained data transfer rates to flash memory, ever manage to work reliably. They just use enormous RAM buffers to handle the few % of memory blocks that write very slowly, I guess?
    Assuming that too - if my testing is valid - though it is writing puny 1K blocks. One of my cards is the beloved SanDisk, the other a just out of the box aData brand card that another user just posted good results with. Both show similar behavior. And given the same pattern of delays from the GPL SD code and the updated MIT BillGr code { both SdFatSdio and SdFatSdioEx }.

    The github.com/greiman/SdFat/tree/master/examples list is about 30 and the base TeensySdioDemo gave good results.

    It is 'known' that using a pre-allocated and blanked file or other means beyond the obvious/simple 'create then write' can result in low latency logging on a good card. There are examples for that.

  9. #9
    There are two levels of delays in writing to an SD card. First there is the overhead imposed by the file system. Imagine the delays you can get when you have to allocate a new cluster. Just finding a free cluster could require a large number of reads. The maximum FAT write time in the SD specification is 750ms.

    Then there are the card delays. For some reason this is the nature of the beast and I have seen the same basic behaviour since 128MB uSD cards. Most writes are reasonably fast but periodically one (or more) will be a lot slower. Why I have no idea. Attached are a couple of plots showing write times (my own SD code for the Teensy 3.6 with no file system involved) when writing a block at a time and when doing an 8K multi-block write. I believe that this was with a 8GB Sandisk.

    Click image for larger version. 

Name:	single.gif 
Views:	13 
Size:	10.7 KB 
ID:	14501Click image for larger version. 

Name:	multi4.gif 
Views:	19 
Size:	9.7 KB 
ID:	14502

    The SD specification states that to achieve the speed class rating you must write in large blocks. 512KB for an SDHC card.

  10. #10
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    7,795
    Not doing FAT and dir entry file size updates each cluster full would save a great deal - thus the pre-allocated empty file use for low latency logging.

    Opened this :: SdFat\examples\LowLatencyLogger
    And it is failing last I was looking at it. I find a file started then it dies and can't remove it on next restart.
    Creating new file
    Erasing all data
    error: writeStart failed
    SD errorCode: 0X65,0X1

  11. #11
    Senior Member+ MichaelMeissner's Avatar
    Join Date
    Nov 2012
    Location
    Ayer Massachussetts
    Posts
    2,921
    Quote Originally Posted by UhClem View Post
    Then there are the card delays. For some reason this is the nature of the beast and I have seen the same basic behaviour since 128MB uSD cards. Most writes are reasonably fast but periodically one (or more) will be a lot slower. Why I have no idea. Attached are a couple of plots showing write times (my own SD code for the Teensy 3.6 with no file system involved) when writing a block at a time and when doing an 8K multi-block write. I believe that this was with a 8GB Sandisk.
    It may be you are seeing the effects of wear leveling that the card does automatically.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •