Forum Rule: Always post complete source code & details to reproduce any issue!
Page 1 of 3 1 2 3 LastLast
Results 1 to 25 of 58

Thread: File abstraction and SdFat integration

  1. #1
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,039

    File abstraction and SdFat integration

    I'm been quiet for the last few days while working on a long-planned change to bring the File class into Teensy's core library, and to remove the Arduino SD library and migrate all SD card use on Teensy to SdFat.

  2. #2
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,039
    If anyone wants to give it a try, there are 3 pieces to grab from github.

    1: In the core library, you'll need this new FS.h header.

    https://github.com/PaulStoffregen/co...r/teensy4/FS.h

    2: You'll need this slightly edited copy of SdFat-beta, to avoid a conflict with FS.h

    https://github.com/PaulStoffregen/SdFat-beta

    3: This replaces the Arduino SD library with a thin wrapper around SdFat, using FS.h's File class.

    https://github.com/PaulStoffregen/SD

    Make sure you download the "Just_Use_SdFat" branch. Best to delete any copy of SD you might have, since this deletes almost all of SD's files.

    This stuff is all very new and probably still pretty rough around the edges. If trying out experimental stuff sounds like fun, now's the time to dive in.

    If downloading lots of files and getting them all in the right places sounds tedious, just wait a few days. I'm going to do more testing, (probably) fix bugs, and then package all this up into 1.54-beta3.

  3. #3
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,039
    If you're wondering why I'm doing all this, here's some of the goals.

    1: Fix the problems where SD and SdFat don't play well together. The goal is you'll be able to use libraries that want SdFat and others that want SD with little or no painful problems. Well, at least in theory. In practice you might have to change "sd" to "SD.sdfs" and other minor tweaks, but the process of using both libraries in the same project should become relatively easy.

    2: All programs using SD will gain SdFat's higher performance and support for long filenames and larger SD cards. The ancient code of SD is completely gone. When you use the SD library, it's actually just calling the same functions of SdFat.

    3: Having the File class in the core library will allow USBHost_t36 to give File class objects for USB mass storage devices. So any library or code that uses File will be able to access either type of media, and other types of media or virtual files in the future.

    At least those are the goals.

  4. #4
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,893
    Just out of curiosity are you planning on extending this to use on FLASH chips?

  5. #5
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,039
    Yes, definitely planning to make at least 1 flash filesystem library use this File base class.

    Hopefully after SD, USBHost_t36 and a flash chip library use this, the API should become pretty stable and those libraries should serve as examples to make it easier to apply to any other library that provides access to media with files.

  6. #6
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,893
    Cool will have to start looking at it - know ESP32 does something with FS as well.

  7. #7
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,893
    Have a few questions now that I am downloaded the updated files per post #2.

    If I run the SD examples should they work? Just ran cardinfo example and got this error if using the SD Card on the T4.1:
    Code:
    CardInfo:40: error: 'BUILTIN_SDCARD' was not declared in this scope
     const int chipSelect = BUILTIN_SDCARD;
    The sdinfo.ino sketch from SDBeta works fine.

    So couple of questions. To use the abstraction layer how to you initialize the card. Guess the next question is if SD lib now working anymore all examples will have to change and anyone using SD lib will have to up their libs.

    So at this point confused? Going through the SD Lib now to see the public functions/

  8. #8
    Senior Member blackketter's Avatar
    Join Date
    May 2015
    Location
    San Francisco
    Posts
    308
    Well done, sir! This is great.

  9. #9
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany NRW
    Posts
    7,069
    What is the impact for the Teensy LC / 3.2 ( Flash+RAM, Speed ) ?


    (p.s. a great decision!)

  10. #10
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,039
    Quote Originally Posted by mjs513 View Post
    CardInfo:40: error: 'BUILTIN_SDCARD' was not declared in this scope
    Opps, fixed.

    https://github.com/PaulStoffregen/SD...ea5a169402f08f


    To use the abstraction layer how to you initialize the card.
    SD.begin(chipselect);

    Or you can access the main SdFat object with "SD.sdfs", if you want to initialize with SdFat with other options not available through the SD library abstraction, you could use something like this:

    SD.sdfs.begin(SdioConfig(DMA_SDIO));

    And if want to create FsFile objects rather than File, so you have access to all of special SdFat functions like truncate() and preAllocate(). For examples, you could write something like this to get DMA mode:

    FsFile myfile = SD.sdfs.open("filename.bin");
    myfile.preAllocate(2000000);

    You can freely mix this direct SdFat usage with File and code means to call the old SD library, since it's just using SdFat now.

    If your program has an instance named "sd", you might do something like "#define sd SD.sdfs" if you don't feel like editing your code in many places.


    Guess the next question is if SD lib now working anymore all examples will have to change and anyone using SD lib will have to up their libs.
    My goal is all programs written for the Arduino SD library will "just work".

    However, fully supporting the Sd2Card, SdVolume, SdFile classes which programs used to query the low-level card info may not be realistic. Still not sure what to do about those...


    Going through the SD Lib now to see the public functions/
    The good news is you won't have to look through very much code! SD.cpp is just 4 lines, and SD.h is 133 lines of mostly inline functions that just call the equivalent SdFat function. The old Arduino SD library truly is gone. It's just a thin compatibility layer so the old SD API can be used to call SdFat functions.

  11. #11
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,893
    @PaulStoffregen

    Just tried the updates and its now working but on a implementation note if you plan to download SD and SDfatbeta with Teensyduino.

    In the Arduino libraries folder there is already a SD library and if you load the the PJRC SD library in the Teensyduino Libraries folder it defaults to the Arduino SD library. I am using 1.54b2 for testing by the way.

    Tried with the standard SD.begin(chipSelect) and SD.sdfs.begin(SdioConfig(DMA_SDIO)); both worked no problem:
    Code:
    Initializing SD card...initialization done.
    
    LOG_0816.csv		4364
    
    OV7670.BMP		614466
    
    System Volume Information/
    
    	WPSettings.dat		12
    
    	IndexerVolumeGuid		76
    
    mtpindex.dat		0
    
    LOG_0001.csv		86201
    
    LOG_0000-afternno.csv		121978
    
    LOG_00001.csv		65318
    
    LOG_62620.csv		183998
    
    done!
    Going to play more with SD

    Oh BTW, think @KurtE mentioned this but Serial.println still prints an extra line - know its small but do you think it can be fixed in 1.54b3?

  12. #12
    Senior Member wwatson's Avatar
    Join Date
    Aug 2017
    Posts
    321
    3: Having the File class in the core library will allow USBHost_t36 to give File class objects for USB mass storage devices. So any library or code that uses File will be able to access either type of media, and other types of media or virtual files in the future.
    This is great Can't wait to work with this and MSC. I have a proof of concept version of a non-blocking MSC that I am just finishing up.

  13. #13
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,039
    Maybe this is a good moment to mention the C++ slicing problems I encountered. Truth is, I mostly think in C (and assembly and analog circuitry) rather than C++. Maybe someone here with more C++ experience might know a better way?

    Arduino's API requires we return a File object, which programs can copy or pass by value into other functions. Any of those File copies must be able to access the media. When File is only for the Arduino SD library, this works because making copies of the File object duplicates all the info needed to actually access the data on the SD card.

    When File becomes a base class, where derived classes add whatever they need to access files on their kind of media, C++ slicing became an issue. Assigning the derived class to a File object allocated in the user's program removes all the info the derived class needs, and calling the virtual functions just runs the copies from File, not the ones in the derived class.

    Maybe there is some elegant C++ way to implement Arduino's File API with derived classes? But this is (so far) the best I could do.

    Click image for larger version. 

Name:	FileClass.jpg 
Views:	8 
Size:	40.9 KB 
ID:	22252

    In this scheme, the File base class has just 1 pointer, which points to an instance of the derived class. In the derived class, instead of the pointer it has a reference count to know how many File objects are point to it.

    The main downside is derived classes have to be allocated on the heap. There's also the slight inefficiency of have 2 levels to access anything. So far (with my admittedly not-so-strong C++ knowledge), this seems to be the only way to make it work. I looked briefly at the ESP8266 and ESP32 code. Seems like they did something similar, maybe even more complicated.

    A minor but annoying issue is the compiler will happily allow you to return a base class instance, which it then slices off when the user assigns it to a File instance. If there isn't some really elegant solution, maybe C++ can offer a way to only allow File's assign operator to accept File objects and give a compile error if the user tries to assign a derived class like SDFile to a File.

    One nice benefit to this is the reference counting. When the last File object referencing the derived class goes out of scope, we can be sure to call the derived class close() function. I know that's something people have wanted to Arduino SD, but each File object can't know how many other coipes may still exist with access to the same physical file on the SD card.

    If anyone has any better C++ suggestions, now is the perfect time! The truth is I'm really an electronics guy who considers C to be a high level language. I can struggle my way through C++ stuff, but I really could use some input from folks who really know C++ well. If there's a better way, I'd sure like to adopt it before we roll this out to lots of end users.

  14. #14
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    12,694
    Quote Originally Posted by mjs513 View Post
    @PaulStoffregen

    Just tried the updates and its now working but on a implementation note if you plan to download SD and SDfatbeta with Teensyduino.

    In the Arduino libraries folder there is already a SD library and if you load the the PJRC SD library in the Teensyduino Libraries folder it defaults to the Arduino SD library. I am using 1.54b2 for testing by the way.

    Tried with the standard SD.begin(chipSelect) and SD.sdfs.begin(SdioConfig(DMA_SDIO)); both worked no problem:
    ...

    Going to play more with SD

    Oh BTW, think @KurtE mentioned this but Serial.println still prints an extra line - know its small but do you think it can be fixed in 1.54b3?
    @mjs513 : noted above it looks like the SD library will be gone except for a stub into SdFat-Beta ::

    p#2 :: 3: This replaces the Arduino SD library with a thin wrapper around SdFat, using FS.h's File class.

    p#10 :: SD.cpp is just 4 lines, and SD.h is 133 lines of mostly inline functions that just call the equivalent SdFat function.

    <edit>: haven't tried any of this yet - maybe the end solution will have version or something to give it precedence when complete.

    @KurtE / @mjs513:: ... @Paul
    the Println() is ugly - it prints like:
    Code:
    size_t Print::println(void)
    {
    	uint8_t buf[2]={'\r', '\n'};
    	return write(buf, 2);
    }
    I suppose the fix is to have the Windows IDE SerMon act like it does under Mac/Linux and turn "\r\n" into a single "\n" for GUI display?

    Or maybe the '\r' and '\n' are not needed for Mac/Linux either and they just happen to eat the Return [ like a good line printer ] then do the New Line without showing a double line feed like on Windows.
    Last edited by defragster; 10-31-2020 at 05:20 AM.

  15. #15
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,039
    Let's discuss newline formats and serial monitor stuff on a different thread.

    At this moment, I'm pretty focused on the File base class and whether this approach to C++ derived classes and Arduino's File API is a good approach.

  16. #16
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,893
    OBJECT SLICING:

    Had no idea about this one until now. So after reading your post I tried to do a little more reading on slicing and solutions. I came across this article which seems pretty good on offering solutions: https://code-examples.net/en/q/430c2 , https://www.codespeedy.com/object-slicing-in-cpp/,

    Think the pointer solution is what I keep coming across but I am out of c++ comfort zone here . Hopefully someone may have a better solution.

    SD Library Conflict issue mentioned in post 11: really a non- issue. I for forgot to rename the lib to SD after I copied it from GITHUB

  17. #17
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,039
    Quote Originally Posted by mjs513 View Post
    SD Library Conflict issue mentioned in post 11: really a non- issue. I for forgot to rename the lib to SD after I copied it from GITHUB
    Whew, that's a relief. I couldn't reproduce it on Linux and really wasn't looking forward to fiddling more with Windows.

  18. #18
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,893
    Quote Originally Posted by PaulStoffregen View Post
    Whew, that's a relief. I couldn't reproduce it on Linux and really wasn't looking forward to fiddling more with Windows.
    Too many distractions here so just realized it this morning - sorry for the confusion.

    But do have another issue you may want to check me on. DATALOGGER: Doesn't seem to working correctly. After letting it run for a few seconds and then checking the file size (should be about 15-20K) only showing 8 or 16 bytes. If the read the file on Windows only showing 1 record - if I try to read the file with Dumpfile the sketch shows card was intizalized correctly but the hangs. How do I know - if I try to load another sketch have to press the PGM button. Just tried it so haven't really looked under the hood yet.

  19. #19
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,039
    Oh, yeah, I believe datalogger isn't working because of this unfinished TODO...

    Code:
            File open(const char *filepath, uint8_t mode = FILE_READ) {
                    oflag_t flags = O_READ;
                    if (mode == FILE_WRITE) flags = O_READ | O_WRITE | O_CREAT;
                    FsFile file = sdfs.open(filepath, flags);
                    // TODO: Arduino's default to seek to end of writable file
                    if (file) return File(new SDFile(file));
                    return File();
            }
    I'll test it later today and fill in that missing piece.

  20. #20
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,893
    thanks Paul. Was just trying to work through the examples and let you know - completely missed the TODO on the first pass.

  21. #21
    Member
    Join Date
    Nov 2012
    Posts
    60
    My 2cents
    I'm not really sure about what the exact nature is of the problem but the 2 things that come to mind are
    1) handles
    2) references https://www.geeksforgeeks.org/pointe...eferences-cpp/

    Handles are used very often in the windows API. Amongst others for files. In its simplest implementation handles are actually pointers disguised in a long. The user doesn't need * or & just a long (little memory usage) and internally you simple convert to a pointer.


    A reference is basically a const pointer to a writeable object. Main advantage is simpler user experience because you do not need & and *

    Best regards
    Jantje

  22. #22
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,039
    I fixed several bugs. Latest code on github should work for the Datalogger, DumpFile and CardInfo examples.

    But CardInfo can't print the root directory because the class name would conflict with SdFat, so I just commented out that last part of the CardInfo example.

  23. #23
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,893
    Quote Originally Posted by PaulStoffregen View Post
    I fixed several bugs. Latest code on github should work for the Datalogger, DumpFile and CardInfo examples.

    But CardInfo can't print the root directory because the class name would conflict with SdFat, so I just commented out that last part of the CardInfo example.
    Well mostly good news.

    Datalogger and Dumpfile are now working

    Cardinfo is giving me the following error:
    Code:
    Initializing SD card...initialization failed. Things to check:
    
    * is a card inserted?
    
    * is your wiring correct?
    
    * did you change the chipSelect pin to match your shield or module?

  24. #24
    Senior Member wwatson's Avatar
    Join Date
    Aug 2017
    Posts
    321
    Quote Originally Posted by mjs513 View Post
    Well mostly good news.

    Datalogger and Dumpfile are now working

    Cardinfo is giving me the following error:
    Code:
    Initializing SD card...initialization failed. Things to check:
    
    * is a card inserted?
    
    * is your wiring correct?
    
    * did you change the chipSelect pin to match your shield or module?
    I was just playing with that same problem. I added this to "SD.h" as a workaround:
    Code:
    #define SD_CARD_TYPE_SD1 0
    #define SD_CARD_TYPE_SD2 1
    #define SD_CARD_TYPE_SDHC 3
    class Sd2Card
    {
    public:
    	bool init(uint8_t speed, uint8_t csPin) {
    #ifdef BUILTIN_SDCARD
    Serial.printf("csPin = %d\n",csPin);
    		if (csPin == BUILTIN_SDCARD) {
    			return SD.sdfs.begin(SdioConfig(FIFO_SDIO));
    			//return SD.sdfs.begin(SdioConfig(DMA_SDIO));
    		} else {
    			return SD.sdfs.begin(csPin);
    		}
    #endif
    
    	}
    	uint8_t type() {
    		return SD.sdfs.card()->type();
    	}
    };
    I wired up one of the PJRC SD Adapters using the standard SPI pins 10,11,12,13. SDInfo.ino works with it but cardInfo.ino still does not.
    Probably missing something.
    Last edited by wwatson; 10-31-2020 at 07:42 PM. Reason: Typo again

  25. #25
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,039
    I'm happy to report Larry Bank's JPEGDEC library is working with the new SD -> SdFat layer.

    Click image for larger version. 

Name:	jpegdec.jpg 
Views:	9 
Size:	83.7 KB 
ID:	22268

    But I did have to make one tiny update to code which assumed the extension would always be .JPG (uppercase).

    https://github.com/PaulStoffregen/JP...abde410c0b907d

    This is going to be a minor compatibility issue... the many programs written for 8.3 uppercase-only file names.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •