File abstraction and SdFat integration

PaulStoffregen

Well-known member
I'm been quiet for the last few days while working on a long-planned change to bring the File class into Teensy's core library, and to remove the Arduino SD library and migrate all SD card use on Teensy to SdFat.
 
If anyone wants to give it a try, there are 3 pieces to grab from github.

1: In the core library, you'll need this new FS.h header.

https://github.com/PaulStoffregen/cores/blob/master/teensy4/FS.h

2: You'll need this slightly edited copy of SdFat-beta, to avoid a conflict with FS.h

https://github.com/PaulStoffregen/SdFat-beta

3: This replaces the Arduino SD library with a thin wrapper around SdFat, using FS.h's File class.

https://github.com/PaulStoffregen/SD

Make sure you download the "Just_Use_SdFat" branch. Best to delete any copy of SD you might have, since this deletes almost all of SD's files.

This stuff is all very new and probably still pretty rough around the edges. If trying out experimental stuff sounds like fun, now's the time to dive in.

If downloading lots of files and getting them all in the right places sounds tedious, just wait a few days. I'm going to do more testing, (probably) fix bugs, and then package all this up into 1.54-beta3.
 
If you're wondering why I'm doing all this, here's some of the goals.

1: Fix the problems where SD and SdFat don't play well together. The goal is you'll be able to use libraries that want SdFat and others that want SD with little or no painful problems. Well, at least in theory. In practice you might have to change "sd" to "SD.sdfs" and other minor tweaks, but the process of using both libraries in the same project should become relatively easy.

2: All programs using SD will gain SdFat's higher performance and support for long filenames and larger SD cards. The ancient code of SD is completely gone. When you use the SD library, it's actually just calling the same functions of SdFat.

3: Having the File class in the core library will allow USBHost_t36 to give File class objects for USB mass storage devices. So any library or code that uses File will be able to access either type of media, and other types of media or virtual files in the future.

At least those are the goals. ;)
 
Yes, definitely planning to make at least 1 flash filesystem library use this File base class.

Hopefully after SD, USBHost_t36 and a flash chip library use this, the API should become pretty stable and those libraries should serve as examples to make it easier to apply to any other library that provides access to media with files.
 
Have a few questions now that I am downloaded the updated files per post #2.

If I run the SD examples should they work? Just ran cardinfo example and got this error if using the SD Card on the T4.1:
Code:
CardInfo:40: error: 'BUILTIN_SDCARD' was not declared in this scope
 const int chipSelect = BUILTIN_SDCARD;

The sdinfo.ino sketch from SDBeta works fine.

So couple of questions. To use the abstraction layer how to you initialize the card. Guess the next question is if SD lib now working anymore all examples will have to change and anyone using SD lib will have to up their libs.

So at this point confused? Going through the SD Lib now to see the public functions/
 
CardInfo:40: error: 'BUILTIN_SDCARD' was not declared in this scope

Opps, fixed.

https://github.com/PaulStoffregen/SD/commit/824a16eb0657a8e7e7868b7063ea5a169402f08f


To use the abstraction layer how to you initialize the card.

SD.begin(chipselect);

Or you can access the main SdFat object with "SD.sdfs", if you want to initialize with SdFat with other options not available through the SD library abstraction, you could use something like this:

SD.sdfs.begin(SdioConfig(DMA_SDIO));

And if want to create FsFile objects rather than File, so you have access to all of special SdFat functions like truncate() and preAllocate(). For examples, you could write something like this to get DMA mode:

FsFile myfile = SD.sdfs.open("filename.bin");
myfile.preAllocate(2000000);

You can freely mix this direct SdFat usage with File and code means to call the old SD library, since it's just using SdFat now.

If your program has an instance named "sd", you might do something like "#define sd SD.sdfs" if you don't feel like editing your code in many places.


Guess the next question is if SD lib now working anymore all examples will have to change and anyone using SD lib will have to up their libs.

My goal is all programs written for the Arduino SD library will "just work".

However, fully supporting the Sd2Card, SdVolume, SdFile classes which programs used to query the low-level card info may not be realistic. Still not sure what to do about those...


Going through the SD Lib now to see the public functions/

The good news is you won't have to look through very much code! SD.cpp is just 4 lines, and SD.h is 133 lines of mostly inline functions that just call the equivalent SdFat function. The old Arduino SD library truly is gone. It's just a thin compatibility layer so the old SD API can be used to call SdFat functions.
 
@PaulStoffregen

Just tried the updates and its now working but on a implementation note if you plan to download SD and SDfatbeta with Teensyduino.

In the Arduino libraries folder there is already a SD library and if you load the the PJRC SD library in the Teensyduino Libraries folder it defaults to the Arduino SD library. I am using 1.54b2 for testing by the way.

Tried with the standard SD.begin(chipSelect) and SD.sdfs.begin(SdioConfig(DMA_SDIO)); both worked no problem:
Code:
Initializing SD card...initialization done.

LOG_0816.csv		4364

OV7670.BMP		614466

System Volume Information/

	WPSettings.dat		12

	IndexerVolumeGuid		76

mtpindex.dat		0

LOG_0001.csv		86201

LOG_0000-afternno.csv		121978

LOG_00001.csv		65318

LOG_62620.csv		183998

done!

Going to play more with SD :)

Oh BTW, think @KurtE mentioned this but Serial.println still prints an extra line - know its small but do you think it can be fixed in 1.54b3?
 
3: Having the File class in the core library will allow USBHost_t36 to give File class objects for USB mass storage devices. So any library or code that uses File will be able to access either type of media, and other types of media or virtual files in the future.

This is great:) Can't wait to work with this and MSC. I have a proof of concept version of a non-blocking MSC that I am just finishing up.
 
Maybe this is a good moment to mention the C++ slicing problems I encountered. Truth is, I mostly think in C (and assembly and analog circuitry) rather than C++. Maybe someone here with more C++ experience might know a better way?

Arduino's API requires we return a File object, which programs can copy or pass by value into other functions. Any of those File copies must be able to access the media. When File is only for the Arduino SD library, this works because making copies of the File object duplicates all the info needed to actually access the data on the SD card.

When File becomes a base class, where derived classes add whatever they need to access files on their kind of media, C++ slicing became an issue. Assigning the derived class to a File object allocated in the user's program removes all the info the derived class needs, and calling the virtual functions just runs the copies from File, not the ones in the derived class.

Maybe there is some elegant C++ way to implement Arduino's File API with derived classes? But this is (so far) the best I could do.

FileClass.jpg

In this scheme, the File base class has just 1 pointer, which points to an instance of the derived class. In the derived class, instead of the pointer it has a reference count to know how many File objects are point to it.

The main downside is derived classes have to be allocated on the heap. There's also the slight inefficiency of have 2 levels to access anything. So far (with my admittedly not-so-strong C++ knowledge), this seems to be the only way to make it work. I looked briefly at the ESP8266 and ESP32 code. Seems like they did something similar, maybe even more complicated.

A minor but annoying issue is the compiler will happily allow you to return a base class instance, which it then slices off when the user assigns it to a File instance. If there isn't some really elegant solution, maybe C++ can offer a way to only allow File's assign operator to accept File objects and give a compile error if the user tries to assign a derived class like SDFile to a File.

One nice benefit to this is the reference counting. When the last File object referencing the derived class goes out of scope, we can be sure to call the derived class close() function. I know that's something people have wanted to Arduino SD, but each File object can't know how many other coipes may still exist with access to the same physical file on the SD card.

If anyone has any better C++ suggestions, now is the perfect time! The truth is I'm really an electronics guy who considers C to be a high level language. I can struggle my way through C++ stuff, but I really could use some input from folks who really know C++ well. If there's a better way, I'd sure like to adopt it before we roll this out to lots of end users.
 
@PaulStoffregen

Just tried the updates and its now working but on a implementation note if you plan to download SD and SDfatbeta with Teensyduino.

In the Arduino libraries folder there is already a SD library and if you load the the PJRC SD library in the Teensyduino Libraries folder it defaults to the Arduino SD library. I am using 1.54b2 for testing by the way.

Tried with the standard SD.begin(chipSelect) and SD.sdfs.begin(SdioConfig(DMA_SDIO)); both worked no problem:
...

Going to play more with SD :)

Oh BTW, think @KurtE mentioned this but Serial.println still prints an extra line - know its small but do you think it can be fixed in 1.54b3?

@mjs513 : noted above it looks like the SD library will be gone except for a stub into SdFat-Beta ::

p#2 :: 3: This replaces the Arduino SD library with a thin wrapper around SdFat, using FS.h's File class.

p#10 :: SD.cpp is just 4 lines, and SD.h is 133 lines of mostly inline functions that just call the equivalent SdFat function.

<edit>: haven't tried any of this yet - maybe the end solution will have version or something to give it precedence when complete.

@KurtE / @mjs513:: ... @Paul
the Println() is ugly - it prints like:
Code:
size_t Print::println(void)
{
	uint8_t buf[2]={'\r', '\n'};
	return write(buf, 2);
}

I suppose the fix is to have the Windows IDE SerMon act like it does under Mac/Linux and turn "\r\n" into a single "\n" for GUI display?

Or maybe the '\r' and '\n' are not needed for Mac/Linux either and they just happen to eat the Return [ like a good line printer ] then do the New Line without showing a double line feed like on Windows.
 
Last edited:
Let's discuss newline formats and serial monitor stuff on a different thread.

At this moment, I'm pretty focused on the File base class and whether this approach to C++ derived classes and Arduino's File API is a good approach.
 
OBJECT SLICING:

Had no idea about this one until now. So after reading your post I tried to do a little more reading on slicing and solutions. I came across this article which seems pretty good on offering solutions: https://code-examples.net/en/q/430c2 , https://www.codespeedy.com/object-slicing-in-cpp/,

Think the pointer solution is what I keep coming across but I am out of c++ comfort zone here :). Hopefully someone may have a better solution.

SD Library Conflict issue mentioned in post 11: really a non- issue. I for forgot to rename the lib to SD after I copied it from GITHUB
 
SD Library Conflict issue mentioned in post 11: really a non- issue. I for forgot to rename the lib to SD after I copied it from GITHUB

Whew, that's a relief. I couldn't reproduce it on Linux and really wasn't looking forward to fiddling more with Windows.
 
Whew, that's a relief. I couldn't reproduce it on Linux and really wasn't looking forward to fiddling more with Windows.
Too many distractions here so just realized it this morning - sorry for the confusion.

But do have another issue you may want to check me on. DATALOGGER: Doesn't seem to working correctly. After letting it run for a few seconds and then checking the file size (should be about 15-20K) only showing 8 or 16 bytes. If the read the file on Windows only showing 1 record - if I try to read the file with Dumpfile the sketch shows card was intizalized correctly but the hangs. How do I know - if I try to load another sketch have to press the PGM button. Just tried it so haven't really looked under the hood yet.
 
Oh, yeah, I believe datalogger isn't working because of this unfinished TODO...

Code:
        File open(const char *filepath, uint8_t mode = FILE_READ) {
                oflag_t flags = O_READ;
                if (mode == FILE_WRITE) flags = O_READ | O_WRITE | O_CREAT;
                FsFile file = sdfs.open(filepath, flags);
                [B]// TODO: Arduino's default to seek to end of writable file[/B]
                if (file) return File(new SDFile(file));
                return File();
        }

I'll test it later today and fill in that missing piece.
 
thanks Paul. Was just trying to work through the examples and let you know - completely missed the TODO on the first pass.
 
My 2cents
I'm not really sure about what the exact nature is of the problem but the 2 things that come to mind are
1) handles
2) references https://www.geeksforgeeks.org/pointers-vs-references-cpp/

Handles are used very often in the windows API. Amongst others for files. In its simplest implementation handles are actually pointers disguised in a long. The user doesn't need * or & just a long (little memory usage) and internally you simple convert to a pointer.


A reference is basically a const pointer to a writeable object. Main advantage is simpler user experience because you do not need & and *

Best regards
Jantje
 
I fixed several bugs. Latest code on github should work for the Datalogger, DumpFile and CardInfo examples.

But CardInfo can't print the root directory because the class name would conflict with SdFat, so I just commented out that last part of the CardInfo example.
 
I fixed several bugs. Latest code on github should work for the Datalogger, DumpFile and CardInfo examples.

But CardInfo can't print the root directory because the class name would conflict with SdFat, so I just commented out that last part of the CardInfo example.
Well mostly good news.

Datalogger and Dumpfile are now working :)

Cardinfo is giving me the following error:
Code:
Initializing SD card...initialization failed. Things to check:

* is a card inserted?

* is your wiring correct?

* did you change the chipSelect pin to match your shield or module?
 
Well mostly good news.

Datalogger and Dumpfile are now working :)

Cardinfo is giving me the following error:
Code:
Initializing SD card...initialization failed. Things to check:

* is a card inserted?

* is your wiring correct?

* did you change the chipSelect pin to match your shield or module?

I was just playing with that same problem. I added this to "SD.h" as a workaround:
Code:
#define SD_CARD_TYPE_SD1 0
#define SD_CARD_TYPE_SD2 1
#define SD_CARD_TYPE_SDHC 3
class Sd2Card
{
public:
	bool init(uint8_t speed, uint8_t csPin) {
[COLOR="#FF0000"]#ifdef BUILTIN_SDCARD
Serial.printf("csPin = %d\n",csPin);
		if (csPin == BUILTIN_SDCARD) {
			return SD.sdfs.begin(SdioConfig(FIFO_SDIO));
			//return SD.sdfs.begin(SdioConfig(DMA_SDIO));
		} else {
			return SD.sdfs.begin(csPin);
		}
#endif[/COLOR]

	}
	uint8_t type() {
		return SD.sdfs.card()->type();
	}
};

I wired up one of the PJRC SD Adapters using the standard SPI pins 10,11,12,13. SDInfo.ino works with it but cardInfo.ino still does not.
Probably missing something.
 
Last edited:
Back
Top