PaulStoffregen
Well-known member
I'm been quiet for the last few days while working on a long-planned change to bring the File class into Teensy's core library, and to remove the Arduino SD library and migrate all SD card use on Teensy to SdFat.
File abstraction and SdFat integration
- Thread starter PaulStoffregen
- Start date
PaulStoffregen
Well-known member
If anyone wants to give it a try, there are 3 pieces to grab from github.
1: In the core library, you'll need this new FS.h header.
https://github.com/PaulStoffregen/cores/blob/master/teensy4/FS.h
2: You'll need this slightly edited copy of SdFat-beta, to avoid a conflict with FS.h
https://github.com/PaulStoffregen/SdFat-beta
3: This replaces the Arduino SD library with a thin wrapper around SdFat, using FS.h's File class.
https://github.com/PaulStoffregen/SD
Make sure you download the "Just_Use_SdFat" branch. Best to delete any copy of SD you might have, since this deletes almost all of SD's files.
This stuff is all very new and probably still pretty rough around the edges. If trying out experimental stuff sounds like fun, now's the time to dive in.
If downloading lots of files and getting them all in the right places sounds tedious, just wait a few days. I'm going to do more testing, (probably) fix bugs, and then package all this up into 1.54-beta3.
1: In the core library, you'll need this new FS.h header.
https://github.com/PaulStoffregen/cores/blob/master/teensy4/FS.h
2: You'll need this slightly edited copy of SdFat-beta, to avoid a conflict with FS.h
https://github.com/PaulStoffregen/SdFat-beta
3: This replaces the Arduino SD library with a thin wrapper around SdFat, using FS.h's File class.
https://github.com/PaulStoffregen/SD
Make sure you download the "Just_Use_SdFat" branch. Best to delete any copy of SD you might have, since this deletes almost all of SD's files.
This stuff is all very new and probably still pretty rough around the edges. If trying out experimental stuff sounds like fun, now's the time to dive in.
If downloading lots of files and getting them all in the right places sounds tedious, just wait a few days. I'm going to do more testing, (probably) fix bugs, and then package all this up into 1.54-beta3.
PaulStoffregen
Well-known member
If you're wondering why I'm doing all this, here's some of the goals.
1: Fix the problems where SD and SdFat don't play well together. The goal is you'll be able to use libraries that want SdFat and others that want SD with little or no painful problems. Well, at least in theory. In practice you might have to change "sd" to "SD.sdfs" and other minor tweaks, but the process of using both libraries in the same project should become relatively easy.
2: All programs using SD will gain SdFat's higher performance and support for long filenames and larger SD cards. The ancient code of SD is completely gone. When you use the SD library, it's actually just calling the same functions of SdFat.
3: Having the File class in the core library will allow USBHost_t36 to give File class objects for USB mass storage devices. So any library or code that uses File will be able to access either type of media, and other types of media or virtual files in the future.
At least those are the goals.
1: Fix the problems where SD and SdFat don't play well together. The goal is you'll be able to use libraries that want SdFat and others that want SD with little or no painful problems. Well, at least in theory. In practice you might have to change "sd" to "SD.sdfs" and other minor tweaks, but the process of using both libraries in the same project should become relatively easy.
2: All programs using SD will gain SdFat's higher performance and support for long filenames and larger SD cards. The ancient code of SD is completely gone. When you use the SD library, it's actually just calling the same functions of SdFat.
3: Having the File class in the core library will allow USBHost_t36 to give File class objects for USB mass storage devices. So any library or code that uses File will be able to access either type of media, and other types of media or virtual files in the future.
At least those are the goals.
PaulStoffregen
Well-known member
Yes, definitely planning to make at least 1 flash filesystem library use this File base class.
Hopefully after SD, USBHost_t36 and a flash chip library use this, the API should become pretty stable and those libraries should serve as examples to make it easier to apply to any other library that provides access to media with files.
Hopefully after SD, USBHost_t36 and a flash chip library use this, the API should become pretty stable and those libraries should serve as examples to make it easier to apply to any other library that provides access to media with files.
mjs513
Senior Member+
Have a few questions now that I am downloaded the updated files per post #2.
If I run the SD examples should they work? Just ran cardinfo example and got this error if using the SD Card on the T4.1:
The sdinfo.ino sketch from SDBeta works fine.
So couple of questions. To use the abstraction layer how to you initialize the card. Guess the next question is if SD lib now working anymore all examples will have to change and anyone using SD lib will have to up their libs.
So at this point confused? Going through the SD Lib now to see the public functions/
If I run the SD examples should they work? Just ran cardinfo example and got this error if using the SD Card on the T4.1:
Code:
CardInfo:40: error: 'BUILTIN_SDCARD' was not declared in this scope
const int chipSelect = BUILTIN_SDCARD;The sdinfo.ino sketch from SDBeta works fine.
So couple of questions. To use the abstraction layer how to you initialize the card. Guess the next question is if SD lib now working anymore all examples will have to change and anyone using SD lib will have to up their libs.
So at this point confused? Going through the SD Lib now to see the public functions/
blackketter
Well-known member
Well done, sir! This is great.
PaulStoffregen
Well-known member
Opps, fixed.
https://github.com/PaulStoffregen/SD/commit/824a16eb0657a8e7e7868b7063ea5a169402f08f
SD.begin(chipselect);
Or you can access the main SdFat object with "SD.sdfs", if you want to initialize with SdFat with other options not available through the SD library abstraction, you could use something like this:
SD.sdfs.begin(SdioConfig(DMA_SDIO));
And if want to create FsFile objects rather than File, so you have access to all of special SdFat functions like truncate() and preAllocate(). For examples, you could write something like this to get DMA mode:
FsFile myfile = SD.sdfs.open("filename.bin");
myfile.preAllocate(2000000);
You can freely mix this direct SdFat usage with File and code means to call the old SD library, since it's just using SdFat now.
If your program has an instance named "sd", you might do something like "#define sd SD.sdfs" if you don't feel like editing your code in many places.
My goal is all programs written for the Arduino SD library will "just work".
However, fully supporting the Sd2Card, SdVolume, SdFile classes which programs used to query the low-level card info may not be realistic. Still not sure what to do about those...
The good news is you won't have to look through very much code! SD.cpp is just 4 lines, and SD.h is 133 lines of mostly inline functions that just call the equivalent SdFat function. The old Arduino SD library truly is gone. It's just a thin compatibility layer so the old SD API can be used to call SdFat functions.
mjs513
Senior Member+
@PaulStoffregen
Just tried the updates and its now working but on a implementation note if you plan to download SD and SDfatbeta with Teensyduino.
In the Arduino libraries folder there is already a SD library and if you load the the PJRC SD library in the Teensyduino Libraries folder it defaults to the Arduino SD library. I am using 1.54b2 for testing by the way.
Tried with the standard SD.begin(chipSelect) and SD.sdfs.begin(SdioConfig(DMA_SDIO)); both worked no problem:
Going to play more with SD
Oh BTW, think @KurtE mentioned this but Serial.println still prints an extra line - know its small but do you think it can be fixed in 1.54b3?
Just tried the updates and its now working but on a implementation note if you plan to download SD and SDfatbeta with Teensyduino.
In the Arduino libraries folder there is already a SD library and if you load the the PJRC SD library in the Teensyduino Libraries folder it defaults to the Arduino SD library. I am using 1.54b2 for testing by the way.
Tried with the standard SD.begin(chipSelect) and SD.sdfs.begin(SdioConfig(DMA_SDIO)); both worked no problem:
Code:
Initializing SD card...initialization done.
LOG_0816.csv 4364
OV7670.BMP 614466
System Volume Information/
WPSettings.dat 12
IndexerVolumeGuid 76
mtpindex.dat 0
LOG_0001.csv 86201
LOG_0000-afternno.csv 121978
LOG_00001.csv 65318
LOG_62620.csv 183998
done!Going to play more with SD
Oh BTW, think @KurtE mentioned this but Serial.println still prints an extra line - know its small but do you think it can be fixed in 1.54b3?
PaulStoffregen
Well-known member
Maybe this is a good moment to mention the C++ slicing problems I encountered. Truth is, I mostly think in C (and assembly and analog circuitry) rather than C++. Maybe someone here with more C++ experience might know a better way?
Arduino's API requires we return a File object, which programs can copy or pass by value into other functions. Any of those File copies must be able to access the media. When File is only for the Arduino SD library, this works because making copies of the File object duplicates all the info needed to actually access the data on the SD card.
When File becomes a base class, where derived classes add whatever they need to access files on their kind of media, C++ slicing became an issue. Assigning the derived class to a File object allocated in the user's program removes all the info the derived class needs, and calling the virtual functions just runs the copies from File, not the ones in the derived class.
Maybe there is some elegant C++ way to implement Arduino's File API with derived classes? But this is (so far) the best I could do.
In this scheme, the File base class has just 1 pointer, which points to an instance of the derived class. In the derived class, instead of the pointer it has a reference count to know how many File objects are point to it.
The main downside is derived classes have to be allocated on the heap. There's also the slight inefficiency of have 2 levels to access anything. So far (with my admittedly not-so-strong C++ knowledge), this seems to be the only way to make it work. I looked briefly at the ESP8266 and ESP32 code. Seems like they did something similar, maybe even more complicated.
A minor but annoying issue is the compiler will happily allow you to return a base class instance, which it then slices off when the user assigns it to a File instance. If there isn't some really elegant solution, maybe C++ can offer a way to only allow File's assign operator to accept File objects and give a compile error if the user tries to assign a derived class like SDFile to a File.
One nice benefit to this is the reference counting. When the last File object referencing the derived class goes out of scope, we can be sure to call the derived class close() function. I know that's something people have wanted to Arduino SD, but each File object can't know how many other coipes may still exist with access to the same physical file on the SD card.
If anyone has any better C++ suggestions, now is the perfect time! The truth is I'm really an electronics guy who considers C to be a high level language. I can struggle my way through C++ stuff, but I really could use some input from folks who really know C++ well. If there's a better way, I'd sure like to adopt it before we roll this out to lots of end users.
Arduino's API requires we return a File object, which programs can copy or pass by value into other functions. Any of those File copies must be able to access the media. When File is only for the Arduino SD library, this works because making copies of the File object duplicates all the info needed to actually access the data on the SD card.
When File becomes a base class, where derived classes add whatever they need to access files on their kind of media, C++ slicing became an issue. Assigning the derived class to a File object allocated in the user's program removes all the info the derived class needs, and calling the virtual functions just runs the copies from File, not the ones in the derived class.
Maybe there is some elegant C++ way to implement Arduino's File API with derived classes? But this is (so far) the best I could do.
In this scheme, the File base class has just 1 pointer, which points to an instance of the derived class. In the derived class, instead of the pointer it has a reference count to know how many File objects are point to it.
The main downside is derived classes have to be allocated on the heap. There's also the slight inefficiency of have 2 levels to access anything. So far (with my admittedly not-so-strong C++ knowledge), this seems to be the only way to make it work. I looked briefly at the ESP8266 and ESP32 code. Seems like they did something similar, maybe even more complicated.
A minor but annoying issue is the compiler will happily allow you to return a base class instance, which it then slices off when the user assigns it to a File instance. If there isn't some really elegant solution, maybe C++ can offer a way to only allow File's assign operator to accept File objects and give a compile error if the user tries to assign a derived class like SDFile to a File.
One nice benefit to this is the reference counting. When the last File object referencing the derived class goes out of scope, we can be sure to call the derived class close() function. I know that's something people have wanted to Arduino SD, but each File object can't know how many other coipes may still exist with access to the same physical file on the SD card.
If anyone has any better C++ suggestions, now is the perfect time! The truth is I'm really an electronics guy who considers C to be a high level language. I can struggle my way through C++ stuff, but I really could use some input from folks who really know C++ well. If there's a better way, I'd sure like to adopt it before we roll this out to lots of end users.
defragster
Senior Member+
@PaulStoffregen
Just tried the updates and its now working but on a implementation note if you plan to download SD and SDfatbeta with Teensyduino.
In the Arduino libraries folder there is already a SD library and if you load the the PJRC SD library in the Teensyduino Libraries folder it defaults to the Arduino SD library. I am using 1.54b2 for testing by the way.
Tried with the standard SD.begin(chipSelect) and SD.sdfs.begin(SdioConfig(DMA_SDIO)); both worked no problem:
...
Going to play more with SD
Oh BTW, think @KurtE mentioned this but Serial.println still prints an extra line - know its small but do you think it can be fixed in 1.54b3?
@mjs513 : noted above it looks like the SD library will be gone except for a stub into SdFat-Beta ::
p#2 :: 3: This replaces the Arduino SD library with a thin wrapper around SdFat, using FS.h's File class.
p#10 :: SD.cpp is just 4 lines, and SD.h is 133 lines of mostly inline functions that just call the equivalent SdFat function.
<edit>: haven't tried any of this yet - maybe the end solution will have version or something to give it precedence when complete.
@KurtE / @mjs513:: ... @Paul
the Println() is ugly - it prints like:
Code:
size_t Print::println(void)
{
uint8_t buf[2]={'\r', '\n'};
return write(buf, 2);
}I suppose the fix is to have the Windows IDE SerMon act like it does under Mac/Linux and turn "\r\n" into a single "\n" for GUI display?
Or maybe the '\r' and '\n' are not needed for Mac/Linux either and they just happen to eat the Return [ like a good line printer ] then do the New Line without showing a double line feed like on Windows.
Last edited:
PaulStoffregen
Well-known member
Let's discuss newline formats and serial monitor stuff on a different thread.
At this moment, I'm pretty focused on the File base class and whether this approach to C++ derived classes and Arduino's File API is a good approach.
At this moment, I'm pretty focused on the File base class and whether this approach to C++ derived classes and Arduino's File API is a good approach.
mjs513
Senior Member+
OBJECT SLICING:
Had no idea about this one until now. So after reading your post I tried to do a little more reading on slicing and solutions. I came across this article which seems pretty good on offering solutions: https://code-examples.net/en/q/430c2 , https://www.codespeedy.com/object-slicing-in-cpp/,
Think the pointer solution is what I keep coming across but I am out of c++ comfort zone here. Hopefully someone may have a better solution.
SD Library Conflict issue mentioned in post 11: really a non- issue. I for forgot to rename the lib to SD after I copied it from GITHUB
Had no idea about this one until now. So after reading your post I tried to do a little more reading on slicing and solutions. I came across this article which seems pretty good on offering solutions: https://code-examples.net/en/q/430c2 , https://www.codespeedy.com/object-slicing-in-cpp/,
Think the pointer solution is what I keep coming across but I am out of c++ comfort zone here
. Hopefully someone may have a better solution.SD Library Conflict issue mentioned in post 11: really a non- issue. I for forgot to rename the lib to SD after I copied it from GITHUB
PaulStoffregen
Well-known member
Whew, that's a relief. I couldn't reproduce it on Linux and really wasn't looking forward to fiddling more with Windows.
mjs513
Senior Member+
Too many distractions here so just realized it this morning - sorry for the confusion.
But do have another issue you may want to check me on. DATALOGGER: Doesn't seem to working correctly. After letting it run for a few seconds and then checking the file size (should be about 15-20K) only showing 8 or 16 bytes. If the read the file on Windows only showing 1 record - if I try to read the file with Dumpfile the sketch shows card was intizalized correctly but the hangs. How do I know - if I try to load another sketch have to press the PGM button. Just tried it so haven't really looked under the hood yet.
PaulStoffregen
Well-known member
Oh, yeah, I believe datalogger isn't working because of this unfinished TODO...
I'll test it later today and fill in that missing piece.
Code:
File open(const char *filepath, uint8_t mode = FILE_READ) {
oflag_t flags = O_READ;
if (mode == FILE_WRITE) flags = O_READ | O_WRITE | O_CREAT;
FsFile file = sdfs.open(filepath, flags);
[B]// TODO: Arduino's default to seek to end of writable file[/B]
if (file) return File(new SDFile(file));
return File();
}I'll test it later today and fill in that missing piece.
My 2cents
I'm not really sure about what the exact nature is of the problem but the 2 things that come to mind are
1) handles
2) references https://www.geeksforgeeks.org/pointers-vs-references-cpp/
Handles are used very often in the windows API. Amongst others for files. In its simplest implementation handles are actually pointers disguised in a long. The user doesn't need * or & just a long (little memory usage) and internally you simple convert to a pointer.
A reference is basically a const pointer to a writeable object. Main advantage is simpler user experience because you do not need & and *
Best regards
Jantje
I'm not really sure about what the exact nature is of the problem but the 2 things that come to mind are
1) handles
2) references https://www.geeksforgeeks.org/pointers-vs-references-cpp/
Handles are used very often in the windows API. Amongst others for files. In its simplest implementation handles are actually pointers disguised in a long. The user doesn't need * or & just a long (little memory usage) and internally you simple convert to a pointer.
A reference is basically a const pointer to a writeable object. Main advantage is simpler user experience because you do not need & and *
Best regards
Jantje
PaulStoffregen
Well-known member
I fixed several bugs. Latest code on github should work for the Datalogger, DumpFile and CardInfo examples.
But CardInfo can't print the root directory because the class name would conflict with SdFat, so I just commented out that last part of the CardInfo example.
But CardInfo can't print the root directory because the class name would conflict with SdFat, so I just commented out that last part of the CardInfo example.
mjs513
Senior Member+
Well mostly good news.
Datalogger and Dumpfile are now working
Cardinfo is giving me the following error:
Code:
Initializing SD card...initialization failed. Things to check:
* is a card inserted?
* is your wiring correct?
* did you change the chipSelect pin to match your shield or module?
wwatson
Well-known member
Well mostly good news.
Datalogger and Dumpfile are now working
Cardinfo is giving me the following error:
Code:Initializing SD card...initialization failed. Things to check: * is a card inserted? * is your wiring correct? * did you change the chipSelect pin to match your shield or module?
I was just playing with that same problem. I added this to "SD.h" as a workaround:
Code:
#define SD_CARD_TYPE_SD1 0
#define SD_CARD_TYPE_SD2 1
#define SD_CARD_TYPE_SDHC 3
class Sd2Card
{
public:
bool init(uint8_t speed, uint8_t csPin) {
[COLOR="#FF0000"]#ifdef BUILTIN_SDCARD
Serial.printf("csPin = %d\n",csPin);
if (csPin == BUILTIN_SDCARD) {
return SD.sdfs.begin(SdioConfig(FIFO_SDIO));
//return SD.sdfs.begin(SdioConfig(DMA_SDIO));
} else {
return SD.sdfs.begin(csPin);
}
#endif[/COLOR]
}
uint8_t type() {
return SD.sdfs.card()->type();
}
};I wired up one of the PJRC SD Adapters using the standard SPI pins 10,11,12,13. SDInfo.ino works with it but cardInfo.ino still does not.
Probably missing something.
Last edited:
PaulStoffregen
Well-known member
I'm happy to report Larry Bank's JPEGDEC library is working with the new SD -> SdFat layer.
But I did have to make one tiny update to code which assumed the extension would always be .JPG (uppercase).
https://github.com/PaulStoffregen/JPEGDEC/commit/72ede8db31aa359b34e4641e4fabde410c0b907d
This is going to be a minor compatibility issue... the many programs written for 8.3 uppercase-only file names.
But I did have to make one tiny update to code which assumed the extension would always be .JPG (uppercase).
https://github.com/PaulStoffregen/JPEGDEC/commit/72ede8db31aa359b34e4641e4fabde410c0b907d
This is going to be a minor compatibility issue... the many programs written for 8.3 uppercase-only file names.