Teensyduino File System Integration, including MTP and MSC

[Mcu32]

So it needs a pointer to the filesystem. Ok. That was my question. Maybe not clear enough.
The assumption was that you don't know the FS at compile time, or it can at least vary. The lib should be able to use all of them, so it need pointers to all possible used FS and needs to store them for internal use - is that correct? Maybe I have a knot in my brain...

[Mcu32]

So I would use an init with a variable number of arguments like void func( ... ) (like printf) that takes all the pointers. Then how do I know which one to use, i.e. if the program specifically wants to write to SD?
Or, is it needed to use a bunch of overloaded functions (one for each FS, and all would have to be called in the main program) and store that info in a additional variable? What would you do?

Edit: or how to handle things like preallocate or quickformat?

[Mcu32]

The question seems to be too confuse..?
Ok, so: How do I use preallocate() with that pointer. Just assume it is needed for fast writes in case of SD. If it is not possible: Again, how would you create the library API that can handle that?

[PaulStoffregen]

Preallocating a file isn't part of the abstraction layer. It's also not in Arduino's SD library API, nor a feature LittleFS offers (as far as I know). Preallocation probably doesn't even make any sense in the context of a copy-on-write filesystem. It's only something you would do with certain types of filesystems where fixed allocation on the media is used (no wear leveling or power loss safety at the filesystem level).

SdFat does offer an API for preallocation. The SD library in Teensyduino 1.54 & 1.55 (and all planned future versions) is just a thin layer on top of SdFat. You can access the underlying SdFat APIs from the SD object. In Arduino, click File > Examples > SD > SdFat_Usage to see how.

But hopefully it's obvious this SD library feature to use SdFat specific APIs isn't something you can do with an abstraction layer like FS.h which aims to give a consistent API for all filesystems regardless of the underlying media.

[KurtE]

Probably also worth mentioning LittleFS provides several classes for different type of media (Flash, NAND Flash, FRAM, volatile RAM, Program memory) and different connectivity (1 bit SPI vs 4 bit QSPI). The abstraction layer allows any of them to be used the same way as SD and MSC.

Yes - which is great... However there are times I wish we had the LittleFS_Q_Guess class, that looks at the chip (if any) soldered to T4.1 and figures out what chip it is and then gets me to the write NAND, NOR, FRAM... As I have been very good at labeling the different ones I have, so I usually have to guess myself

Edit: Sort of in the way SDFat can be configured for FatFile, ExFat, or FSFat (which does both)...

[PaulStoffregen]

We could add a LittleFS_QSPI class which contains instances of both LittleFS_QSPIFlash and LittleFS_QPINAND. It's begin() function would just call both of those instances' begin() functions and store in a variable which one was successful. Then all its other functions would just call the functions from whichever instance is in use.

The same could be done for a LittleFS_SPI class which has instances of LittleFS_SPIFlash, LittleFS_SPIFram, LittleFS_SPINAND.

Convenience would cost more code size and slight runtime overhead.

[KurtE]

We could add a LittleFS_QSPI class which contains instances of both LittleFS_QSPIFlash and LittleFS_QPINAND. It's begin() function would just call both of those instances' begin() functions and store in a variable which one was successful. Then all its other functions would just call the functions from whichever instance is in use.

The same could be done for a LittleFS_SPI class which has instances of LittleFS_SPIFlash, LittleFS_SPIFram, LittleFS_SPINAND.

Convenience would cost more code size and slight runtime overhead.

That is what I was thinking...
Or maybe actually just one wrapper: like LittleFS_SPI(uint8_t pin=LITTLEFS_BUILTIN);
i.e. like SD where special define that says use your build int one... Which only works for T4.1... But either way would work.

Code size: Not sure how much difference currently. I believe that if you use any of the library, all of it is brought into the executable. The exception to this that I have seen is for those libraries which are built in a similar way to the core... Or more specifically those libraries, whose library.properties includes the option:

dot_a_linkage=true

Found this out when we were playing with a version of the ILI9341_fonts library that if our sketch used any font in this library, it brought all of them into the hex file... The problem I have had in past with linking with .a files is order can be very important and not sure how well Arduino addresses this.

Edit: Overhead... Probably pretty negligible as compared to the speed of things like writes on some of these chips.

[luni]

The question seems to be too confuse..?
Again, how would you create the library API that can handle that?

As Paul already mentioned, the price for abstracting an interface is that you need to stick to the smallest common denominator. RTTI would kind of help but is way too expensive for Microcontrollers. If you really need to get information about the original type of the passed in file system you can use templates or for simple cases just overload the corresponding functions like:

#include "Arduino.h"
#include "FS.h"
#include "LittleFS.h"
#include "sd.h"

// library code ------------------------------------------------------

// general case-------------------------------------------------------

void myWrite(FS& filesystem, const char* filename)
{
    Serial.println("General handler, works for all FS derived filesystems");

    File myfile = filesystem.open(filename, FILE_WRITE_BEGIN);
    if (myfile)
    {
        myfile.println("This is test data written to a file.");
        Serial.println("file written");
    }
    else
    {
        Serial.println("error opening file for write");
    }
}

// override in case you need something special done for certain filesystems ----------------

void myWrite(SDClass& filesystem, const char* filename)
{
    Serial.println("SD specific handler");

    Serial.printf("Bytes per Cluster: %d\n", filesystem.sdfs.bytesPerCluster()); // do somthing specific to SD cards...
    myWrite((FS&)filesystem, filename);                                          // call the general handler for standard work
}

// usage --------------------------------------------------------------------------

void setup()
{
    while (!Serial) {}

    LittleFS_RAM lfsram;
    lfsram.begin(1024);

    SD.begin(BUILTIN_SDCARD);

    myWrite(lfsram, "testfile");
    myWrite(SD, "testfile"); // overloaded function will be called
}

void loop()
{
}

[PaulStoffregen]

Ok, finally here is the FS addition in the core library.

https://github.com/PaulStoffregen/co...81566ec4a950ec

And here are updates to the SD library to use it, and to automatically use the RTC

https://github.com/PaulStoffregen/SD...6bc62c094f07ee

https://github.com/PaulStoffregen/SD...1fd26405c4d67b

[KurtE]

The question seems to be too confuse..?
Ok, so: How do I use preallocate() with that pointer. Just assume it is needed for fast writes in case of SD. If it is not possible: Again, how would you create the library API that can handle that?

Again warning, I am only writing psuedo code here, and doing such things can have big negative consequences on different file systems...
And may not work, but on old systems used to do something like:

 file = myfs.open("bigfile.txt", FILE_WRITE);
  file.seek(my_pre_size, SeekSet);
  file.write("\0", 1); // note some systems needed you to actually write... Others did ot
  file.seek(0, SeekSet); 
...

When you are done you now still have two options: You could leave the file at the full extent
by simply closing the file.

Or you could truncate the file, again it has been awhile, I don't remember for sure if truncate value is from current or absolute size... assume absolute.
If so you could do:
uint64_t pos = file.position();
Or I think you could do pos = fille.seek(0, SeekCur);
and then pass pos to truncate, before close...

But again this MIGHT help SD, it might really slow you down with LittleFS on some media...

As for Format... We don't have that directly into FS... We do have some stuff we are using in this integration code with MTP and our PFS file system code that we added in format stuff, that we know which FS it is and call off to code that is either in LittleFS or in code in our PFSstuff... But that is a different subject

[mjs513]

Ok, finally here is the FS addition in the core library.

https://github.com/PaulStoffregen/co...81566ec4a950ec

And here are updates to the SD library to use it, and to automatically use the RTC

https://github.com/PaulStoffregen/SD...6bc62c094f07ee

https://github.com/PaulStoffregen/SD...1fd26405c4d67b

Ok - looks like you have been busy.

[PaulStoffregen]

I haven't touched LittleFS yet. Hoping you can do it and send a pull request? The core library now as breakTime() which converts the 32 bit RTC number to the format getCreateTime() and getModifyTime() use, and makeTime() to convert to the 32 bit number in setCreateTime() and setModifyTime().

Edit: maybe also copy the new date printing stuff from SD listfiles example into LittleFS ListFiles.

[mjs513]

I haven't touched LittleFS yet. Hoping you can do it and send a pull request? The core library now as breakTime() which converts the 32 bit RTC number to the format getCreateTime() and getModifyTime() use, and makeTime() to convert to the 32 bit number in setCreateTime() and setModifyTime().

Edit: maybe also copy the new date printing stuff from SD listfiles example into LittleFS ListFiles.

Was looking at that this morning and playing around with it in a separate sketch just to see . Going to look now and will send the pr later if thats ok

[KurtE]

Was looking at that this morning and playing around with it in a separate sketch just to see . Going to look now and will send the pr later if thats ok

Sounds good, I am now going to try to build FS_Integration MTP code
with SD_MTP-logger sketch and
start updating ...

Note: It build now, as it does not have the #defnie FS_FILE_SUPPORT_DATES
Which I was using to trigger to enable the date time code.

We may still need something similar... as for example until MTP is integrated as part of Teensyduino and also maybe MSC... They may need to run on previous releases and know they don't have that functionality

[mjs513]

Sounds good, I am now going to try to build FS_Integration MTP code
with SD_MTP-logger sketch and
start updating ...

Note: It build now, as it does not have the #defnie FS_FILE_SUPPORT_DATES
Which I was using to trigger to enable the date time code.

We may still need something similar... as for example until MTP is integrated as part of Teensyduino and also maybe MSC... They may need to run on previous releases and know they don't have that functionality

I took out the #defnie FS_FILE_SUPPORT_DATES, and have getModifyTime and getCreateTime working with my test sketch. Getting ready to put in the new set functions. For UsbMscFat guess that next on the list.

Assuming you are changing the MTP getModifyDatTime etc functions to match the new core functions?

[PaulStoffregen]

When LittleFS updates are merged, I'll package up 1.56-beta1. Then you can can use "#if TEENSYDUINO >= 156" to check.

My hope is to merge all this stuff "soon", so this sort of version checking won't be needed long-term.

[KurtE]

For testing right now, I just put that #define into MTP...

But wondering now if I should store stuff locally as the structure or as uint32_t... I started down the structure route but I think I will end up with the uint32_t...

[mjs513]

When LittleFS updates are merged, I'll package up 1.56-beta1. Then you can can use "#if TEENSYDUINO >= 156" to check.

My hope is to merge all this stuff "soon", so this sort of version checking won't be needed long-term.

Just sent you the PR for LittleFS.

[mjs513]

For testing right now, I just put that #define into MTP...

But wondering now if I should store stuff locally as the structure or as uint32_t... I started down the structure route but I think I will end up with the uint32_t...

Stored the uint32_t's and then just converted them as needed with either breakTime or MakeTime.

[KurtE]

Stored the uint32_t's and then just converted them as needed with either breakTime or MakeTime.

That is what I am working on... Just making my way through several of the places that use it.

Sorry going slower than I should, but took advantage of sun being out and went for walk

[PaulStoffregen]

When using the new DateTimeFields in MTP, please keep in mind it is slightly different than TimeElements. It follows C library struct tm format. The month is 0-based, not 1-12 as in TimeElements. The year is offset by 1900, not 1970.

// DateTimeFields follows C library "struct tm" convention, but uses much less memory
typedef struct  {
        uint8_t sec;   // 0-59
        uint8_t min;   // 0-59
        uint8_t hour;  // 0-23
        uint8_t wday;  // 0-6, 0=sunday
        uint8_t mday;  // 1-31
        uint8_t mon;   // 0-11
        uint8_t year;  // 70-206, 70=1970, 206=2106
} DateTimeFields;

[mjs513]

That is what I am working on... Just making my way through several of the places that use it.

Sorry going slower than I should, but took advantage of sun being out and went for walk

Yeah about the same here - want to do a sanity check on that PR to make sure.

EDIT: ok it looks right for the changes I made so let the fun begin.

[KurtE]

When using the new DateTimeFields in MTP, please keep in mind it is slightly different than TimeElements. It follows C library struct tm format. The month is 0-based, not 1-12 as in TimeElements. The year is offset by 1900, not 1970.

// DateTimeFields follows C library "struct tm" convention, but uses much less memory
typedef struct  {
        uint8_t sec;   // 0-59
        uint8_t min;   // 0-59
        uint8_t hour;  // 0-23
        uint8_t wday;  // 0-6, 0=sunday
        uint8_t mday;  // 1-31
        uint8_t mon;   // 0-11
        uint8_t year;  // 70-206, 70=1970, 206=2106
} DateTimeFields;

Good heads up... I need to update some more of my WIP for this.
How important is the wday field?

I noticed the 1900 vs 1970...

I have converted code but now debugging...
But I know from this, my conversion to and from MTP dates needs more adjustment:

       case MTP_PROPERTY_DATE_MODIFIED:      //0xDC09:
          {
            #ifdef MTP_SUPPORT_MODIFY_DATE
              // String is like: YYYYMMDDThhmmss.s
              uint32_t dt;
              DateTimeFields dtf;

              if (storage_->getModifyTime(p1, dt)) {
                // going to use the buffer name to output
                breakTime(dt, dtf);
                snprintf(name, MAX_FILENAME_LEN, "%04u%02u%02uT%02u%02u%02u", 
                      dtf.year+1900, dtf.mon, dtf.mday, dtf.hour, dtf.min, dtf.sec);
                writestring(name);
                printf("Modify (%x)Date/time:%s\n", dt, name);
                break;
              }
            #endif
          }

And the reverse function as well...

[KurtE]

Ok My MTP_T4/FS_Integrated fork/branch has been updated to this newer date/time format.

I believe most of the changes are now working with the new updates to FS and date/Time....

Paul - I should probably be able to get rid of the setSyncProvider stuff with the MTP stuff...

[wwatson]

@Paul @KurtE @mjs513 - Been working on updating mscFS.h/cpp. No luck yet. I have attached a zip file with the files below.

The compile error:

/home/wwatson/arduino-1.8.16/hardware/teensy/../tools/arm/bin/arm-none-eabi-g++ -c -O2 -g -Wall -ffunction-sections -fdata-sections -nostdlib -MMD -std=gnu++14 -fno-exceptions -fpermissive -fno-rtti -fno-threadsafe-statics -felide-constructors -Wno-error=narrowing -mthumb -mcpu=cortex-m7 -mfloat-abi=hard -mfpu=fpv5-d16 -D__IMXRT1062__ -DTEENSYDUINO=155 -DARDUINO=10816 -DARDUINO_TEENSY41 -DF_CPU=600000000 -DUSB_SERIAL -DLAYOUT_US_ENGLISH -I/tmp/arduino_build_798727/pch -I/home/wwatson/arduino-1.8.16/hardware/teensy/avr/cores/teensy4 -I/home/wwatson/Arduino/libraries/UsbMscFat-main/src -I/home/wwatson/arduino-1.8.16/hardware/teensy/avr/libraries/USBHost_t36 -I/home/wwatson/arduino-1.8.16/hardware/teensy/avr/libraries/SdFat/src -I/home/wwatson/arduino-1.8.16/hardware/teensy/avr/libraries/SPI /tmp/arduino_build_798727/sketch/listfilesUSB.ino.cpp -o /tmp/arduino_build_798727/sketch/listfilesUSB.ino.cpp.o
In file included from /home/wwatson/Arduino/libraries/UsbMscFat-main/examples/listfilesUSB/listfilesUSB.ino:14:0:
/home/wwatson/Arduino/libraries/UsbMscFat-main/src/mscFS.h:191:27: error: 'dateTime' has not been declared
   FsDateTime::setCallback(dateTime);
                           ^
/home/wwatson/Arduino/libraries/UsbMscFat-main/src/mscFS.h:191:35: warning: ISO C++ forbids declaration of 'setCallback' with no type [-fpermissive]
   FsDateTime::setCallback(dateTime);
                                   ^
/home/wwatson/Arduino/libraries/UsbMscFat-main/src/mscFS.h:191:35: error: invalid use of '::'
Using library UsbMscFat-main at version 1.0.0 in folder: /home/wwatson/Arduino/libraries/UsbMscFat-main 
Using library USBHost_t36 at version 0.1 in folder: /home/wwatson/arduino-1.8.16/hardware/teensy/avr/libraries/USBHost_t36 
Using library SdFat at version 2.0.5-beta.1 in folder: /home/wwatson/arduino-1.8.16/hardware/teensy/avr/libraries/SdFat 
Using library SPI at version 1.0 in folder: /home/wwatson/arduino-1.8.16/hardware/teensy/avr/libraries/SPI 
Error compiling for board Teensy 4.1.

I am using the latest version of arduino and TD along with a brand new clean install of Ubuntu 20.04. Installed the updated cores and SD libraries. It seems like the compiler is not recognizing the dateTime() function? Probably missing something again