Forum Rule: Always post complete source code & details to reproduce any issue!
Page 3 of 15 FirstFirst 1 2 3 4 5 13 ... LastLast
Results 51 to 75 of 362

Thread: Teensyduino File System Integration, including MTP and MSC

  1. #51
    Member
    Join Date
    Sep 2021
    Posts
    74
    So it needs a pointer to the filesystem. Ok. That was my question. Maybe not clear enough.
    The assumption was that you don't know the FS at compile time, or it can at least vary. The lib should be able to use all of them, so it need pointers to all possible used FS and needs to store them for internal use - is that correct? Maybe I have a knot in my brain...

  2. #52
    Member
    Join Date
    Sep 2021
    Posts
    74
    So I would use an init with a variable number of arguments like void func( ... ) (like printf) that takes all the pointers. Then how do I know which one to use, i.e. if the program specifically wants to write to SD?
    Or, is it needed to use a bunch of overloaded functions (one for each FS, and all would have to be called in the main program) and store that info in a additional variable? What would you do?

    Edit: or how to handle things like preallocate or quickformat?

  3. #53
    Member
    Join Date
    Sep 2021
    Posts
    74
    The question seems to be too confuse..?
    Ok, so: How do I use preallocate() with that pointer. Just assume it is needed for fast writes in case of SD. If it is not possible: Again, how would you create the library API that can handle that?

  4. #54
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    25,092
    Preallocating a file isn't part of the abstraction layer. It's also not in Arduino's SD library API, nor a feature LittleFS offers (as far as I know). Preallocation probably doesn't even make any sense in the context of a copy-on-write filesystem. It's only something you would do with certain types of filesystems where fixed allocation on the media is used (no wear leveling or power loss safety at the filesystem level).

    SdFat does offer an API for preallocation. The SD library in Teensyduino 1.54 & 1.55 (and all planned future versions) is just a thin layer on top of SdFat. You can access the underlying SdFat APIs from the SD object. In Arduino, click File > Examples > SD > SdFat_Usage to see how.

    But hopefully it's obvious this SD library feature to use SdFat specific APIs isn't something you can do with an abstraction layer like FS.h which aims to give a consistent API for all filesystems regardless of the underlying media.

  5. #55
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,577
    Quote Originally Posted by PaulStoffregen View Post
    Probably also worth mentioning LittleFS provides several classes for different type of media (Flash, NAND Flash, FRAM, volatile RAM, Program memory) and different connectivity (1 bit SPI vs 4 bit QSPI). The abstraction layer allows any of them to be used the same way as SD and MSC.
    Yes - which is great... However there are times I wish we had the LittleFS_Q_Guess class, that looks at the chip (if any) soldered to T4.1 and figures out what chip it is and then gets me to the write NAND, NOR, FRAM... As I have been very good at labeling the different ones I have, so I usually have to guess myself

    Edit: Sort of in the way SDFat can be configured for FatFile, ExFat, or FSFat (which does both)...

  6. #56
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    25,092
    We could add a LittleFS_QSPI class which contains instances of both LittleFS_QSPIFlash and LittleFS_QPINAND. It's begin() function would just call both of those instances' begin() functions and store in a variable which one was successful. Then all its other functions would just call the functions from whichever instance is in use.

    The same could be done for a LittleFS_SPI class which has instances of LittleFS_SPIFlash, LittleFS_SPIFram, LittleFS_SPINAND.

    Convenience would cost more code size and slight runtime overhead.

  7. #57
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,577
    Quote Originally Posted by PaulStoffregen View Post
    We could add a LittleFS_QSPI class which contains instances of both LittleFS_QSPIFlash and LittleFS_QPINAND. It's begin() function would just call both of those instances' begin() functions and store in a variable which one was successful. Then all its other functions would just call the functions from whichever instance is in use.

    The same could be done for a LittleFS_SPI class which has instances of LittleFS_SPIFlash, LittleFS_SPIFram, LittleFS_SPINAND.

    Convenience would cost more code size and slight runtime overhead.
    That is what I was thinking...
    Or maybe actually just one wrapper: like LittleFS_SPI(uint8_t pin=LITTLEFS_BUILTIN);
    i.e. like SD where special define that says use your build int one... Which only works for T4.1... But either way would work.

    Code size: Not sure how much difference currently. I believe that if you use any of the library, all of it is brought into the executable. The exception to this that I have seen is for those libraries which are built in a similar way to the core... Or more specifically those libraries, whose library.properties includes the option:
    Code:
    dot_a_linkage=true
    Found this out when we were playing with a version of the ILI9341_fonts library that if our sketch used any font in this library, it brought all of them into the hex file... The problem I have had in past with linking with .a files is order can be very important and not sure how well Arduino addresses this.

    Edit: Overhead... Probably pretty negligible as compared to the speed of things like writes on some of these chips.

  8. #58
    Senior Member
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    1,626
    Quote Originally Posted by Mcu32 View Post
    The question seems to be too confuse..?
    Again, how would you create the library API that can handle that?
    As Paul already mentioned, the price for abstracting an interface is that you need to stick to the smallest common denominator. RTTI would kind of help but is way too expensive for Microcontrollers. If you really need to get information about the original type of the passed in file system you can use templates or for simple cases just overload the corresponding functions like:

    Code:
    #include "Arduino.h"
    #include "FS.h"
    #include "LittleFS.h"
    #include "sd.h"
    
    // library code ------------------------------------------------------
    
    // general case-------------------------------------------------------
    
    void myWrite(FS& filesystem, const char* filename)
    {
        Serial.println("General handler, works for all FS derived filesystems");
    
        File myfile = filesystem.open(filename, FILE_WRITE_BEGIN);
        if (myfile)
        {
            myfile.println("This is test data written to a file.");
            Serial.println("file written");
        }
        else
        {
            Serial.println("error opening file for write");
        }
    }
    
    // override in case you need something special done for certain filesystems ----------------
    
    void myWrite(SDClass& filesystem, const char* filename)
    {
        Serial.println("SD specific handler");
    
        Serial.printf("Bytes per Cluster: %d\n", filesystem.sdfs.bytesPerCluster()); // do somthing specific to SD cards...
        myWrite((FS&)filesystem, filename);                                          // call the general handler for standard work
    }
    
    // usage --------------------------------------------------------------------------
    
    void setup()
    {
        while (!Serial) {}
    
        LittleFS_RAM lfsram;
        lfsram.begin(1024);
    
        SD.begin(BUILTIN_SDCARD);
    
        myWrite(lfsram, "testfile");
        myWrite(SD, "testfile"); // overloaded function will be called
    }
    
    void loop()
    {
    }

  9. #59
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    25,092
    Ok, finally here is the FS addition in the core library.

    https://github.com/PaulStoffregen/co...81566ec4a950ec

    And here are updates to the SD library to use it, and to automatically use the RTC

    https://github.com/PaulStoffregen/SD...6bc62c094f07ee

    https://github.com/PaulStoffregen/SD...1fd26405c4d67b

  10. #60
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,577
    Quote Originally Posted by Mcu32 View Post
    The question seems to be too confuse..?
    Ok, so: How do I use preallocate() with that pointer. Just assume it is needed for fast writes in case of SD. If it is not possible: Again, how would you create the library API that can handle that?
    Again warning, I am only writing psuedo code here, and doing such things can have big negative consequences on different file systems...
    And may not work, but on old systems used to do something like:

    Code:
     file = myfs.open("bigfile.txt", FILE_WRITE);
      file.seek(my_pre_size, SeekSet);
      file.write("\0", 1); // note some systems needed you to actually write... Others did ot
      file.seek(0, SeekSet); 
    ...
    When you are done you now still have two options: You could leave the file at the full extent
    by simply closing the file.

    Or you could truncate the file, again it has been awhile, I don't remember for sure if truncate value is from current or absolute size... assume absolute.
    If so you could do:
    uint64_t pos = file.position();
    Or I think you could do pos = fille.seek(0, SeekCur);
    and then pass pos to truncate, before close...

    But again this MIGHT help SD, it might really slow you down with LittleFS on some media...

    As for Format... We don't have that directly into FS... We do have some stuff we are using in this integration code with MTP and our PFS file system code that we added in format stuff, that we know which FS it is and call off to code that is either in LittleFS or in code in our PFSstuff... But that is a different subject

  11. #61
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    7,558
    Quote Originally Posted by PaulStoffregen View Post
    Ok, finally here is the FS addition in the core library.

    https://github.com/PaulStoffregen/co...81566ec4a950ec

    And here are updates to the SD library to use it, and to automatically use the RTC

    https://github.com/PaulStoffregen/SD...6bc62c094f07ee

    https://github.com/PaulStoffregen/SD...1fd26405c4d67b
    Ok - looks like you have been busy.

  12. #62
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    25,092
    I haven't touched LittleFS yet. Hoping you can do it and send a pull request? The core library now as breakTime() which converts the 32 bit RTC number to the format getCreateTime() and getModifyTime() use, and makeTime() to convert to the 32 bit number in setCreateTime() and setModifyTime().

    Edit: maybe also copy the new date printing stuff from SD listfiles example into LittleFS ListFiles.

  13. #63
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    7,558
    Quote Originally Posted by PaulStoffregen View Post
    I haven't touched LittleFS yet. Hoping you can do it and send a pull request? The core library now as breakTime() which converts the 32 bit RTC number to the format getCreateTime() and getModifyTime() use, and makeTime() to convert to the 32 bit number in setCreateTime() and setModifyTime().

    Edit: maybe also copy the new date printing stuff from SD listfiles example into LittleFS ListFiles.
    Was looking at that this morning and playing around with it in a separate sketch just to see . Going to look now and will send the pr later if thats ok

  14. #64
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,577
    Quote Originally Posted by mjs513 View Post
    Was looking at that this morning and playing around with it in a separate sketch just to see . Going to look now and will send the pr later if thats ok
    Sounds good, I am now going to try to build FS_Integration MTP code
    with SD_MTP-logger sketch and
    start updating ...

    Note: It build now, as it does not have the #defnie FS_FILE_SUPPORT_DATES
    Which I was using to trigger to enable the date time code.

    We may still need something similar... as for example until MTP is integrated as part of Teensyduino and also maybe MSC... They may need to run on previous releases and know they don't have that functionality

  15. #65
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    7,558
    Quote Originally Posted by KurtE View Post
    Sounds good, I am now going to try to build FS_Integration MTP code
    with SD_MTP-logger sketch and
    start updating ...

    Note: It build now, as it does not have the #defnie FS_FILE_SUPPORT_DATES
    Which I was using to trigger to enable the date time code.

    We may still need something similar... as for example until MTP is integrated as part of Teensyduino and also maybe MSC... They may need to run on previous releases and know they don't have that functionality
    I took out the #defnie FS_FILE_SUPPORT_DATES, and have getModifyTime and getCreateTime working with my test sketch. Getting ready to put in the new set functions. For UsbMscFat guess that next on the list.

    Assuming you are changing the MTP getModifyDatTime etc functions to match the new core functions?

  16. #66
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    25,092
    When LittleFS updates are merged, I'll package up 1.56-beta1. Then you can can use "#if TEENSYDUINO >= 156" to check.

    My hope is to merge all this stuff "soon", so this sort of version checking won't be needed long-term.

  17. #67
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,577
    For testing right now, I just put that #define into MTP...

    But wondering now if I should store stuff locally as the structure or as uint32_t... I started down the structure route but I think I will end up with the uint32_t...

  18. #68
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    7,558
    Quote Originally Posted by PaulStoffregen View Post
    When LittleFS updates are merged, I'll package up 1.56-beta1. Then you can can use "#if TEENSYDUINO >= 156" to check.

    My hope is to merge all this stuff "soon", so this sort of version checking won't be needed long-term.
    Just sent you the PR for LittleFS.

  19. #69
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    7,558
    Quote Originally Posted by KurtE View Post
    For testing right now, I just put that #define into MTP...

    But wondering now if I should store stuff locally as the structure or as uint32_t... I started down the structure route but I think I will end up with the uint32_t...
    Stored the uint32_t's and then just converted them as needed with either breakTime or MakeTime.

  20. #70
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,577
    Quote Originally Posted by mjs513 View Post
    Stored the uint32_t's and then just converted them as needed with either breakTime or MakeTime.
    That is what I am working on... Just making my way through several of the places that use it.

    Sorry going slower than I should, but took advantage of sun being out and went for walk

  21. #71
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    25,092
    When using the new DateTimeFields in MTP, please keep in mind it is slightly different than TimeElements. It follows C library struct tm format. The month is 0-based, not 1-12 as in TimeElements. The year is offset by 1900, not 1970.

    Code:
    // DateTimeFields follows C library "struct tm" convention, but uses much less memory
    typedef struct  {
            uint8_t sec;   // 0-59
            uint8_t min;   // 0-59
            uint8_t hour;  // 0-23
            uint8_t wday;  // 0-6, 0=sunday
            uint8_t mday;  // 1-31
            uint8_t mon;   // 0-11
            uint8_t year;  // 70-206, 70=1970, 206=2106
    } DateTimeFields;

  22. #72
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    7,558
    Quote Originally Posted by KurtE View Post
    That is what I am working on... Just making my way through several of the places that use it.

    Sorry going slower than I should, but took advantage of sun being out and went for walk
    Yeah about the same here - want to do a sanity check on that PR to make sure.

    EDIT: ok it looks right for the changes I made so let the fun begin.

  23. #73
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,577
    Quote Originally Posted by PaulStoffregen View Post
    When using the new DateTimeFields in MTP, please keep in mind it is slightly different than TimeElements. It follows C library struct tm format. The month is 0-based, not 1-12 as in TimeElements. The year is offset by 1900, not 1970.

    Code:
    // DateTimeFields follows C library "struct tm" convention, but uses much less memory
    typedef struct  {
            uint8_t sec;   // 0-59
            uint8_t min;   // 0-59
            uint8_t hour;  // 0-23
            uint8_t wday;  // 0-6, 0=sunday
            uint8_t mday;  // 1-31
            uint8_t mon;   // 0-11
            uint8_t year;  // 70-206, 70=1970, 206=2106
    } DateTimeFields;
    Good heads up... I need to update some more of my WIP for this.
    How important is the wday field?

    I noticed the 1900 vs 1970...

    I have converted code but now debugging...
    But I know from this, my conversion to and from MTP dates needs more adjustment:
    Code:
           case MTP_PROPERTY_DATE_MODIFIED:      //0xDC09:
              {
                #ifdef MTP_SUPPORT_MODIFY_DATE
                  // String is like: YYYYMMDDThhmmss.s
                  uint32_t dt;
                  DateTimeFields dtf;
    
                  if (storage_->getModifyTime(p1, dt)) {
                    // going to use the buffer name to output
                    breakTime(dt, dtf);
                    snprintf(name, MAX_FILENAME_LEN, "%04u%02u%02uT%02u%02u%02u", 
                          dtf.year+1900, dtf.mon, dtf.mday, dtf.hour, dtf.min, dtf.sec);
                    writestring(name);
                    printf("Modify (%x)Date/time:%s\n", dt, name);
                    break;
                  }
                #endif
              }
    And the reverse function as well...

  24. #74
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,577
    Ok My MTP_T4/FS_Integrated fork/branch has been updated to this newer date/time format.

    I believe most of the changes are now working with the new updates to FS and date/Time....

    Paul - I should probably be able to get rid of the setSyncProvider stuff with the MTP stuff...

  25. #75
    Senior Member wwatson's Avatar
    Join Date
    Aug 2017
    Posts
    538
    @Paul @KurtE @mjs513 - Been working on updating mscFS.h/cpp. No luck yet. I have attached a zip file with the files below.

    The compile error:
    Code:
    /home/wwatson/arduino-1.8.16/hardware/teensy/../tools/arm/bin/arm-none-eabi-g++ -c -O2 -g -Wall -ffunction-sections -fdata-sections -nostdlib -MMD -std=gnu++14 -fno-exceptions -fpermissive -fno-rtti -fno-threadsafe-statics -felide-constructors -Wno-error=narrowing -mthumb -mcpu=cortex-m7 -mfloat-abi=hard -mfpu=fpv5-d16 -D__IMXRT1062__ -DTEENSYDUINO=155 -DARDUINO=10816 -DARDUINO_TEENSY41 -DF_CPU=600000000 -DUSB_SERIAL -DLAYOUT_US_ENGLISH -I/tmp/arduino_build_798727/pch -I/home/wwatson/arduino-1.8.16/hardware/teensy/avr/cores/teensy4 -I/home/wwatson/Arduino/libraries/UsbMscFat-main/src -I/home/wwatson/arduino-1.8.16/hardware/teensy/avr/libraries/USBHost_t36 -I/home/wwatson/arduino-1.8.16/hardware/teensy/avr/libraries/SdFat/src -I/home/wwatson/arduino-1.8.16/hardware/teensy/avr/libraries/SPI /tmp/arduino_build_798727/sketch/listfilesUSB.ino.cpp -o /tmp/arduino_build_798727/sketch/listfilesUSB.ino.cpp.o
    In file included from /home/wwatson/Arduino/libraries/UsbMscFat-main/examples/listfilesUSB/listfilesUSB.ino:14:0:
    /home/wwatson/Arduino/libraries/UsbMscFat-main/src/mscFS.h:191:27: error: 'dateTime' has not been declared
       FsDateTime::setCallback(dateTime);
                               ^
    /home/wwatson/Arduino/libraries/UsbMscFat-main/src/mscFS.h:191:35: warning: ISO C++ forbids declaration of 'setCallback' with no type [-fpermissive]
       FsDateTime::setCallback(dateTime);
                                       ^
    /home/wwatson/Arduino/libraries/UsbMscFat-main/src/mscFS.h:191:35: error: invalid use of '::'
    Using library UsbMscFat-main at version 1.0.0 in folder: /home/wwatson/Arduino/libraries/UsbMscFat-main 
    Using library USBHost_t36 at version 0.1 in folder: /home/wwatson/arduino-1.8.16/hardware/teensy/avr/libraries/USBHost_t36 
    Using library SdFat at version 2.0.5-beta.1 in folder: /home/wwatson/arduino-1.8.16/hardware/teensy/avr/libraries/SdFat 
    Using library SPI at version 1.0 in folder: /home/wwatson/arduino-1.8.16/hardware/teensy/avr/libraries/SPI 
    Error compiling for board Teensy 4.1.
    I am using the latest version of arduino and TD along with a brand new clean install of Ubuntu 20.04. Installed the updated cores and SD libraries. It seems like the compiler is not recognizing the dateTime() function? Probably missing something again
    Attached Files Attached Files
    Last edited by wwatson; 09-25-2021 at 08:46 PM. Reason: Show llistfilesUSB.ino error

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •