Teensyduino File System Integration, including MTP and MSC

KurtE · Sep 8, 2021

As has been mentioned in several threads, a few of us have been experimenting with for awhile now.
This is sort of a continuation of the thread: https://forum.pjrc.com/threads/6633...-LittleFS-UsbMSCFat-to-work-with-each-other-8

Note these first few posts here are just first cut, that we will hopefully refine as we go.

Put simply: there are a lot of interesting bits and pieces of functionality associated with File Systems and the like, that we (or at least I) think would be great if they all worked together and at least basic versions of them were integrated into Teensyduino. How much actually makes it there and when... TBD...

Some pieces of this have already been done over the last few releases. Like having a new basic FS/File base class, which we now have an SD and LittleFS updated to use it and in the builds. Likewise portions of MTP and MSC are using this as well, although not yet fully integrated into the builds.

Some of the features I personally think that would be great to enhance and integrate into the system include things like:

a) Support Dates and Times on File systems - SDFat that ships with Teensyduino already supports it.
b) MTP integration
c) MSC Integration
d) some form of File system notifications. Like if running MTP on PC and user deletes a file there... Sketch may want to know about it. Dito if sketch changes files, MTP may want to know about it...
e) Try to make it easy for sketches to use: Hopefully as simple as include a couple of library headers: define a couple of objects, have your setup: call a couple of init/setup/begin methods, and then have your loop call: mtpd.loop(); And of course setup your USB type to include MTP... And likewise for MSC.

Updated 10/28/21 links from @wwatson p#366

Code:

[B][URL="https://github.com/KurtE/MTP_Teensy"]github.com/KurtE/MTP_Teensy[/URL]
[URL="https://github.com/KurtE/USBHost_t36/tree/FS_Integration_MSC"]github.com/KurtE/USBHost_t36/tree/FS_Integration_MSC[/URL]
[URL="https://github.com/KurtE/SdFat/tree/GPT_disks"]github.com/KurtE/SdFat/tree/GPT_disks[/URL]
[URL="https://github.com/KurtE/SD"]github.com/KurtE/SD[/URL]
[/B]
Then as of today I followed this link:
[URL="https://forum.pjrc.com/threads/68560-LittleFS-SPI-and-QSPI-wrappers"]forum.pjrc.com/threads/68560-LittleFS-SPI-and-QSPI-wrappers[/URL] post #3 which pointed me to this sketch:
[URL="https://github.com/KurtE/MTP_Teensy/blob/main/examples/SD_SPI_QSPI_MTP-logger/SD_SPI_QSPI_MTP-logger.ino"]github.com/KurtE/MTP_Teensy/blob/main/examples/SD_SPI_QSPI_MTP-logger/SD_SPI_QSPI_MTP-logger.ino[/URL]

KurtE has moved all needed files from UsbMscFat to  USBHost_t36-FS_integration_MSC and updated them so you will not need to use UsbMscFat any more. It's more self contained now for MSC and maybe other options. The SdFat-GPT_disks library is updated to work with GTP and extended partitions which I have not worked with yet.

9/30/21 : p#135 - status update >> Teensyduino-File-System-Integration-including-MTP-and-MSC

Current state of things. @mjs513 and myself have a set of updated libraries, that we have been making changes in. Most(all) of them we are doing the work in branches called FS_Integration:

cores: with https://github.com/KurtE/cores/tree/FS_Integration
MTP: https://github.com/KurtE/MTP_t4/tree/FS_Integration
MSC: https://github.com/KurtE/UsbMscFat/tree/FS_Integration
LittleFS: https://github.com/Mjs513/LittleFS/tree/FS_Integration
SD: https://github.com/KurtE/SD/tree/FS_Integration

These are are all setup to with support for File object to support dates/times - API's/methods have not been approved so subject to change. But current stuff does show dates and time.

Update(9/21/2021) LittleFS has been updated to support creation and modification dates.

Example output from my Locked T4 on PC

Again this posting will hopefully be updated by all involved, either directly or indirectly
Kurt

EDIT: warning the MTP_Teensy code currently requires the new Teensybuino beta 1.57
Will soon update this page again with more details

KurtE · Sep 8, 2021

Direct or Related issues and things to investigate:

Note: Currently working/testing mostly with T4.x

MTP integration with Serial and Emulated Serial and maybe other USB types
a) The only semi-setup in builds is the emulated Serial. At least for me this always loses all of the early Serial output code. And I am not sure if it ever satisfies the !Serial case... Need to try again... Having better luck with secondary not official USB MTP with Serial.
b) MTP With Serial: Using updated for Teensy4 usb_desc.h from my fork/branch... Updated slightly version worked a little more consistently at startup on some boards... Not sure yet with this locked T4... A lot of this depends on startup timing.

larger SD Cards with Fat32 on them take a real long time to see how much space is used on that disk. By long time I mean with a 32gb card I have seen times like 7-12 seconds to read the entire FAT in and count bits...
With MSC - we have been trying some different tricks for this, like reading the disk asynch and the like. Plus the Fat32 standard has an Information Sector where the value can be cached... However SDFat does nothing with this sector other than I think to semi initialize it if they format the disk. With the MSC stuff with the SD sub-library fslib. which is called pfslib where we added functions to read and write the information sector... Not sure if anything is using this.

Also by default with the SDFat library. Everytime you call for sdfs.clusterCount() or the calls that call this, you have the multi-second read the whole... fat...
This can be reduced to only once if the option: #define MAINTAIN_FREE_CLUSTER_COUNT 1
Default is 0)
This option is in the top level directory of SdFat\src directory in the SdFatConfig.h file

PFSLIb - As I mentioned, in the above section, this is currently in MSC project. It is more a slightly extended version of the FSLib in SDFat project. We created it as there are some things that the owner of SDFat did not want added to SDFat as they may not be totally specific to most SDCards. We are now using some of these features now with our MTP integration of the SDCard, so we may want to find a better home for it. Of course if all the libraries are integrated it may not matter much.

LittleFs does not currently support dates and times - We have not yet added dates and times to LittleFS, partially as we are waiting to see if Dates and Times will be accepted into the File object. Assuming it is done, there are notes on how it can be done up on the LittleFS github project. Including the Issue: Manage generic attibutes
With the support for that issue in the now closed Pull request: implement generic attribute implementation
Updated 9/21/2021: - LittleFS (https://github.com/mjs513/LittleFS/tree/FS_Integration) now supports creation and modification date and time.

LittleFS Slow Write and MTP - Or rephrased MTP times out if operations take longer than expected. If we copy a file to some of the LittleFS systems which take a long time when it does a write, this can cause the host to timeout. We have tried a few different approaches, but none are great...

Dates/Times and setup sketch - Several times we though our sketch was not working, that we were garbage dates and times. Then need to remember to set up a sync provider.
So in my case with MTP stuff, I added a call in setSyncProvider() in the MTPD::Begin method. but maybe this should be auto-magically setup if for example you are using any FS...

FS and MTP Notifications - Two parts
a) On MTP user creates/deletes/modifies a file in the Storage - The MTP subsystem gets notified of this, but the client program may want to know this information. Like your configuration file was updated.

b) The sketch changes uses the FS associated with one of the storages shown on the PC, like they create a new file. MTP should know about this such that the updates show up on the PC. Currently we do have hacks that the client code can directly tell MTP that the FS changed...

Supporting MTP on different platforms:
a) Windows - Windows 10 (and probably most others) have MTP support built in...

b) Ubuntu - I am not sure if there is direct support or if you need to install stuff. I have played with it some with installing: libmtp-dev
I see something about installing snapd to copy android files... So not sure it that will work

c) MAC - Not sure no luck yet... I did see a package you can purchase that supports some stuff...

More to come...

KurtE · Sep 8, 2021

<place holder> testing results

wwatson · Sep 8, 2021

@KurtE - I know you and @mjs513 put a lot of work into using partitions and formatting. Do you think these would be practical additions? If I am right LittleFS does not support partitions yet or if at all.

defragster · Sep 8, 2021

<Place Holder> - Cool! ...

BriComp · Sep 8, 2021

defragster said:
<Place Holder> - Cool! ...

What's all this <Place Holder> rubbish?

defragster · Sep 8, 2021

BriComp said:
What's all this <Place Holder> rubbish?

This is likely to be a thread that lasts for some time as the updates aren't planned be in TD 1.55. - As Sr+ members can return to edit posts - the place holders allow returning to TOP OF THREAD to find the current state and details over time as it develops and consumes the many posts and pages that are likely to follow. In KurtE's case he was planning ahead to allow posts as noted for that evolution. I was poking fun in a way hoping to be 'First' - but do expect to follow and test and may have something interesting to update some weeks from now - or that post can be taken over as needed by Sr+ or Paul.

KurtE · Sep 8, 2021

@BriComp - As @defragster mentioned... In fact I just put some stuff into 2nd post. Things like: Asking an Fat32 partition on an SDCard for how clusters are in use, could take a very long time. There are ways we could work on this to make it work
reasonably...

@wwatsom - Partitions and LittleFS - No idea... That is would one want to partition an SD Card for example and then format one or more partitions with LittleFS?

wwatson · Sep 8, 2021

@KurtE - With the limited size of LittleFS devices it does not seem practical to me

KurtE · Sep 8, 2021

@Paul @mjs513 and others...

Expanding on my first issue in the 2nd post about MTP working with Serial or emulated Serial.

That is how MTP and Serial or Emulated Serial don't like to work with each other very well and sketch startup time.

I am having a little better luck with my experiment with changing the MTP Disk with emulated Serial, when I reordered the Interfaces in the usb_desc.h Maybe it is only circumstantial...

What I am wondering is how the USB client code works along with the PC, where for example if we have multiple USB interfaces:

In this case when MTP starts up, the host sends out a request message: of Start a Session, and waits for a response from us.

In the mean time, our code may be waiting for the Serial object to be available before we continue... Or I usually have a timeout of 5 seconds.

Currently the MTP code does not respond to the start session, until all of the setup() code completes and the loop code calls off to the mtpd.loop...
Again note: sometime wish some devices this may be awhile. like > 10 seconds.

Sometime about then the emulated serial output will start output at that time, other times it may not output anything, until I enter something .

In my last run the output started like:

Code:

5373 CMD: 1004(GET_STORAGE_IDS)l: 12 T:3
5373 RESP:2001(RSP:OK)l: 12 T:3
5373 CMD: 1005(GET_STORAGE_INFO)l: 16 T:4 : 10001
65537 0 name:Program
65537 0 name:Program
5373 RESP:2001(RSP:OK)l: 16 T:4 : 10001
5373 CMD: 1005(GET_STORAGE_INFO)l: 16 T:5 : 20001

Note: in my current code here the CMD/RESP lines the first number is millis(), so the first message response of OK was sent just before this and was lost but the next message received was 5+ seconds into the run, and that was with my bypassing the 10 seconds to get the used space of the SD Card...

Sorry I know this may be unclear, but not sure best way to ask this.

Simply put, can slow processing of inputs on one USB Interface case the Serial code to not receive the special message saying that the Serial port is there for an extended period of time?

Again I know that is probably unclear!

Edit: Next thing I may try on this, is to have the while(!Serial....) code

Is to maybe at startup, add in the code like we do in the background when we format drives, which is start intervalTimer and if it receives any messages, it responds back with MTP_RESPONSE_DEVICE_BUSY
...

wwatson · Sep 8, 2021

KurtE said:
@Paul @mjs513 and others...

Expanding on my first issue in the 2nd post about MTP working with Serial or emulated Serial.

That is how MTP and Serial or Emulated Serial don't like to work with each other very well and sketch startup time.

I am having a little better luck with my experiment with changing the MTP Disk with emulated Serial, when I reordered the Interfaces in the usb_desc.h Maybe it is only circumstantial...

What I am wondering is how the USB client code works along with the PC, where for example if we have multiple USB interfaces:

In this case when MTP starts up, the host sends out a request message: of Start a Session, and waits for a response from us.

In the mean time, our code may be waiting for the Serial object to be available before we continue... Or I usually have a timeout of 5 seconds.

Currently the MTP code does not respond to the start session, until all of the setup() code completes and the loop code calls off to the mtpd.loop...
Again note: sometime wish some devices this may be awhile. like > 10 seconds.

Sometime about then the emulated serial output will start output at that time, other times it may not output anything, until I enter something .

In my last run the output started like:

Code:

5373 CMD: 1004(GET_STORAGE_IDS)l: 12 T:3 5373 RESP:2001(RSP:OK)l: 12 T:3 5373 CMD: 1005(GET_STORAGE_INFO)l: 16 T:4 : 10001 65537 0 name:Program 65537 0 name:Program 5373 RESP:2001(RSP:OK)l: 16 T:4 : 10001 5373 CMD: 1005(GET_STORAGE_INFO)l: 16 T:5 : 20001

Note: in my current code here the CMD/RESP lines the first number is millis(), so the first message response of OK was sent just before this and was lost but the next message received was 5+ seconds into the run, and that was with my bypassing the 10 seconds to get the used space of the SD Card...

Sorry I know this may be unclear, but not sure best way to ask this.

Simply put, can slow processing of inputs on one USB Interface case the Serial code to not receive the special message saying that the Serial port is there for an extended period of time?

Again I know that is probably unclear!

Edit: Next thing I may try on this, is to have the while(!Serial....) code

Is to maybe at startup, add in the code like we do in the background when we format drives, which is start intervalTimer and if it receives any messages, it responds back with MTP_RESPONSE_DEVICE_BUSY
...

I Started working with this today. Created a sketch to test with using only SD. I see what you are seeing. First time after init there is the delay getting used space. Then after that no delay. Even after creating a new file and getting used space. The used size space seems to be correct and there is no delay. Then deleting the file and getting used space, no delay. Have to go back and explore SdFat again.

mjs513 · Sep 9, 2021

Some added status on examples in the MTP library:

Currently SD, LittleFS, UsbMSC, Cores FS-Integration branches are relatively stable (by relatively I mean we aren't making continuous changes to those libraries), MTP on the hand is in a state of flux as we try to improve user setup, timing and add additional functions.

If you look in the MTP src folder you will see several additional classes that have been added:

SDMTPClass - this class added a simpler was to create SD Cards in MTP, added the ability to detect when a SD card has been inserted into either the Builtin_SDCard reader or an external SD Card reader with a Detect Pin such as: https://www.adafruit.com/product/254 transparently to the user. This is illustrated in 2 examples:

Code:

[B]SD_MTP-logger
SD_Program_MTP-logger[/B]

USB_MSC_MTP - this class took all the code for a MSC drive that was originally in MTP and moved it it into its own class so simply calling off a single function will add the MSC drive.

Code:

[B]USB_MTP-logger[/B]

LFS_MTP_Callback - LittleFS class that moves the old callback code for the format setup into a call back class that configures it for you.

Code:

[B]LFS_SPI_MTP_Simple_Datalogger
LFS_Program_MTP_Simple_Datalogger[/B]

Two additional functions have been added to MTP to tell it where to put the index file:

Code:

  void setIndexFile(File * index_file=nullptr);  // allow application to pass in index_file to be used. 
  bool setIndexStore(uint32_t storage=0);  // or can specify specif store in list defaults to 0

You will probably see these in use in the examples so I made not of the here.

MTP-test.ino - MTP-test has now been updated to allow logging to any drive, list directory, dump the log file, format selected drive etc. Still retains allowing multiple configuration of flash drives, Memory and SD cards. [UPDATED]

NOTE: Examples these are changing so be warned to check back frequently.

KurtE · Sep 9, 2021

Thanks @mjs513 - might be good if we can move this message up to the top (more or less)

In my last post/update yesterday, I mentioned something I want to try to see if it helps with the Serial/MTP startup (both real serial and emulation).

In another new method we added mtpd.begin(); - this method tries to remove the stuff of checking to see if pieces are in core, and initialize callbacks...
The method also sets up a time sync provider.

Add a call to code we added for doing format of MTP storages on the PC. In those cases we start an interval timer which checks for MTP messages, and if received it responds device busy... I may try adding a call into there, to start this up... And maybe in loop check to see if this is running and disable it...

KurtE · Sep 9, 2021

Quick update:

I have been hacking around and pushed up a new MTP branch: FS_Integration_debug
Which I have been playing with different approaches to try to get MTP and MTP with Serial to come up quicker with Serial working.

While at it I added some quick and dirty methods to MTP to set the Serial Terminal Serial object... I then update Storage to use it as well as my SD Wrapper file to use it...

So for example can now easily see differences of using Serial4 instead of Serial...

I also added in the code start an IntervalTimer when I start the MTP, that checks for messages an tries to respond. By default I was answering I am busy... But that was causing MTP on PC to give up..
So I have it handle a few messages up front...

However I was missing the real early Debug output, example from the MTP Interval timer before Serial would startup...

So I hacked up a quick and dirty class I added to my Test sketch:

Code:

// Quick and Dirty memory Stream...
class RAMStream : public Stream {
  public:
  // overrides for Stream
  virtual int available() {return (tail_ - head_); }
  virtual int read() {return (tail_ != head_)? buffer_[head_++] : -1;  }
  virtual int peek() {return (tail_ != head_)? buffer_[head_] : -1;  }

  // overrides for Print
  virtual size_t write(uint8_t b) {
    if (tail_ < sizeof(buffer_)) {
      buffer_[tail_++] = b;
      return 1;
    }
    return 0;
  }

  enum {BUFFER_SIZE=32768};
  uint8_t buffer_[BUFFER_SIZE];
  uint32_t head_ = 0;
  uint32_t tail_ = 0;
};

Which I set at startup to MTP to talk to, and after Serial says it is valid, I swap Serial back and then dump out all of the output that was captured... The calls is not very eloquent, but helps get the data...

I also have my call back for SD Cards Used space hacked up to call off to information sector which is not correct but fast... need to work in how and when to switch to slower version...

I am now having better luck with the MTP (serial emulation)... Where it comes up real quick and the Debug Serial terminal properly restarts... Not as much with the MTP with Serial...

Still more debugging to go. But in case anyone is interested in the current debug output with simulated Serial:

Code:

[COLOR="#FF0000"]*** Start Interval Timer ***
350 CMD: 1002(OPEN_SESSION)l: 16 T:0 : 1
350 RESP:2001(RSP:OK)l: 16 T:0 : 1
400 CMD: 1001(GET_DEVICE_INFO)l: 12 T:1
400 RESP:2001(RSP:OK)l: 12 T:1
450 CMD: 1014(GET_DEVICE_PROP_DESC)l: 16 T:2 : d402
450 RESP:2001(RSP:OK)l: 16 T:2 : d402
[/COLOR]
D:\GitHub\MTP_t4\examples\SD_Program_MTP-logger\SD_Program_MTP-logger.ino Sep  9 2021 12:56:48
468 Initializing MTP Storage list ...Date: 9/9/2021 12:57:5
sd_addFilesystem: 0 2000744c Program 20004788 2000744c
Trying to open SPI config: 8 9 0 16000000
Total Size: 31902400512 Used Size: 5242880
addFSToStorage: Added FS
sd_addFilesystem: 1 20006f60 SPI8 2000741c 20006f60
Ram Drive of size: 65536 initialized
sd_addFilesystem: 2 20004508 RAM 200045d0 20004508
Set Storage Index drive to 2
481 SD initialized.

Menu Options:
	1 - List Drives (Step 1)
	2 - Select Drive for Logging (Step 2)
	l - List files on disk
	e - Erase files on disk
	s - Start Logging data (Restarting logger will append records to existing log)
	x - Stop Logging data
	d - Dump Log
	r - Reset MTP
	h - Menu

*** end Interval Timer ***
759 CMD: 1004(GET_STORAGE_IDS)l: 12 T:3
759 RESP:2001(RSP:OK)l: 12 T:3
759 CMD: 1005(GET_STORAGE_INFO)l: 16 T:4 : 10001
65537 0 name:Program
65537 0 name:Program
759 RESP:2001(RSP:OK)l: 16 T:4 : 10001
759 CMD: 1005(GET_STORAGE_INFO)l: 16 T:5 : 20001


{{{{{{{{{ SDMTPClass::totalSizeCB called 20004940 1 536899424 cs:8 ft:32
0


}}}}}}}}} SDMTPClass::usedSizeCB called 20004940 1 536899424 cs:8 ft:32
us:5242880 t:0
131073 1 name:SPI8


{{{{{{{{{ SDMTPClass::totalSizeCB called 20004940 1 536899424 cs:8 ft:32
0


}}}}}}}}} SDMTPClass::usedSizeCB called 20004940 1 536899424 cs:8 ft:32
us:5242880 t:0
131073 1 name:SPI8
760 RESP:2001(RSP:OK)l: 16 T:5 : 20001
760 CMD: 1005(GET_STORAGE_INFO)l: 16 T:6 : 30001
196609 2 name:RAM
196609 2 name:RAM
760 RESP:2001(RSP:OK)l: 16 T:6 : 30001
760 CMD: 9801(GET_OBJECT_PROPS_SUPPORTED)l: 16 T:7 : 3000
760 RESP:2001(RSP:OK)l: 16 T:7 : 3000
760 CMD: 9801(GET_OBJECT_PROPS_SUPPORTED)l: 16 T:8 : 3001
760 RESP:2001(RSP:OK)l: 16 T:8 : 3001

Note the CMD/RESP lines are showing the millis(), which in this case the code was up and running with an Explorer window showing three storages:
The lines in RED were the ones captured with the quick and dirty Serial object...

KurtE · Sep 11, 2021

Quick note: running on Ubuntu... We will need to work through some of the differences to make it work reasonably under Ubuntu:
Note: I believe I used the command

Code:

sudo apt-get install mtp-tools

To install some support on 20.04

I may need to play to see if this gives the latest libmtp stuff.

Was able to get it to come up with sketch with: Program, SD, RAM drives.
Shows up as a SHOTWELL or some such name device in the Files sketch.

The Three storages all show up with the same name: Teensy MTP Disk
Right now it looks like it is not grabbing the files properly. Looks Like the code is telling Ubuntu about them.
From before I know that Events, were not working correctly...

But at least it shows some stuff...

wwatson · Sep 12, 2021

@ALL - Spent the last few days playing with TeensyEXT4. NO GO! other than reading the superblock. This was trying to avoid using an image file and use MSC directly. Not happening. Still do not understand EXT4 completely yet but I think lwext4 is specifically designed to use an image file. That will be the next attempt. After that I will bail on this project if it does not work. Hate failures

KurtE · Sep 13, 2021

wwatson said:
@ALL - Spent the last few days playing with TeensyEXT4. NO GO! other than reading the superblock. This was trying to avoid using an image file and use MSC directly. Not happening. Still do not understand EXT4 completely yet but I think lwext4 is specifically designed to use an image file. That will be the next attempt. After that I will bail on this project if it does not work. Hate failures

Sounds interesting... But don't know anything much about EXT4.. Like what is an image file?

wwatson · Sep 13, 2021

KurtE said:
Sounds interesting... But don't know anything much about EXT4.. Like what is an image file?

This is the project I started a while back and had not worked on for quite a while.

https://github.com/wwatson4506/TeensyEXT4

It's based on lwext4 found here:

https://github.com/gkostka/lwext4

An image file is a file that is used as a file system. It is formatted as a file system then opened for read/write as a file storage device. The image file can be formatted as any file system type you like an can be used on a Windows or Linux file system. In this case the image file is created and formatted with the Linux EXT4 style formatting.

Now that I think about it this would probably be better for transporting the file between storage devices. SD, MSC etc...

KurtE · Sep 15, 2021

Now that TD 1.55 is released

I have started updating several of the FS_Integration forks/branches to be up to date with the current released...
That is I rebased the changes to the current master/Main/Juse_Use_SdFat of the main libraries.

So far I have updated: Cores, SD
I have not checked MTP/MSC or LittleFS.
To see what later changes we have not rebased against.

@Paul and all - Now that 1.55 is released.
I will probably first issue a PR for Cores, Maybe two.

First one: Properly initialize the fill MTP Interface... That is the MTP Interface event endpoint is not initialized in cores...
And not have code added into MTP library to do it after the fact...

KurtE · Sep 16, 2021

Quick update from yesterday:

LittleFS was also updated to build with Dates and Times defined within FS.h... Still does not maintain them... Someone who understands LittleFS better should probably do it.

I also merged in my debug branch of MTPD into my FS_Integration branch. What parts of this will and should be preserved here I am not sure... We may want to experiment with some alternative ways to deal with the issues...

Sort of two major issues:

a) The current released code waits until the whole sketch is initialized and calls mtpd.loop(), before it services any requests from the host computer. If you wait too long, the host computer becomes very unhappy and mtp does not work...
b) Fat32 drives of any larger size take forever to compute their free/used space.. As MTP wants to know this information for each storage... Some of these drives are taking something like 10 seconds on SPI SD drives.

The code that I merged in does a couple of things

1) I added the ability to tell MTP to write to a different stream other than Serial.... I had some of the other files/classes respect this... Thought might want something like this anyway as maybe we want to write somewhere else... In test sketch I use this to initially use a quick an dirty memory stream to capture MTP setup stuff before USB Serial is initialized... Then set it to Serial and dump out what I captured early.

2) I have a new method: mtpd.begin() we added to remove some of the #if .. stuff and the like to now also create an interval timer that checks for MTP messages from the host and responds. There are only a few messages it gives valid data, others it tells the host that it is busy...

3) My wrapper class for SD has Callback when MTP asks for used space that if it is FAT32, it tries to bypass the main code. Right now it tries to read from the Information sector, which SD library does not update nor look at... Need to rework this somehow... Would be nice if we could get SD to semi-maintain this data...

Anyway have one test sketch using all of this: SD_Program_MTP-logger, which shows some of this stuff in action:

Code:

*** Start Interval Timer ***
2351 CMD: 1002(OPEN_SESSION)l: 16 T:0 : 1
2351 RESP:2001(RSP:OK)l: 16 T:0 : 1
2401 CMD: 1001(GET_DEVICE_INFO)l: 12 T:1
2401 RESP:2001(RSP:OK)l: 12 T:1
2451 CMD: 1014(GET_DEVICE_PROP_DESC)l: 16 T:2 : d402
2451 RESP:2001(RSP:OK)l: 16 T:2 : d402

D:\GitHub\MTP_t4\examples\SD_Program_MTP-logger\SD_Program_MTP-logger.ino Sep 16 2021 05:58:53
2470 Initializing MTP Storage list ...Date: 16/9/2021 5:59:1
sd_addFilesystem: 0 2000744c Program 20004788 2000744c
Trying to open SPI config: 8 9 0 16000000
Total Size: 66002944 Used Size: 22044672
addFSToStorage: Added FS
sd_addFilesystem: 1 20006f60 SPI8 2000741c 20006f60
Ram Drive of size: 65536 initialized
sd_addFilesystem: 2 20004508 RAM 200045d0 20004508
Set Storage Index drive to 2
2505 SD initialized.

Menu Options:
	1 - List Drives (Step 1)
	2 - Select Drive for Logging (Step 2)
	l - List files on disk
	e - Erase files on disk
	s - Start Logging data (Restarting logger will append records to existing log)
	x - Stop Logging data
	d - Dump Log
	r - Reset MTP
	h - Menu

*** end Interval Timer ***
2740 CMD: 1004(GET_STORAGE_IDS)l: 12 T:3
2740 RESP:2001(RSP:OK)l: 12 T:3
2741 CMD: 1005(GET_STORAGE_INFO)l: 16 T:4 : 10001
65537 0 name:Program
65537 0 name:Program
2741 RESP:2001(RSP:OK)l: 16 T:4 : 10001
2741 CMD: 1005(GET_STORAGE_INFO)l: 16 T:5 : 20001

The memory stream here heped me to see what the host was asking for before we had Serial up and running...
In this run, it was quick enough startup that I did not have to respond busy to get ids:

Code:

2740 CMD: 1004(GET_STORAGE_IDS)l: 12 T:3
2740 RESP:2001(RSP:OK)l: 12 T:3

Some of the runs I do...

But again maybe some of this should be done differently... More later

mjs513 · Sep 21, 2021

LittleFS Now Supports File Creation and Modification Dates/Times

As noted in Post #2 we used the custom attributes (lfs_setattr and lfs_getattr) to assign file create and modification dates. Dates are stored in TimeLib format using the attribute commands and converted to FS_DATE/FS_TIME for use with MTP.

An example sketch (LittleFS_Program_Simple_Datalogger-dates is provided to extract the dates and times if so desired without using MTP using SdFat functions such as FS_MONTH. Example:

Code:

      uint16_t c_date; uint16_t c_time;
      entry.getCreateDateTime(&c_date, &c_time);
      Serial.printf("    Created: %d-%02d-%02d %02d:%02d:%02d\n", FS_YEAR(c_date), FS_MONTH(c_date), FS_DAY(c_date), FS_HOUR(c_time), FS_MINUTE(c_time), FS_SECOND(c_time));
      entry.getModifyDateTime(&c_date, &c_time);
      Serial.printf("    LastWrite: %d-%02d-%02d %02d:%02d:%02d\n", FS_YEAR(c_date), FS_MONTH(c_date), FS_DAY(c_date), FS_HOUR(c_time), FS_MINUTE(c_time), FS_SECOND(c_time));

.

KurtE · Sep 21, 2021

mjs513 said:
LittleFS Now Supports File Creation and Modification Dates/Times

As noted in Post #2 we used the custom attributes (lfs_setattr and lfs_getattr) to assign file create and modification dates. Dates are stored in TimeLib format using the attribute commands and converted to FS_DATE/FS_TIME for use with MTP.

An example sketch (LittleFS_Program_Simple_Datalogger-dates is provided to extract the dates and times if so desired without using MTP using SdFat functions such as FS_MONTH. Example:

Code:

uint16_t c_date; uint16_t c_time; entry.getCreateDateTime(&c_date, &c_time); Serial.printf(" Created: %d-%02d-%02d %02d:%02d:%02d\n", FS_YEAR(c_date), FS_MONTH(c_date), FS_DAY(c_date), FS_HOUR(c_time), FS_MINUTE(c_time), FS_SECOND(c_time)); entry.getModifyDateTime(&c_date, &c_time); Serial.printf(" LastWrite: %d-%02d-%02d %02d:%02d:%02d\n", FS_YEAR(c_date), FS_MONTH(c_date), FS_DAY(c_date), FS_HOUR(c_time), FS_MINUTE(c_time), FS_SECOND(c_time));

.

This is great stuff!

Thought I would mention that we have experimented with it, both from sketches that are simply using LittleFS... We also tested it with MTP stuff as well.

So locally when you create a file with FILE_WRITE or FILE_WRITE_BEGIN, if file is created the created attribute is added and likewise each time you open it this way the modified attribute is updated.

In addition to this, when for example you copy a file there in MTP, the API is in place to allow MTP to specify the creation date and the modification date, which you see in the screen shot.

But again warning: This is all using our addition of Date/Time stuff to the File object within FS.h... These members are the same names as the ones currently contained within SDFat.
But they are all subject to change.

Code:

#ifdef FS_FILE_SUPPORT_DATES
	// These will all return false as only some FS support it.
  	virtual bool getAccessDateTime(uint16_t* pdate, uint16_t* ptime) {
  		return (f) ? f->getAccessDateTime(pdate, ptime) : false;
  	}
  	virtual bool getCreateDateTime(uint16_t* pdate, uint16_t* ptime) {
  		return (f) ? f->getCreateDateTime(pdate, ptime) : false;
  	}
  	virtual bool getModifyDateTime(uint16_t* pdate, uint16_t* ptime) {
  		return (f) ? f->getModifyDateTime(pdate, ptime) : false;
  	}
  	virtual bool timestamp(uint8_t flags, uint16_t year, uint8_t month, uint8_t day,
                 uint8_t hour, uint8_t minute, uint8_t second) {
  		return (f) ? f->timestamp(flags, year, month, day, hour, minute, second) : false;
  	}
#endif

Note the TimeStamp method has a field flags which we have not defined here. They are currently the same as the ones within SDFat:

Code:

	//static const uint8_t T_ACCESS = 1;
		/** set the file's creation date and time */
		static const uint8_t T_CREATE = 2;
		/** Set the file's write date and time */
		static const uint8_t T_WRITE = 4;

I could easily see this method being changed to instead be 3 different methods like:

Code:

virtual bool setAccessDateTime(uint16_t date, uint16_t time);

KurtE · Sep 23, 2021

@Paul @mjs513 and all:

setSyncProvider - Would be nice if programs that use File system Dates and Times just worked without having to know about the time/date glue in the Time library.

That is the dates and times will be screwed up if something does not call setSyncProvider()...

Note: Mentioned in post #2...

First I am wondering why we default in time.cpp:
we have: getExternalTime getTimePtr; // pointer to external sync function

Which I am sort of surprised that it is not explicitly set to nullptr... But get set in the uninitialized variable clear to zero....

But wondering why we don't just have a simple function in this function like:

time_t default_get_Time() {return Teensy3Clock.get();}

and initialize the getTimePtr to be this function?

That way not every sketch or library has to deal with what if it is not set?

Alternatively I see everyone of our FS classes in their constructor or the like, either always calling setSyncProvider, or maybe
doing an extern of the getTimePtr and if NULL setting their own...

luni · Sep 23, 2021

Just as a side note:
Does changing line 242 in time.cpp to

Code:

getExternalTime getTimePtr = []{return (time_t) Teensy3Clock.get();};  // pointer to external sync function, default syncs to the Teensy3Clock

work? (It sets the default value to your proposed function without having to explicitly define it)

KurtE · Sep 23, 2021

luni said:
Just as a side note:
Does changing line 242 in time.cpp to

Code:

getExternalTime getTimePtr = []{return (time_t) Teensy3Clock.get();}; // pointer to external sync function, default syncs to the Teensy3Clock

work? (It sets the default value to your proposed function without having to explicitly define it)

Good question, you are much more up on the newer c++ stuff... Most of my learning was from long ago...

But if works it would be great!

Teensyduino File System Integration, including MTP and MSC

Senior Member+

Senior Member+

Senior Member+

Well-known member

Senior Member+

Well-known member

Senior Member+

Senior Member+

Well-known member

Senior Member+

Well-known member

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Well-known member

Senior Member+

Well-known member

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Well-known member

Senior Member+