Many TLAs: MTP MSC FS SD SDFat LittleFS UsbMSCFat to work with each other 8)

[KurtE]

@mjs513 - pushed up changes to MTP and MSC to build with current stuff... have not tested yet

 

[mjs513]

We could do it that way and ditch the formatter function.

As for the items highlighted in red I changed them back to:

        PFsFatFormatter::format(partVol, fat_type, sectorBuffer, &Serialx);
        PFsExFatFormatter::format(partVol, sectorBuffer, &Serial);

in the last PR. Probably what its complaining about is the extra Serial in PFsFatFormatter::format.

Do we now have the ability to insert/append an ExFat partition?

That's up next. Have a few errands to run today.

 

[mjs513]

@mjs513 - pushed up changes to MTP and MSC to build with current stuff... have not tested yet

Ok will try it later.

EDIT ---- Which branch am I using?

EDIT2 === never mind test-msc-parts . Just gave it a try and it works. So up next is checking exFAT.

 

[mjs513]

@KurtE
Got it exFat changes almost working. Stuck here because my mind isn't working:

      m_capacityMB = (m_sectorCount + SECTORS_PER_MB - 1)/SECTORS_PER_MB;

ExFat don't know about sectors/MB. Tried m_sectorCount*bytes.sector/1000000 but not sure it worked or have other issue,

Have some more debugging to do before I send the files

 

[KurtE]

Hi @mjs513 the Sectors per MB is just a hard coded value is PFsFatFormatter.cpp

const uint16_t BYTES_PER_SECTOR = 512;
const uint16_t SECTORS_PER_MB = 0X100000/BYTES_PER_SECTOR;

 

[mjs513]

Ok so it is the same as for FAT - wasn't 100% sure. Had other copy paste issues as well.

Just pushed the update to you. Check create partition (at end at least) and format whole drive and seems to work. Delete partition works of course.

 

[KurtE]

Sounds good.

Will try to test some later today.... Maybe about time again to merge back into @wwaston master branch?

 

[mjs513]

Sounds good.

Will try to test some later today.... Maybe about time again to merge back into @wwaston master branch?

Yeah - think we made enough changes that its time to merge again. Still can't figure out whats going on with FAT and used space.

 

[KurtE]

Yeah - think we made enough changes that its time to merge again. Still can't figure out whats going on with FAT and used space.

PR Issued. https://github.com/wwatson4506/UsbMscFat/pull/5

Will see what other things we should fix.

 

[mjs513]

Cool at least we all will have a common baseline.

You mean lets see what else I can break

 

[wwatson]

@Kurte @ mjs513 - Did the PR. Have not found any problems yet so far.

I have gone down with another back injury and have not had time to play with this much with the medication I am taking. Hopefully this will change next week and I will be able test more but for now here is the the latest toy I have been playing with.

lwext4:

Teensy EXT4 file system testing

parent_blockdev = 536872984
ext4_mbr
mscInit()
<<< msReadBlocks(0 1 200)
ext4_mbr_scan:
mbr_entry 0:
 offeset: 0x100000, 1MB
 size:    0x3b9b00000, 15259MB
mbr_entry 1:
 offeset: 0x3b9c00000, 15260MB
 size:    0x37aa00000, 14250MB
mbr_entry 2:
	empty/unknown
mbr_entry 3:
	empty/unknown
mbr_scan() Passed

I have finally got my modified version of lwext4 talking to a thumb drive through MSC. It is using a small wrapper to access needed info and basic functions from MSC. The only tests so far are initialization and reading the MBR and partitions on the USB thumb drive formatted as ext4 (two partitions). There are several test functions from lwext4 that I am going to add to test with. It is a start

Right now it is working with one hard coded MSC drive (msDrive1), not multiple drives. This is going to take some time. Trying still to decide to convert to C++ or just do wrapper functions

Once I have the lwext4 tests working I will set it up on GitHub.

 

[wwatson]

@All - Fixed the 'CopyFilesUSB.ino' sketch. It now works with the latest version of UsbMscFat. Updated on GitHub.

 

[mjs513]

@wwatson
Sorry to hear about your back - sounds bad. Have backproblems too but not as bad as yours it sounds like.

Sounds like you are making progress with lwext4 - but sounds challenging

 

[wwatson]

Thanks Mike - Thank goodness there are Laptops At least I can still work on ext4 while I'm down. The biggest hurdle right now is being able to debug the C code. I'm working on that right now. There is a lot more to it than EXFAT or FAT32. Doing a lot of research as I go.

Will post when I have the basic tests working.

 

[KurtE]

Afternoon/evening all - Still have not made it to town...

So playing with MTP support... What do you think?



So far I have only setup to pass through the Modify Date, can also do create date.
Not too bad: I have only had to hit: (cores)FS.h, LittleFS, SD, MSC, and MTP... The LittleFS files are showing up with some bogus date stuff, will need to figure out how best to handle ones that don't actually support dates... Actually maybe it does, but I have not looked.

Back to playing

 

[mjs513]

Take a read through this - there is a link for an example: https://github.com/littlefs-project/littlefs/issues/23

Nice job on getting timestamps working - see what happens when I don't bother you with problems.

 

[wwatson]

@All - Hit the brick wall Working with lwext4 I am to a point where I can initialize, get stats about a USB drive and mount it. But that's it!! Any other kind of access fails to the point of wiping out the formatting on the drive. So far just working with USB thumb drives. And maybe this is the warning to look at:

Lwext4 is an excellent choice for SD/MMC card, USB flash drive or any other wear
leveled memory types. However it is not good for raw flash devices.

Probably missing something. Going to try with USB to Sata drives to see what happens. Have tried with several different thumb drives and I get varying results. Makes me think it is a problem with the mix of C and C++ and my great programming skills.
Plan for the worst, hope for the best I guess. Got plenty of time to play right now

 

[KurtE]

@wwatson - First I hope you back is getting better. And your lwext4 sounds interesting. I have not had as much interest in this FS mainly because it is not used on Windows.... Although I do stuff on some Linux boxes like Linux... So would be be nice be able to read the disks for example that we plug into an RPI... At some point, you might put up what you have and some of us may play along, if you think that might help some.

@mjs513 - That link looks interesting. Have you tried the stuff in it to see, how hard it might be to have a modify date? Also is his stuff already in LittleFS, I know that the issue you linked to was linked to by another issue/PR which looks like parts of stuff were merged.

Actually I think we get a lot more done, when were up bouncing ideas and code off of each other! Most of the updates to get the stuff to show dates is pretty straight forward. The real issue is:
a) (@Paul) Can we add the Date Time methods to the File object in FS.h?

That is I added:

#define FS_FILE_SUPPORT_DATES 1
... (File class)
#ifdef FS_FILE_SUPPORT_DATES
	// These will all return false as only some FS support it.
  	virtual bool getAccessDateTime(uint16_t* pdate, uint16_t* ptime) {
  		return (f) ? f->getAccessDateTime(pdate, ptime) : false;
  	}
  	virtual bool getCreateDateTime(uint16_t* pdate, uint16_t* ptime) {
  		return (f) ? f->getCreateDateTime(pdate, ptime) : false;
  	}
  	virtual bool getModifyDateTime(uint16_t* pdate, uint16_t* ptime){
  		return (f) ? f->getModifyDateTime(pdate, ptime) : false;
  	}
  	virtual bool timestamp(uint8_t flags, uint16_t year, uint8_t month, uint8_t day,
                 uint8_t hour, uint8_t minute, uint8_t second){return false;}
#endif

Likewise in SD.h

#ifdef FS_FILE_SUPPORT_DATES
	// These will all return false as only some FS support it.
  	virtual bool getAccessDateTime(uint16_t* pdate, uint16_t* ptime) {
  		return sdfatfile.getAccessDateTime(pdate, ptime);
  	}
  	virtual bool getCreateDateTime(uint16_t* pdate, uint16_t* ptime) {
  		return sdfatfile.getCreateDateTime(pdate, ptime);
  	}
  	virtual bool getModifyDateTime(uint16_t* pdate, uint16_t* ptime) {
  		return sdfatfile.getModifyDateTime(pdate, ptime);
  	}
  	virtual bool timestamp(uint8_t flags, uint16_t year, uint8_t month, uint8_t day,
                 uint8_t hour, uint8_t minute, uint8_t second) {
  		return sdfatfile.timestamp(flags, year, month, day, hour, minute, second);
  	}
#endif

So with that, you can now get the Dates and times Off of SD drives.
And in UsbMscFat...\msc.FS.h similar:

#ifdef FS_FILE_SUPPORT_DATES
	// These will all return false as only some FS support it.
	virtual bool getAccessDateTime(uint16_t* pdate, uint16_t* ptime) {
		return mscfatfile.getAccessDateTime(pdate, ptime);
	}
	virtual bool getCreateDateTime(uint16_t* pdate, uint16_t* ptime) {
		return mscfatfile.getCreateDateTime(pdate, ptime);
	}
	virtual bool getModifyDateTime(uint16_t* pdate, uint16_t* ptime) {
		return mscfatfile.getModifyDateTime(pdate, ptime);
	}
	virtual bool timestamp(uint8_t flags, uint16_t year, uint8_t month, uint8_t day,
               uint8_t hour, uint8_t minute, uint8_t second) {
		//return mscfatfile.timestamp(year, month, day, hour, minute, second);
		return false;
	}
#endif

To support the dates and time in the MSC based files.

Currently in LittleFs I added

#ifdef FS_FILE_SUPPORT_DATES
	// These will all return false as only some FS support it.
  	virtual bool getAccessDateTime(uint16_t* pdate, uint16_t* ptime){
  		*pdate = 0; *ptime = 0; return false;
  	}
  	virtual bool getCreateDateTime(uint16_t* pdate, uint16_t* ptime){
  		*pdate = 0; *ptime = 0; return false;
  	}
  	virtual bool getModifyDateTime(uint16_t* pdate, uint16_t* ptime){
  		*pdate = 0; *ptime = 0; return false;
  	}
  	virtual bool timestamp(uint8_t flags, uint16_t year, uint8_t month, uint8_t day,
                 uint8_t hour, uint8_t minute, uint8_t second){return false;}
#endif

So they always fail...

And then I needed to add the glue in MTP to store the dates and times, plus call off to get them, plus enable the Object Attributes....

Now for more playing. May try to look at littleFS for those dates and like, unless someone beats me to it

 

[mjs513]

@mjs513 - That link looks interesting. Have you tried the stuff in it to see, how hard it might be to have a modify date? Also is his stuff already in LittleFS, I know that the issue you linked to was linked to by another issue/PR which looks like parts of stuff were merged.

Only found that yesterday after you posted about date-time in LFS. I did just look to see if the custom attribute stuff was merged and it looks like it was but haven't tried it yet. Been distracted with the MicroMod board.

Think you might beat me to it this time - still playing with MM stuff.

PS> Right now they are working on my roof and sound like thunder so even more distractions.

 

[KurtE]

@mjs513 - I think I will do a couple of experiments and see what happens. What is the worst that can happen, the chip melt
But again I have not done much with LittleFs on the inside, mostly bothered you with the problems I ran into

The thoughts are: It looks like files can have attributes.
In LittleFS.h We set: config.attr_max = 0;

Which I believe may not mean what others are expecting, I believe it means: it is limited to the value of: #define LFS_ATTR_MAX 1022
So more than enough.

Then I maybe try some functions like:

// Returns the size of the attribute, or a negative error code on failure.
// Note, the returned size is the size of the attribute on disk, irrespective
// of the size of the buffer. This can be used to dynamically allocate a buffer
// or check for existance.
lflfs_ssize_t lfs_getattr(lfs_t *lfs, const char *path,
        uint8_t type, void *buffer, lfs_size_t size);

// Set custom attributes
//
// Custom attributes are uniquely identified by an 8-bit type and limited
// to LFS_ATTR_MAX bytes. If an attribute is not found, it will be
// implicitly created.
//
// Returns a negative error code on failure.
int lfs_setattr(lfs_t *lfs, const char *path,
        uint8_t type, const void *buffer, lfs_size_t size);

Then maybe try to define a new attribute like
#define LITTLEFS_ATTR_MODDATE 42
So use that as the type field, and try to store 4 bytes for DATE/TIME in same format as SD stuff...

Then for now maybe change the close() method in littleFS to call off to get the date/time and set this value
before calling the lfs_file_close, which I believe the close method on files open for write will then store out the attributes...

Plus change my do nothing getModifyDateTime to call to get this attribute...

Plus I need to look more in MTP that for example if you copy a file into the MTP directory, how to hopefully preserve the modify date of the actual file...

 

[mjs513]

@KurtE
Never a bother - its more fun and efficient when we all work together.

Any scroll down to the bottom of the link I posted. Think it has what you may need to start.

EDIT: Still cleaning off the old paper that I think had several layers of junk up there from before me and still banging away to get it off.

 

[KurtE]

@mjs513 - So far have not started that part on LittleFS yet.

Now that I have dates and Times showing, it is obvious we are not preserving any of the dates/times when we for example copy a picture from your PC to a drive hosted by MTP.

The Send Object code (SendObjectInfo has an objectInfo structure that we use to know things like the file name, the size...
The date sent does include the create and modify date info in it, but we throw it away...

So Just added code to retrieve that data from the information given us. So far just showing in debug text:

DATA:100c(SEND_OBJECT_INFO)l: 168 T:31 : 0 3000 5222 3000 0
SendObjectInfo: 131073 4294967295 20005220: 0 3000 0 5222 3000 0 0 0 d8 f2 18 0 0 0 0 : halloween.jpg
Read DateTime string: 20210402T102729.0
>> date: 5282 2021 4 2
>> time: 536e 10 27 29
Created: 5282 536e
Read DateTime string: 20041101T215234.0
>> date: 3161 2004 11 1
>> time: ae91 21 52 34
Modified: 3161 ae91

So now I need to modify/add storage code to allow me to pass these value in to the storage code, where will hold in node, and when send complete probably need to look at those and tell the FS to set the dates. At least the modify date. Create maybe correct for current date/time

 

[wwatson]

@KurtE - Thanks. I have been working with lwext4 and found that the major part of my problems were with the code being riddled with straight printf's. In the config file there were options to turn off debugging and using printf. After I disabled those the code became repeatable. There is still a lot of things that are not working but I can mount and get the stats. Tomorrow I will put it up on GitHub so you guy's can can look at it and try out what I have done. I have all the links to various versions of lwext4 that I am using for reference. I will add those to GitHub. I know everybody is busy with MM. Will continue to play with this Much more to be done and learn about.

 

[KurtE]

@mjs513 @PaulStoffregen @WMXZ @wwatson ...

LittleFS - I have not started yet, maybe today on trying to add the Modify date...

I had another distraction on this (and external) and now that I show dates and time for Modify and Create, It is then obvious we are NOT capturing and maintaining the dates, when someone does something like Copy a file on the PC to a Storage under MTP...

So I put the work in to support it... Or at least some of it. Still probably WIP.

Example I copied my T4.1 extended card jpg to storage on the Teensy:


As you maybe can see the Modify date is preserved, and Windows told me the Create date which was used which is today...

The SendObjectInfo has the dates as part of it, which we were not using, I now extract them,

CMD: 100c(SEND_OBJECT_INFO)l: 20 T:190 : 20001 ffffffff
DATA:100c(SEND_OBJECT_INFO)l: 176 T:190 : 0 3000 60109 3000 0
SendObjectInfo: 131073 4294967295 20005220: 0 3000 0 60109 3000 0 0 0 568 35c 18 0 0 0 0 : T4.1-Cardlike.jpg
Read DateTime string: [COLOR="#FF0000"]20210403T062400.0[/COLOR]
>> date: 5283 2021 4 3
>> time: 3300 6 24 0
Created: 5283 3300
Read DateTime string: [COLOR="#FF0000"]20210124T062334.0[/COLOR]
>> date: 5238 2021 1 24
>> time: 32f1 6 23 34
Modified: 5238 32f1
RESP:2001(RSP:OK)l: 24 T:190 : 20001 ffffffff 2b
CMD: 100d(SEND_OBJECT)l: 12 T:191
MTPD::SendObject: len:393481
 # USB Packets: 768 total: 16 avg ms: 0 max: 1
 # Write: 49 total:208 avg ms: 4 max: 65
>>>Total Time: 234403

When the send object completes, I then update the Storage Record with these dates (Note: I added these fields to records). And I called through the SDFat code newer method:
timestamp which allows me to set these.

Although doing a full refresh does not look like these dates are preserved. So still debugging. Probably something simple like I did not open the file for write or the like.

But question still is, do we want the level of functionality?
Again I am using added methods on the File object in FS to get and set dates...

Edit: Fixed the date/times not being saved

 

[KurtE]

Now if anyone would like to try out my current WIP code, I have pushed up all of the changes to the appropriate libraries with all of them using a new branch: FS_DATES

This includes: cores (FS.h), SD (note my fork created SD-1), LittleFS, UsbMscFat, MTP_t4

@Paul if this looks like something we can try getting into next beta: maybe should make slight mods to FS.h...
The current changes are:

add: #define FS_FILE_SUPPORT_DATES 1

and add:

#ifdef FS_FILE_SUPPORT_DATES
	// These will all return false as only some FS support it.
  	virtual bool getAccessDateTime(uint16_t* pdate, uint16_t* ptime) {
  		return (f) ? f->getAccessDateTime(pdate, ptime) : false;
  	}
  	virtual bool getCreateDateTime(uint16_t* pdate, uint16_t* ptime) {
  		return (f) ? f->getCreateDateTime(pdate, ptime) : false;
  	}
  	virtual bool getModifyDateTime(uint16_t* pdate, uint16_t* ptime) {
  		return (f) ? f->getModifyDateTime(pdate, ptime) : false;
  	}
  	virtual bool timestamp(uint8_t flags, uint16_t year, uint8_t month, uint8_t day,
                 uint8_t hour, uint8_t minute, uint8_t second) {
  		return (f) ? f->timestamp(flags, year, month, day, hour, minute, second) : false;
  	}
#endif

Again these are the date/Time functions added to SdFat that are in the current beta

What is not here is the flags for timestamp which are in SDFAT as:
T_WRITE, T_CREATE, and I think T_MODIFY.

Could/should maybe instead add something like:
enum TimeStamp {
...

Thoughts