Many TLAs: MTP MSC FS SD SDFat LittleFS UsbMSCFat to work with each other 8)

Just a follow to what @KurtE mentioned on naming conventions. No real preference but thought maybe using the following would be more intuitive:
Code: 
USBHost myusb;
USBHub hub1(myusb);
USBHub hub2(myusb);

USBDrive myDisk1(myusb);
USBDrive myDisk2(myusb);

USBFilesystem myFilesystem1(myusb);
USBFilesystem myFilesystem2(myusb);
USBFilesystem myFilesystem3(myusb);
USBFilesystem myFilesystem4(myusb);
USBFilesystem myFilesystem5(myusb);

USBDriveController works for me as well if that is more consistent with the way other classes work.

Do agree with Kurt when he mentioned:
Not sure about having myFilesystem1 saying which drive it is on, as I may want to be able to handle...
Click to expand...
I have had multiple USB drives with multiple FS's on each and could be an issue. SD cards on USB - had bad experiences with multiple partitions on SD Cards but they do work.
 
Anyone else have feelings about the class names? Is USBDrive & USBFilesystem clear enough? (really, I'm too close to the code right now to be objective on this....)
 
Works for me... I was torn between USBFilesystem or simply USBFS or to be like SDFat have it be USBFat...

But no real strong feelings so if those two work, go for it!
 
I'm wondering if we really need the UsbFs class. It's pretty much just added a layer that gives a bunch of error message printing. Maybe just using an instance of FsVolume would be enough?
 
PaulStoffregen said:
I'm wondering if we really need the UsbFs class. It's pretty much just added a layer that gives a bunch of error message printing. Maybe just using an instance of FsVolume would be enough?
Click to expand...

Neutral on naming here - as long as there are examples exposing the needed interface.

Mike noted SD issue with partitions(volumes?). Seeing 'FsVolume' that looks like it might cover those as well? So taking out yet another layer of USB something seems cleaner.
 
defragster said:
Neutral on naming here - as long as there are examples exposing the needed interface.

Mike noted SD issue with partitions(volumes?). Seeing 'FsVolume' that looks like it might cover those as well? So taking out yet another layer of USB something seems cleaner.
Click to expand...

Not really SD but sd cards at least some of them - but they do work - probably not good for smaller cards.
 
defragster said:
So taking out yet another layer of USB something seems cleaner.
Click to expand...

Indeed, I've been ruthlessly removing layers and deleting code. It's now down to just 3 classes (device, file, filesystem) and only a few hundred lines, not including the long list of sense code names.

https://github.com/PaulStoffregen/UsbMscFat/tree/UsbMscFat/src

However, a couple hundred lines of volume label stuff from PFsLib went into SdFat's FsLib, and that's building on top of Kurt's recent work which brought partition support and other stuff into FsLib.

I also left out a lot of code for formatting filesystems. Will work on restoring those features (probably with different API) after the merge to USBHost_t36.
 
PaulStoffregen said:
I'm wondering if we really need the UsbFs class. It's pretty much just added a layer that gives a bunch of error message printing. Maybe just using an instance of FsVolume would be enough?
Click to expand...

I am not sure, it has been awhile, and I know some of it may have been to deal with other abstractions. FsVolume may be sufficient, would need to verify things like it handles both FatVolume and ExFatVolume.

Also need to make sure it knows the difference between USB and SD assuming that behavioral differences are kept.

Again with things like Format.... On SD, you format the whole card into one volume (Like the PC app SDFormatter), where on most every other type of storage, you want format to only format the actual partition...

But again it may just have been glue to work with what was the current abstractions.
 
I just looked at FsVolume and it looks like it does address both, for example

Code: 
  uint32_t fatStartSector() const {
    return m_fVol ? m_fVol->fatStartSector() :
           m_xVol ? m_xVol->fatStartSector() : 0;

Same way with did it in pfsVolume but if I remember right we also had to add a few additional functions to get what we needed to address extFatFormat and FatFormatter code.
 
@All - As far as the sense codes go and that big list of names I have only seen one instance where I used it to discover that a PNY 64Gig thumb drive had magically write protected itself. It was the only time a sense code has ever been returned on any of the USB drives I have worked with. This was common failure with this thumb drive. I am not so sure there is a good reason to keep that in. Unless you guy's think it necessary? Most common failures of MSC devices are formatting and init timing that I have seen. It would get rid of some code bloat.
 
UsbMscFat has finally been merged into USBHost_t36.

https://github.com/PaulStoffregen/USBHost_t36/commit/ec185ec5c8226269a57e0678c0bb685fe8cb5b9a

This stuff still remains to be done....

1: Classes still named msController and MSCClass

2: Still using begin() to start up

3: Formatting stuff missing, need to consider better API...

4: Only ListFiles example brought over so far

5: FsBlockDeviceInterface readSectorsCallback() & writeSectorsCallback() needed for performance

6: Need to check attribution & MIT license headers
 
Good Morning Paul,

I synced up, but am waiting until you give the go ahead and try things out... still distracted with some other things.
 
KurtE said:
Good Morning Paul,

I synced up, but am waiting until you give the go ahead and try things out... still distracted with some other things.
Click to expand...

Quick update, I did a quick and dirty check to see how bad the sketch I was playing with, that allowed me to plug in a CircuitPython board and connect up through the MSC stuff (Fat12).

There are the obvious things to change, like class names and header files...

Ran into not having the PFsLib object which we were using to dump the mbr: (pfsLIB.mbrDmp(&mscDrive, (uint32_t) - 1, Serial);

Looks like the class: MSCClass
Does not have a bool operator on it...


Now will try to look to see if the PFSLib stuff for Fat12 made it over...
 
KurtE said:
Quick update, I did a quick and dirty check to see how bad the sketch I was playing with, that allowed me to plug in a CircuitPython board and connect up through the MSC stuff (Fat12).

There are the obvious things to change, like class names and header files...

Ran into not having the PFsLib object which we were using to dump the mbr: (pfsLIB.mbrDmp(&mscDrive, (uint32_t) - 1, Serial);

Looks like the class: MSCClass
Does not have a bool operator on it...


Now will try to look to see if the PFSLib stuff for Fat12 made it over...
Click to expand...

Pretty much found the same things here when I was trying to update the listfiles sketch to do a chkmscdrive in post https://forum.pjrc.com/threads/6633...-each-other-8)?p=305168&viewfull=1#post305168. Which needs things like pfsLIB.mbrDmp, and pfsLIB.listpartitions.
 
Also might be good if there was an example showing where you plug in a drive which has multiple volumes, how that is handled in the new object classes.

That is where in SDFat probably that walks the MBR and/or GPT tables and initializes the N different Volume/FileSystem objects. Note: I can also see that at some point where not all of these volumes have to be Fat based...

Before I think it was the USBMSCDevice class that did this work.
 
I'll merge stuff into FsLib soon. Would be faster if you send a pull request, but probably best if I do it so I'm familiar with the SdFat changes.

On the printing functions, I'd rather put them into USBHost_t36 to minimize the amount of change in SdFat. Likewise for some of the fairly stand-alone formatting stuff. Eventually we'll probably need to include some or all of Bill's future SdFat changes, which becomes more difficult the farther Teensy's SdFat changes drift from Bill's design. Especially creating any new classes within FsLib is a diverging path I'd like to avoid.

For anyone following this conversation with curiosity about what we've changed in Teensy's fork of SdFat, I've been trying to keep the README file updated with a high-level summary of all our modifications. If there's anything I've forgotten to put on that list, please remind me or send a pull request to add it.
 
KurtE said:
Paul - The PFSLib stuff added to support Fat12 was not migrated into FatLib
Click to expand...

It seems to work with FAT12.

I added a DriveInfo example, adapted from UsbMscFat MSCDriveInfo. Here's what it prints when I connect a Teensy 4.0 with Circuit Python, which I previously plugged into my Linux desktop to copy a PDF onto the drive so it would have something recognizable besides just the Circuit Python default.

Code: 
Initializing USB drive 1...
Device 1 Info:
       connected: 1
     initialized: 1
   USB Vendor ID: 239a
  USB Product ID: 8086
      HUB Number: 0
        HUB Port: 0
  Device Address: 1
Removable Device: YES
        VendorID: PJRC    
       ProductID: Teensy 4.0      
      RevisionID: 1.0 
         Version: 2
    Sector Count: 2040
     Sector size: 512
   Disk Capacity: 1044480 Bytes

Files:
2098-01-01 00:00          0 .fseventsd/
  2098-01-01 00:00          0 no_log
2098-01-01 00:00          0 .metadata_never_index
2098-01-01 00:00          0 .Trashes
2098-01-01 00:00         22 code.py
2098-01-01 00:00          0 lib/
2098-01-01 00:00         95 boot_out.txt
2022-05-04 08:48     284152 max11615.pdf

Volume name: CIRCUITPY
Volume type: FAT12
Cluster Size: 512 bytes
Volume size: 1024512 bytes
 Space used: 286208 bytes


Initializing USB drive 2...
initialization failed with code: No USB drive detected, plugged in?
USB drive error: 0x28,0x0 --> Type: NO_SENSE Cause: NO ADDITIONAL SENSE INFORMATION
 
Looking at this more closely, it seems PFsLib/PFsVolume.cpp has an optimized version of freeClusterCount().

However, we also have basically that same optimization to the original freeClusterCount() in FatLib/FatPartition.cpp. But there are a couple issues.

1: The read function name is different, readSectorsCallback() versus readSectorsWithCB(). msController currently isn't implementing readSectorsCallback().

2: Only FAT16 & FAT32 are currently using readSectorsCallback(). FAT12 is done with Bill's original code which reads 1 sector at a time.

The net result seems to be everything working, though performance will not be as good.

We also have a writeSectorsCallback() which was created to speed up FAT32 formatting (mostly motivated by shared SPI). It's also not yet implemented by msController.
 
Good Morning All

I just updated to the latest change - nice add for print partition table. So decided to give it a go with 2 USB drives - one formatted as a MBR drive with 2 exFAT partitions, the second is a GPT formatted drive with a FAT32+exFAT partition. Yes I know things are still changing. Seems like for the normal MBR it is working but for the GPR is having a issue. But when I looked at the new printPartionTable looks like its missing that piece:
Code: 
Initializing USB drive 1...
Device 1 Info:
       connected: 1
     initialized: 1
   USB Vendor ID: 13fe
  USB Product ID: 6300
      HUB Number: 1
        HUB Port: 3
  Device Address: 2
Removable Device: YES
        VendorID:         
       ProductID: USB DISK 3.0    
      RevisionID: PMAP
         Version: 6
    Sector Count: 242075647
     Sector size: 512
   Disk Capacity: 123942731264 Bytes

Partition Table
	part,boot,bgnCHS[3],type,endCHS[3],start,length
exFAT:	1,0,0x20,0x21,0x0,0x7,0xFE,0xFF,0xFF,8064,121074785
	 < unused area starting at: 121082849 length 7 >
exFAT:	2,0,0x20,0x21,0x0,0x7,0xFE,0xFF,0xFF,121082856,120991737
pt_#0:	3,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
pt_#0:	4,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
	 < unused area starting at: 242074593 length 1054 >

Volume name: exFAT0
Volume type: FAT64
Cluster Size: 131072 bytes
Volume size: 61987094528 bytes
 Space used: 70778880 bytes

Files:
2022-04-27 08:07       135897 WM2204270371_invoice - LCSC.COM.pdf
2022-04-27 18:28        84305 613e-cy8JgL._AC_SL1000_.jpg
2022-04-27 18:29        33878 51ZIZAVX8ZL._AC_.jpg


Initializing USB drive 2...
Device 2 Info:
       connected: 1
     initialized: 1
   USB Vendor ID: 13fe
  USB Product ID: 6300
      HUB Number: 1
        HUB Port: 4
  Device Address: 3
Removable Device: YES
        VendorID:         
       ProductID: USB DISK 3.0    
      RevisionID: PMAP
         Version: 6
    Sector Count: 242075647
     Sector size: 512
   Disk Capacity: 123942731264 Bytes

Partition Table
	part,boot,bgnCHS[3],type,endCHS[3],start,length
pt_#238:	1,0,0x0,0x2,0x0,0xEE,0xFE,0xBF,0xDB,1,4294967295
pt_#0:	2,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
pt_#0:	3,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
pt_#0:	4,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
	 < unused area starting at: 0 length 242075647 >

Volume name: 
Volume type: FAT32
Cluster Size: 32768 bytes
Volume size: 30977032192 bytes
 Space used: 2326528 bytes

Files:

EDIT: Also just noticed FAT12 is missing as well:
Code: 
    case 1:
      Serialx.print(F("FAT12:\t"));
      break;
 
Last edited:
Good morning all,

Looks like making some progress :D

Of course I would plug in a complicated drive ;)
I have a couple of external SSDs that I have been using for experimenting, like:
https://smile.amazon.com/gp/product/B01N5IB20Q
Connected with something like:
https://smile.amazon.com/gp/product/B07S9CKV7X

The one I plugged in I think is earlier ones I purchased, like only 120GB...
Code: 
Initializing USB drive 1...
initialization failed with code: No USB drive detected, plugged in?
USB drive error: 0x28,0x0 --> Type: NO_SENSE Cause: NO ADDITIONAL SENSE INFORMATION

Initializing USB drive 1...
Device 1 Info:
       connected: 1
     initialized: 1
   USB Vendor ID: 174c
  USB Product ID: 55aa
      HUB Number: 0
        HUB Port: 0
  Device Address: 1
Removable Device: NO
        VendorID: asmedia 
       ProductID: ASMT1051        
      RevisionID: 0   
         Version: 6
    Sector Count: 234441647
     Sector size: 512
   Disk Capacity: 120034123264 Bytes

Partition Table
	part,boot,bgnCHS[3],type,endCHS[3],start,length
FAT32:	1,0,0x20,0x21,0x0,0xC,0xFE,0xFF,0xFF,2048,32768000
exFAT:	2,0,0x20,0x21,0x0,0x7,0xFE,0xFF,0xFF,32770048,65536000
Extend:	3,0,0xFE,0xFF,0xFF,0xF,0xFE,0xFF,0xFF,98306048,136132608
pt_#0:	4,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
	 < unused area starting at: 234438656 length 2991 >

Volume name: PRI_FAT32
Volume type: FAT32
Cluster Size: 32768 bytes
Volume size: 16773021696 bytes
 Space used: 316080128 bytes

Files:
2021-12-04 07:08     489461 zarathustra.mp3
2021-12-03 08:24    2014737 Away_in_a_Manger.mp3
2021-12-03 08:24    3944449 Dont_Rain_on_My_Parade.mp3
2021-12-03 09:18    2985790 Welcome_Christmas.mp3
2021-12-04 07:09     553004 odd1.wav
2021-12-03 09:29    7669613 256K.OPU
2021-12-04 07:09    3177823 Candyman.aac


Initializing USB drive 2...
initialization failed with code: No USB drive detected, plugged in?
USB drive error: 0x28,0x0 --> Type: NO_SENSE Cause: NO ADDITIONAL SENSE INFORMATION

Sort of a more complex setup with two primary partitions, and an extended partition, which has two more partitions on it...
screenshot.jpg
This is one of the disks I used to debug the MBR and extended... I think one of the others is setup with GPT...

Quick update: I plugged in the one configured for GPT instead of MBR...

Code: 
Initializing USB drive 1...
initialization failed with code: Check USB drive format.
 
Last edited:
mjs513 said:
@PaulStoffregen - @KurtE - @wwatson.

Just noticed something that may be important.

The main branch for USBMscFat for the MBRdmp (https://github.com/wwatson4506/UsbM...7693a9501646bb5f0/src/PFsLib/PFsLib.cpp#L265_ ) function has not been updated to reflect the changes that we have made in the decouple branch (https://github.com/wwatson4506/UsbM...b39e4e03e733bcd530/src/PFsLib/PFsLib.cpp#L240 )
Click to expand...

Paul - I am not sure if you ever looked at the branches decouple or the one that is based off of it decouple_new_sdfat?
It looked like you created your own new branch: with_PFslib that you worked in for a bit and then you worked in the main UsbMscFat branch and
then I think you split from there and some parts to SDFat and others to USBHost_t36...

So question will be if you did not start from our more updated branches, we may need to go through source differences as there may be lots of things we fixed earlier.

While the decouple_new_sdfat branch is only three ahead of what was our UsbMscFat branch:

It's code was based off of the USBHost_T36 branch FS_Integration_MSC which had lots of changes in it.

So again not clear what the current code base is based on and if we need to go back through all of those changes to find things we fixed..
 
You must log in or register to reply here.
Back
Top