Many TLAs: MTP MSC FS SD SDFat LittleFS UsbMSCFat to work with each other 8)

[mjs513]

I have seen the changing volume label do that. What I was going to try next is to simply put in a delay after I do set volume label and see if that helps.

Mike I will take a look at the format stuff hopefully in a little bit. Right now still figuring out coordinates for drawing centered text... Also OBD2 stuff...

Thanks Kurt.

What are you trying to do with OBD2 = did you make you own or did you get an off the shelf one?

 

[KurtE]

Sorry I know that I was off topic,

What are you trying to do with OBD2 = did you make you own or did you get an off the shelf one?

Off shelf from Amazon - figuring out Check engine on Truck... Does not like batteries.

 

[wwatson]

I have seen the changing volume label do that. What I was going to try next is to simply put in a delay after I do set volume label and see if that helps.

Mike I will take a look at the format stuff hopefully in a little bit. Right now still figuring out coordinates for drawing centered text... Also OBD2 stuff...

@KurtE Tried that to all the way up to delay(1000). Did not help. Ran into the same issue with the copyFilesUSB.ino sketch a while back when checking for SDIO card being available. When I would hot plug a SD card it would take time for it to become available.

 

[wwatson]

@All - Well, working with ExFat formatting is really an eye opener. Started of with @mjs513's version and quickly messed that up big time. Decided to actually learn about the layout of ExFat then start playing with formatting. It seems to be all about the offsets to the different parts of ExFat for each partition. Ran out of time this weekend

 

[mjs513]

@All - Well, working with ExFat formatting is really an eye opener. Started of with @mjs513's version and quickly messed that up big time. Decided to actually learn about the layout of ExFat then start playing with formatting. It seems to be all about the offsets to the different parts of ExFat for each partition. Ran out of time this weekend

Yep = went through pretty much the same steps you did. The references in post 146 are pretty good and show the layout and offsets. I actually downloaded WinHex so I could examine the MBR and the layout. Pretty much the key is what I was having the major problems with: m_relativeSectors, which relates to partitionOffset. Once you have that set up right think everything else will fall into place.

Here are a couple of screen shots for a 2 partition drive:



The offset is in bytes so for the first partition it starts at:
1048576 / 512 (Bytes/sector) = 2048 sectors

And for reference 2048 happens to be m_relativesectors from the MBR table which I can not get for some reason. As I mentioned it hardfaults every time.

 

[wwatson]

@mjs513 - That is exactly what I getting when formatting a device in Linux. offset = 2048. Not sure yet where 8192 meg offset comes from when using the SdFat ExFatFormatter.

I started my own sketch and external ExFat formatting library to test with and was able to get it to create the partitions using manual partitionOffsets, sectorCount and bool addPart parameters. The partitions were recognized correctly but the actual file system was not. Tried to make it as generic as I could for now.

 

[KurtE]

@wwatson @mjs513 - Pushed up change to m UsbMscFat VolumeName_SD branch

That I think fixes the changing volume label then trying to reread the device fails...

After my write of the volume label I then told the device to do a syncDevice() operation which so far has made things happy...

Wondering if some of your formatting stuff may be hit by that as well?

 

[KurtE]

Morning (at least for me) all:

Any good ExFat to look at/ play with?

For now I might play some more more with Free cluster stuff and MTP. May end up doing more of the bypass the SDFat code for some of this to see if that helps. Still wish we could get some simple changes into the main line code to make it easier.

 

[mjs513]

Morning (at least for me) all:

Any good ExFat to look at/ play with?

For now I might play some more more with Free cluster stuff and MTP. May end up doing more of the bypass the SDFat code for some of this to see if that helps. Still wish we could get some simple changes into the main line code to make it easier.

Morning all

@KurtE - right now neither @wwatson seem to be having much luck. For me its trying to read the MBR in PFsExFormatter like I do when I am using PFsFatFormatter. I think I explained in post #147. Basically I am getting stuck with this code:

m_dev->readSector(0, (uint8_t*)&mbr

It causes a hard fault every time - not sure why.

 

[mjs513]

@mjs513 - That is exactly what I getting when formatting a device in Linux. offset = 2048. Not sure yet where 8192 meg offset comes from when using the SdFat ExFatFormatter.

I started my own sketch and external ExFat formatting library to test with and was able to get it to create the partitions using manual partitionOffsets, sectorCount and bool addPart parameters. The partitions were recognized correctly but the actual file system was not. Tried to make it as generic as I could for now.

8192 is the clusterheapstartsector. It comes from 2 places ExFatVolume.h and it gets calculated in ExFatFormatter as well. When I did the dump:

From either partition or from ExFatVolume I get:

    m_sectorsPerCluster:256
    m_sectorsPerClusterShift:8
    m_relativeSectors:4031758080
[COLOR="#FF0000"]    m_clusterHeapStartSector: 8192
[/COLOR]    m_fatLength: 3840
    m_fatStartSector: 4096
    m_fatType: 64
    m_clusterCount: 486368
    m_totalSectors: 570704772

. Which gives a clusterHeapStartSector of 8192 but when it gets manually calculated you get something way off.

As to your second point - ExFat seems more complicated - you can get the MBR set up correctly but then the rest gets messed up as you found.

Here are a couple of additional references I was looking at:
1. https://wiki.osdev.org/ExFAT
2. https://github.com/MicrosoftDocs/win32/blob/docs/desktop-src/FileIO/exfat-specification.md

 

[KurtE]

@mjs513 I brought in your code from the posting mentioned.

Has the format function changed from then. The compiler gave me a strong warning in the format function:

bool PFsExFatFormatter::format(BlockDeviceInterface *dev, uint8_t part, PFsVolume &partVol, uint8_t* secBuf, print_t* pr) {
#if !PRINT_FORMAT_PROGRESS
(void)pr;
#endif  //  !PRINT_FORMAT_PROGRESS
[COLOR="#FF0000"]  MbrSector_t* mbr;[/COLOR]
  ExFatPbs_t* pbs;
  DirUpcase_t* dup;
  DirBitmap_t* dbm;
  DirLabel_t* label;

  uint8_t vs;

  m_secBuf = secBuf;
  m_pr = pr;
  m_dev = dev;
  m_part = part;
  
  Serial.println(part);
  
  //if (!m_dev->readSector(0, (uint8_t*)&mbr)) {
  //  Serial.print("\nread MBR failed.\n");
  //  //errorPrint();
  //  return false;
  //}
  //MbrPart_t *pt = &mbr->part[part];
  [COLOR="#FF0000"]
  MbrPart_t *pt = &mbr->part[part];[/COLOR]

The compiler warns me that you are using an unitialized variable (pointer) mbr... Line 75

 

[KurtE]

Note: I edited the start of the method like:

bool PFsExFatFormatter::format(BlockDeviceInterface *dev, uint8_t part, PFsVolume &partVol, uint8_t* secBuf, print_t* pr) {
#if !PRINT_FORMAT_PROGRESS
(void)pr;
#endif  //  !PRINT_FORMAT_PROGRESS
  MbrSector_t mbr;
  ExFatPbs_t* pbs;
  DirUpcase_t* dup;
  DirBitmap_t* dbm;
  DirLabel_t* label;

  uint8_t vs;

  m_secBuf = secBuf;
  m_pr = pr;
  m_dev = dev;
  m_part = part;
  
  Serial.println(part);
  
  if (!m_dev->readSector(0, (uint8_t*)&mbr)) {
    Serial.print("\nread MBR failed.\n");
    //errorPrint();
    return false;
  }
  MbrPart_t *pt = &mbr.part[part];

When I run it now I see (only showing part after I typed in 2 for partition number)

Initialize USB drive...UsbBase::mscBegin called 20003de0 1 1
    After usbDriveBegin
USB drive 1 is present.
Partition 2 valid:1
ExFatFormatter - WIP
2
    m_sectorsPerCluster:64
    m_sectorsPerClusterShift:6
    m_relativeSectors:10291200
    m_clusterHeapStartSector: 10292096
    m_fatLength: 704
    m_fatStartSector: 10291328
    m_fatType: 64
    m_clusterCount: 84914
    m_totalSectors: 5435392

Sectors/cluserShift: 8, Sectors/Cluster: 256, FatLength 8192, FatOffset: 20
Partition Offset: 16384
clusterHeapOffset: 16384, clusterCount: 21104, volumeLength 5419008
    m_relativeSectors:16384
    m_totalSectors:5419008
Format failed
Press any key to run again

 

[mjs513]

@KurtE
Ok - what did you change - I can't see the difference? Now it works! Guess now have to leave displays and get back to this. It would always fault at m_dev->readSector(0, (uint8_t*)&mbr) even if I had MbrPart_t *pt = &mbr.part[part] it would keep telling me to use -> ? Confused as usual.


EDIT: Ok see what you changed - I am bad ain't I!

 

[KurtE]

@KurtE

EDIT: Ok see what you changed - I am bad ain't I!

Been there, done that!

Edit: Sometimes nice to build with Sublime-text and the Tset stuff. It jumped out at me... Once the build is done you can use the F4 key to walk through messages. Like I am still seeing:


Note: Now that we have a reasonable Size program as part of the build, can probably remove my older version of it from the Tset stuff...

 

[KurtE]

More on the topic of MTP integration...

I am still trying to think of the best way to integrate some of these different types of storages with MTP...


Currently we are running with sort of a hack to allow me for example on the PC select format on one of the storages and then have sketch code called that figures out which storage and does the appropriate code.
Note the format is not part of FS object yet (but maybe sometime....)

But I may also for example on some devices want to do something special for things like get free space as on for example large FAT32 devices the default code may take like 5 seconds... But we have short cut code.

So again what is the best way to integrate this.

Could for example maybe extend the storage object class, and have sub-classes for some of the different storage types.
Like could have different one for LittleFS versus PSFs... We already have separate FS classes for them. But question is
should the setup for adding the filesystem to storage. Currently we just pass in FS...
So would need way to keep list of different sub-classes...

Or potentially we could define some callback interface, that gets called for doing special things. Note: some of these special things code optionally include other things, like letting us know files changed on the storage...
Example we could extend the addFileSystem:
To maybe something like: virtual uint32_t addFilesystem(FS &filesystem, const char *name, MTP_StorageCallback *cb = nullptr, uint32_t token=0 )=0;

Where we might have members like:
bool format(uint32_t token);
uint64_t freSpace(uint32_t token);
...

Where I would probably stash a this pointer of the FS in the token and cast it back in the callback to know which FS it is...

Or could keep adding different callbacks...

Suggestions? I am sort of leaning toward a callback interface as I think it might be less disruptive to code and probably easier to expand...

 

[mjs513]

@KurtE - if is all in a callback it would be easier. Otherwise as you think of things....

Right now format assumes you already have a formatted drive with the partitions set up the way you want them. At some point in time probably need options to format whole drive and then create partitions but right just trying to get whats there working with existing code. Thinking more about if you hit format from MTP for the partition you selected.

@wwatson is already working on add partitions etc so ---.

 

[mjs513]

Afternoon all

Made some progress on exFAT formatting. Got it working for a single partition where windows still recognizes the drive and doesn't create extra unallocated areas before and after the partition like it would do if I just used SDFat's calculated offsets. Basically changed it to use the same partition information that the is read by ExPatition:

  // Determine partition layout.  
  m_relativeSectors = getLe32(pt->relativeSectors);
  
  // Determine partition layout.
  for (m = 1, vs = 0; m && sectorCount > m; m <<= 1, vs++) {}
  sectorsPerClusterShift = vs < 29 ? 8 : (vs - 11)/2;
  //1 << n is the same as raising 2 to the power n
  sectorsPerCluster = 1UL << sectorsPerClusterShift;
  
  //The FatLength field shall describe the length, in sectors, of each FAT table
  //   At least (ClusterCount + 2) * 2^2/ 2^BytesPerSectorShiftrounded up to the nearest integer
  //   At most (ClusterHeapOffset - FatOffset) / NumberOfFats rounded down to the nearest integer
  //fatLength = 1UL << (vs < 27 ? 13 : (vs + 1)/2);
  fatLength = partVol.fatLength();
  
  //The FatOffset field shall describe the volume-relative sector offset of the First FAT
  //   At least 24, which accounts for the sectors the Main Boot and Backup Boot regions consume
  //   At most ClusterHeapOffset - (FatLength * NumberOfFats), which accounts for the sectors the Cluster Heap consumes
  //fatOffset = fatLength;
  fatOffset = partVol.fatStartSector() - m_relativeSectors;
  
  //The PartitionOffset field shall describe the media-relative sector offset of the partition which hosts the given exFAT volume
  //partitionOffset = 2*fatLength;
  partitionOffset = m_relativeSectors;
  
  //The ClusterHeapOffset field shall describe the volume-relative sector offset of the Cluster Heap
  //   At least FatOffset + FatLength * NumberOfFats, to account for the sectors all the preceding regions consume
  //   At most 2^32- 1 or VolumeLength - (ClusterCount * 2^SectorsPerClusterShift), whichever calculation is less
  //clusterHeapOffset = 2*fatLength;
  clusterHeapOffset = partVol.clusterHeapStartSector() - m_relativeSectors;
  
  //The ClusterCount field shall describe the number of clusters the Cluster Heap contains
  //   (VolumeLength - ClusterHeapOffset) / 2SectorsPerClusterShiftrounded down to the nearest integer, which is exactly the number of clusters which can fit between the beginning of the Cluster Heap and the end of the volume
  //   232- 11, which is the maximum number of clusters a FAT can describe
  //clusterCount = (sectorCount - 4*fatLength) >> sectorsPerClusterShift;
  clusterCount = partVol.clusterCount();
  
  //The VolumeLength field shall describe the size of the given exFAT volume in sectors
  //   At least 2^20/ 2^BytesPerSectorShift, which ensures the smallest volume is no less than 1MB
  //   At most 264- 1, the largest value this field can describe
  volumeLength = clusterHeapOffset + (clusterCount << sectorsPerClusterShift);

The extra comments are taken from the MS spec. Think I may still have a problem with VolumeLength field but haven't sorted that out yet.

The funny thing is that when I tried formatting the 2 partitions MTP still will recognize and read the partition even though Windows does not?

Attaching what I have at this point in case you want to give it a try.

By the way found another good reference on exFAT forensics:
https://digital-forensics.sans.org/summit-archives/2010/10-exfat-ham.pdf

 

[wwatson]

@mjs513 - Sorry I have not had time to play with this much this week. My supervisor went on vacation so I had to take over. Have to go back to work and load a late truck

I did however have time to start working with the same section that determines the partition layout. You zoomed ahead Downloaded your latest. Hopefully will have some time this week to try it out.

 

[KurtE]

@mjs513 @wwatson ... - I just pushed up some changes to MTP_t4/test_msc_parts

Plus UsbMscFat/VolumeName_SD branch.

Where I introduced a call back interface to storage, that for example I moved the format code to. plus there is a callback for getting the totalsize and freesize through the callback.

Right now using the PFSVolume code parts I have for free clusters, so far appears to be fast enough for MTP to come up. I still may try in Fat32 case to read the sector and maybe then add some additional call backs to know when files change or the like such that maybe can add code to write the updated information sector back out.

Maybe soon can try adding the Format code for MSC drives...

 

[KurtE]

@mjs513 Morning for me, Afternoon for you

PFsExFatFormatter.cpp - I brought down latest stuff but having compile errors, maybe need updated PFsVolume.h code?

C:\Users\kurte\Documents\Arduino\libraries\UsbMscFat\src\PFsLib\PFsExFatFormatter.cpp: In member function 'bool PFsExFatFormatter::format(BlockDeviceInterface*, uint8_t, PFsVolume&, uint8_t*, print_t*)':
C:\Users\kurte\Documents\Arduino\libraries\UsbMscFat\src\PFsLib\PFsExFatFormatter.cpp:89:54: error: 'class PFsVolume' has no member named 'bytesPerSector'
  Serial.printf("    m_bytesPerSector: %u\n", partVol.bytesPerSector());
                                                      ^
C:\Users\kurte\Documents\Arduino\libraries\UsbMscFat\src\PFsLib\PFsExFatFormatter.cpp:91:60: error: 'class PFsVolume' has no member named 'rootDirectoryCluster'
  Serial.printf("    m_rootDirectoryCluster: %u\n", partVol.rootDirectoryCluster());
                                                            ^
C:\Users\kurte\Documents\Arduino\libraries\UsbMscFat\src\PFsLib\PFsExFatFormatter.cpp:92:59: error: 'class PFsVolume' has no member named 'bytesPerSectorShift'
  Serial.printf("    m_bytesPerSectorShift: %u\n", partVol.bytesPerSectorShift());
                                                           ^

Will check to see if maybe did not update something. Also strange that I can still compile the MTP

EDIT: I checked in my version that removed these errors...

I changed:

  Serial.printf("    m_rootDirectoryCluster: %u\n", partVol.rootDirectoryCluster());

to:

	Serial.printf("    m_rootDirectoryCluster: %u\n", partVol.getExFatVol()->rootDirectoryCluster());

EDIT I also uploaded changes again to both libraries, where I added a method to the PFsVolume to return the partition number.

I then added the format callback for MSC drives to try to do a format: I am not sure how much of it is working as MTP is not totally happy after but....

//=============================================================================
// try to get the right FS for this store and then call it's format if we have one...
// Here is for MSC Drives (SDFat)
PFsFatFormatter FatFormatter;
PFsExFatFormatter ExFatFormatter;
uint8_t  sectorBuffer[512];


uint8_t MSCMTPCB::formatStore(MTPStorage_SD *mtpstorage, uint32_t store, uint32_t user_token, uint32_t p2, bool post_process)
{
  // Lets map the user_token back to oint to our object...
  Serial.printf("Format Callback: user_token:%x store: %u p2:%u post:%u \n", user_token, store, p2, post_process);


  if (msc[user_token].mscfs.fatType() == FAT_TYPE_FAT12) {
    Serial.printf("    Fat12 not supported\n");  
    return MTPStorageInterfaceCB::FORMAT_NOT_SUPPORTED;
  }

  // For all of these the fat ones will do on post_process
  if (!post_process) return MTPStorageInterfaceCB::FORMAT_NEEDS_CALLBACK;

  if (msc[user_token].mscfs.fatType() == FAT_TYPE_EXFAT) {
      Serial.println("ExFatFormatter - WIP");
      ExFatFormatter.format(msc[user_token].mscfs.usbDrive(), msc[user_token].mscfs.part(), msc[user_token].mscfs, sectorBuffer, &Serial);
  } else {
      FatFormatter.format(msc[user_token].mscfs.usbDrive(), msc[user_token].mscfs.part(), msc[user_token].mscfs, sectorBuffer, &Serial);
  }
  return MTPStorageInterfaceCB::FORMAT_SUCCESSFUL;
}

Let me know what you think and/or if should pull it out for now.

 

[mjs513]

Evening @KurtE

I had added:

partVol.rootDirectoryCluster());

to PFsVolume but must have forgotten to include it in the updated zip.

That was a good idea to add: " PFsVolume to return the partition number ". Will have to pull it in.

FAT32 seems to be working but now I just ran a test and its back to taking forever to format the second partition on the drive. Have to go back and sort that one out.

I definitely need to add you updated code!

Let me know what you think and/or if should pull it out for now.

Torn on the FAT32 - but definitely pull the ExFAT. The closer I get to thinking I got it fixed the farther I get from the solution.

 

[KurtE]

@mjs513 - Good morning...

Looking at the last code I had/have it looks like PFsFormatter.cpp has the calls to do the formatting commented out. Need to see if there are any more recent versions...

Edit: I think I have the one posted in #167

 

[mjs513]

@mjs513 - Good morning...

Looking at the last code I had/have it looks like PFsFormatter.cpp has the calls to do the formatting commented out. Need to see if there are any more recent versions...

@KurtE - Good morning as well

Post #137 has the FatFormmater code for when I said it was working. As for ExFAT - still working on that one. Can't seem to break the code for multiple partitions as I said.

 

[mjs513]

@KurtE
Ignore last post. Here is a bit of cleaned up copy and an updated PFsVolum.h as well that I am using.

 

[KurtE]

Actually rereading the code :0 I see I missed the formatting... That is my eyes went directly to the commented out code, but skipped right over the line above it:

  rtn = m_sectorCount < 0X400000 ? makeFat16() :makeFat32();
  //if(partVol.fatType() == 16) {
//	writeMsg("format makeFAT16\r\n");  
//	rtn = makeFat16();
//  } else if(partVol.fatType() == 32) {
//	writeMsg("format makeFAT2\r\n");  
//	rtn = makeFat32();
//  }	else {
//	  rtn = false;
//  }

Now to try it again... Note: I did try it yesterday out of my MTP thing and my USB Stick was completely corrupted... Windows did not even want to repartitions it... Only after I used the SDFormatter app with the rewrite the disk did it behave again... Probably coincidence...