Many TLAs: MTP MSC FS SD SDFat LittleFS UsbMSCFat to work with each other 8)

mjs513 · Mar 11, 2021

@KurtE
Can't figure out what the cause is - makes no sense especially since it works quickly for 1st partition and takes forever for the second partition. Have no clue everything looks like. Hate these kinds of problems.

Very cool you got setVolName working for FAT16 and FAT32. Going to download and give it a try between trips to the garage

Its around 65deg today so taking advantage of it

mjs513 · Mar 11, 2021

@KurtE

Added setVolumeLabel to the formatter so that it reads the volume label on start and then calls setVolumeLabel prior to returning:

On first load:

Code:

msc # Partition Table
	part,boot,bgnCHS[3],type,endCHS[3],start,length
FAT32:	1,0,0x20,0x21,0x0,0xC,0xFE,0xFF,0xFF,2048,33187840
FAT32:	2,0,0xFE,0xFF,0xFF,0xB,0xFE,0xFF,0xFF,33189888,29362176
pt_#0:	3,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
pt_#0:	4,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
Partition 0 valid:1
Partition 1 valid:1
Partition 2 valid:0
Partition 3 valid:0
0:>> Fat32: 
$$$ PFsVolume::getVolumeLabel(32): 53910 32768
Volume name:(V1FAT32) Partition Total Size:16965599232 Used:131072 time us: 253623
    Free Clusters: API: 517745 by CB:517745 time us: 253366
    Free Clusters: Info: 517745 time us: 1868
1:>> Fat32: 
$$$ PFsVolume::getVolumeLabel(32): 33641646 512
Volume name:(v2FAT32) Partition Total Size:14802134016 Used:12884881408 time us: 1829123
    Free Clusters: API: 3744634 by CB:3744634 time us: 1828990
    Free Clusters: Info: 28910415 time us: 4242
done...
Enter 0, 1 , 2, or 3 for Partition or Enter to Bypass

After formatting 1st partition:

Code:

$$$ PFsVolume::getVolumeLabel(32): 53910 32768
Volume name:(V1FAT32)
PFsFatFormatter::format................
Sector Count: 33187840, Sectors/MB: 2048
Partition Capacity (MB): 16205
Fat Type: 32
    m_dataStart:53910
    m_sectorsPerCluster:64
    m_relativeSectors:2048

format makeFAT2
 MAKEFAT32
partType: 12, fatStart: 45820, totalSectors: 33187840
Writing FAT ................................
.....
Format Done
Press any key to run again
Initialize USB drive...UsbBase::mscBegin called 20003ea0 1 1
    After usbDriveBegin
USB drive 1 is present.

msc # Partition Table
	part,boot,bgnCHS[3],type,endCHS[3],start,length
FAT32:	1,0,0x20,0x21,0x0,0xC,0xFE,0xFF,0xFF,2048,33187840
FAT32:	2,0,0xFE,0xFF,0xFF,0xB,0xFE,0xFF,0xFF,33189888,29362176
pt_#0:	3,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
pt_#0:	4,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
Partition 0 valid:1
Partition 1 valid:1
Partition 2 valid:0
Partition 3 valid:0
0:>> Fat32: 
$$$ PFsVolume::getVolumeLabel(32): 53910 32768
Volume name:(V1FAT32) Partition Total Size:16965599232 Used:32768 time us: 253623
    Free Clusters: API: 517748 by CB:517748 time us: 253367
    Free Clusters: Info: 4294967295 time us: 1868
1:>> Fat32: 
$$$ PFsVolume::getVolumeLabel(32): 33641646 512
Volume name:(v2FAT32) Partition Total Size:14802134016 Used:12884881408 time us: 1829123
    Free Clusters: API: 3744634 by CB:3744634 time us: 1828990
    Free Clusters: Info: 28910415 time us: 4242
done...
Enter 0, 1 , 2, or 3 for Partition or Enter to Bypass

So it looks like the function is working properly. Even checked in Win Explorer to see if the vol label is correct.

mjs513 · Mar 11, 2021

@KurtE
Just gave the sketch a try and it seems to be working for my 32GB 2 FAT32 partions, here is the name change for the first partition:

Code:

Press any key to run again
Or enter new Volume label: <dev:12s> <Par> Name

Try to set device:2000234c part:1 to:newFAT32

$$$ PFsVolume::getVolumeLabel(32): 34816 8192
Previous Volume name:(V1FAT32)
$$$ PFsVolume::getVolumeLabel(32): 34816 8192
56 31 46 41 54 33 32 20 20 20 20 08 00 00 00 00 00 00 00 00 00 00 17 45 6B 52 00 00 00 00 00 00 :V1FAT32    ............EkR......
42 20 00 49 00 6E 00 66 00 6F 00 0F 00 72 72 00 6D 00 61 00 74 00 69 00 6F 00 00 00 6E 00 00 00 :B .I.n.f.o...rr.m.a.t.i.o...n...
01 53 00 79 00 73 00 74 00 65 00 0F 00 72 6D 00 20 00 56 00 6F 00 6C 00 75 00 00 00 6D 00 65 00 :.S.y.s.t.e...rm. .V.o.l.u...m.e.
53 59 53 54 45 4D 7E 31 20 20 20 16 00 8E 16 45 6B 52 6B 52 00 00 17 45 6B 52 03 00 00 00 00 00 :SYSTEM~1   ....EkRkR...EkR......
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :................................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :................................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :................................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :................................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :................................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :................................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :................................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :................................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :................................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :................................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :................................
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 :................................
Set new name *** succeeded ***
Initialize USB drive...UsbBase::mscBegin called 20002800 1 1
    After usbDriveBegin
USB drive 1 is present.

msc # Partition Table
	part,boot,bgnCHS[3],type,endCHS[3],start,length
FAT32:	1,0,0x20,0x21,0x0,0xC,0xFE,0xFF,0xFF,2048,33187840
FAT32:	2,0,0xFE,0xFF,0xFF,0xB,0xFE,0xFF,0xFF,33189888,29362176
pt_#0:	3,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
pt_#0:	4,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
Partition 0 valid:1
Partition 1 valid:1
Partition 2 valid:0
Partition 3 valid:0
0:>> Fat32: 
$$$ PFsVolume::getVolumeLabel(32): 34816 8192
Volume name:(newFAT32) Partition Total Size:16975396864 Used:32768 time us: 1012498
    Free Clusters: API: 2072188 by CB:2072188 time us: 1012491
    Free Clusters: Info: 2072191 time us: 1992
1:>> Fat32: 
$$$ PFsVolume::getVolumeLabel(32): 33641646 512
Volume name:(v2FAT32) Partition Total Size:14802134016 Used:12884880896 time us: 1829123
    Free Clusters: API: 3744635 by CB:3744635 time us: 1828990
    Free Clusters: Info: 28910415 time us: 4242

Initialize SD card...initialization failed.

done...
Press any key to run again
Or enter new Volume label: <dev:12s> <Par> Name

mjs513 · Mar 11, 2021

Not sure what happened looks like posts got crossed - volumeName sketch test results was suppose to be before Formatter.

KurtE · Mar 11, 2021

Just pushed up update for support for ExFat...

Some things I am unsure of, is if I can reuse previously deleted items.... I think they are marked with the first byte high bit not set.

The EX appears to work, although sometimes scanning right after I do that, it fails. But then I do it again and it works OK... Maybe need to add delay in sketch.

mjs513 · Mar 11, 2021

@KurtE
Just downloaded your latest changes and just notice something strange with the 2nd fat32 partition. Its showing used of 13GB of 14GB total. According to windows its 2k? First volume is correct for used and total.

Code:

0:>> Fat32: Volume name:(V1FAT32) Partition Total Size:16975396864 Used:131072 time us: 253748
    Free Clusters: API: 518044 by CB:518044 time us: 253491
    Free Clusters: Info: 518046 time us: 1743
1:>> Fat32: Volume name:(v2FAT32) [B][COLOR="#FF0000"]Partition Total Size:14802134016 Used:12884881408 time us: 1829123[/COLOR][/B]
    Free Clusters: API: 3744634 by CB:3744634 time us: 1828990
    Free Clusters: Info: 28910415 time us: 4242

wwatson · Mar 11, 2021

@KurtE, @mjs513 - I just downloaded the latest version of UsbMscFat-VolumeName_SD. Compiled and got an error saying:

Code:

In file included from /home/wwatson/Arduino/libraries/UsbMscFat-VolumeName_SD/src/USBFat.h:34:0,
                 from /home/wwatson/Arduino/libraries/UsbMscFat-VolumeName_SD/src/mscFS.h:27,
                 from /home/wwatson/Arduino/libraries/UsbMscFat-VolumeName_SD/examples/VolumeName/VolumeName.ino:8:
/home/wwatson/Arduino/libraries/UsbMscFat-VolumeName_SD/src/PFsLib/PFsLib.h:33:29: fatal error: PFsFatFormatter.h: No such file or directory
compilation terminated.

.

I looked in the PFsLib folder and noticed that PFsFatFormatter.h was named PfsFatFormatter.h. Probably not a problem with Windows but Linux is a bit picky that way

Now compiles ok and I am going to check it out.

mjs513 · Mar 11, 2021

@wwatson
Recommend not running it - something got way messed up - what was working is not working any more and I am tired of staring at it - back to MBR getting messed up and not working. Let me mess up my drives and cards. If you are ambitious you can take a look at what I did.

KurtE · Mar 11, 2021

For the fun of it I thought I would try it out on the MAC.

Looks like MACs do not come with any support for MTP... Not sure if anyone else has tried it? If so which application package?

wwatson · Mar 11, 2021

@mjs513 - I have three different sticks and several cards that I use for testing purposes. I can take a quick look at it tonight and this weekend hopefully is mine and I can get caught up with you and KurtE's changes. It looks like major progress to me

sbfreddie · Mar 12, 2021

KurtE said:
For the fun of it I thought I would try it out on the MAC.

Looks like MACs do not come with any support for MTP... Not sure if anyone else has tried it? If so which application package?

Kurt:
I have an app called SyncMate that claims to support MTP on the Mac (MacOS 10.15.7). I have not tried to use it yet, but your development here is the reason I purchased it.

Keep up the good work!

Thanks,
Ed

mjs513 · Mar 12, 2021

@KurtE - @wwatson
I think I got it working again I hope.

Anyway did a few minor changes and added it @KurtE's setVolumeLabel so it rewrites the existing volume label after formatting is complete and seems to be working. Right now its limited to FAT16 and FAT32's with Bill's current format limitations: <=2GB it gets formatted as FAT16, >2GB and <=32GB it gets formatted as FAT32 period. SectorCount limit test has been commented out because of different formats (let me know if this becomes a problem).

Going forward think I may change it to more generic so you only have to pass in partition number to get formatted without the pointer to the volume - will have to do a partition begin in the format to make that work

I am attaching the updated files and example for you all to test if you want to play.

KurtE · Mar 12, 2021

@mjs513 good morning.

Downloaded and built and reformatted two partitions on my USB stick

Next I think I will partially redo the fast free count code (callbacks) and add in some form of timeout. There have been a few times when either it hung or crashed....
So will probably put something in, like if we don't get any more sectors in some time, have the code be able to bail out...

mjs513 · Mar 12, 2021

KurtE said:
@mjs513 good morning.

Downloaded and built and reformatted two partitions on my USB stick

Next I think I will partially redo the fast free count code (callbacks) and add in some form of timeout. There have been a few times when either it hung or crashed....
So will probably put something in, like if we don't get any more sectors in some time, have the code be able to bail out...

Yeah - only 2 days to fix what was working

cool on the bailout - things changing fast

KurtE · Mar 12, 2021

@mjs513 and others, I checked in changes the the USBHOst and the MSC code to do timeouts...

Warning I also updated your formatter code that was in previous message... Hope that was OK...

mjs513 · Mar 12, 2021

KurtE said:
@mjs513 and others, I checked in changes the the USBHOst and the MSC code to do timeouts...

Warning I also updated your formatter code that was in previous message... Hope that was OK...

Not a problem - glad you did. I just closed the open PR. Actually working on the exFormatter. Think I know where the problem is but not sure what is causing the hardfault. When I do this:

Code:

  if (!m_dev->readSector(0, (uint8_t*)&mbr)) {
    Serial.print("\nread MBR failed.\n");
    //errorPrint();
    return false;
  }
  
  MbrPart_t *pt = &mbr.part[part];

in ExFatFormatter it causes a hardfault. Oh - if FatFormatter that code snippet works.

KurtE · Mar 12, 2021

mjs513 said:
Not a problem - glad you did. I just closed the open PR. Actually working on the exFormatter. Think I know where the problem is but not sure what is causing the hardfault. When I do this:

Code:

if (!m_dev->readSector(0, (uint8_t*)&mbr)) { Serial.print("\nread MBR failed.\n"); //errorPrint(); return false; } MbrPart_t *pt = &mbr.part[part];

in ExFatFormatter it causes a hardfault. Oh - if FatFormatter that code snippet works.

Did not know there was an open one... Probably as I am not the library owner, I only see those PRs that have @kurte in them...

I need to play more with the Hard fault stuff. Trying to remember if Paul pulled it in to the latest betas? Or do I need to do something still to enable it.

mjs513 · Mar 12, 2021

@KurtE
Keep forgetting about problem with PRs when you are not the owner

Say you the time, Paul never pulled in the changes I had to manually make the changes from Frank's fork if I remember correctly.

wwatson · Mar 12, 2021

@KurtE and @mjs513 - I am so far behind you guy's.

@mjs513 and others, I checked in changes the the USBHOst and the MSC code to do timeouts...

Warning I also updated your formatter code that was in previous message... Hope that was OK...

Which branch of USBHost_t36? Is it MSC_read_sectors_cb? And VolumeName_SD?

Sorry

mjs513 · Mar 13, 2021

wwatson said:
@KurtE and @mjs513 - I am so far behind you guy's.

Which branch of USBHost_t36? Is it MSC_read_sectors_cb? And VolumeName_SD?

Sorry

@wwatson - no need to be sorry I get confused as well. But you are right those are the two branches you should be using.

mjs513 · Mar 13, 2021

Morning All - especially now that I have had my coffee

While working on ExFatFormatter which I am having issues with I came across a couple interesting things while doing a google search.

1. WinHEX - http://www.winhex.com/winhex/
2 ExFAT presentation - https://events.static.linuxfound.org/images/stories/pdf/lceu11_munegowda_s.pdf, actually pretty interesting.
3. Demystifing ExFAT - https://rshullic.files.wordpress.co...icrosoft-extended-file-system-exfat-v1-00.pdf

mjs513 · Mar 13, 2021

@KurtE - @wwatson - @All

I am stumped on getting ExFormatter working. When I dump data for a 2 partition drive I see:

Code:

msc # Partition Table
	part,boot,bgnCHS[3],type,endCHS[3],start,length
exFAT:	1,0,0x20,0x21,0x0,0x7,0xFE,0xFF,0xFF,2048,124516352
exFAT:	2,0,0xFE,0xFF,0xFF,0x7,0xFE,0xFF,0xFF,124518400,125218816
pt_#0:	3,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
pt_#0:	4,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0

where:

Code:

relativeSectors for part 1 is 2048
relativeSectors for part 2 is 124518400

but because I can't seem to read the mbr in PFsExFormatter with:

Code:

  //if (!m_dev->readSector(0, (uint8_t*)&mbr)) {
  //  Serial.print("\nread MBR failed.\n");
  //  //errorPrint();
  //  return false;
  //}
  //MbrPart_t *pt = &mbr->part[part];

- it causes a hard fault its not getting the correct values:

Code:

m_relativeSectors:4031758080
m_relativeSectors:2997053506

anyway if anyone has any ideas I am attaching a zip of files I changed in PFs. Also as a note I did add a few extra calls in PFsVolume.h:

Code:

  /** \return the cluster heap start sector. */
  uint32_t clusterHeapStartSector() const {	  
	return m_xVol ? m_xVol->clusterHeapStartSector() : 0;
   }
  /** \return the FAT length in sectors */
  uint32_t fatLength() const {
	  return m_xVol ? m_xVol->fatLength() : 0;
   }
   /** \return the power of two for sectors per cluster. */
   uint8_t  sectorsPerClusterShift() const {
	   return m_xVol ? m_xVol->sectorsPerClusterShift() : 0;;
   }

hope I did this right.

wwatson · Mar 13, 2021

@KurtE - @mjs513

Finally got setup to test. Started out with VolumeName.ino. Everything works great except for one small issue the SDIO card that probably goes deeper than my understanding of how SDIO works.

When I change the volume label on the SDIO card I get this:

Code:

Initialize SD card...initialization failed.

If I do a reset it will show that the card is present and that the volume name was changed.
This is what I added to volumeName.ino as a workaround:

Code:

#if 1
  sd.begin(SD_CONFIG);
#endif

  if (!sd.begin(SD_CONFIG)) {
    Serial.println("initialization failed.\n");
  } else {
    Serial.println("SD card is present.\n");

to make it initialize the SDIO card without having to do a reset.

Next is playing with formatting.

mjs513 · Mar 13, 2021

@wwatson = @KurtE

Fat16/Fat32 formatting looks like it is working. ExFat has issues as I mentioned.

The only files you need to play with exfat in the PFsLib zip are:

Code:

PFsExFatFormatter.h
PFsExFatFormatter.cpp
PFsVolume.h
and
PFsLib.h

KurtE · Mar 13, 2021

I have seen the changing volume label do that. What I was going to try next is to simply put in a delay after I do set volume label and see if that helps.

Mike I will take a look at the format stuff hopefully in a little bit. Right now still figuring out coordinates for drawing centered text... Also OBD2 stuff...

Many TLAs: MTP MSC FS SD SDFat LittleFS UsbMSCFat to work with each other 8)

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Well-known member

Senior Member+

Senior Member+

Well-known member

Well-known member

Senior Member+

Attachments

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Well-known member

Senior Member+

Senior Member+

Senior Member+

Attachments

Well-known member

Senior Member+

Senior Member+