Teensyduino File System Integration, including MTP and MSC

Ok guess its my turn. while @KurtE was working on all that I started looking at formatting again. Unfortunately a lot of whats in fatFormat and exFatFormat is dependent on the final API and where needed data elements are can be found. For instance, just as a start I did have to
1. Modify findPartitions function to return a couple of extra data elements - makes it equivalent to what was in getPartitionInfo in the pfsLib library.
2. Also remembered that we exposed a few extra data elements in FsVolume than what Bill has currently in SdFat. Specifically,
Code:
fatLength.
sectorsPerClusterShift
clusterHeapStartSector.
Not sure how to handle this. Do we want to see if Bill is willing to add it or we keep it as a delta?

This is really preliminary and probably wont make it into beta2 as things are changing too much to finalize.
 
@Paul, @KurtE, @mjs513 - Decided to test the latest with USBhost_t36. I tested with TD1.57B1 first Paul's latest and then Kurt's latest. found no issues.

Here is output from Kurt's active branch of USBHost_t36. Tested with two thumb drives one with a single EXFAT partition and another with two partitions, EXFATP1 and FAT32P2.

Here is the output:
Code:
 === Drive index 0 found ===
       connected: 1
     initialized: 1
   USB Vendor ID: 0951
  USB Product ID: 1666
      HUB Number: 1
        HUB Port: 1
  Device Address: 2
Removable Device: YES
        VendorID: Kingston
       ProductID: DataTraveler 3.0
      RevisionID:     
         Version: 6
    Sector Count: 241660915
     Sector size: 512
   Disk Capacity: 123730388480 Bytes

Partition Table
        part,boot,bgnCHS[3],type,endCHS[3],start,length
exFAT:  1,0,0x4,0x1,0x4,0x7,0xFE,0xC2,0xFF,2048,241657856
pt_#0:  2,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
pt_#0:  3,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
pt_#0:  4,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
         < unused area starting at: 241659904 length 1011 >

=== File system (0) ===
        Manufacturer: Kingston
        Product: DataTraveler 3.0
        Serial Number: 4CEDFB74A418F4A0D96A029
Volume name: 128GEXFAT
Volume type: FAT64
Cluster Size: 131072 bytes
Volume size: 123723579392 bytes
 Space used: 419168256 bytes  (91 ms to compute)

Files:
2019-01-01 00:00     56863276 armaged.wav
2019-01-01 00:07     43025728 DeffLepardAnimal.wav
2019-01-01 00:07      7502713 StatusQuoWhateverYouWant.mp3
2019-01-01 00:09     35639166 ThePaperboysMolinos.wav
2019-01-01 00:10     36381082 ThePaperboysMolinosI'veJustSeenaFace.wav
2019-01-01 00:56     94519514 YouareLazyJimmyBarnesJoeBonamassa.wav
2022-01-01 00:00     32768000 32MEGfile.dat
2019-01-01 00:00     13166660 YouareLazyJimmyBarnesJoeBonamassa.mp3
2019-01-01 00:52     32768000 bench.dat
2022-01-01 00:00     32768000 test.txt
2022-01-01 00:00     32768000 copy.txt
2022-01-01 00:00         2043 datalog.txt


 === Drive index 1 found ===
       connected: 1
     initialized: 1
   USB Vendor ID: 0951
  USB Product ID: 1666
      HUB Number: 1
        HUB Port: 4
  Device Address: 3
Removable Device: YES
        VendorID: Kingston
       ProductID: DataTraveler 3.0
      RevisionID:     
         Version: 6
    Sector Count: 30218841
     Sector size: 512
   Disk Capacity: 15472046592 Bytes

Partition Table
        part,boot,bgnCHS[3],type,endCHS[3],start,length
exFAT:  1,0,0x4,0x1,0x4,0x7,0xFE,0xC2,0xFF,2048,17928192
FAT32:  2,0,0xFE,0xC2,0xFF,0xB,0xFE,0xC2,0xFF,17930240,12288000
pt_#0:  3,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
pt_#0:  4,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
         < unused area starting at: 30218240 length 601 >

=== File system (1) ===
        Manufacturer: Kingston
        Product: DataTraveler 3.0
        Serial Number: D067E516489AF1B0B60F06F
Volume name: EXFATP1
Volume type: FAT64
Cluster Size: 32768 bytes
Volume size: 9176088576 bytes
 Space used: 103710720 bytes  (34 ms to compute)

Files:
2022-02-21 14:13        16976 pathname
2022-02-21 14:13          974 pathname.c
2022-02-21 13:23         2352 pathname.o
2021-11-20 14:13        14916 playFSwav.cpp
2021-11-20 14:11         3659 playFSwav.h
2022-03-05 14:58        17184 qsortStructArray
2022-03-05 15:01         1384 qsortStructArray.c
2022-03-05 14:58         3224 qsortStructArray.o
2022-03-05 14:20        17352 qsortTest
2022-03-05 11:16         4418 qsortTest.c
2022-03-05 09:35         5776 qsortTest.o
2021-11-23 18:25       319806 Screenshot at 2021-11-23 17-25-06.png
2021-12-05 18:39    102879394 YoureLazyJimmyBarnesJoeBonamassa.wav

=== File system (2) ===
Volume name: FAT32P2
Volume type: FAT32
Cluster Size: 4096 bytes
Volume size: 6279168000 bytes
 Space used: 334946304 bytes  (0 ms to compute)

Files:
2019-01-01 00:03   56862764 armaged.wav
2019-01-01 00:02   43025216 DeffLepardAnimal.wav
2019-01-01 00:03   60148386 StatusQuoWhateverYouWant.wav
2019-01-01 00:03   35638654 ThePaperboysMolinos.wav
2019-01-01 00:03   36380570 ThePaperboysMolinosI'veJustSeenaFace.wav
2019-01-01 00:02  102878370 YoureLazyJimmyBarnesJoeBonamassa.wav


 *** enter any key to run again ***

Looks like things are progressing :)
 
Good morning Paul and all:

Status update for MSC integration into USBHost:

I currently have A Pull request: https://github.com/PaulStoffregen/USBHost_t36/pull/89
Which added in a lot of the functionality we had in the earlier integration attempt: Details from the PR:
@mjs513 and myself have been extending the beta1 code to add some additional stuff. We still need to go through and rework some of the examples and test more, but thought I would go ahead and issue a PR with current WIP as to give you a head start on seeing what has changed so far:
First pass to add the functionality we were using with the previous
integration pass and work toward beta 2...
Still retaining some support for the Beta1 way of starting up a filesystem:
but added USBFilesystem::end() which closes the file system allows you to reuse the object.
Added bool operator to USBFilesystem know if object is valid.
Added USBFSBase class that works like hid class or the like, that USBFilesystem derives from. Which allows us to have
multiple Filesystem implementations.
Added code to create list of these, with ini() method added to Filesystem object to add it to a free list.
When these objects are claimed, I remove them from this list and add them to list connected to drive. When drive
goes away, I end() these objects and then put them back on free list. Note: I am contemplating not linking and unlinking
but simply when walk whole lists each time.
Added (maybe temporary) version of DriveInfo that tries to walk through partitions of a drive and print out information.
Extended USBDrive::findPartition, including:
a) support extended partitions
b) reworked the GPT walking to not bail on first not known GUID but instead all GUIDS, until empty entry found
b1) The enumeration does not check the guid. Instead returns it and lets caller decide if they understand the guid
c) returns a little more information needed for format.
d) return code was updated from bool to int, where 0 is no FS found, and other values to know if MBR, GPT, EXT partition
Enumeration: Added method to USBDrive::startFilesystems
Also added USBDrive::filesystemsStarted() method as ran into issues like, the DriveInfoAll sketch would enumerate all of the disks and Filesystems, and then wait to see if you want to run it again. Well suppose you unplug one disk and plug in another, we would still end up with same number of drives... So caching which disk we enumerated would know that something had changed. Now the enumeration code sets state and drive going way unsets it...
Which first does begin() to make sure drive is ready, it then keeps calling findPartition until it receives
back a return of 0.
It then calls off to claim for each of the free USBFilesystem objects until one claims it or none...
If FS claimed it removes the item from list and adds to the drives list.
When drive goes away, it walks it's list and calls another member of the new FS base class which the current
one calls end(). It also then removes the item from drive list and puts back on free.
Since I issued the above Pull Request we (mostly Mike) have been working on format support. I have now pushed up new branch based on the branch associated with the PR:
https://github.com/kurte/USBHost_t36/tree/fs_integration_part2_format

This includes a drop of a new class that Mike developed that contains the Format code. This code is still being updated and tested. Yesterday I added it as it’s own individual class into the USBHost library (in that branch), I then added the Format method to the USBFilesystem class. It was defaulting to the base classes empty return false method.

This implementation, creates an instance of the format class and then calls off to the a simplified format member which will try to format the volume to the same type (if possible), there may be some limitations depending on size of volume.

I did run into some issues, with this, in that like the SD format code, I malloc the buffer to use. Ran into issue that the ReadSectors/WriteSectors methods did not handle this. And it corrupted the disks. So added in the check and calls to arm_dcache_ … Which is now working.

As I mentioned we have a couple of sketches updated to use the newer enumeration, including with MTP_Teensy, and these now call off to use the format without changes to these sketches.

Still being tested, and fixed.

For example the next thing to fix, is an issue, that Suppose the Format does change the Filesystem information. I am currently holding a state variable in the USBDrive object that says the drive was enumerated. So my user code will not go through and enumerate again and for example won't in test cases show updated Partition information. So I probably need to update that state, when we enumerate again, I need to know that I am
already holding onto a USBFilesystem object for that Partition. So probably don't try to claim another one. Probably want to add new method that says updatePartition, which will give it a chance to change things.

Note: in this case I had already updated the USBFilesystem object at the completion of the format... So will see what all needs to be done.
 
Good after noon all.
Using Kurt's latest branch with formatting decided to test the changes with my MTP_TEST_INTEGRITY sketch (not pushed yet with the updates)> Here is a dump of the storage list and what it looks like windows explorer
Code:
Dump Storage list(13)
store:0 storage:10001 name:PgmIndx fs:2001210c pn:    (In program mem)
store:1 storage:20001 name:BUILTIN fs:20014b98 pn:    (BUILTIN sd Card)
store:2 storage:30001 name:EXT1 fs:2001506c pn:		  (Card on audio shield)
store:3 storage:40001 name:RAM0 fs:20014990 pn:EXTMEM
store:4 storage:50001 name:RAM1 fs:20014a5c pn:EXTMEM
store:5 storage:60001 name:PROGM fs:200147ec pn:PROGRAM
store:6 storage:70001 name:QSPI fs:200148b8 pn:W25Q128JV*M (DTR)
store:7 storage:80001 name:MSC0 fs:200122ac pn:		  (MSC0/1 on USB: MBR fat32+exFAT)
store:8 storage:90001 name:MSC1 fs:20012754 pn:
store:9 storage:a0001 name:MSC2-GPTFAT32 fs:20012bfc pn:   (MSC2/3 on USB: GPT fat32+exFAT)
store:10 storage:b0001 name:MSC3 fs:200130a4 pn:
store:11 storage:c0001 name:MSC4-Part1 fs:2001354c pn:     (MS4/5 on USB SD Card: MBE fat32+exFAT)
store:12 storage:d0001 name:MSC5-Part2 fs:200139f4 pn:

Capture.jpg

Have not done exhaustive testing but it looking good - for several drives I tried the updated formatting code and seems to work (note changes to FsVolume no longer needed). Copying files and directories seem to be working to usb drives as well. Have also tested play back of audio files in windows from USB drives and that seem to be working.
 
Nice work! So, to catch up using TD 1.57b1:
MAIN :: github.com/KurtE/MTP_Teensy
and
github.com/kurte/USBHost_t36/tree/fs_integration_part2_format

With optional edit to CORES to support SERIAL still required under the prior overloaded 'everything' option? { will have to dig that up }

Yes - all of the above.
Assumes Beta 1 install
Plus the trees you mentioned.

Note: I have a couple of USB examples converted. Both my CircuitMicroPython example plus under simplified... examples I have example 5...

Still debugging some trying to tell user list was updated... code.

Paul - In all of your not free time! also at some point would be interesting to talk events/notification... Like when a drive is inserted, can other sub-systems and user sketch get some form of notification.

Like New USBFilesystem object is claimed, have ability for MTP to optionally automatically add the MTP storage list....

But first would be good to get this level of functionality working properly and migrated back into the master branch.
 
Yes - all of the above.
Assumes Beta 1 install
Plus the trees you mentioned.
...

Thanks Kurt, it builds.
Will have to get back to what was in progress in Feb and start with some simple Xfer examples for confirming valid file transfer.

Browsing simple folders on T_4.1 SD works.
Opening a TXT file of 244 or 756 bytes is FAST.

Three folders with a 500 byte file show that file to take some LONG time 'Calculating' before the file opens: Measures 30 seconds delay
>> "This PCTeensyMakeFiles 22048JanSRC_o222\bbb.txt"
 
Does this mean anything as a complete issue indication:
Code:
$$$ Failed to read Index record: 15 bytes Read: 0
$$$ Failed to read Index record: 16 bytes Read: 0
$$$ Failed to read Index record: 15 bytes Read: 0

On current over large SD tried to copy what I thought was a smaller subdirectory and this over filled TyComm 24 MB buffer repeating those type lines ... and kept going long after for many non stop minutes.
Total 107999 files of Size 1088487493 Bytes
Bytes Used: 28699525120, Bytes Total:128144375808
> Had to restart the Teensy to stop it as I could not get Edge Browser to respond while that was running.

Will start with clean SD card and just the files from MakeFiles.ino to verify and test for base function and perhaps any simple repro's.
 
@defragster
Hi Tim
A little confused are you transfering files from the PC to the Builtinsd Card over MTP. I just transferred 104MB of audio files to the BUILTIN SDCard in about 14 seconds?

Writing you 'B'ig File:
Code:
Start Big write of 25124864 Bytes............................................................
Big write /0_bigfile.txt took 24.38 Sec for 25124864 Bytes : file3.size()=25122816
	Big write KBytes per second 1030.55 

Bytes Used: 43122688, Bytes Total:63831015424

Delete with read verify all #bigfile's
	Verify /0_bigfile.txt bytes 25122816 : ..................................................	GOOD! >>  bytes 25122816
	Big read&compare KBytes per second 901.68

and writing Kurt's Index file:
Code:
	Big write KBytes per second 360.94
 
@defragster
Hi Tim
A little confused are you transfering files from the PC to the Builtinsd Card over MTP. I just transferred 104MB of audio files to the BUILTIN SDCard in about 14 seconds?
...
The problem relates to simple Notepad text file open using MTP when size is 500 bytes.

SD card on T_4.1 holds files created up to Jan/Feb 2022. The Makefiles folders and files.
> Going to set that Samsung 128GB SD aside.
> Moved to a different SD card behavior reproduces on 500 byte file: See next post

From PC with T_4.1 connected and running.

DoubleClick the .TXT file:
bbb.txt: 30 second Delayed 'Calculating' "Windows Dialog" open to notepad, size 244 bytes
aaa.txt: Fast open to notepad, size 244 bytes
ccc.txt: Fast open to notepad, size 756 bytes

Will get a fresh SD card and run Makefiles.ino to get a minimal set of those files ... done with repro, details in next post after Makefile.ino updated on GitHub.
 
Only change to github.com/Defragster/.../MakeFiles.ino was an #ifdef.
Starting a T_4.1 with a usably empty SD card running MTP Experimental Makefile.ino create files with "t - Test Files write 2204"
> confirm Windows explorer can browse the folders/files
> go to MTP root and copy all files to 'Empty' Windows disk folder
> remove SD card from T_4.1 and mount on PC and copy the same files to "MTP_COPY" folder and DIFF the two folders.

New card had here some binary files - they won't copy to Win 11 PC folder from MTP_T_4.1_SDcard.
> it has 220 binary files - ZIP is 29.4 MB so will get to the fact that it copies all 220 files to an Empty folder.
> this recurred on WHOLE SD or Single copy of just that folder
- > copy of one of the single file like 'P3B391.BIN' does work.

Copy of Whole SD to PC presented this error before completion:
MTPtoPC_ErrorPoolEmpty.png

And lots of missing and erroneous files after copy.
Source DIR as read from SD NATIVE on PC:
Total Files Listed:
514 File(s) 103,846,889 bytes
68 Dir(s) 531,372,298,240 bytes free
SD card image copy on PC after MTP copy:
Total Files Listed:
232 File(s) 34,428,321 bytes
65 Dir(s) 531,372,335,104 bytes free

Noticed the 5/16/2022 files on the SD card are stored dated as TOMORROW 5/17/2022, though the TimeStamp is correct?
05/16/2022 08:59 PM <DIR> ..
05/16/2022 08:34 PM 124,928 124928_21.txt

Versus
05/16/2022 08:36 PM <DIR> .
05/16/2022 08:41 PM <DIR> ..
05/17/2022 08:34 PM 124,928 124928_21.txt

Seeing that files in "Directory of D:\tmp\TD_1.57\SDCOPY_2204_16May\Huge\D0.10" have sizes from another folder not expected 400K files:
MTP_400K_FileBreak.jpg
> Valid source on RIGHT

This file takes 30 seconds to open to notepad (file content is valid) from MTP_T_4.1: "D:\tmp\TD_1.57\SDCOPY_2204_16May\D0.25\500.txt"
--> Wait is accompanied by Windows dialog: "0% complete" ... " Time remaining: Calculating..."
> Folder has 24 other files of 501 to 524 and they all open and read to notepad in under a second
View attachment 500.txt

Too much else to question with missing folders and corrupt files ... will have to find a simple test case if this doesn't lead to enough to find some low level flaw.
 
Paul - In all of your not free time! also at some point would be interesting to talk events/notification... Like when a drive is inserted, can other sub-systems and user sketch get some form of notification.

Yeah, been thinking about this recently.... we really do need better notification of filesystem / device connect & removal. It really wants to be integrated at the FS class level so the API is consistent.

Might be a little while until I can do much. Yesterday PJRC was down to just me & Robin due to Covid19. We both again this morning tested negative (as far as we know neither of us has ever had it) and looks like we'll have 1 person back today from quarantine, so production can get moving again at least partially without Robin & I doing it all.

I still have the USB host set address delay stuff on my workbench. Is see you sent a PR. Might get that merged before I butcher the timer code, just in case it would end up conflicting.

We're facing more parts shortages which are going to draw away a lot of my time soon. I'm really not happy about it.
 
Yeah, been thinking about this recently.... we really do need better notification of filesystem / device connect & removal. It really wants to be integrated at the FS class level so the API is consistent.

Might be a little while until I can do much. Yesterday PJRC was down to just me & Robin due to Covid19. We both again this morning tested negative (as far as we know neither of us has ever had it) and looks like we'll have 1 person back today from quarantine, so production can get moving again at least partially without Robin & I doing it all.

I still have the USB host set address delay stuff on my workbench. Is see you sent a PR. Might get that merged before I butcher the timer code, just in case it would end up conflicting.

We're facing more parts shortages which are going to draw away a lot of my time soon. I'm really not happy about it.

Yikes - I hope you all stay well! Also keeping fingers crossed on port shortages!

I just merged the branch with the Format code back into the branch with the open PR: https://github.com/KurtE/USBHost_t36/tree/fs_integration_part2
https://github.com/PaulStoffregen/USBHost_t36/pull/89

Yes - it would be interesting to have some consistent notifications for FS objects. May also want to extend to Drive objects as well. Although we may want to wait and see how well we can handle drive insertions and automatically enumerate.

Also in the the Drive's case, we might consider to also generalize this in USBHost. That is should we add some form of notification or callbacks that libraries/sketches can subscribe to, to find out when new USB Host devices are inserted or removed?

But most of this can wait until later!

Again fingers crossed!
 
@PaulStoffregen
All I can say is ditto to what Kurt said. Parts shortages I think is going to be a way of life for the next year and may get worse. Sorry to hear about PJRC getting hit with Covid-19 but sounds like that will be getting better soon. Guess the saying is true - when it rains it pours.
 
@defragster - not sure about everything else but I just copied a 1.1MB .bin file to an ext sd card without issues, then took the card out and was able to transfer back to the pc. Was also able to transfer it to the PC through MPT so sounds like something strange going on with your setup.
 
Hi @defragster

Quick update - as I mentioned in the last post I merged back to the one _part2 branch and deleted the _part2_format.

At some point soon I will take a closer look at the issues you are seeing with your test cases. Hopefully we will be able to
resolve them. It would help if you have some simple test case(s) that illustrate them.

Looks like most of it is on the MTP side, which is great.

But right now I am sort of focused on the MSC part, with trying to get it more of it fleshed out for the next beta...

With your MTP/SD type examples, it will be interesting to see if the issues with data loss and the like and similarly with time to open files with app,
to figure out if the problems are with the MTP part and/or with the SD/SDFAt side.

Now back to playing
 
The problem relates to simple Notepad text file open using MTP when size is 500 bytes.

SD card on T_4.1 holds files created up to Jan/Feb 2022. The Makefiles folders and files.
> Going to set that Samsung 128GB SD aside.
> Moved to a different SD card behavior reproduces on 500 byte file: See next post

From PC with T_4.1 connected and running.

DoubleClick the .TXT file:
bbb.txt: 30 second Delayed 'Calculating' "Windows Dialog" open to notepad, size 244 bytes
aaa.txt: Fast open to notepad, size 244 bytes
ccc.txt: Fast open to notepad, size 756 bytes

Will get a fresh SD card and run Makefiles.ino to get a minimal set of those files ... done with repro, details in next post after Makefile.ino updated on GitHub.

Ok catching up now after the dentist. This is what I currently have on my ext 64gb samsung sd card:
Code:
Storage Index 2 Name: EXT1 Selected

 Space Used = 19660800
Filesystem Size = 63831015424
Directory
---------
mtpindex.dat                          0
111/
  aaa.txt                             244
  bbb.txt                             500
  ccc.txt                             756
222/
  aaa.txt                             244
  bbb.txt                             500
  ccc.txt                             756
333/
  aaa.txt                             244
  bbb.txt                             500
  ccc.txt                             756
ManyD8/
  D0/
    100.txt                           100
    1300.txt                          1300
    1700.txt                          1700
    2100.txt                          2100
    500.txt                           500
    900.txt                           900
    D1/
      100.txt                         100
      1300.txt                        1300
      1700.txt                        1700
      2100.txt                        2100
      500.txt                         500
      900.txt                         900
      D2/
        100.txt                       100
        1300.txt                      1300
        1700.txt                      1700
        2100.txt                      2100
        500.txt                       500
        900.txt                       900
        D3/
          100.txt                     100
          1300.txt                    1300
          1700.txt                    1700
          2100.txt                    2100
          500.txt                     500
          900.txt                     900
          D4/
            100.txt                   100
            1300.txt                  1300
            1700.txt                  1700
            2100.txt                  2100
            500.txt                   500
            900.txt                   900
            D5/
              100.txt                 100
              1300.txt                1300
              1700.txt                1700
              2100.txt                2100
              500.txt                 500
              900.txt                 900
              D6/
                100.txt               100
                1300.txt              1300
                1700.txt              1700
                2100.txt              2100
                500.txt               500
                900.txt               900
                D7/
                  100.txt             100
                  1300.txt            1300
                  1700.txt            1700
                  2100.txt            2100
                  500.txt             500
                  900.txt             900
MoreFD/
  D0/
    10125.txt                         10125
    125.txt                           125
    2125.txt                          2125
    4125.txt                          4125
    6125.txt                          6125
    8125.txt                          8125
    D1/
      10125.txt                       10125
      125.txt                         125
      2125.txt                        2125
      4125.txt                        4125
      6125.txt                        6125
      8125.txt                        8125
      D2/
        10125.txt                     10125
        125.txt                       125
        2125.txt                      2125
        4125.txt                      4125
        6125.txt                      6125
        8125.txt                      8125
        D3/
          10125.txt                   10125
          125.txt                     125
          2125.txt                    2125
          4125.txt                    4125
          6125.txt                    6125
          8125.txt                    8125
ccc.txt                               756
NINA_W102-1.7.5.bin                   1160192
System Volume Information/
  WPSettings.dat                      12
  IndexerVolumeGuid                   76
If I go into the 111 directory from MTP and open up the three files (244, 500, 756 bytes) :
Code:
DoubleClick the .TXT file:
[B]bbb.txt: Fast open to notepad, size 500 bytes
[/B]aaa.txt: Fast open to notepad, size 244 bytes
ccc.txt: Fast open to notepad, size 756 bytes

as a double check I went into ManyD8 dircectory and drilled down to D7 and opened up the 500 byte file:
500.txt - fast open with Notepad

Would double check to make sure you are using @KurtE's last changes in his _part2 USBHost_36 library and UsbMscFat from Paul's repository list. MTP - don't think there were any changes I am Using @KurtE's MTP_Teensy library (main branch - https://github.com/KurtE/MTP_Teensy).
 
RE: Posts 1188,1190,1191,1192
I'll sync to the latest and confirm when I get back to it ... later today.

> assume Mike/Kurt are running Win 10? : Win 11 here on desktop, will have to try on Win 10 laptop.

Try to make a simpler repro case/steps to some degree. But a pile of files on SD only pulling off a select set should work, unless the storage info is confused somehow where folders a-z and various sub files A-Z get mis mapped?
The PRIMES binary files are just that, will see about adding in to MakeFiles.ino a way to dynamically create a consistent set of those without having to deal with a 30MB ZIP copy. Very odd those files ended up failing to exist after copy.
Have to try Makefiles.ino file set on SD put direct on PC copied back to T_4.1 and then "v - Verify Files"
 
Ok catching up now after the dentist. This is what I currently have on my ext 64gb samsung sd card:
...
as a double check I went into ManyD8 dircectory and drilled down to D7 and opened up the 500 byte file:
500.txt - fast open with Notepad

Would double check to make sure you are using @KurtE's last changes in his _part2 USBHost_36 library and UsbMscFat from Paul's repository list. MTP - don't think there were any changes I am Using @KurtE's MTP_Teensy library (main branch - https://github.com/KurtE/MTP_Teensy).

@mjs513: That looks like an old fileset pre MakeFiles "2204", please Purge and recreate with "t" command on start.
The verify scheme and mechanics of file content have evolved. Old file sets are not reproducible/verifiable - and current Makefiles does not make ManyD8 using "t" - though it can with other options.

Using current: "This PC\Teensy\MakeFiles 2204\ManyD10\D0.5"
It contains files:
Code:
 Directory of D:\tmp\TD_1.57\[B]SDCOPY_2204_16May\ManyD10\D0.5[/B]
05/16/2022  08:34 PM             1,012 1012.txt
05/16/2022  08:34 PM             1,524 1524.txt
05/16/2022  08:34 PM             2,036 2036.txt
05/16/2022  08:34 PM             2,548 2548.txt
05/16/2022  08:34 PM               500 500.txt
05/16/2022  08:58 PM    <DIR>          D1.5
> Each of those 5 files ( with 500 base and added multiple of 512 bytes ) takes 30 seconds to open on Windows 10 from T_4.1 w/MTP Experimental.
--> Not yet updated from github ... gotta run
--> Moved T_4.1 to wife's Win 10 desktop and the same 30 second 'Calculating' appeared for files sized 500 and 2548. {so not my Win 11 machine}
 
RE: Posts 1188,1190,1191,1192
I'll sync to the latest and confirm when I get back to it ... later today.

> assume Mike/Kurt are running Win 10? : Win 11 here on desktop, will have to try on Win 10 laptop.
Yes running Windows 10.... My computer is not Windows 11 compatible... They do not support my processor...
 
Updated to the fs_integration_part2 branch of USBHost_t36.

Seems to have made some change?

> Full disk copy interrupted 5 times with p#1186 Error dialog "... pool is empty"
- > deselected partially copied folders and copied remainder after each failure
- > lots of missing and wrong sized files

Half of the PRIMES folder copied - though most files are wrong length. Screen shot of CodeCompare also shows the length on the p#1194 as copied from MTP_SD to Win11_HDD:
17May_500_Primes_BAD.jpg
> Resulting length of p#1194 5 files 500 bytes + mult of 512 bytes are all wrong from "ManyD10\D0.5"
> below that 120 of the 220 PRIMES files exist by NAME, but NONE are the full files size

Made a batch file to replicate a single BIN file : 01/01/2021 12:00 AM 397,312 P3B391.BIN
to 220 of them total to replicate the 'logging' the T_4.1 did that holds data wanted back on the PC.

> Start T_4.1 and folder "\1_PRIMES" is present - do not open/visit the folder first
>> drag 1_PRIMES to PC_HDD :: Results in EMPTY folder
--> This folder had NOT been opened so T_4.1 had not discovered/reported the directory data?

> Start T_4.1 and folder "\1_PRIMES" is present
> Double click the MTP_SD folder to see there are ONLY 167 files ( one of which is the BAT file ) Showing? {not 221}: This gets the T_4.1 to report the directory to the PC?
>> Select available files, drag files selected in "\1_PRIMES" to PC_HDD:: Results in 167 files that have the correct size and Content, except FOUR are 187.50 KB instead of 388 KB and corrupted
--> 54 of the actual 221 files do not COPY at all to the PC_HDD

If interested to see: {MakeFiles.ino not required}
> Download 1_Primes.zip to PC folder and extract to PC_HDD:
Code:
[ATTACH]28404._xfImport[/ATTACH]

> In folder "1_PRIMES" Double click the "makemore.bat"
--> "FOR /L %%y IN (2, 1, 220) DO copy P3B391.BIN P3B391.%%y.BIN"
> That provides "reference folder" of 221 files to copy to SD card to "\1_PRIMES"
> "Eject" and place that SD card into T_4.1 running 'MTP Experimental'
--> as above 'select and drag' folder "\1_PRIMES" to PC_HDD, before 'visiting' directory
-> is the HDD folder Empty?
--> as above "visit the PC_HDD directory"
-> How many of the 221 files are displayed?
-> Select 'All files' and drag them to an empty PC_HDD folder
--> How many files are present?
-> File/"CodeCompare" copied folder to "reference folder"
--> Are all files exact match?

This process may work with Fewer or Smaller files - but it is something that was 'on hand' from having Teensy do 'some logging work'.

** Focus on SD card due to ease (still a PIA) of moving media back and forth for known file fill and compare. Also, this is likely the fastest and 'common' use case {Native BUILTIN_SDCARD} and once it works - ideally all other media will work when the USB_XFER is working
 
@defragster - @KurtE
Sorry got distracted but think I (we) figured out the issue. Downloaded the Makefile2204.ino from github and ran the example using a 128Gb SD Card and saw the same problems that you identified, i.e.,
1. The library, drive or media pool is empty
2. Looking at the files on the SD Card saw some weirdness in terms of incomplete files or file sizes until I restarted the Teensy.

After some playing a thought struck me that the mtpindex file is getting corrupted by writing so many small files and directories to the SD Card since you are using a single device to hold both the files and index file. So I modified your sketch to create a drive in program memory before the SD Card so the index file would be maintained on a different device (usually do this with my mtp_test_integrity sketch):
Code:
#include "LittleFS.h"
LittleFS_Program lfsProg; // Used to create FS on the Flash memory of the chip
static const uint32_t file_system_size = 1024 * 1024 * 1;
and in setup
Code:
  // Lets add the Program memory version:
  // checks that the LittFS program has started with the disk size specified
  if (lfsProg.begin(file_system_size)) {
    MTP.addFilesystem(lfsProg, "PgmIndx");
  } else {
    Serial.println("Error starting Program Flash storage");
  }

Then I re-ran your Makefiles2204 sketch and all files looked good on the SD Card and then did the transfer to a directory on the PC and all problems that you identified seem to be resolved.

You might want to give this a try. In my testing I transferred about 20Mb of data in 3 directories and about 20 audio files and didn't have a problem going in either direction.
 
I did similar to what @mjs513 did.

Note I ran the original sketch, created files, tried to copy to pc, received same error message. Took ssd to pc, ran chkdsk and showed corruption
, which it fixed.

Modified simple mtp sd sketch to add Pgm disk… and copy worked.
 
@defragster - @KurtE
Sorry got distracted but think I (we) figured out the issue. Downloaded the Makefile2204.ino from github and ran the example using a 128Gb SD Card and saw the same problems that you identified, i.e.,
1. The library, drive or media pool is empty
2. Looking at the files on the SD Card saw some weirdness in terms of incomplete files or file sizes until I restarted the Teensy.

...

I did similar to what @mjs513 did.

Note I ran the original sketch, created files, tried to copy to pc, received same error message. Took ssd to pc, ran chkdsk and showed corruption
, which it fixed.

Modified simple mtp sd sketch to add Pgm disk… and copy worked.

Thanks, that is a good tip and needs attention to work right for dynamic file create and utility.

Seemed something like that {beyond my under the covers inattention} given the 1_Primes folder Not-Visited versus Visited ... though still incomplete - and those file did exist on first appearance of those files given they came preinstalled from PC_create/copy, especially back to the first time this SD came into use where original "PRIMES" has been sitting for about 14 months, and may have pre-existed the first creation of the "mtpindex file"

Is there a way to 'refresh/ mtpindex file'? Sort of like 'myFile.close()' ?

Also, odd even when it worked the p#1196 failed to find 221 files - but logged only 167 of them?

For now I'll look to see if doing this cleans up current 'gross' testing.
 
Made the p#1197 LFS PROG edit and it created that drive.
Removed 'mtpindex file' from SD card while in PC
Programmed T_4.1 with that SD card in it - so on Restart, first seen with MakeFiles.ino with that file on PROG disk.
T_4.1 arrived with PROG device online as well as the SD
> did not execute any menu or change code in MakeFiles.ino - just presented MTP to PC_Host

Copy of 1_Primes completed and seemed good based on Dir data of just that subfolder.
> Indeed ALL files properly compare with CodeCompare

Returned to ROOT and copied All folders to empty PC_HDD folder.
That was at 4:54 PM and now 5:18 PM so it started well and now 24 minutes later it is showing this:
17May_LongCopy.png

Has been showing 1:45 done for past few minutes.
No Error messages in TyComm and no Windows Dialogs other than 'Copying'.

Now minutes later it jumped to '18 minutes' as the progress bar raced to '5 seconds remaining'

Completed without errors, a 'minute' later.
Unpowered T_4.1 and removed SD card.
SD Card in PC Disk Check: No Errors

Made SD copy to PC for File Compare to the MTP_T_4.1 file folder copy. Much better but still some errors. The bottom 'Sys Vol Info' not expected, so okay on that.
> Two PRIMES files are again 387.50 KB and not 388 KB { RIGHT side of image is CORRECT source }
> 500 byte files are stored as 512 Bytes in three folders
> Similar issues in the ManyD10 subfolders for 500 Bytes and those of 500 Bytes + multiple of 512 added bytes.
-->> Visiting two of those files sees them each take 30 seconds to load into Notepad:
> This PC\Teensy\MakeFiles 2204\ManyD10\D0.5\D1.5\D2.5\D3.5\D4.5\D5.5\D6.5\D7.5\D8.5\D9.5\500.txt
>This PC\Teensy\MakeFiles 2204\D0.25\500.txt
17May_PROGidx_Errs.jpg

> still seeing above errors
> mtpindex is fragile and out of sync
 
Back
Top