Teensyduino File System Integration, including MTP and MSC

Quick update: I just merged in my experiment branch of MTP_Teensy back into the main branch.

With this, if you are running on T3.5/3.6 or T4.x - that uses the Record block code. I added a simple Memory FileSystem , which can create exactly one file. Sort of like earlier version of Frank's MemoryFile stuff.
Except this one allocated/reallocates the buffer on the fly when needed.

You can tell the storage system to use it by setting the storage to use to index -2...

But the main reason I added it, was I kept running into cases where for example I have a couple of storage units I add to the MTP list. And lets say the first one was my SD Card adapter, and suppose that the slot is empty. And if the user goes and wants to enumerate the 2nd storage, the system would just fail as it could not create the index file.

This delta detects that the storage file was not created and instead creates the index file on this FS...
Which I think is working pretty well. Note: I left in some debug messages for now, that shows when memory is allocated for this file and when it needs to realloc the memory to grow... An interesting thing is that my current configuration with the record blocks code is keeping a cache of 4 blocks in memory already, and I found for example a SD card of 60+ files, that I still did not actually need to allocate the buffer for this memory file.


I also fixed a few other probable bugs in the MTP code while doing this. When we did a format, the code was not telling the host after it completed to do a reset, but we killed our own cache... Now I have the code after the format completes, send the reset device event to host. Also the reset did not properly clear my high water mark for the block file, so could run into some issues.

There are still some issues that we need to investigate, that we have seen while testing some of this code.

As part of the testing I added the mtp_sd_debug sketch for now... Also it was extended to have 2 SD drives defined.

a) Format of SD cards, does not always properly reset the MTP stuff in some cases. And we see something like:
screenshot.jpg
Which is wrong, but other times it would look correct:
screenshot3.jpg

After @mjs513 found the issue with two 64MB SD cards, but would not replicate on my machine where I was running a 32mb or 16 or 8mb, I tried a 128MB SD in the SDIO and I showed the failure state.

Best Guess Issue: Our code in SD.format() has code in it to try to restart the file system with the new setup. And my guess is that this is not working fully for ExFat on SDIO...


b) Card detection - I am not sure if it is currently not working properly, ie. maybe some parts not implemented with the updated SDFat and/or we are not calling anything to see...
Will try adding code to detect this tomorrow... But I know we would like to have this done automatically without sketch code having to do the checking and...

That is all for now
 
Ran with the updated MTP__MAIN code and SD_MTP_DEBUG.INO

1: First without debug=2, pulled copy MTP to HDD, did 'c' and 'p' - no debug output of course
2: Built SD_MTP_DEBUG.INO with debug=2, pulled another copy MTP to HDD, did 'c' and 'p' - have saved output in txt file.

3: Pulled SD from Teensy to HDD to have a copy of what it holds.

File compare of 3::1 is worse than recent where a single file was bad
File compare of 3::2 is similar?
File compare of 1::2 is not good either.
Seems like the 12 bytes of 500 .vs. 512 has returned?
This is #3::#1 compare, i.e. SD 'true copy' on left and MTP copy #1 on right:
19June12Bytes.jpg

Windows Disk Check gave 768KB file0000.chk - seems to be remains of the index file? But bigger than previous?

That and debug output in this file: View attachment 19JuneMTPd.zip
 
Ran with the updated MTP__MAIN code and SD_MTP_DEBUG.INO
...
Thanks Tim,

Will have to take a look again, but with the Famous Last words: "My changes should not have made any difference here" ;)

The Ram Index file will do some output, so if you see that, then it might be involved. And unless you forced it (-2 FS Index) ... It will only be used if it could not open the index file.
And with that the top level will show RED for the first one... And with previous code double clicking on the second SD card, would have failed...

The debug looks like:
Code:
Failed to open Index file on storage number 0 try memory File
!!!IndexMemFS::open
	$$0x70200028 = malloc(2048)
	$$0x70200028 = realloc(0x70200028, 4096)
	$$0x70200028 = realloc(0x70200028, 6144)
	$$0x70200028 = realloc(0x70200028, 8192)
	$$0x70200028 = realloc(0x70200028, 10240)
	$$0x70200028 = realloc(0x70200028, 12288)
	$$0x70200028 = realloc(0x70200028, 14336)
	$$0x70200028 = realloc(0x70200028, 16384)
	$$0x70200028 = realloc(0x70200028, 18432)
So this run with sd with your 490+ files grew to 18K for the index.

Again if this index was not used. The only changes were related to doing a format of a drive. Which before was wrong. It cleared the index file, but never
told the host about it, so host would ask about object like number 518 and there was nothing left in index for it... So instead I do the OK on format
and then send the Device Reset event, which should start a new session and then that will clear the old index.

And fixed some compile bugs with different options. Found if you set debug=2 it was not compiling... Likewise found if you built for T3.2 it was not building.....

BUT... who knows ...
Maybe need to double check the deltas... All was combined into one change so should be easy to spot if I accidentally mucked up something else.
 
defragster said:
Ran with the updated MTP__MAIN code and SD_MTP_DEBUG.INO

Using my mtp_test_integrity sketch and using an index file in program mem I first ran you makeFiles using the 't' option and verified the files on the Builtin_sd card. I then copied them to "SD1" and then from "SD1" down to my PC. I pulled out the SD Card (from the BUiltIN_SDCARD slot) and did a Compare between the files on the PC and the SD Card - saw no differences.

Now if I reconfigure and use @KurtE's new memFile when I copy files from SDIO to SD1:
Code:
!!!IndexMemFS::open
    >> After mkdir
    >> After OpenFileByIndex
    >> After mkdir
    >> After OpenFileByIndex
	$$0x7000000c = malloc(2048)
	$$0x7000000c = realloc(0x7000000c, 4096)
    >> After mkdir
    >> After OpenFileByIndex
    >> After mkdir
    >> After OpenFileByIndex
    >> After mkdir
    >> After OpenFileByIndex
    >> After mkdir
    >> After OpenFileByIndex
    >> After mkdir
    >> After OpenFileByIndex
	$$0x7000000c = realloc(0x7000000c, 6144)
	$$0x7000000c = realloc(0x7000000c, 8192)
    >> After mkdir
    >> After OpenFileByIndex
    >> After mkdir
    >> After OpenFileByIndex
    >> After mkdir
    >> After OpenFileByIndex
    >> After mkdir
    >> After OpenFileByIndex
    >> After mkdir
    >> After OpenFileByIndex
    >> After mkdir
    >> After OpenFileByIndex
    >> After mkdir
    >> After OpenFileByIndex
    >> After mkdir
    >> After OpenFileByIndex
	$$0x7000000c = realloc(0x7000000c, 10240)
	$$0x7000000c = realloc(0x7000000c, 12288)
    >> After mkdir
    >> After OpenFileByIndex
    >> After mkdir
    >> After OpenFileByIndex
	$$0x7000000c = realloc(0x7000000c, 14336)
	$$0x7000000c = realloc(0x7000000c, 16384)
    >> After mkdir
    >> After OpenFileByIndex
	$$0x7000000c = realloc(0x7000000c, 18432)
    >> After mkdir
    >> After OpenFileByIndex
	$$0x7000000c = realloc(0x7000000c, 20480)
    >> After mkdir
    >> After OpenFileByIndex
    >> After mkdir
    >> After OpenFileByIndex
	$$0x7000000c = realloc(0x7000000c, 22528)
	$$0x7000000c = realloc(0x7000000c, 24576)

and then do a compare of the SDIO card on the PC to the downloaded files from SD1 it still works - no errors:
Capture.PNG
 
Odd not repro at all?

>> will hope to have time to reproduce with updated code.

The file set on SD card was 'probably': makefiles(2206): but unknown cmds :(
> should create and add to a "mfLog.txt" for tracking the 'reproducible' content creation.

<EDIT> Posted with makefiles using LFS Flash not SD
This is SD card summary
Code:
Total 1642 files of Size 14896998 Bytes	 Verify Errors 0	2260.560500 msec
Bytes Used: 225968128, Bytes Total:125062610944

Should also find/add Teensy SD Format for Fresh start in repro steps with MakeFiles.
 
Last edited:
Main branch has that stuff in it...

New branch is a WIP to detect when SD Cards are inserted and removed. It is now detecting, but not sure if certain cases, maybe in SD/SDFat itself are being updated in certain cases.

That is next on my investigate list.
 
@defragster

I am using a 32GB SD Card in SDIO and without using you LFS option. And yes I am using Test Files write 2206 option

I downloaded your latest version and got this error about writing files to the disk

Code:
Make DD File:ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/D7.5/1012	File size=1012=1012	 us=152160    end:zE. 187x  97D89
Make DD File:ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/D7.5/1524	File size=1524=1524	 us=152740    end:zE. 188x  97DE7
Make DD File:ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/D7.5/2036	File size=2036=256	 us=14019788    end: >> WRONG << zE. 189x  97E65

 ================================ Media Size=16777216	Used Size=16777216	 us=21334148   KB/sec=2.67   Bytes written=58412
	Make DeepDir :TEST
	Make Dir failed!TEST/D0.28
Make DD File:TEST/D0.28/495 >> Fail Open << 
Make DD File:TEST/D0.28/496 >> Fail Open << 
Make DD File:TEST/D0.28/497 >> Fail Open << 
Make DD File:TEST/D0.28/498 >> Fail Open << 
Make DD File:TEST/D0.28/499 >> Fail Open << 
Make DD File:TEST/D0.28/500 >> Fail Open << 
Make DD File:TEST/D0.28/501 >> Fail Open << 
Make DD File:TEST/D0.28/502 >> Fail Open << 
Make DD File:TEST/D0.28/503 >> Fail Open << 
Make DD File:TEST/D0.28/504 >> Fail Open << 
Make DD File:TEST/D0.28/505 >> Fail Open << 
Make DD File:TEST/D0.28/506 >> Fail Open << 
Make DD File:TEST/D0.28/507 >> Fail Open << 
Make DD File:TEST/D0.28/508 >> Fail Open << 
Make DD File:TEST/D0.28/509 >> Fail Open << 
Make DD File:TEST/D0.28/510 >> Fail Open << 
Make DD File:TEST/D0.28/511 >> Fail Open << 
Make DD File:TEST/D0.28/512 >> Fail Open << 
Make DD File:TEST/D0.28/513 >> Fail Open << 
Make DD File:TEST/D0.28/514 >> Fail Open << 
Make DD File:TEST/D0.28/515 >> Fail Open << 
Make DD File:TEST/D0.28/516 >> Fail Open << 
Make DD File:TEST/D0.28/517 >> Fail Open << 
Make DD File:TEST/D0.28/518 >> Fail Open << 
Make DD File:TEST/D0.28/519 >> Fail Open << 
Make DD File:TEST/D0.28/520 >> Fail Open << 
Make DD File:TEST/D0.28/521 >> Fail Open << 
Make DD File:TEST/D0.28/522 >> Fail Open << 

 ================================ Media Size=16777216	Used Size=16777216	 us=120120   KB/sec=115.75   Bytes written=14238
	Make DeepDir :
	Make Dir failed!/D0.28
Make DD File:/D0.28/495 >> Fail Open << 
Make DD File:/D0.28/496 >> Fail Open << 
Make DD File:/D0.28/497 >> Fail Open << 
Make DD File:/D0.28/498 >> Fail Open << 
Make DD File:/D0.28/499 >> Fail Open << 
Make DD File:/D0.28/500 >> Fail Open << 
Make DD File:/D0.28/501 >> Fail Open << 
Make DD File:/D0.28/502 >> Fail Open << 
Make DD File:/D0.28/503 >> Fail Open << 
Make DD File:/D0.28/504 >> Fail Open << 
Make DD File:/D0.28/505 >> Fail Open << 
Make DD File:/D0.28/506 >> Fail Open << 
Make DD File:/D0.28/507 >> Fail Open << 
Make DD File:/D0.28/508 >> Fail Open << 
Make DD File:/D0.28/509 >> Fail Open << 
Make DD File:/D0.28/510 >> Fail Open << 
Make DD File:/D0.28/511 >> Fail Open << 
Make DD File:/D0.28/512 >> Fail Open << 
Make DD File:/D0.28/513 >> Fail Open << 
Make DD File:/D0.28/514 >> Fail Open << 
Make DD File:/D0.28/515 >> Fail Open << 
Make DD File:/D0.28/516 >> Fail Open << 
Make DD File:/D0.28/517 >> Fail Open << 
Make DD File:/D0.28/518 >> Fail Open << 
Make DD File:/D0.28/519 >> Fail Open << 
Make DD File:/D0.28/520 >> Fail Open << 
Make DD File:/D0.28/521 >> Fail Open << 
Make DD File:/D0.28/522 >> Fail Open << 

 ================================ Media Size=16777216	Used Size=16777216	 us=120049   KB/sec=115.82   Bytes written=14238
	Make DeepDir :
	Make Dir failed!/D0.25
Make DD File:/D0.25/500 >> Fail Open << 
Make DD File:/D0.25/501 >> Fail Open << 
Make DD File:/D0.25/502 >> Fail Open << 
Make DD File:/D0.25/503 >> Fail Open << 
Make DD File:/D0.25/504 >> Fail Open << 
Make DD File:/D0.25/505 >> Fail Open << 
Make DD File:/D0.25/506 >> Fail Open << 
Make DD File:/D0.25/507 >> Fail Open << 
Make DD File:/D0.25/508 >> Fail Open << 
Make DD File:/D0.25/509 >> Fail Open << 
Make DD File:/D0.25/510 >> Fail Open << 
Make DD File:/D0.25/511 >> Fail Open << 
Make DD File:/D0.25/512 >> Fail Open << 
Make DD File:/D0.25/513 >> Fail Open << 
Make DD File:/D0.25/514 >> Fail Open << 
Make DD File:/D0.25/515 >> Fail Open << 
Make DD File:/D0.25/516 >> Fail Open << 
Make DD File:/D0.25/517 >> Fail Open << 
Make DD File:/D0.25/518 >> Fail Open << 
Make DD File:/D0.25/519 >> Fail Open << 
Make DD File:/D0.25/520 >> Fail Open << 
Make DD File:/D0.25/521 >> Fail Open << 
Make DD File:/D0.25/522 >> Fail Open << 
Make DD File:/D0.25/523 >> Fail Open << 
Make DD File:/D0.25/524 >> Fail Open << 

 ================================ Media Size=16777216	Used Size=16777216	 us=118069   KB/sec=105.87   Bytes written=12800
	Make DeepDir :Huge
	Make Dir failed!Huge/D0.10
Make DD File:Huge/D0.10/400000 >> Fail Open << 
Make DD File:Huge/D0.10/400000 >> Fail Open << 
Make DD File:Huge/D0.10/400000 >> Fail Open << 
Make DD File:Huge/D0.10/400000 >> Fail Open << 
Make DD File:Huge/D0.10/400000 >> Fail Open << 
Make DD File:Huge/D0.10/400000 >> Fail Open << 
Make DD File:Huge/D0.10/400000 >> Fail Open << 
Make DD File:Huge/D0.10/400000 >> Fail Open << 
Make DD File:Huge/D0.10/400000 >> Fail Open << 
Make DD File:Huge/D0.10/400000 >> Fail Open << 

 ================================ Media Size=16777216	Used Size=16777216	 us=112552   KB/sec=34706.18   Bytes written=4000000
MakeData done.

----	Task 't' complete!	----

 DISK FULL?
        
D0.20 /                                                                                                                                                                    
DONE RESULTS GOOD HERE 20 Files(s)	 with 20 files of Size 81920 Bytes	48.768912 ms
        
D0.30 /                                                                                                                                                                                                                                                    
DONE RESULTS GOOD HERE 30 Files(s)	 with 30 files of Size 952320 Bytes	162.355058 ms
        
D0.40 /                                                                                                                                                                                                                                                                                                                                    
DONE RESULTS GOOD HERE 40 Files(s)	 with 40 files of Size 4997120 Bytes	552.729065 ms
        
D0.60 /                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
DONE RESULTS GOOD HERE 60 Files(s)	 with 60 files of Size 3870720 Bytes	634.163818 ms
        
ManyD10 /no count DIR
        
D0.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	6.549532 msec

D1.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	8.648413 msec

D2.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	9.648873 msec

D3.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	11.650767 msec

D4.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	13.651083 msec

D5.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	14.654478 msec

D6.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	16.575843 msec

D7.5 /                        
2036.txt	XXXX	 File name Size 2036 MISMATCH != size of 0:            
	  DONE RESULTS BAD HERE  	XXX 'b XX	  MISSING 1 FILE(S)  
 2 File Error(s)  
	 with 4 files of Size 3036 Bytes	11.821247 ms
                                        
°FILE: mtpindex.dat sz=0    
Total 190 files of Size 9958456 Bytes	 Verify Errors 2	1502.443352 msec
Bytes Used: 16777216, Bytes Total:16777216

----	Task 'v' complete!	----

 DISK FULL?

Doing the same with the previous version of the sketch:
Code:
 ================================ Media Size=31902400512	Used Size=20185088	 us=1512892   KB/sec=2581.98   Bytes written=4000000
MakeData done.

----	Task 't' complete!	----
        
D0.20 /                                                                                                                                                                    
DONE RESULTS GOOD HERE 20 Files(s)	 with 20 files of Size 81920 Bytes	20.717430 ms
        
D0.40 /                                                                                                                                                                                                                                                                                                                                    
DONE RESULTS GOOD HERE 40 Files(s)	 with 40 files of Size 4997120 Bytes	466.443903 ms
        
D0.60 /                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
DONE RESULTS GOOD HERE 60 Files(s)	 with 60 files of Size 3870720 Bytes	379.679783 ms
        
D0.30 /                                                                                                                                                                                                                                                    
DONE RESULTS GOOD HERE 30 Files(s)	 with 30 files of Size 952320 Bytes	103.297283 ms
        
ManyD10 /no count DIR
        
D0.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	4.809175 msec

D1.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	5.001795 msec

D2.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	5.000888 msec

D3.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	4.987115 msec

D4.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	4.981368 msec

D5.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	4.979342 msec

D6.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	4.991235 msec

D7.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	4.993702 msec

D8.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	5.119688 msec

D9.5 /                                            
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	4.864952 ms
                                                
TEST /no count DIR
        
D0.28 /                                                                                                                                                                                                                                    
DONE RESULTS GOOD HERE 28 Files(s)	 with 28 files of Size 14238 Bytes	21.784330 ms
            
D0.28 /                                                                                                                                                                                                                                    
DONE RESULTS GOOD HERE 28 Files(s)	 with 28 files of Size 14238 Bytes	21.277497 ms
        
D0.25 /                                                                                                                                                                                                            
DONE RESULTS GOOD HERE 25 Files(s)	 with 25 files of Size 12800 Bytes	19.418430 ms
        
Huge /no count DIR
        
D0.10 /                                                                                    
DONE RESULTS GOOD HERE 10 Files(s)	 with 10 files of Size 4000000 Bytes	359.086603 ms
            
System Volume Information /no count DIR
        
°FILE: WPSettings.dat sz=12        
°FILE: IndexerVolumeGuid sz=76        
Total 293 files of Size 14019644 Bytes	 Verify Errors 0	1454.572343 msec
Bytes Used: 20185088, Bytes Total:31902400512

----	Task 'v' complete!	----
 
@KurtE - okay will got back to MAIN - misunderstood that code was edited since that last update and newest was that linked branch.

@defragster

I am using a 32GB SD Card in SDIO and without using you LFS option. And yes I am using Test Files write 2206 option

I downloaded your latest version and got this error about writing files to the disk

...

Darn .. I think I saw that the other day and torn with other tasks - I didn't get back to it :(

There are two cases of:
Code:
    myFile = DISK.open(szPath, FILE_WRITE);
    if ( !myFile ) {
      Serial.print(" >> Fail Open << \t <<<<<<");
      break;
    }

For creation and it seems one of them got broken when I changed who does path printing where to clean up for the MakeRoot case ...

...
 
Mike, ...

Opps - seems it was fixed here and working - just not updated. :rolleyes:

github.com/Defragster/T4LockBeta/tree/main/MakeFiles

TESTING: 't', 'b', 's' cmds results in no create errors and no 'v'erify errors:
Code:
Total 1167 files of Size 14448850 Bytes	 Verify Errors 0	1996.544283 msec
Bytes Used: 164233216, Bytes Total:125062610944

And : "W"ipe, 't', 'r', 'b', 's', 'v' :gives no errors and:
Code:
Total 1657 files of Size 14497850 Bytes	 Verify Errors 0	2239.919350 msec
Bytes Used: 228458496, Bytes Total:125062610944
 
Mike, ...

Opps - seems it was fixed here and working - just not updated. :rolleyes:

github.com/Defragster/T4LockBeta/tree/main/MakeFiles

TESTING: 't', 'b', 's' cmds results in no create errors and no 'v'erify errors:
Code:
Total 1167 files of Size 14448850 Bytes	 Verify Errors 0	1996.544283 msec
Bytes Used: 164233216, Bytes Total:125062610944

And : "W"ipe, 't', 'r', 'b', 's', 'v' :gives no errors and:
Code:
Total 1657 files of Size 14497850 Bytes	 Verify Errors 0	2239.919350 msec
Bytes Used: 228458496, Bytes Total:125062610944

Ok that worked
Code:
 ================================ Media Size=31902400512	Used Size=20086784	 us=1078072   KB/sec=3623.37   Bytes written=4000000
MakeData done.

----	Task 't' complete!	----
        
D0.20 /                                                                                                                                                                    
DONE RESULTS GOOD HERE 20 Files(s)	 with 20 files of Size 81920 Bytes	20.550878 ms
        
D0.40 /                                                                                                                                                                                                                                                                                                                                    
DONE RESULTS GOOD HERE 40 Files(s)	 with 40 files of Size 4997120 Bytes	448.074238 ms
        
D0.60 /                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
DONE RESULTS GOOD HERE 60 Files(s)	 with 60 files of Size 3870720 Bytes	365.986132 ms
        
D0.30 /                                                                                                                                                                                                                                                    
DONE RESULTS GOOD HERE 30 Files(s)	 with 30 files of Size 952320 Bytes	99.561345 ms
        
ManyD10 /no count DIR
        
D0.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	4.931843 msec

D1.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	5.006277 msec

D2.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	4.997070 msec

D3.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	4.977583 msec

D4.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	4.994243 msec

D5.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	4.970763 msec

D6.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	4.998930 msec

D7.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	4.995590 msec

D8.5 /                                                
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	5.124850 msec

D9.5 /                                            
DONE RESULTS GOOD HERE 5 Files(s)	 with 5 files of Size 7620 Bytes	4.845907 ms
                                                
TEST /no count DIR
        
D0.28 /                                                                                                                                                                                                                                    
DONE RESULTS GOOD HERE 28 Files(s)	 with 28 files of Size 14238 Bytes	22.177132 ms
            
D0.28 /                                                                                                                                                                                                                                    
DONE RESULTS GOOD HERE 28 Files(s)	 with 28 files of Size 14238 Bytes	22.265605 ms
        
D0.25 /                                                                                                                                                                                                            
DONE RESULTS GOOD HERE 25 Files(s)	 with 25 files of Size 12800 Bytes	19.411712 ms
        
Huge /no count DIR
        
D0.10 /                                                                                    
DONE RESULTS GOOD HERE 10 Files(s)	 with 10 files of Size 4000000 Bytes	346.617738 ms
        
Total 291 files of Size 14019556 Bytes	 Verify Errors 0	1405.560870 msec
Bytes Used: 20086784, Bytes Total:31902400512

----	Task 'v' complete!	----

not sure how you getting 1642 files on SD Card - what are you running.

EDIT: I am only testing with 't" option -Test Files write 2206
 
@defragster - ok you answered my question as I was typing it

Code:
Total 1656 files of Size 14497750 Bytes	 Verify Errors 0	2492.506248 msec
Bytes Used: 67928064, Bytes Total:31902400512

----	Task 'v' complete!	----
Running your 't', 's', 'b' and 'v' commands I get only 1656 while you show 1657????
 
@mjs513 : That was running on 16MB FLASH: Media Size=16,777,216

================================ Media Size=16777216 Used Size=16777216 us=112552 KB/sec=34706.18 Bytes written=4000000

It ran out of ROOM! See:
Code:
[B]#define USE_SDIO_SD[/B]
//#define USE_PSRAM
//#define TEST_QSPI // Typical NOR FLASH
//#define TEST_QSPI_NAND // NAND Flash

Latest code update will have the indication of 'Disk Full' after create CMD {if remaining under 2048 bytes}. And it ABORTS added creations at that point.
> I could add an "if ( Abort )" test after at that point as well as a Large create can fail with over 2048 bytes left.

It also prints what MEDIA in use - but those messages are lost in setup() before:
Code:
...
  delay(2000);
  Serial.println("initialization done.");
 
@KurtE - @defragster

Just tried coping those 1656 files from SD1 to SD2 using mtp_test_integrity 3 different ways - ExtMem, PrgMem and MemFile for where the index file is located.

All three ways failed to copy the files - only about 32MB of 64MB was able to be written before Windows gave me the Beeping sound and MTP stop transferring files. Didn't see anything on why in with the Debug turned on either:

This was the last messages I saw in the monitor before it died:
Code:
loop:97483 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:25f1 : d75 dc08 (CREATED)
97483 RESP:2001(RSP:OK)l: 12 T:25f1
loop:97483 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:25f2 : d75 dc09 (MODIFIED)
97483 RESP:2001(RSP:OK)l: 12 T:25f2
loop:97484 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:25f3 : d76 dc04 (SIZE)
97484 RESP:2001(RSP:OK)l: 12 T:25f3
loop:97484 CMD: 1008(GET_OBJECT_INFO)l: 16 T:25f4 : d76
97484 RESP:2001(RSP:OK)l: 12 T:25f4
loop:97485 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:25f5 : d76 dc08 (CREATED)
97485 RESP:2001(RSP:OK)l: 12 T:25f5
loop:97485 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:25f6 : d76 dc09 (MODIFIED)
97485 RESP:2001(RSP:OK)l: 12 T:25f6
loop:97486 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:25f7 : d77 dc04 (SIZE)
97486 RESP:2001(RSP:OK)l: 12 T:25f7
loop:97486 CMD: 1008(GET_OBJECT_INFO)l: 16 T:25f8 : d77
97486 RESP:2001(RSP:OK)l: 12 T:25f8
loop:97486 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:25f9 : d77 dc08 (CREATED)
97486 RESP:2001(RSP:OK)l: 12 T:25f9
loop:97487 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:25fa : d77 dc09 (MODIFIED)
97487 RESP:2001(RSP:OK)l: 12 T:25fa
loop:97487 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:25fb : d78 dc04 (SIZE)
97487 RESP:2001(RSP:OK)l: 12 T:25fb
loop:97487 CMD: 1008(GET_OBJECT_INFO)l: 16 T:25fc : d78
97488 RESP:2001(RSP:OK)l: 12 T:25fc
loop:97488 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:25fd : d78 dc08 (CREATED)
97488 RESP:2001(RSP:OK)l: 12 T:25fd
loop:97488 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:25fe : d78 dc09 (MODIFIED)
97488 RESP:2001(RSP:OK)l: 12 T:25fe
 
Problem was using LFS FLASH that was too small. With command 'b'ig?
> except testing here used size is only 9MB of 64MB ????

Back on 64MB LFS Flash I see those errors - that do NOT repro on SD using the same 'b' command with the same code ???

Gotta run ... but using SD card that should work
 
This isn't right - in moving path define this one got mismanaged:

Works - wrong on SD - should be making unique files in a folder - not re-using file name:
Code:
 ================================ Media Size=125062610944	Used Size=393216
Make DF File:  ManyF/111.128.txt	File size=100=100	 us=3033    end:za.   1x      4
Make DF File:  ManyF/111.128.txt	File size=101=101	 us=7029    end:za.   2x      9
Make DF File:  ManyF/111.128.txt	File size=102=102	 us=19873    end:za.   3x      E
Make DF File:  ManyF/111.128.txt	File size=103=103	 us=6980    end:za.   4x     13

Fails on LFS Flash:
Code:
 ================================ Media Size=67108864	Used Size=131072
Make DF File: >> iDW Fail Open << 	ManyF/111.128.txt
Make DF File: >> iDW Fail Open << 	ManyF/111.128.txt
Make DF File: >> iDW Fail Open << 	ManyF/111.128.txt
Make DF File: >> iDW Fail Open << 	ManyF/111.128.txt
Make DF File: >> iDW Fail Open << 	ManyF/111.128.txt

Verify compalins ... but about file size - not count as DirName malformed:
Code:
111.128 /            
111.128.txt	XXXX	 File name Size 111 MISMATCH != size of 227:

... arggh ... gotta run
 
Sorry I have been distracted... I am currently hacking on trying to get SD media detection working without having the sketch have to add their own code to it...

That part appears to be working, although it is pretty specific to SD and at some point maybe want more generic solution...

But in addition to this just trying to get get the system to work when we detect an SD has removed and likewise when it has been inserted And we have detected it.

For example with released B3, and you have code that detects it, like in some of our sketches (or this stuff) and we send out a MTP Reset event. MTP then calls back to
us asking for information again, which drives... And the total size and used/free size...
And the updated SDFat code appears to be returning the old cached data...

I put a hack into SD to get around this:
Code:
int64_t usedSize() {
		if (!cardPreviouslyPresent) return (uint64_t)0;
		return (uint64_t)(sdfs.clusterCount() - sdfs.freeClusterCount())
		  * (uint64_t)sdfs.bytesPerCluster();
	}
	uint64_t totalSize() {
		if (!cardPreviouslyPresent) return (uint64_t)0;
		return (uint64_t)sdfs.clusterCount() * (uint64_t)sdfs.bytesPerCluster();
	}
If we don't think the drive is there return 0...

But, I then tried with I had a 32MB Fat32 volume installed with 490+ files in it, which I had enumerated. I then removed it... I then inserted a different disk 128mb with very little on it.
I then tried to enumerate it and the code faulted:
Code:
loop:125199 CMD: 9801(GET_OBJECT_PROPS_SUPPORTED)l: 16 T:7 : 3001
125199 RESP:2001(RSP:OK)l: 12 T:7
loop:132933 CMD: 1007(GET_OBJECT_HANDLES)l: 24 T:8 : 10001 0 ffffffff
CrashReport:
  A problem occurred at (system time) 14:4:29
  Code was executing from address 0x3FF0
  CFSR: 8200
	(PRECISERR) Data bus error(address in BFAR)
	(BFARVALID) Accessed Address: 0x2001F
  Temperature inside the chip was 47.00 °C
  Startup CPU clock speed is 600MHz
  Reboot was caused by auto reboot after fault or bad interrupt detected
Added SD card using built in SDIO, or given SPI CS
Add **SDClass** file system
Added SD2 card using built in SDIO, or given SPI CS
Add **SDClass** file system
Add **FS** file system

Code:
C:\Users\kurte\AppData\Local\Temp\arduino_build_SD_MTP_debug.ino>addr2line -e SD_MTP_debug.ino.elf 0x3FF0
C:\Users\kurte\Documents\Arduino\libraries\SdFat\src\FatLib/FatPartition.h:218
Code:
 bool syncDevice() {
[COLOR="#FF0000"]    return m_blockDev->syncDevice();[/COLOR]
  }

But again this is an ExFat partition now so that is not right...
My guess is the code in bool SDClass::mediaPresent()
that does: ret = sdfs.restart();

That restart() may not be functioning like it was before SDFat update to latest...
 
This is with 'b'ig create - not designed to work on LFS as written.

Updated with added process debug notes noting fail to create DIR. Detected fail to create DIR and exited without streaming cascading failures.
github.com/Defragster/T4LockBeta/tree/main/MakeFiles

Problem is on LFS folders cannot be two deep?
Code:
 ================================ Media Size=67108864	Used Size=8519680	 us=8713146   KB/sec=0.00   Bytes written=0
	Make Dir failed!ManyF/111.128
...
	Make Dir failed!ManyF/111.124

This works on SD card:
Code:
 ================================ Media Size=125062610944	Used Size=393216
	Made DF Dir:ManyF/111.128
Make DF File:  ManyF/111.128/100.txt	File size=100=100	 us=4532    end:za.   1x      4
 
...
Code:
C:\Users\kurte\AppData\Local\Temp\arduino_build_SD_MTP_debug.ino>addr2line -e SD_MTP_debug.ino.elf 0x3FF0
C:\Users\kurte\Documents\Arduino\libraries\SdFat\src\FatLib/FatPartition.h:218
Code:
 bool syncDevice() {
[COLOR="#FF0000"]    return m_blockDev->syncDevice();[/COLOR]
  }

But again this is an ExFat partition now so that is not right...
My guess is the code in bool SDClass::mediaPresent()
that does: ret = sdfs.restart();

That restart() may not be functioning like it was before SDFat update to latest...

Did some more tracing... Debug code and the like.
And sure enough issues with: OK SD was removed, and the FS is no longer valid, I put in code to call end() on the FS, to see if that helps...

So then we plug in new SD, and then ask it to enum...
Well the cached file (_file) was pointing to the root of that drive and when it tries to rebuild the index. The enum code wants to open a new file
... And first it tries to close whatever what opened there before.

and the SDFat FatFile::close, then decides to sync the device and flush out any caches... But the data is garbage and screwy things happen including crash.

I know we sort of solved this before, but...
 
@KurtE: would having an 'Unmount/Safe Remove' option allow closing disk files and then pulling the MTP drive and doing MTP.reset be a cleaner or good option?

@defragster - ok you answered my question as I was typing it

Code:
Total 1656 files of Size 14497750 Bytes	 Verify Errors 0	2492.506248 msec
Bytes Used: 67928064, Bytes Total:31902400512

----	Task 'v' complete!	----
Running your 't', 's', 'b' and 'v' commands I get only 1656 while you show 1657????

Cool there was an answer!

The Off by 1: maybe the .dat file, or when Windows formats it leaves its hidden folder and file, or a leftover FOUNDxxxx.CHK file from disk repair?

This github.com/Defragster/T4LockBeta/tree/main/MakeFiles should be cleaner and have some details info shown.
Added 'L'og display of CMD's issued, where each command executed is recorded now in file: mfLog.txt. Can also open from MTP drive to see what created the image at hand.
 
This is with 'b'ig create - not designed to work on LFS as written.

Updated with added process debug notes noting fail to create DIR. Detected fail to create DIR and exited without streaming cascading failures.
github.com/Defragster/T4LockBeta/tree/main/MakeFiles

Problem is on LFS folders cannot be two deep?
......

This works on SD card:...

Not sure that statement is correct that QSPI Flash can only write 2 dir deep. I just ran your updated test sketch using the 't' command and you can see clearly it created the several levels of directories until the disk got full on the 16MB QSPI flash I have on the T4.1. It looks like it wrote about 190 of the 290+ files before the disk became full. More than likely its because LFS using the Larger Block sizes of 32K so you are not going to be able to put a ton of small files on them before it get full. This was discussed previously.

Code:
Make DD File:  ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/1012.txt	File size=1012=1012	 us=147217    end:zE. 177x  979DD
Make DD File:  ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/1524.txt	File size=1524=1524	 us=148308    end:zE. 178x  97A3B
Make DD File:  ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/2036.txt	File size=2036=2036	 us=152498    end:zE. 179x  97AB9
Make DD File:  ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/2548.txt	File size=2548=2548	 us=150506    end:zE. 180x  97B57
Make DD File:  ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/500.txt	File size=500=500	 us=155296    end:zE. 181x  97B75
Make DD File:  ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/1012.txt	File size=1012=1012	 us=154970    end:zE. 182x  97BB3
Make DD File:  ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/1524.txt	File size=1524=1524	 us=151133    end:zE. 183x  97C11
Make DD File:  ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/2036.txt	File size=2036=2036	 us=151481    end:zE. 184x  97C8F
Make DD File:  ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/2548.txt	File size=2548=2548	 us=153773    end:zE. 185x  97D2D
Make DD File:  ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/D7.5/500.txt	File size=500=500	 us=153280    end:zE. 186x  97D4B
Make DD File:  ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/D7.5/1012.txt	File size=1012=1012	 us=154179    end:zE. 187x  97D89
Make DD File:  ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/D7.5/1524.txt	File size=1524=1524	 us=156002    end:zE. 188x  97DE7
Make DD File:  ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/D7.5/2036.txt	File size=2036=256	 us=14020298    end: >> WRONG << zE. 189x  97E65
	========================= Media Size=16777216	Used Size=16777216	 us=21016835   KB/sec=2.71   Bytes written=58412
	Make DeepDir :TEST


	!!!!  PRIOR OPERATION WAS ABORTED !!!!!!!
	========================= Media Size=16777216	Used Size=16777216	 us=108554
	Make DeepDir :


	!!!!  PRIOR OPERATION WAS ABORTED !!!!!!!
	========================= Media Size=16777216	Used Size=16777216	 us=306
	Make DeepDir :


	!!!!  PRIOR OPERATION WAS ABORTED !!!!!!!
	========================= Media Size=16777216	Used Size=16777216	 us=251
	Make DeepDir :Huge


	!!!!  PRIOR OPERATION WAS ABORTED !!!!!!!
	========================= Media Size=16777216	Used Size=16777216	 us=108196
MakeData done.

----	Task 't' complete!	========================= Media Size=16777216	Used Size=16777216	 us=1306

 DISK FULL?


	!!!!  PRIOR OPERATION WAS ABORTED !!!!!!!
 
Not sure that statement is correct that QSPI Flash can only write 2 dir deep. I just ran your updated test sketch using the 't' command and you can see clearly it created the several levels of directories until the disk got full on the 16MB QSPI flash I have on the T4.1. It looks like it wrote about 190 of the 290+ files before the disk became full. More than likely its because LFS using the Larger Block sizes of 32K so you are not going to be able to put a ton of small files on them before it get full. This was discussed previously.

...

There was a note way back about depth support on 'some media' : perhaps mis-remembered, or it changed ... in scanning it seemed others had depth, but not seen just then.

Though still the same 'creation' works on SD that fails there. Adding path prints and log DIR creation path, or fail it showed SD could add path, but LFS_Flash could not ?

Flash on this T_4.1 is 64MB - so not running out of space like a 16MB would.

... have a much ignored external task that needs attention - would rather do this, or go to the dentist :confused: ... but it is due ...
 
Did some more tracing... Debug code and the like.
And sure enough issues with: OK SD was removed, and the FS is no longer valid, I put in code to call end() on the FS, to see if that helps...

and the SDFat FatFile::close, then decides to sync the device and flush out any caches... But the data is garbage and screwy things happen including crash.

I know we sort of solved this before, but...

This morning I pushed up a change to MTP_Teensy (try_auto_detect_sd_media branch) which appears to be working better with the Have my 32mb sd card in SD1, start up sketch
double click to enumerate the lots of files... Pull it out after enumeration done, insert a 32mb SD card, go back to Teensy root, and then double slick on SD1 and it enumerate the new SD
card, which I like a little better ;) than faulting the Teensy.

I also have a few changes to the SD library which help some, but not sure if needed...

That is, I changed:
Code:
	uint64_t usedSize() {
		[COLOR="#FF0000"]if (!cardPreviouslyPresent) return (uint64_t)0;[/COLOR]
		return (uint64_t)(sdfs.clusterCount() - sdfs.freeClusterCount())
		  * (uint64_t)sdfs.bytesPerCluster();
	}
	uint64_t totalSize() {
		[COLOR="#FF0000"]if (!cardPreviouslyPresent) return (uint64_t)0;[/COLOR]
		return (uint64_t)sdfs.clusterCount() * (uint64_t)sdfs.bytesPerCluster();
	}
So if the begin failed or if the last detect said no device, then return 0 for these two...
Wonder if many of the other wrapper functions should do the same.

Also in the function
Code:
bool SDClass::mediaPresent()
{
	//Serial.print("mediaPresent: ");
	bool ret;
	SdCard *card = sdfs.card();
//	Serial.printf("mediaPresent: card:%x cs:%u cd:%u\n", (uint32_t)card, csPin_, cdPin_);
	if (card) {
		if (cardPreviouslyPresent) {
			#ifdef BUILTIN_SDCARD
			uint32_t s;
			if (csPin_ == BUILTIN_SDCARD) {
				#if defined(__MK64FX512__) || defined(__MK66FX1M0__)
				card->syncDevice();
				#endif  // defined(__MK64FX512__) || defined(__MK66FX1M0__)
				s = card->status();
			} else s = 0xFFFFFFFF;
			#else
			const uint32_t s = 0xFFFFFFFF;
			#endif
			if (s == 0xFFFFFFFF) {
				// see if we have digital pin to bypass...
				if (cdPin_ < NUM_DIGITAL_PINS) ret = digitalRead(cdPin_);
				else {
					// SPI doesn't have 32 bit status, read CID register
					cid_t cid;
					ret = card->readCID(&cid);
				}
				//Serial.print(ret ? "CID=ok" : "CID=unreadable");
			} else if (s == 0) {
				// assume zero status means card removed
				// bits 12:9 are card state, which should
				// normally be 101 = data transfer mode
				//Serial.print("status=offline");
				ret = false;
				[COLOR="#FF0000"]sdfs.end();[/COLOR]
				#ifdef _SD_DAT3
				if (csPin_ == BUILTIN_SDCARD) 
					pinMode(_SD_DAT3, INPUT_PULLDOWN);
				#endif
			} else {
...
Which I am not sure if it helped or not.

I have not pushed these SD functions to github, but could...
 
There was a note way back about depth support on 'some media' : perhaps mis-remembered, or it changed ... in scanning it seemed others had depth, but not seen just then.

Though still the same 'creation' works on SD that fails there. Adding path prints and log DIR creation path, or fail it showed SD could add path, but LFS_Flash could not ?

Flash on this T_4.1 is 64MB - so not running out of space like a 16MB would.

... have a much ignored external task that needs attention - would rather do this, or go to the dentist :confused: ... but it is due ...

@defragster - here is a snapshot of my flash after I ran the t option and it failed because disk was full. NOTE the many sub directories that were created:
Capture.PNG
kind of shows that LFS can create sub-directories. Yes I believe at one time there was an issue but it has since been fixed for quite a while.
 
Back
Top