Teensyduino File System Integration, including MTP and MSC

Did a sync of the latest - just the MTP sources? Saw this new bad behavior:

500.txt files of 500 bytes - and those of 500B + multiple of 512B all ended up on PC from T_4.1 copy with 12 bytes of garbage chars.

And the SD2202 transfer that was under 4 minutes took over 35 minutes to complete.

<edit>
-Copy of a single 500.txt file seems to work.
-Folder 'D9' copied with one copy of only 500.txt works
-DID Repro the following copy just D9 folder from Second SD - so not a faulty or slow SD card.

But go to: This PC\Teensy\SD Card\SD2202\ManyD10\D0\D1\D2\D3\D4\D5\D6\D7\D8
Copy D9 from Teensy to a PC Folder.
> Takes TOO long timewise
> Did this a second time with same result - SerMon spew below.
> PC folder copy of D9 looks like this - each '.txt' file 12 bytes too big:
Code:
 Directory of D:\tmp\TD_1.56\8Jan\a\D9
01/07/2022  10:31 PM             1,024 1012.txt
01/07/2022  10:31 PM             1,536 1524.txt
01/07/2022  10:31 PM             2,048 2036.txt
01/07/2022  10:31 PM             2,560 2548.txt
01/07/2022  10:31 PM               512 500.txt
Here is repro SerMon output:
Code:
loop:7565628 CMD: 1009(GET_OBJECT)l: 16 T:9a9e : cf4
7565639 RESP:2001(RSP:OK)l: 12 T:9a9e
MTP_class::Loop usb_mtp_status 19 != 0x1 reset
loop:7596115 CMD: 1009(GET_OBJECT)l: 16 T:9a9f : cf0
7596126 RESP:2001(RSP:OK)l: 12 T:9a9f
MTP_class::Loop usb_mtp_status 19 != 0x1 reset
loop:7626661 CMD: 1009(GET_OBJECT)l: 16 T:9aa0 : cf1
7626671 RESP:2001(RSP:OK)l: 12 T:9aa0
MTP_class::Loop usb_mtp_status 19 != 0x1 reset
loop:7657224 CMD: 1009(GET_OBJECT)l: 16 T:9aa1 : cf2
7657234 RESP:2001(RSP:OK)l: 12 T:9aa1
MTP_class::Loop usb_mtp_status 19 != 0x1 reset
loop:7687812 CMD: 1009(GET_OBJECT)l: 16 T:9aa2 : cf3
7687822 RESP:2001(RSP:OK)l: 12 T:9aa2
MTP_class::Loop usb_mtp_status 19 != 0x1 reset
 
Last edited:
Just now committed a fix for ZLP transmit, which should fix this problem.

https://github.com/KurtE/MTP_Teensy/commit/a87873cafdafa6e784af535770bdefd8caa30e08

Those missing ZLP also cause a massive slowdown. I believe you'll see the speed is much better with this fix. :)

Sounds great... Is it time to start playing yet :D

Since you were/are busy with the MTP_Teensy files, I thought I would again take a look again at how it uses the Storage Index file... I have already posted and emailed about some of the things I believe should be enhanced or reworked.

But thought I might first look at a few basic things and wondering how best address:

Example: If we look at the Storage Record we have:
Code:
#define MAX_FILENAME_LEN 256
...
	struct Record {
	  uint32_t parent;
	  uint32_t child; // size stored here for files
	  uint32_t sibling;
	  uint32_t dtModify;
	  uint32_t dtCreate;
	  uint8_t isdir;
	  uint8_t scanned;
	  uint16_t store; // index int physical storage (0 ... num_storages-1)
	  char name[MAX_FILENAME_LEN];
	};
So right now every record is 280 bytes long, and if we are writing these out to one or more of our stores like SD one at a time, 280 is not a multiple of sector or cluster so ...

First question MAX_FILENAME_LEN is 256 what does this correspond to? And how is it used?

Yes on Windows a file name component can be 255 characters long, and before current windows 10 stuff, MAXPATHLEN was 260. With more recent versions of windows 10, at least optionally this restriction of 260 has been removed...
Have not looked yet at MAC or Linux...

Note: there are several places in the code, like:
Code:
void MTPStorage::OpenFileByIndex(uint32_t i, uint32_t mode)
{
	bool file_is_open = sd_isOpen(file_);  // check to see if file is open
	if (file_is_open && (open_file_ == i) && (mode_ == mode)) {
		return;
	}
	char filename[MAX_FILENAME_LEN];
That if a storage item actually had a file name of the max length, we could not build the path name here as we want to start of with /
i.e. the 260 in windows was to handle: c:\<255 charcters><\0>

Obviously we could add in fudge here and again have MAX_FILENAME_LEN and MAX_PATH_LEN

My guess is that on average file names are probably less than 32 bytes long, So lots of waste in the index list.

Also: then you look at some of our FS Implementations like LittleFS. The code is hard coded that the full path is 128 bytes.

Example:
Code:
virtual File openNextFile(uint8_t mode=0) {
		if (!dir) return File();
		struct lfs_info info;
		do {
			memset(&info, 0, sizeof(info)); // is this necessary?
			if (lfs_dir_read(lfs, dir, &info) <= 0) return File();
		} while (strcmp(info.name, ".") == 0 || strcmp(info.name, "..") == 0);
		//Serial.printf("ONF::  next name = \"%s\"\n", info.name);
		char pathname[128];
		strlcpy(pathname, fullpath, sizeof(pathname));
		size_t len = strlen(pathname);
		if (len > 0 && pathname[len-1] != '/' && len < sizeof(pathname)-2) {
			// add trailing '/', if not already present
			pathname[len++] = '/';
			pathname[len] = 0;
		}
		strlcpy(pathname + len, info.name, sizeof(pathname) - len);

With SD and SDFat - we have LFN turned on so each element can be 255 characters long, Not sure if there is anywhere in the library code that builds a pathname... So may not be issue.
I would assume same for MSC except have not looked yet to see if code builds pathnames from components...

So simple questions come to mind:
a) Should FS class have members to allow us to know what max file and path lengths are?
b) should LittleFS 128 byte be updated?
c) Should MTP handle maximums? both path and filename?

d) In the first pass, would it be OK to cheat and limit slightly:
That is suppose I did something like:
Reduce the MAX_FILENAME_LEN to something like:
Code:
#define MAX_FILENAME_LEN (256-24)
Then with no other changes the record would be 256 bytes so at least it would map to normal fs sizes... Would be curious to see how that might change how much time is spent reading/writing the index file...

Could probably reduce the 28 bytes down, example maybe limit the number of files we will handle per storage to 65536 (maybe 65535) and we generate: the actual object ids using the numbers stored here plus:
combine store, scanned, isdir into one uint8_t. Could reduce this overhead to by maybe about 9 bytes... Might try that later..

Maybe first will instrument the code to see how often we do reads/writes to underlying FS try caching N records (maybe 8 records). Note: might not cache on low memory machines like tlc/32... And then see
how much this might cut down on the number of read/writes to fs...

Sorry probably too much rambling here.
 
A little hint, that might be useful (or not..)

GCC supports arrays like this:

char buf[strlen(filename)+1];

So, you can define buffers with the exact size, do not waste anything (on the stack), and it is a way to prevent overflows. The size does _not_ have to be constant at compile-time as long it is on the stack.
For a strlcat() for eaxample, ust calculate the size before defining the array..
For the struct Record, however, this is not possible. But i guess there is a clever way to do it, by just spending a byte or two to store the length.. (like Arduino strings, or better pascal-like strings)

+ just by just storing the string with the actual length after the record.
 
Last edited:
char buf[strlen(filename)+1];

I don't want to start a religion war but VLAs are discussed quite controversial. The main argument against them goes like this: If you don't know the size at compile time VLAs might generate a stack overflow. So, you probably know the max len or want to restrict the array size to some max value. In this case you can as well use an array with the max size directly.

I personally like the simple syntax but never dared to use them ;-) For stuff like filenames the chances for a stack overflow are probably negligible anyway.

Edit: There is also void *alloca(size_t size) which allocates memory like malloc but uses the stack. Same arguments apply...
 
It needs _always_ less stack than allocating the max len every time :)

But yes, the correct way would be to use the heap.
The Ardunio disciples' unfounded fear of this goes back to the AVRs with a few hundred bytes of RAM.
And today they forget that _no_ fragmentation occurs if you do it right and put free() after malloc() without having a "permanent" malloc() in between - which is almost never the case anyway, and there is a way to work around it in most cases if you don't get too stupid.
 
Is it time to start playing yet :D

Yes. I'm done making major changes in MTP_Teensy.cpp. Well, with the caveat that the event stuff might get similar work in a few weeks. But major changes that break github merge should be settled down for now.

You'll see MTP_Teensy.cpp looks very different than a week ago. Almost all the code is now hardware independent, except for just a few low-level USB functions. I put all the high-level stuff at the top, then the major MTP commands, followed by the boilerplate stuff, a layer for platform independent read/write functions, and near the end the low-level USB stuff. Hopefully this new structure makes the code easier to read, but it might take a little adjustment if you were used to seeing all the T3 stuff followed by almost everything duplicated for T4.

The old TRANSMIT macro is gone. Now every MTP command is serviced by a function that just runs once. Everything done for each MTP command can now be read as just a normal straight flow from start to end within its function. For some of them, like GetObjectPropValue(), this does add many lines to compute the data size which will be transmitted and get the data ready before sending.

I added comments in the main loop() function to document the container input/output that gets passed to all commands, and the expected format of "return_code" (which now uses 20 bits, 16 for MTP response code and 4 to specify the number of params to transit in the response). I also put comments above each command function with a quick summary of the cmd param, data phase, and response params, with the MTP spec page numbers to get more detail.

Hopefully all this will make the code easier for anyone to read and understand.



First question MAX_FILENAME_LEN is 256 what does this correspond to? And how is it used?
....
My guess is that on average file names are probably less than 32 bytes long, So lots of waste in the index list.
....
Also: then you look at some of our FS Implementations like LittleFS. The code is hard coded that the full path is 128 bytes.

Yeah, we currently have inconsistent maximum filename lengths in LittleFS, SdFat, MTP, and USB MSC. I'm personally not too worried about this, as long as we properly truncate to avoid buffer overflows. But if everyone wants to talk of filename lengths, go right ahead. Maybe at some point we'll "standardize" all these libraries on the same maximum size.
 
Just now committed a fix for ZLP transmit, which should fix this problem.

https://github.com/KurtE/MTP_Teensy/commit/a87873cafdafa6e784af535770bdefd8caa30e08

Those missing ZLP also cause a massive slowdown. I believe you'll see the speed is much better with this fix. :)

Indeed those 500 byte anomalies are GONE!

Speed good - back to 4 minutes total to HDD not the 35 minute problem. ALSO to RAMDRIVE it was under 2 minutes with SD2202 image on fresh start including the DirWalk of ~40 seconds!

The DIRWALK is suspect as one folder created but no populated in either SD2201 or SD2202 :: This PC\Teensy\SD Card\SD2202\ManyD10\D0\D1\D2\D3\D4\D5\D6\D7\D8\D9
> For that 10th folder D9, the files are not copied
> Walking down the PATH on MTP Teensy there is NO SerMon spew until that folder is reached then the files in that folder are 'Discovered' it seems:
Code:
loop:376886 CMD: 1007(GET_OBJECT_HANDLES)l: 24 T:9a60 : 10001 0 d0
  >> 3312: 0 0 0 208 0 1012 1641594672 1641669816 1012.txt
  >> 3313: 0 0 0 208 3312 1524 1641594672 1641669816 1524.txt
  >> 3314: 0 0 0 208 3313 2036 1641594672 1641669816 2036.txt
  >> 3315: 0 0 0 208 3314 2548 1641594672 1641669816 2548.txt
  >> 3316: 0 0 0 208 3315 500 1641594672 1641669816 500.txt
376921 RESP:2001(RSP:OK)l: 12 T:9a60
loop:376922 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:9a61 : cf4 dc02 (FORMAT)
376922 RESP:2001(RSP:OK)l: 12 T:9a61
loop:376923 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:9a62 : cf4 dc03 (PROTECTION)
376923 RESP:2001(RSP:OK)l: 12 T:9a62
loop:376923 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:9a63 : cf4 dc04 (SIZE)
376923 RESP:2001(RSP:OK)l: 12 T:9a63
loop:376923 CMD: 1008(GET_OBJECT_INFO)l: 16 T:9a64 : cf4
376924 RESP:2001(RSP:OK)l: 12 T:9a64
loop:376925 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:9a65 : cf4 dc01 (STORAGE_ID)
376925 RESP:2001(RSP:OK)l: 12 T:9a65
loop:376925 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:9a66 : cf4 dc07 (OBJECT NAME)
376925 RESP:2001(RSP:OK)l: 12 T:9a66
loop:376925 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:9a67 : cf4 dc08 (CREATED)
376926 RESP:2001(RSP:OK)l: 12 T:9a67
loop:376926 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:9a68 : cf4 dc09 (MODIFIED)
376926 RESP:2001(RSP:OK)l: 12 T:9a68
loop:376926 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:9a69 : cf4 dc0b (PARENT)
376927 RESP:2001(RSP:OK)l: 12 T:9a69
loop:376927 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:9a6a : cf4 dc41 (PERSISTENT_UID)
376927 RESP:2001(RSP:OK)l: 12 T:9a6a
loop:376927 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:9a6b : cf4 dc44 (NAME)
376927 RESP:2001(RSP:OK)l: 12 T:9a6b
loop:376928 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:9a6c : cf3 dc02 (FORMAT)
376929 RESP:2001(RSP:OK)l: 12 T:9a6c
loop:376929 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:9a6d : cf3 dc03 (PROTECTION)
376929 RESP:2001(RSP:OK)l: 12 T:9a6d
loop:376929 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:9a6e : cf3 dc04 (SIZE)
376929 RESP:2001(RSP:OK)l: 12 T:9a6e
loop:376930 CMD: 1008(GET_OBJECT_INFO)l: 16 T:9a6f : cf3
376930 RESP:2001(RSP:OK)l: 12 T:9a6f
...

EDIT: ignore
 
Last edited:
Recent copies have all been FROM MTP Teensy SD card to PC.

Time to go the other way : PC to T_4.1 SD card.
> started BIG of course since the SD was empty copied LOTS of folders of all recent TEST that is 21 MB not the 3MB single folder.

It ran for some time and terminated copy with this:
Code:
...
952: 0 0 1 437 951 490 1641728087 1641325242 490.txt
953: 0 0 1 437 952 491 1641728087 1641325242 491.txt
954: 0 0 1 437 953 492 1641728088 1641325242 492.txt
955: 0 0 1 437 954 493 1641728088 1641325242 493.txt
956: 0 0 0 0 0 0 0 0 
957: 0 0 0 0 0 0 0 0 
958: 0 0 0 0 0 0 0 0 
959: 0 0 0 0 0 0 0 0 
960: 0 0 0 0 0 0 0 0 
961: 0 0 0 0 0 0 0 0 
962: 0 0 0 0 0 0 0 0 
963: 0 0 0 0 0 0 0 0 
964: 0 0 0 0 0 0 0 0 
965: 0 0 0 0 0 0 0 0 
966: 0 0 0 0 0 0 0 0 
967: 0 0 0 0 0 0 0 0 
968: 0 0 0 0 0 0 0 0 
288421 RESP:2001(RSP:OK)l: 24 T:2fb6 : 10001 1b5 3c8
loop:288421 CMD: 100d(SEND_OBJECT)l: 12 T:2fb7
SendObject: 506 bytes, id=3c8
SendObject complete
288425 RESP:2005(RSP:OPERATION_NOT_SUPPORTED)l: 12 T:2fb7
loop:288428 CMD: 1005(GET_STORAGE_INFO)l: 16 T:2fb8 : 10001
65537 0 name:SD Card
288432 RESP:2001(RSP:OK)l: 12 T:2fb8

It made two of the folders before quitting. Selected 3 others and did a Drag to Copy and Windows says:
"Copy File "X" Cannot copy aaa.txt The device has either stopped responding or has been disconnected."

Odd the end of SerMon Spew is NEW but similar to above:
Code:
...
950: 0 0 1 437 949 488 1641728086 1641325242 488.txt
951: 0 0 1 437 950 489 1641728086 1641325242 489.txt
952: 0 0 1 437 951 490 1641728087 1641325242 490.txt
953: 0 0 1 437 952 491 1641728087 1641325242 491.txt
954: 0 0 1 437 953 492 1641728088 1641325242 492.txt
955: 0 0 1 437 954 493 1641728088 1641325242 493.txt
956: 0 0 0 0 0 0 0 0 
957: 0 0 0 0 0 0 0 0 
958: 0 0 0 0 0 0 0 0 
959: 0 0 0 0 0 0 0 0 
960: 0 0 0 0 0 0 0 0 
961: 0 0 0 0 0 0 0 0 
962: 0 0 0 0 0 0 0 0 
963: 0 0 0 0 0 0 0 0 
964: 0 0 0 0 0 0 0 0 
965: 0 0 0 0 0 0 0 0 
966: 0 0 0 0 0 0 0 0 
967: 0 0 0 0 0 0 0 0 
968: 0 0 0 0 0 0 0 0 
969: 0 0 0 0 0 0 0 0 
970: 0 0 0 0 0 0 0 0 
    >> After mkdir
    >> After OpenFileByIndex
580437 RESP:2001(RSP:OK)l: 24 T:2fc5 : 10001 3c9 3ca
loop:580437 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:2fc6 : 3ca dc02 (FORMAT)
580437 RESP:2001(RSP:OK)l: 12 T:2fc6
loop:580437 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:2fc7 : 3ca dc01 (STORAGE_ID)
580437 RESP:2001(RSP:OK)l: 12 T:2fc7
loop:580437 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:2fc8 : 3ca dc07 (OBJECT NAME)
580438 RESP:2001(RSP:OK)l: 12 T:2fc8
loop:580438 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:2fc9 : 3ca dc0b (PARENT)
580438 RESP:2001(RSP:OK)l: 12 T:2fc9
loop:580438 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:2fca : 3ca dc41 (PERSISTENT_UID)
580438 RESP:2001(RSP:OK)l: 12 T:2fca
loop:580438 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:2fcb : 3ca dc44 (NAME)
580438 RESP:2001(RSP:OK)l: 12 T:2fcb
loop:580439 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:2fcc : 3ca dc03 (PROTECTION)
580439 RESP:2001(RSP:OK)l: 12 T:2fcc
loop:580439 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:2fcd : 3ca dc04 (SIZE)
580439 RESP:2001(RSP:OK)l: 12 T:2fcd
loop:580439 CMD: 1008(GET_OBJECT_INFO)l: 16 T:2fce : 3ca
580439 RESP:2001(RSP:OK)l: 12 T:2fce
loop:580440 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:2fcf : 3ca dc08 (CREATED)
580440 RESP:2001(RSP:OK)l: 12 T:2fcf
loop:580440 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:2fd0 : 3ca dc09 (MODIFIED)
580440 RESP:2001(RSP:OK)l: 12 T:2fd0
loop:654483 CMD: 1005(GET_STORAGE_INFO)l: 16 T:2fd1 : 10001
65537 0 name:SD Card
654487 RESP:2001(RSP:OK)l: 12 T:2fd1
 
Starting again empty SDIO SD on T_4.1
Just copy of SD2202 underway From PC to T_4.1:
> Started saying some few minutes? 4 or 6
> after 6 minutes Win Explorer now suggesting 30 minutes for the copy ... this is similar to time reported on prior p#934
> Sound was OFF - seeing batches of blocks of ' 0 0 0 0 0 0 0 0 0 ' similar to above and hearing CHIMES with sound on
-- Chimes seems to come in rapid batches and a quick trill as they overlap each other?
> Now over 9 minutes suggestion is 6 minutes remaining ... at at 10 min saying 5 mins
> Sounds continue intermittent 'bip, bip, bip' never a complete sound but suggesting errors
--> For REF the copy PC to SD used to be PERFECT before the rewrite work began :(

OPPS - think I had focus on COPY progress message box and hit ENTER trying to type here in Browser
> Copy stopped at 14 mins showing about 2 minutes to go while there were lots of Sounds Chimes
Seems this is what I did before in p#934 as Sermon ended with:
Code:
...
2974: 0 0 0 0 0 0 0 0 
2975: 0 0 0 0 0 0 0 0 
2976: 0 0 0 0 0 0 0 0 
1029155 RESP:2001(RSP:OK)l: 24 T:3f7b : 10001 8d5 ba0
loop:1029155 CMD: 100d(SEND_OBJECT)l: 12 T:3f7c
SendObject: 690 bytes, id=ba0
SendObject complete
1029160 RESP:2005(RSP:OPERATION_NOT_SUPPORTED)l: 12 T:3f7c
loop:1029209 CMD: 1005(GET_STORAGE_INFO)l: 16 T:3f7d : 10001
65537 0 name:SD Card
1029217 RESP:2001(RSP:OK)l: 12 T:3f7d

Need to start with smaller subset ... and not interrupt. The SD back on PC shows no miscompare files, just missing files that is no conclusive with the CANCEL ...

Started again on clean SD from SD2202.
> Copied ManyD10 - seemed to quickly complete
> Started copy of ManyF and Sermon is running and the Copy Dialog/Box time is going up to 27 minutes.
-> The Explorer VIEW of the Teensy CLOSED - but the copy dialog persists?
Teensy still shows in the left Navigation panel and I can get to the drive and copy continues 'About 30 min...' and no error chimes yet

NOTE: ERROR CHIMES as soon as I hit POST! SerMon is surging and chiming

Has problems starting here?
Code:
...
953: 0 0 1 84 952 844 1641731452 1641594620 844.txt
954: 0 0 1 84 953 845 1641731453 1641594620 845.txt
955: 0 0 1 84 954 846 1641731453 1641594620 846.txt
956: 0 0 0 0 0 0 0 0 
957: 0 0 0 0 0 0 0 0 
958: 0 0 0 0 0 0 0 0 
959: 0 0 0 0 0 0 0 0 
960: 0 0 0 0 0 0 0 0 
961: 0 0 0 0 0 0 0 0 
962: 0 0 0 0 0 0 0 0 
963: 0 0 0 0 0 0 0 0 
964: 0 0 0 0 0 0 0 0 ...

Problems for some time - then it resumes ...

<EDIT>: RESULTS
Out of 3072 files in ManyF there were 2201 files NOT COPIED onto the SD from the PC
Here is the default SerMon output as captured: View attachment ManyF_Fail.txt

According to CodeCompare on the Expected Source and Result ManyF folder these files are wholly missing: View attachment missing.txt
 
Last edited:
I will look at this late tonight or early tomorrow. Would really help me if this problem could be reproduced with a smaller test. This test is probably much too large for the protocol analyzer to get a useful capture.

But if the problem is "only" that we're hitting some sort of limit in memory or the MTP_Storage class, rather than more USB issues like ZLP handling (ZLP receive may indeed need work), then it might only happen with large tests. Sheer guesswork & speculation right now.
 
I will look at this late tonight or early tomorrow. Would really help me if this problem could be reproduced with a smaller test. This test is probably much too large for the protocol analyzer to get a useful capture.

But if the problem is "only" that we're hitting some sort of limit in memory or the MTP_Storage class, rather than more USB issues like ZLP handling (ZLP receive may indeed need work), then it might only happen with large tests. Sheer guesswork & speculation right now.

Indeed - not simple cases.
> #933 is From T_4.1 to PC
> #935 is From PC to T_4.1

Was hoping setup (or sermon output) triggered some knowing speculation or limited guesswork, 'Sheer' is a different story.

Luckily it is SD based - at least it allows known start data and positive verify back on PC with SD access.
> repro refinement still "Sheer guesswork & speculation" to craft and arrange a triggering alternate fileset/test.

>>If somebody could replicate on LINUX {or another non Win11 machine} it might confirm at least it is not just a 'Windows' issue.
- if the steps presented aren't clear enough let me know.
> Github sources Kurte:MTP_Teensy current as of now, from 11 hours ago.
> T_4.1 in use with SDIO SD card and Example_3_simple_SD.ino

Given independent repro feedback I can proceed knowing it isn't a local OS/User error.
 
FYI: Anyone not updated to Win 11 - good plan to stay on Win 10 a bit perhaps. New i7 w/SSD is really painfully sluggish for File Explorer work :(

Powered up Win 10 laptop - will update and try from here for repro. Will take a bit to transfer ...
 
Sorry have been distracted from trying to many different tests today.

I have been playing with a straw man WIP record cache with 8 records (except on T3.2 turned off).
Not sure about LC? Are we supporting it? As it is shown in boards.txt..

Code is up in new branch: https://github.com/KurtE/MTP_Teensy/tree/cache_storage_records

Only changes to Storage files and this branch is up to date with main branch.

Only one sketch updated to add a couple of debug commands, to be able to toggle cache on and off, plus it keep track on how many reads were requested and how many went to fs to get the data... Ditto for write.

The cache is currently configured for 8 records. It has secondary structure with which item, is it dirty and index into storage array. The last part was because right now keeping the list in the order of the last items read/or written.
So I end up moving that data around in the structure.

Thinking of alternative one that does not require copying the data around,
Instead maybe keep a logical weighted value for each record in the list, and when we read or write and not in this cache, then choose the one with the lowest logical value to replace.
Could do something like: Each look into the list:
When we scan the list for object_index x, for all of the items in the list that are not X subtract 1 from value
If it is in the list, for reads maybe increment by 5 and writes 10... (as writes take longer to some storages...)...

Again just playing and see what it does
 
Sorry have been distracted from trying to many different tests today.

I have been playing with a straw man WIP record cache with 8 records (except on T3.2 turned off).
Not sure about LC? Are we supporting it? As it is shown in boards.txt..
...
Again just playing and see what it does

Interesting to see how it helps. Bulk testing here so far not likely to help given once and done Moving data Off or Onto?

Bumped laptop to 2021 Fall Win 10 from Spring. Arduino drive copied external to transfer ...
 
File something; something.open("thisfile.txt"); something.isOpen(); (something.isOPen(): 'class File' has no member named 'isOpen') or if(something); (No Effect).

Is there a example of proper usage of .isOpen() on an open or closed file or directory???

Thanks :)
 
Win 10 laptop building and running - seems the same on above repro 993 and 995.

Re #993 Repro on Win 10 laptop: YES
> Put SD2202 on SD into T_4.1 {Also copy of SD2201}
- copy folder and the D9 files are missing {Same in copy of SD2201}

Re #995 Repro on Win 10 laptop w/ManyF: YES
> MANY files missing from folder copy to PC from T_4.1

NOTE on SerMon during LONG COPY: The long list of files reprints fully on each next added file! And it shows this line of ZEROs it seems when added files FAIL:
Code:
...
952: 0 0 1 84 951 843 1641768516 1641594620 843.txt
953: 0 0 1 84 952 844 1641768516 1641594620 844.txt
954: 0 0 1 84 953 845 1641768517 1641594620 845.txt
955: 0 0 1 84 954 846 1641768518 1641594620 846.txt
956: 0 0 0 0 0 0 0 0 
957: 0 0 0 0 0 0 0 0 
958: 0 0 0 0 0 0 0 0 
959: 0 0 0 0 0 0 0 0 
960: 0 0 0 0 0 0 0 0 
961: 0 0 0 0 0 0 0 0 
962: 0 0 0 0 0 0 0 0 
963: 0 0 0 0 0 0 0 0 
964: 0 0 0 0 0 0 0 0 
965: 0 0 0 0 0 0 0 0 
966: 0 0 0 0 0 0 0 0 
967: 0 0 0 0 0 0 0 0 
968: 0 0 0 0 0 0 0 0 
969: 0 0 0 0 0 0 0 0 
970: 0 0 0 0 0 0 0 0 
971: 0 0 0 0 0 0 0 0 
972: 0 0 0 0 0 0 0 0 
973: 0 0 0 0 0 0 0 0 
974: 0 0 0 0 0 0 0 0 
975: 0 0 0 0 0 0 0 0 
976: 0 0 0 0 0 0 0 0 
977: 0 0 0 0 0 0 0 0 
978: 0 0 0 0 0 0 0 0 
979: 0 0 0 0 0 0 0 0 
980: 0 0 0 0 0 0 0 0 
981: 0 0 0 0 0 0 0 0 
982: 0 0 0 0 0 0 0 0 
983: 0 0 0 0 0 0 0 0 
984: 0 0 0 0 0 0 0 0 
373985 RESP:2001(RSP:OK)l: 24 T:2fd5 : 10001 54 3d8
loop:373985 CMD: 100d(SEND_OBJECT)l: 12 T:2fd6
SendObject: 875 bytes, id=3d8
SendObject complete
373989 RESP:2005(RSP:OPERATION_NOT_SUPPORTED)l: 12 T:2fd6
loop:374205 CMD: 100c(SEND_OBJECT_INFO)l: 20 T:2fd7 : 10001 54
SendObjectInfo: 10001 54 Dataset len=156
File "876.txt" size:876 Created:61db664f Modified:61d8befe
 read consumed all data (TODO: how to check ZLP)
84: 0 1 1 83 0 985 1641768230 1641768230 111
985: 0 0 1 84 984 0 0 0 876.txt
0: 0 1 1 -1 0 83 0 0 /
20: 0 0 0 0 0 0 1546300826 1546300826 mtpindex.dat
21: 0 1 0 0 20 0 1641338864 1641338864 System Volume Information...
 
Last edited:
Good very early morning all.

Moved away from @defragster's sd2201 test set to test the flash chips. Copying a lot of small files (1024) to flash chips may not be the best where we have erase blocks sizes of 64K for flash and 128K for NAND chips. So instead copied the following directories from the PC to the Win1G, Win2G NAND chips and a subset to the sflash6 on the memboard and back to the PC all successfully. File sizes matched and music files opened and played correctly in Windows Groove Music:
Code:
Sample Test Files ~73mb/
  2001/
    calculations.wav                  426300
    completed.wav                     276460
    dangerous_to_remain.wav           372892
    enough_info.wav                   513388
    functional.wav                    237356
    one_moment.wav                    202236
    operational.wav                   772140
    sorry_dave.wav                    791164
    stop.wav                          200844
  Audacity/
    Away_in_a_Manger.mp3              2014737
    Dont_Rain_on_My_Parade.mp3        3944449
    Take_My_Breathe_Away.mp3          5740819
    Welcome_Christmas.mp3             2985790
  Candyman.aac                        3177823
  Dont Rain on My Parade.mp3          3944449
  FLAC/
    T1_1024.FLA                       9802802
    T1_128.FLA                        11126659
    T1_256.FLA                        10415954
    T1_512.FLA                        10007370
  SDTEST1.raw                         8393684
  odd1.mp3                            46888
  odd1.wav                            553004
  zarathustra.mp3                     489461
and
Code:
LFS_CPY/ ~67MB
  Picture1.png                        1037815
  SDTEST1.wav                         16788062
  SDTEST2.wav                         16426210
  SDTEST3.wav                         13617870
  SDTEST4.wav                         17173664
  calculations.wav                    426812
  cccc.py                             2440
  completed.wav                       276972
  dangerous_to_remain.wav             373404
  datalog.txt                         1187
  enough_info.wav                     513900
  functional.wav                      237868
  mtpindex.dat                        0
  odd1.wav                            553516
  one_moment.wav                      202748
  operational.wav                     772652
  sorry_dave.wav                      791676
  stop.wav                            201356

Other functions tested:
1. Format from MTP: works for all NOR and NAND flash chips on the memory board
2. FAIL: Format works work SDIO card but an issue is still persisting where after format Used and Unused Space is not updated for the SD card so it turns red in Windows Explorer and you can not access it anymore.
3. Copy the Sample Test Files folder from the Win1G to a 128GB jump drive:
a. Part way through the transfer Windows Progress Bar closes with a Beep but the transfer continues until complete.
b. Once completed I have to issue a manual 'r' device reset and refresh windows explorer for the jump drive for the Sample Test Files directory to appear.
c Bottom line the transfer does work.
 
Good very early morning all.

Moved away from @defragster's sd2201 test set to test the flash chips. Copying a lot of small files (1024) to flash chips may not be the best where we have erase blocks sizes of 64K for flash and 128K for NAND chips. So instead copied the following directories from the PC to the Win1G, Win2G NAND chips

...

c Bottom line the transfer does work.

Very AWESOME 'bottom line' with working transfers. No fun having trashed data on an 'opaque' FLASH chip that is harder to DIFF and debug.

Though I expect the 'simple' stuff will work - based on how well the larger SD2202 works. There are just some edge cases, and with SD being faster and more 'transparent' I'm looking to test there first.

The new file content better human readable in SD2202 is better - and line breaks allow CodeCompare to pinpoint discrepancies as it breaks and compares on newline.

Code in MakeFiles.ino allows creation of files of any size and number. And selective directories can be made and copied selectively.
> and for you Mike - as noted - I did a #define for the "SD" as: #define DISK SD. That should allow use more easily on any compatible FS.

> I had the thought of adding a 'zero growSize' option to make files of same size and on seeing the '0' growSize param, extend the base filename instead with the 'numFiles' value.
> Also thought of doing like I did in "LittleFS Integrity" make a func to scan the file for expected content and file size in a DirWalk - but it would need 'info' to detect missing files to be conclusive. Perhaps a count.txt in each folder holding the number of files written there.

Also there was that PC APP that does MTP posted some while back in recent days [ p#902 ]. I built it - but didn't look into it yet to see if it could programmatically do an MTP DirWalk with Verify of returned content. It offers to ask for 'DIR of all files' in one option and has another to take the 'ID' returned to 'ask for file content'
> With programmatic 'real time' verify of transfer it could HALT on errors and that might allow "protocol analyzer to get a useful capture" at the point of failure the traffic would stop.
 
I started testing with Android File Transfer for MacOS.

First, fixed a bug which was causing a wrong GetStorageIDs response.

https://github.com/KurtE/MTP_Teensy/commit/e8e127245dbefbcaead748f6ec457312856089df

Windows doesn't seem to mind, but it was causing Android File Transfer to give up. When it does, looks like Android File Transfer has a bug where it mistakenly retransmits the prior command, but never tries to receive any reply. Then on our side, we have a less-than-stellar situation where any reply we tried to send can stay buffered forever if the host stops sending IN tokens, which later causes Android File Transfer to fail very differently on the next run when it unexpectedly gets data we tried to send the last time the program ran. Very confusing!

With this fixed come the issue of startup. Looks like we have a sort of race condition between enumeration complete and MTP.begin() called. On Windows, we use a timer to answer requests before loop() is getting called. On Mac, it "works" but the first automatic run of Android File Transfer see no files. Then running manually again works.

I was able to copy a JPEG file from Teensy to the Mac and then open it. But copying files from Mac to Teensy doesn't seem to work. Haven't investigated yet...


screenshot.jpg
 
Last edited:
Looks like everyone has been busy...

I started testing with Android File Transfer for MacOS.
...
With this fixed come the issue of startup. Looks like we have a sort of race condition between enumeration complete and MTP.begin() called. On Windows, we use a timer to answer requests before loop() is getting called. On Mac, it "works" but the first automatic run of Android File Transfer see no files. Then running manually again works.

I was able to copy a JPEG file from Teensy to the Mac and then open it. But copying files from Mac to Teensy doesn't seem to work. Haven't investigated yet...

Yes, I added the timer earlier on, to see what messages we absolutely had to respond to in a timely manor at startup, or nothing USB works...

That was including Serial output... Still sort of still that way. Which is why some of the sketches were setup where I capture all serial output to a memory stream, and after MTP.begin() is called we dump the contents of this stream to Serial.

And especially with multiple storages especially some like SD cards that can take a long while for the list of storages to initialize, this can take a long time. Better now with the Fat32 code for getting used/free space, but still...


It would be great to have a better approach for startup... As I believe I mentioned here or email or...

Wondering if we should setup to have the main MTP USB code be able to and maybe responsible for automatically responding to some of the initial standard messages that the host asks for.

Things like: MTP_OPERATION_GET_DEVICE_INFO, MTP_OPERATION_GET_DEVICE_PROP_DESC,

It also appeared to need for us to respond to start a session: MTP_OPERATION_OPEN_SESSION

After that I was able to get away with answering we are busy...
The main one that we would hit next was: MTP_OPERATION_GET_STORAGE_IDS

Which if we responded we are busy, the window might come up with no items in it... If we hit F5 or the like to refresh it would then get the list...

So again not sure if this was/is the best way to tell host that we are not ready yet.

Also how to better have USB with MTP be, such that it does not interfere with other USB starting up. With the Interval timer stuff at least SEREMU appears to come up, although mostly losing all out output until MTP is happy.
When we were setting up to also support USB Serial (think that still should be option), if I remember correctly, it would get most all of the early Serial output, but the code for handling things like: while(!Serial);
would not work. Which is why some examples have or had things like: while(!Serial && !Serial.available()) ;
And you would have to hit something like enter to have it get through...
 
I'm not seeing any obvious errors with the protocol analyzer during startup. It might be that everything is "working" but MacOS is much quicker than Windows and gets empty data before we're ready, and then we never send events to say it should check again. Maybe? It's too bad the analyzer can't also capture a video feed with precise time sync to the data stream. The host sends so much stuff and usually I spend a lot of time trying to guess what it was attempting to do at various points.

Sadly, I need to step back from MTP for a day or two. Some urgent PJRC business stuff (which I mostly blew off last week while overhauling MTP_Teensy.cpp) is calling for my attention. If you want to play in the new code without any merge conflicts, now's the perfect time. :)
 
Sorry have been distracted from trying to many different tests today.

I have been playing with a straw man WIP record cache with 8 records (except on T3.2 turned off).
Not sure about LC? Are we supporting it? As it is shown in boards.txt..

Code is up in new branch: https://github.com/KurtE/MTP_Teensy/tree/cache_storage_records

Only changes to Storage files and this branch is up to date with main branch.

Only one sketch updated to add a couple of debug commands, to be able to toggle cache on and off, plus it keep track on how many reads were requested and how many went to fs to get the data... Ditto for write.

The cache is currently configured for 8 records. It has secondary structure with which item, is it dirty and index into storage array. The last part was because right now keeping the list in the order of the last items read/or written.
So I end up moving that data around in the structure.

Thinking of alternative one that does not require copying the data around,
Instead maybe keep a logical weighted value for each record in the list, and when we read or write and not in this cache, then choose the one with the lowest logical value to replace.
Could do something like: Each look into the list:
When we scan the list for object_index x, for all of the items in the list that are not X subtract 1 from value
If it is in the list, for reads maybe increment by 5 and writes 10... (as writes take longer to some storages...)...

Again just playing and see what it does

Sorry I didn't respond to this last night. Right now I am using a 4MB index file using LittleFS_Program on the TMM as a result of all the devices I am testing with - if you look closely at the debug out everytime you open or write a file the whole index file has to get read which contains the directories for all the devices - this can get way too large and add time to the process. So your idea of streamling the index file and how to use it would be great. Haven't started playing with the changes you posted yet.

@PaulStoffregen - @defragster
At Tim's suggestion I went ahead and tried his sketch to makefiles directly on the Win2G chip and to my surprise it failed to created second level directories:
Code:
root:
  |
  |- level 1
      |- level 2 (FAILS TO CREATE)

So I tested it from withing LFS_SPI_USAGE sketch:
Code:
  Serial.println("\n---------------");
  Serial.println("Create a directory and a subfile");
  bool test = myfs.mkdir("structureData1/111");
and sure enough test returns false - failed to create the directory but does work with:
Code:
  Serial.println("\n---------------");
  Serial.println("Create a directory and a subfile");
  bool test = myfs.mkdir("structureData1");

Just thought you should know. There is also no way to do change directory so can not even try that approach.
 
@Paul - good luck! I can imagine you have your hands pretty full!

@all - Sort of wondering what the normal usage patterns that are expected to work with MTP? For me, I see it as a way to get a few files back and forth between host and Teensy, not one that for example to the host works like a full filesystem that apps on the Host can manipulate...


Sorry sort of follow on to yesterdays ramblings

<<< Ramble Start >>>
Or another way to phrase it, is should we design/test around a setup with maybe one or two storage added (maybe 3 or 4) and that likely the user will typically walk through typically many 50-100 files, maybe in cases like 100s, with a structure not likely to be deeper than 2-5 levels... Or are we trying to develop a setup where the user will have 20 storage added each with maybe 10s of thousands of files, that go like 20 levels deep in directory structure... My gut says the simpler setup. If we were going for larger setups, I am not sure MTP is the best route.
Why I am asking some of this is again trying to understand is how to cleanup the storage list, which I mentioned recently. That is for example running a sample run, I browsed a few directories and dumped the debug data:
Code:
Dump Storage list(9)
store:0 storage:10001 name:Program fs:20005a1c
store:1 storage:20001 name:RAM fs:20005ccc
store:2 storage:30001 name:QNAND fs:20005bc8
store:3 storage:40001 name:NAND_3 fs:20005d98
store:4 storage:50001 name:NAND_4 fs:20006060
store:5 storage:60001 name:Flash_5 fs:20006328
store:6 storage:70001 name:Flash_6 fs:200065f0
store:7 storage:80001 name:SD_Builtin fs:20006ecc
store:8 storage:90001 name:SD_SPI fs:2000739c

Dump Index List
0: 0 1 1 -1 0 38 0 0 /
1: 1 1 0 -1 0 0 0 0 /
2: 2 1 0 -1 0 0 0 0 /
3: 3 1 0 -1 0 0 0 0 /
4: 4 1 0 -1 0 0 0 0 /
5: 5 1 0 -1 0 0 0 0 /
6: 6 1 0 -1 0 0 0 0 /
7: 7 1 1 -1 0 28 0 0 /
8: 8 1 0 -1 0 0 0 0 /
20: 7 0 0 7 0 32012 1637295642 1634128928 foo.txt
21: 7 0 0 7 20 8732 1637295648 1633279812 FS.h
22: 7 0 0 7 21 22014964 1637739436 1636096816 T4LargeIndexedTestfile.txt
23: 7 0 0 7 22 21 977443208 977443208 test1.txt
24: 7 0 0 7 23 0 1641466548 0 mtpindex.dat
25: 7 1 0 7 24 0 1637296742 1637296742 foo
26: 7 1 1 7 25 34 1641747544 1641747544 333
27: 7 1 1 7 26 31 1641747544 1641747544 111
28: 7 1 1 7 27 37 1641747544 1641747544 222
29: 7 0 0 27 0 244 1641747544 1641637052 aaa.txt
30: 7 0 0 27 29 500 1641747544 1641637052 bbb.txt
31: 7 0 0 27 30 756 1641747544 1641637052 ccc.txt
32: 7 0 0 26 0 244 1641747544 1641637052 aaa.txt
33: 7 0 0 26 32 500 1641747544 1641637052 bbb.txt
34: 7 0 0 26 33 756 1641747544 1641637052 ccc.txt
35: 7 0 0 28 0 244 1641747544 1641637052 aaa.txt
36: 7 0 0 28 35 500 1641747544 1641637052 bbb.txt
37: 7 0 0 28 36 756 1641747544 1641637052 ccc.txt
38: 0 0 0 0 0 0 1639834176 1639834176 mtpindex.dat
Note the ones above 20 are actual storage units and already split those out on their own.
But for each of the others there is a 280 byte record. For each of these records: example 37: 7 0 0 28 36 756 1641747544 1641637052 ccc.txt

It is in storage 7(SD_Builtin) It is Not a directory and as such has not be scanned(children enumerated), it's parent is 28, next sibling is 36, Next field 756 is either first child (if directory) or size, then we have create/modify dates and the names.

Currently we use a File to store this index, where we do a seek to (record-20)*280 and read/write 280 bytes. As long as the Teensy is running an MTP session, the ID's must be maintained. That is at any time for example MTP may choose to ask for information on #29... AFAIK - There is no way for us to know what branches of storage objects the host might have open. For example it could be the user on the host closed all of the windows showing any MTP data and we won't know it and as such we need to maintain list through the entire session.

One exception to this, is if WE send an Reset event to the host, they will send us a message to start new session, and that point we can toss all of this.

<<< Ramble End or maybe slightly less rambling ;) >>>

Playing with caching of items currently 8:
Might help with things like SendObject: that creates a record and writes it, then file is received, which we read it back in and write it again with updated data (dates, sizes)

If you try to open #37, we generate full path: so have to read #37, #28, #7 (store always in memory), so hopefully if done a few times, we keep #28 also in memory and not have to seek/read...
But if directory structure is 15 levels deep, this won't help, may hurt... I could change setting to more items, but...

Wondering about if it will make sense to break up this storage file, like one per store, or if when we addFilesystem, if it makes sense to optionally say if this one should manage it's own and/or is the default one to use if ones don't handle themself... or ???

Short term - either with caching or not caching... If some of these test cases are failing: Wonder if running into issue of storage file not working. Earlier I did.
So there is debug code in mtp_storage, like:
Code:
bool MTPStorage::WriteIndexRecord(uint32_t i, const Record &r)
{
	OpenIndex();
	mtp_lock_storage(true);
	bool write_succeeded = true;
	if (i < MTPD_MAX_FILESYSTEMS) {
		// all we need is the first child pointer and if it was scanned
		store_first_child_[i] = r.child;
		store_scanned_[i] = r.scanned;
	} else {
		index_.seek((i - MTPD_MAX_FILESYSTEMS) * sizeof(r));		
		size_t bytes_written = index_.write((char *)&r, sizeof(r));
		if (bytes_written != sizeof(r)) {
			DBGPrintf(F("$$$ Failed to write Index record: %u bytes written: %u\n"), i, bytes_written);
			write_succeeded = false;
		}
	}
	mtp_lock_storage(false);
	return write_succeeded;
}
Which if debug level is high enough >1 the DBGPrintf will output... Currently set to 0...
So could either:
a) change like 35: #define DEBUG 0
b) change this DBGPrintf into an output that does not depend on debug level. Also there is code to test if read fails as well...

But first more coffee and time to play
 
On the question of anticipated usage, I would imagine nearly all uses will have no more than 4 filesystems "stores", and the majority will probably use just 1. Likewise, I would imagine most people will put a few dozen files in the root directory and probably not use subdirectories at all. For those using hierarchy, it's pretty unlikely the sorts of projects made for microcontrollers will nest more than 2 or 3 levels deep.

I do believe these very large tests have value in possibly catching rare bugs which we might miss with only smaller tests.

But when it comes to optimizing performance, we should probably focus on the likely usage scenarios. If the "large" tests show correct but slow behavior, that's probably fine. We have limited resources, especially RAM. People use Teensy for real-time projects where a slower file transfer that imposes less latency on real-time processes is probably better than one which competes faster by hogging the CPU.

Minimizing disruption to libraries like Audio, Servo, Wire, Serial1-8, OctoWS2811 during file transfers is one of the most important design goals, far more important than completing the transfer as fast as possible. The whole reason we're doing MTP rather than MSC (and the same reason Android switched from MSC to MTP) is so we can continue operating normally while the host reads and writes files.

I know this skirts all the more specific questions, but you did ask about expected uses and goals. Hopefully this helps?
 
Back
Top