Teensyduino File System Integration, including MTP and MSC

@KurtE
I downloaded your cache_storage_records and took it for a test run using the TMM + MemoryBoard (Q512+N01+N02). Decided to take it for a stress though and copied 1 of directories in Tim's ManyF directory(1024 files ranging from 40-1423bytes)
Code:
1. N02 Storage:
   a. Stopped the coping after a while since it was going to take a couple hours to transfer the files from the PC to N02 but..
   b. 9 of the 232 files that did copy over had 0 bytes but all the failures had the following error message
          File "1120.txt" size:1120 Created:61e14e3c Modified:61dbed2e
          read consumed all data (TODO: how to check ZLP)
          MTP_class::Loop usb_mtp_status 19 != 0x1 reset

2. Next up was the W25Q512 storage:
   a.  Coping was alot faster and i let it run to completion
   b.  19 of the 1024 files failed to copy (showed 0 bytes) - same error message.

3. N01 NAND
   a.   Copy of the 1024 taking over 2 hours 
   b.   Ok got tired of waiting for the complete transfer.
   c.   14 files of 250 files failed to copy.

Using a different memory board:
Code:
store:2 storage:30001 name:Adesto fs:2000c1b4 pn:AT25SF041
store:3 storage:40001 name:Q64 fs:2000c290 pn:W25Q64JV*Q/W25Q64FV
store:4 storage:50001 name:Q256JWIM fs:2000c36c pn:W25Q256JW*M

Code:
4.  W25Q256JWIW
   a.  Again got tired waiting for copy to finish
   b.  196 files with 0 bytes out of 896
   c.   245 files with wrong file sizes
   d/  PROBABLY BECAUSE DISK WAS FULL ON TRANSFER:
          Space Used = 33554432
           Filesystem Size = 33554432

5. W25Q64
  a. Can only hold 256 of the files before the disk is full.
  b. which is only about 280k
  c.  Formatted the chip again and transferred a 2MB mp3
[CODE] Space Used = 2162688
Filesystem Size = 8388608
Directory
---------
Away_in_a_Manger.mp3                  2014737
Explorer shows only 48Kb and looks like 1/4 full?
Tried playing it and yes it was truncated:
Something strange in the neighborhood as they say:
Code:
loop:56831 CMD: 1009(GET_OBJECT)l: 16 T:24 : 10
MTP_class::Loop usb_mtp_status 19 != 0x1 reset
loop:57269 CMD: 1009(GET_OBJECT)l: 16 T:25 : 10
57287 RESP:2001(RSP:OK)l: 12 T:25


[/CODE]
 
Last edited:
That is correct :)
Just a quick update:

I added code in to handle the rename of files to different name lengths and did a little testing. It is not foolproof, but it tries to pack as many records it can into each buffer,
but reserves like minimum of 64 bytes for names to be renamed. Note: I round up space for names to 4 byte increments as to keep structures in area to be 4 byte aligned.

I have merged all of my commits into one and I then rebased my branch to the current main branch...
It still builds..

I am thinking of merging soon, still things that will want to change, but...
 
@KurtE
As a follow up to post #1001 I reverted back to the main branch of MTP_Teensy to see what explorer showed for the 2MB MP3 file. The MP3 file shows correctly in Windows Explorer, i.e., about 2MB and will play the whole song on the PC.

@PaulStoffregen - @KurtE - @defragster
In this last test with the MTP main branch I tried copying those 1024 files from the ManyF directory to the Q64 chip but would only fit 123 of them before the disk was full. So I changed the Q64 back to sector erase from 64k Block erase and all 1024 files fit on the Q64. All files were copied and the correct sizes showed for all files. And only about 50% disk usage:
Code:
 Space Used = 4513792
Filesystem Size = 8388608

So the question is do we revert back to sector erase or leave as is except for the smaller chips!


EDIT: Changed the W25Q512 to sector erase as well and reran the copy of all those files with the same result as for the Q256 chip. All files transferred with no errors or data loss.
Changed the Q256JW to use Sector Erases as well and coping speed went up exponentially, actually at the point where I can wait for the transfer to complete of the files from the PC to the Flash. All files copied successfully from the PC to the chip - file sizes matched
 
Last edited:
@KurtE
As a follow up to post #1001 I reverted back to the main branch of MTP_Teensy to see what explorer showed for the 2MB MP3 file. The MP3 file shows correctly in Windows Explorer, i.e., about 2MB and will play the whole song on the PC.

@PaulStoffregen - @KurtE - @defragster
In this last test with the MTP main branch I tried copying those 1024 files from the ManyF directory to the Q64 chip but would only fit 123 of them before the disk was full. So I changed the Q64 back to sector erase from 64k Block erase and all 1024 files fit on the Q64. All files were copied and the correct sizes showed for all files. And only about 50% disk usage:
Code:
 Space Used = 4513792
Filesystem Size = 8388608

Changed the Q256JW to use Sector Erases as well and coping speed went up exponentially, actually at the point where I can wait for the transfer to complete of the files from the PC to the Flash. All files copied successfully from the PC to the chip - file sizes matched

That is a Huuge difference ... space (50% free versus over 700% short!) utilization and speed!

BTW: I bugged koromix for 1062 uploads timing out ( long erase times or MTP online ) - he upped the timeout and posted a link in the ISSUE on TyQT's git I downloaded but not tried it yet ... got a wife's coworker computer over lunch to add to the mix here ... MalwareBytes found 38 things in 42 seconds ... hopefully it works when it gets home like it does here ...
> This is when the TSET "T" option is used to have TeensyLoader do the upload and it doesn't complete in time. Seems Paul extended the wait time as well for that/
 
Last edited:
That is a Huuge difference ... space (50% free versus over 700% short!) utilization and speed!

BTW: I bugged koromix for 1062 uploads timing out ( long erase times or MTP online ) - he upped the timeout and posted a link in the ISSUE on TyQT's git I downloaded but not tried it yet ... got a wife's coworker computer over lunch to add to the mix here ... MalwareBytes found 38 things in 42 seconds ... hopefully it works when it gets home like it does here ...
> This is when the TSET "T" option is used to have TeensyLoader do the upload and it doesn't complete in time. Seems Paul extended the wait time as well for that/

Think for chips up to and including the Q256 we should go back to sector erase and keep the block erases for the large size chips - maybe
 
@KurtE
As a follow up to post #1001 I reverted back to the main branch of MTP_Teensy to see what explorer showed for the 2MB MP3 file. The MP3 file shows correctly in Windows Explorer, i.e., about 2MB and will play the whole song on the PC.

@PaulStoffregen - @KurtE - @defragster
In this last test with the MTP main branch I tried copying those 1024 files from the ManyF directory to the Q64 chip but would only fit 123 of them before the disk was full. So I changed the Q64 back to sector erase from 64k Block erase and all 1024 files fit on the Q64. All files were copied and the correct sizes showed for all files. And only about 50% disk usage:
Code:
 Space Used = 4513792
Filesystem Size = 8388608

So the question is do we revert back to sector erase or leave as is except for the smaller chips!


EDIT: Changed the W25Q512 to sector erase as well and reran the copy of all those files with the same result as for the Q256 chip. All files transferred with no errors or data loss.
Changed the Q256JW to use Sector Erases as well and coping speed went up exponentially, actually at the point where I can wait for the transfer to complete of the files from the PC to the Flash. All files copied successfully from the PC to the chip - file sizes matched
Think fixed... Problem was I reduced the size of some of the fields, like parent and sibling from 32 bits to 16 bits... But I also did it for child. Problem is when not a directory, child is overloaded to hold the size of the file...

So increased that one back to 32 bits...
 
So the question is do we revert back to sector erase or leave as is except for the smaller chips!

We need to keep in mind these tests copying thousands of tiny files are not the expected usage pattern. Slow performance and inefficient use of space are an acceptable result. If we can improve performance in ways that benefit all cases, then of course we should. But when we have to male decisions that trade off performance for some cases in preference to others, we need to keep in mind which cases matter and which are not important.

Writing a small number of files, or even 1 large file, which fills nearly all space is the case we need to consider for optimization. We're using 64K erase on Winbond NOR flash chips because previous tests copying to used flash (where all writes must do erase) transferred too slowly and Windows would give up.
 
With existing LFS disk params:

Edited MakeFiles here to make files on the 2Gb flash:
Code:
Total 127 files of Size 7766016 Bytes
Bytes Used: 17301504, Bytes Total:265289728

That is 6 4KB files and 120 63KB files

The files were made on empty Media with Windows showing: 252 MB free of 253 MB - while the T_4.1 was busy making the files.

They are now in 2 folders and viewing them with explorer and going back up to top level device label - that '252 MB free' has not been updated - same as noted with format.

It took a long time to show the Explorer view of the folder with 120 files. Again, to note - it is doing an N^2 Sermon printing of the names. File 1, then 1&2, then 1&2&3, ... until finishes sending to Windows.

Using Win Explorer to delete the folder with 6 and 120 files resulted in it seeing the Teensy non responsive doing the 120 folder. It was still alive going into the folder and selecting all 120 files - the delete then completed showing the folder empty.
> Going up a level to delete the 'empty folder':
Code:
D0.60 could not be deleted. The device has wither stopped responding or has been disconnected.

SerMon UI is not responsive to PrintDirectory ... Windows Explorer still shows as if the folder is present, but empty now.

Slight Mod to created files - works for creation:
Code:
Total 150 files of Size 9902080 Bytes
Bytes Used: 20971520, Bytes Total:265289728
using:
Code:
  MakeDeepDirs( szStart[0], 1, 20, 4096, 0 );
  showMediaSpace();
  MakeDeepDirs( szStart[0], 1, 40, 122*1024, 0 );
  showMediaSpace();
  MakeDeepDirs( szStart[0], 1, 60, 63*1024, 0 );
  showMediaSpace();
  MakeDeepDirs( szStart[0], 1, 30, 31*1024, 0 );
20 4K
40 122K
60 63K
30 31K
 
We need to keep in mind these tests copying thousands of tiny files are not the expected usage pattern. Slow performance and inefficient use of space are an acceptable result. If we can improve performance in ways that benefit all cases, then of course we should. But when we have to male decisions that trade off performance for some cases in preference to others, we need to keep in mind which cases matter and which are not important.

Writing a small number of files, or even 1 large file, which fills nearly all space is the case we need to consider for optimization. We're using 64K erase on Winbond NOR flash chips because previous tests copying to used flash (where all writes must do erase) transferred too slowly and Windows would give up.

Indeed, for 256MB media thousands of files doesn't make sense.

Numbers in p#1011 for Flash seem reasonable for usable space and number of files if it gives more timely responsive system.

I could see some lag on the creation ( By T_4.1 on Flash ) in groups - 3-4 files seemed to progress, then a pause. I should add an elapsed time for ref to catch the slow spots - maybe the batches are when SerMon gets fed.
 
Last edited:
We need to keep in mind these tests copying thousands of tiny files are not the expected usage pattern. Slow performance and inefficient use of space are an acceptable result. If we can improve performance in ways that benefit all cases, then of course we should. But when we have to male decisions that trade off performance for some cases in preference to others, we need to keep in mind which cases matter and which are not important.

Writing a small number of files, or even 1 large file, which fills nearly all space is the case we need to consider for optimization. We're using 64K erase on Winbond NOR flash chips because previous tests copying to used flash (where all writes must do erase) transferred too slowly and Windows would give up.

As to your first point I agree with you, writing 1000 small files is not expected usage pattern for flash chips. There were a couple of reasons was looking at erase size again: (1) trying to determine why when I wrote the 1024 files was it was failing to copy some files from the PC to the flash, and (2) when I started testing the W25Q64 chip saw that the overhead was so high that only about 300K would fit on a 8MB chip. Think I satisfied myself regarding item 1.

As for Item 2 I did a bit more testing this morning and last night. For I use pattern I could think that I would store images on the flash which I have done in the past. For this I copied approximately 7.4MB of png and jpgs and a gif to both the Q64 and the Q256.

In the case of the Q64:
a. 64K Block erase:
Code:
Storage Index 3 Name: Q64 Selected

 Space Used = 8323072
Filesystem Size = 8388608
Note that the disk is just about full and does show red in the Windows Explorer

b. 4096 Sector Erase
Code:
 Space Used = 7442432
Filesystem Size = 8388608
Here we have a bit more room, maybe 880K almost a 1Meg which is the more than likely the difference in filesystem overhead.

When I copied the files to the Q256 saw the same delta's. Transfer time didn't seem to be noticeable - no I didn't time it, sorry, was on the list of things to do.''

After this test there doesn't appear to be that much benefit changing back to sector erases the Q256 unless you "Writing a small number of files, or even 1 large file, which fills nearly all space is the case we need to consider for optimization". As for the Q64 not entirely sure we don't want to tweak that to maybe use 32K or 4K?????

Yes you are right we converted to using 64K erase on NOR flash chips because windows was timing out on transfers but that was also before the re-write of MTP_Teensy.

EDIT: Changed Q64 to 32k:
Code:
 Space Used = 7929856
Filesystem Size = 8388608
does recover some space
 
Think fixed... Problem was I reduced the size of some of the fields, like parent and sibling from 32 bits to 16 bits... But I also did it for child. Problem is when not a directory, child is overloaded to hold the size of the file...

So increased that one back to 32 bits...

Morning all

Just downloaded the cache branched and seems to be fixed and working. File sizes are correct, images display correctly and mp3's play to completion without a problem
 
Yes you are right we converted to using 64K erase on NOR flash chips because windows was timing out on transfers but that was also before the re-write of MTP_Teensy.

I didn't do anything in the rewrite to address Windows becoming impatient if the write speed is under ~85 Kbytes/sec
 
I didn't do anything in the rewrite to address Windows becoming impatient if the write speed is under ~85 Kbytes/sec

Funny whatever changed seems to have drastically improved the issue - haven't had a problem like we saw in the past when testing with the sector erases yesterday. In the past hit them right away with stalls or maybe I just didn't hit those cases.
 
Last edited:
With MP3 or AAC you hear - with some wrong data - only a short "blip" (if at all). This requires concentrated listening.
Better than using audio as a test would be to use a CRC.
 
I didn't do anything in the rewrite to address Windows becoming impatient if the write speed is under ~85 Kbytes/sec

In this case and like many others with these individual components plus the integration, I wonder if there any one specific generalization of what the expected user patterns will be, and as such how to prioritize things.

Example with LittleFS on Q64. If your primary usage for this chip is to upload files to it reliably using MTP, than yes the larger record sizes may make total sense. If however you never use MTP on this device and/or you only use MTP to download files from this device... Than maybe the ability to store more files may very likely have a higher priority.

Then there are people like me, who want it all :D I want as much of the space usable as possible, I want it as fast as possible and hope the machine can handle it ;) It may also depend on your tolerance level for failure/recovery.

Again depending on situations, there are several possible things one could try: If one went to a smaller size record.

a) In SendObject can we keep the connection from totally dying when we don't keep up? (USB State change)? That is do we recover?

b) With smaller size records do we fail if all of the blocks are clean, i.e. don't need to be erased? If so, are there other approaches like:
1) User is happy enough knowing that if the format the storage first they can easily transfer.
2) Can LittleFs be changed to give priority to using previously erased blocks?
3) Can we optionally setup a background/idle task, that cleans up dirty free blocks...
...

c) Can I define a way to cheat :D) example suppose I have a PSRAM, and would know none of the files I wish to store on this chip or chips, will not exceed lets say 2mb... Suppose I tell the system that it is OK
to use a 2mb buffer on the PSRAM, then could have it receive all of the data into large buffer and write it out at whatever speed that device wants...

...

Now back to playin
 
@KurtE - @PaulStoffregren

There seems to be 2 issues that I have run across with while testing all these issues:
1. Files with long file names do not copy to the flash drives. This only applies to Flash - USB and SD have no issue with long file names. I did try to change it in LFS but to no avail. Seems like the max file name length is 24. Currently LFS has a name length set to 39 so its coming from someplace else

2. If I do a format of the SDIO card - after format is fails to shows disk full with no indication of free / total space in explorer or if I try and do print directory from the sketch.?
 
Yes, Frank is correct, at least a CRC should be used. I believe Defraster's testing involves a full binary compare of the copied files against the originals.

But the (old) problem of going too slowly (under 85 kbytes/sec) was unmistakable. It was never about playing an audio file (which was plenty fast enough). That particular issue happened while copying from Windows to Teensy. While trying to copy the file, after some time Windows would give up and show a dialog box error. After that error, nothing would work at all using Teensy's files from Windows until the USB was disconnected and reconnected.

I believe we now have all the LittleFS classes able to sustain over 85 kbytes/sec speed, which involved using 32K or 64K erase sizes on the slow NOR flash chips. The downside is even a tiny file allocates at least 32K or 64K, so copying lots of them can quickly fill up the chip with wasted space. I'm pretty sure that's a problem we're just going to have to accept on slow NOR flash chips, as 4K sectors are just too slow.

The ZLP issues (hopefully now all fixed) would manifest as files having less data than they should, or some files not copied at all. I believe much of the verification for those cases used numbers for the filename, where you could easily see if the number of bytes matched the filename.
 
With MP3 or AAC you hear - with some wrong data - only a short "blip" (if at all). This requires concentrated listening.
Better than using audio as a test would be to use a CRC.

Frank - as Paul said you are correct the issue was that the file copied to the flash was of the correct size (checked by doing a printDirectory with file size) but MTP was showing the wrong file size so when I tried to play the file it Windows thougth it was an mp3 of shorter length - this is just my guess. Kurt in post #1009 identified the issue and corrected so now all is good after he fixed his fork.

But it is good to know what happens with a mp3 or aac when you have wrong data.

As Paul mentioned I have been doing a lot of coping of files of all sizes and types from the PC to Teensy flash (NORs and NANDs) and since the changes to 64K erase/block sizes haven't had any issues except as noted when coping alot of small files.
 
But the (old) problem of going too slowly (under 85 kbytes/sec) was unmistakable. It was never about playing an audio file (which was plenty fast enough). That particular issue happened while copying from Windows to Teensy. While trying to copy the file, after some time Windows would give up and show a dialog box error. After that error, nothing would work at all using Teensy's files from Windows until the USB was disconnected and reconnected.

I believe we now have all the LittleFS classes able to sustain over 85 kbytes/sec speed, which involved using 32K or 64K erase sizes on the slow NOR flash chips. The downside is even a tiny file allocates at least 32K or 64K, so copying lots of them can quickly fill up the chip with wasted space. I'm pretty sure that's a problem we're just going to have to accept on slow NOR flash chips, as 4K sectors are just too slow.
Yep - been there and pulled hair out... But is it still true with current code? Two parts:
a) LittleFS on these chips will have problems with to many dirty blocks... - Would still be true.
b) If error USB will not recover? If host is not happy: will we see:
Code:
bool MTP_class::receive_bulk(uint32_t timeout) { // T4
  if (usb_mtp_status != 0x01) return false;
And then the SendObject( code will bail, and hopefully we reset back to ...status = 0x1;
Does the USB recover?

If it does than maybe bumping up to 32K may not be a clear win...

Regardless maybe at a minimum, there should be some warnings/suggestions anywhere we talk about LittleFS and supported chips, something like: that if at all possible you should avoid using these chips...
 
On a different integration subject:

Suppose, I setup a Teensy with lets say an ILI9341 display (could be music), that I have as a electronic picture frame or some such thing. Suppose it cycles through the pictures contained within some directory on some storage.
And 95 percent of the time it is running off of Battery or 5v wall wart...

And only occasionally I this device in to USB port, to change the set of image files contained on the storage using MTP (delete some, add some new ones)...
And then I unplug it from USB...

There are probably many questions, options to think about, like:

Can the sketch detect when the USB Port is connected, like: if (MTP) { ok we have mtp }
Likewise when disconnected?

When/What code should handle the init and when. That is does the sketch need to do all of the initialization up front, or only when the usb is connected? All of our examples do everything up front.

Buffers and the like: maybe I want to use the DMAMEM for SendObject or the like when connected. When not connected, maybe I use it as a secondary frame buffer for the display that I use for transitions from one picture to another..

Code may go into different display mode when usb connected...

EDIT: Can the sketch control if MTP works or not. Example suppose I wish to have the screen change, when USB is plugged in and prompt the user for a password, as I would not like my dogs (Laik and Annie) to change which pictures are shown. As maybe Laik thinks I have too many pictures of Annie on it and not enough of him :)

...

Probably lots of other issues/opportunities ...

Guessing won't know, until try a skeleton sketch.

Thoughts
 
Last edited:
Seems like the max file name length is 24.

I tried but could not reproduce this issue.

First I renamed a file on the SD card to "06abc678901234567890123456789.jpg". Then I copied to the Windows desktop, then copied it to the SPI flash chip. I rebooted Teensy. It was still there on both with all 33 chars in the filename. Then I deleted the file from the desktop, and copied from the SPI flash to desktop again. Still all 33 chars.

capture.png


Is there any chance your flash chip got formatted with a copy of LittleFS set to only 24 char filename limit? I believe LittleFS uses the smaller of its config and whatever was used at the time the media was formatted, so even if you've changed the source code, you'll probably still see the smaller size until the media is reformatted.
 
Back
Top