MTP file size limits.

Can GetObjectPartial support 64 bits?
Simple answer, I have no idea.
The description includes:
The operation is identical to GetObject, except that the second and third parameters contain the offset in bytes and the number of bytes to obtain starting from the offset. If the entire object is desired, starting from the offset in the second parameter, the third parameter may be set to 0xFFFFFFFF. The first response parameter shall contain the actual number of bytes of the object sent, not including any wrappers or overhead structures
So as you mentioned the offset can not be > 4gb... So far I have not found anyone who calls GetPartialObject, so not sure.
But for sure it should at least understand that the local file is > 4gb
 
Just a quick update. Been playing with the latest set of changes from this morning and still having issues with 4 and 5gb file transfers. I am currently using a win 11 pc using ide 2.3.3 with 0.60.3 installed.

1. From sdcard takes about 8.5 minutes for a 5gb file and about an hour from the usb stick!

2. Transferring from either a SDCard or USB Stick a 4Gb or 5gb files transferred but doing a file compare showed the two files did not match the original files.

Used winmerge to compare files and https://myjob.page/tools/test-files to generate the test files.

1730558980230.png

1730559009736.png


for testing using a micromod on Paul's micromod test board we used years ago.
 
Both Windows 11 Media Player and whatever unnamed media player Ubuntu 24.04 Gnome Files defaults to using cause GetPartialObject to be requested.
 
1. From sdcard takes about 8.5 minutes for a 5gb file and about an hour from the usb stick!
I can confirm the timings!

:eek: >1 hour to copy over MSC to USB SSD drive. I have not waited that long yet...
May play around some with how it is output to see if it can be sped up some without major changes...

Both Windows 11 Media Player and whatever unnamed media player Ubuntu
On the one sound file I tried on Windows, it downloaded the whole thing. However running it on RPI5 the VLC media player is doing partials.
So will experiment more.

EDIT: I am trying a copy operation from SD to USB drive. And I should probably time it, but it probably took > 15 minutes and it is doing reads/writes of 4kb... So any speedup probably needs to be done within MSC.

Edit 2, looks like I also need to check the copy code...


1730561277329.png
1730561370735.png

As it looks like the sizes don't match...
 
Last edited:
Not sure same thread stuff, or different thread, but I know that the MSC code is a LOT slower than the the exFat SDIO card, and I believe it also has limits of 4gb on a file. Not sure if by bug or ...

I updated one of @mjs513 test programs for mtp, that allows you to generate a large file. I made it such that it could generate >4GB file.
That code is pretty simple, I just modified a few places to be uint64_t instead of uint32_t...
Code:
void test_write_file(int ch) {
  test_file_write_size = CommandLineReadNextNumber(ch, test_file_write_size);
  test_file_size = CommandLineReadNextNumber(ch, test_file_size);
  if (ch >= ' ') {
    Serial.setTimeout(0);
    int cb = Serial.readBytesUntil( '\n', test_file_name, sizeof(test_file_name));
  }
  File testFile; // Specifes that dataFile is of File type

  myfs->remove(test_file_name);  // try to remove files before as to force it to reallocate the file...
 
  testFile = myfs->open(test_file_name, FILE_WRITE_BEGIN);
  if (!testFile) {
    Serial.printf("Failed to open %s\n", test_file_name);
    return;
  }
  Serial.printf("Start write file: %s length:%llu Write Size:%u\n", test_file_name, test_file_size, test_file_write_size); 

  uint32_t cb_write = test_file_write_size;
  uint64_t cb_left = test_file_size;
  uint8_t fill_char = '0';
  test_file_buffer[cb_write - 2] = '\n'; // make in to text...
  uint32_t percent_left_prev = (cb_left * 100ul) / test_file_size;
  elapsedMillis em;
  while (cb_left) {
    if (cb_left < cb_write) cb_write = cb_left;
    memset(test_file_buffer, cb_write - 1, fill_char);
    testFile.write(test_file_buffer, cb_write);
    fill_char = (fill_char == '9')? '0' : fill_char + 1;
    cb_left -= cb_write;
    uint32_t percent_left = (cb_left * 100ul) / test_file_size;
    while (percent_left < percent_left_prev) {
        percent_left_prev--;
        Serial.write('.');
    }

  } 
  testFile.close();
  Serial.printf("\nTime to write: %u\n", (uint32_t)em);
 
}
Note: on the SSD USB drive run I did not have the quick and dirty code to print out '.'s I added it as to make sure we were
still making progress.
Code:
Start write file: large_test_file.txt length:5000000000 Write Size:512
Time to write: 3156663
Yep that is 52.6 minutes to create the file

I then ran the same thing to the SD Card:
Storage Index 1 Name: BUILTIN Selected
Code:
Start write file: large_test_file.txt length:5000000000 Write Size:512
....................................................................................................
Time to write: 223827
Or about: 3.7 minutes.

And then checked to see what was created:
The MSC drive was < 4GB
1730575833752.png

where as the SD card was the correct size
1730575897116.png


I am guessing that USB/MSC has issue of 32 bit versus 64 bit at some point(s) in the code
 
Not sure if better to edit previous post, or new one here. I added code to test if the number bytes written did not equal the number
of bytes requested and bail and print info...
Code:
    size_t cb_written = testFile.write(test_file_buffer, cb_write);
    if (cb_written != cb_write) {
        uint64_t cb_output = test_file_size - cb_left;
        Serial.printf("\n\!Write failed cb left:%llu  output:%llu(0x%llx)\n", cb_left, cb_output, cb_output);
        break;
    }
This time I tried writing 4096 bytes at a time instead of 512.
And it failed:
Code:
Storage Index 3 Name: MSC0 Selected
Start write file: large_test_file.txt length:5000000000 Write Size:4096
......................................................................................
!Write failed cb left:705036800  output:4294963200(0xfffff000)

Time to write: 398205
But it was a lot quicker with 4096 bytes per write:
It failed after about 4GB was written in about 6.6 minutes.

Next up, verify that this 256GB SSD drive is formated as exfat
Actually it might be Fat32... will try to switch it and or drive different one
 
Last edited:
Assuming we get to the end and have MTP fully support 64-bit file sizes, would it be possible to actually put the MTP bits into Teensydunio 1.60?
 
would it be possible to actually put the MTP bits into Teensydunio 1.60?

Definitely not ready for 1.60 in its current state. I'm diving into the code now to really look at what's needed.

Support for larger than 4GB files is not a huge concern (at least for 1.60...) I'm also not really worried about MSC slow speed for huge files, as long as it's reliable.
 
Successfully copied a 6GB file from a SD card (FAT64 format) to my Linux desktop. Didn't time it, but took several minutes. SHA256 and full file compare confirms every byte correctly received.

FWIW, turns out this 64GB Sandisk card was actually FAT32 format. Had to reformat the card before I could put the 6GB file onto it.
 
Successfully copied a 6GB file from a SD card (FAT64 format) to my Linux desktop. Didn't time it, but took several minutes. SHA256 and full file compare confirms every byte correctly received.

FWIW, turns out this 64GB Sandisk card was actually FAT32 format. Had to reformat the card before I could put the 6GB file onto it.
Yep - My 128GB SDCards were exfat... So was able to upload, others failed and left the disk with size like 3.9GB... I just confirmed I could create a 5GB file on my other SSD that I reformatted to exFat... Now to try to download that file.
With SDCard expect about 8.5 minutes. For USB something over an hour.
1730582398522.png

While this transfer is happening, we cannot receive (or send any messages) to MTP host, As the endpoint is busy...
We have similar issue with copy/move operations, like from one storage to another, as again the write takes a long time.
But will see how well it works
 
While this transfer is happening, we cannot receive (or send any messages) to MTP host, As the endpoint is busy...
We have similar issue with copy/move operations, like from one storage to another, as again the write takes a long time.
But will see how well it works
Yep noticed the same thing. Still trying to get everything synched on my win 10 machine
 
If you wanted to try a different USB mass storage implementation, I have an MTP sample in my repo. I added extra handling to try to make it gracefully handle the device being hotplugged although there are still issues that can only be addressed internally in the MTP code. But I suspect the slow speed is mostly due to reading/writing small chunks of data at a time, the intermediate buffer really needs to be at least 32KB.
 
Still wondering if FS.h should have a member function to return the file system type. Sort of along the SDFat methed fatType which
returns which type of FAT.

Would be nice if it would return: FAT, exFAT, LittleFS, Other or some such thing. (FAT could be broken down to FAT16 and FAT32 if desired),
But in the case of size passed in of 0xfffffffful, we could then see if the FS is exFat or maybe other with warning....
 
But I suspect the slow speed is mostly due to reading/writing small chunks of data at a time, the intermediate buffer really needs to be at least 32KB.
Yep the current upload/download tries to write out each packet as received (or sent) so either 500 (first) and 512 after that.

We had other implementations before, where we would cache up the data into larger buffers, but it was problematic on how much as if the write took too long, and we had not read in the next buffer, the MTP Host would get mad and drop us. So always was trying to find a happy
medium.
 
I have developed a wildlife image-capture app for the FLIR Lepton thermal imaging camera. It is designed to capture about 8 38Kbyte frames /second when an object above a threshold temperature is in the frame. I must have set the threshold too low in the last 36-hour session, as I recorded about 7.4GB of data. The data was recorded on a 128GB micro-sd card formatted for EX-FAT.

The 7.4GB number is the size reported by the Teensy SD directory command. Time and date are good. I haven't opened the rain-resistant case to pull the card and read the data with a PC SD card reader, but have been using MTP to get the files.

Here's the issue: The PC MTP folder view shows the file to be only 3.4GB in length. A transfer to my PC resulted in a 3.4GB file.

I guess the question is: Can Teensy MTP handle files larger than 4GB?

If teensy MTP is indeed limited to 4GB files, I can add code to close a file and open a new file when the length approaches 4GB.

I would strongly suggest to keep the files size below a maximum that's far below 4GB. I suppose those files are to be moved around and that's slow for big files. Also if it's 1 file you have to wait until it's downloaded totally. When you have several, you can start processing the first files while downloading the others.

Quite possibly you can have 2 limits:
- a lower soft one which is used when you're app stops recording (threshold lower) --> no problems with lost frames.
- a hard one when approaching 1,9 GB (for example)
 
@KurtE - Could I talk you into creating another branch from the large_files branch? I'll use it to do the stuff I was planning before the chip shortages hit. I normally use only the simple git command and github website, so syncing back to your repository will be much simpler (for me) if the branch starts created on your repository.

About MTP in general...

So always was trying to find a happy medium.

Indeed, and my main goal before merging this into the core library is the trade-off between optimizing the transfer performance versus blocking the user's application. As currently implemented, a call to MTP.loop() can take many seconds or even minutes if the host wants to transfer a large file. I'm going to restructure the code as a state machine, so each call to MTP.loop() moves at most 1 more block.


Still wondering if FS.h should have a member function to return the file system type.

FS is meant to abstract the filesystem, so adding this sort of API pretty much goes against the primary purpose of FS.

If we do need more functions in FS or File, they need to be carefully considered. So my main question is why do you want to query the type of underlying filesystem? What will you actually do with that information? Is it only to know whether files larger than 2^32 bytes are supported? I saw a mention of file attributes (hidden, system, etc). Is there something else?

Regarding FS, also on my near term goals is moving the MSC presence stuff into USBHost_t36, under the FS.mediaPresent(). Likewise for SD / SdFat, we already have FS.mediaPresent() but perhaps it's not yet fully up to the task of usage from MTP?
 
I saw a mention of file attributes (hidden, system, etc). Is there something else?
Retrieving/assigning the volume label, for one thing. Also retrieving the max supported filename length and checking if the FS is read-only without actually attempting to write something.
What I would propose is one simple extendable general purpose function:
Code:
virtual int ioctl(uint32_t function_num, const void* in_buffer, size_t in_length, void* out_buffer, size_t out_length) { return -1;}

Any class that inherits class FS could override this if desired, or they could just keep the default implementation.
Values for function_num under a certain threshold would be reserved for "well defined" purposes, like the ones already mentioned - getting/setting the volume label, getting the max supported filesize, file attribute handling etc. Values over the threshold would be free for anyone to use for any purpose.
 
@KurtE - Could I talk you into creating another branch from the large_files branch? I'll use it to do the stuff I was planning before the chip shortages hit. I normally use only the simple git command and github website, so syncing back to your repository will be much simpler (for me) if the branch starts created on your repository.
Sorry, I took yesterday late afternoon and evening off. I created: pauls_branch

I hear you about git commands. These days I normally do most stuff using the github desktop. Which you can now install on linux as well.
I even have it installed on my RPI5.

Note: yesterday I pushed up some new versions of the sendObject and getObject - that is #if on a #define in the file as well as only
on T4.x. It uses the 4k static buffer (disk_buffer_) we had defined for T4.x but was not in use. With this:
The sending or receiving a 5GB file went from > hour to about 10 minutes. It would need lots more testing, before merging into main...

FS is meant to abstract the filesystem, so adding this sort of API pretty much goes against the primary purpose of FS.
Yes and No... In the same way why does the SD library allow you access to SDFat?
But here are some of the things that code like MTP might want to know about the underlying file system.
a) cluster size - probably not the best name, but more like preferred write size. That is 4K works a lot better on many fat systems than 512 bytes, but littleFS maybe not so much.
b) Max file name length
c) Max file size
...

Alternative to this is to allow some form of callouts, to the sketch that can influence such things.

Indeed, and my main goal before merging this into the core library is the trade-off between optimizing the transfer performance versus blocking the user's application. As currently implemented, a call to MTP.loop() can take many seconds or even minutes if the host wants to transfer a large file. I'm going to restructure the code as a state machine, so each call to MTP.loop() moves at most 1 more block.
Sometimes it is hard to find the right tradeoffs, as each of us may have different priorities for different things in their sketch. For example, someone who plugs in their Teensy to a PC to download their log files may want that done as fast as possible, as to not have to wait around for a long period of time, whereas someone else, may simply want to copy a new song to their jukebox app and not want it to interfere listening to their current music.

That is where I really wish we had some form of tasking, where one could set the priorities of each task, in much the same way one can now set
the priority of different interrupts....

Now back to playing... And merging
 
@PaulStoffregen I re-forked the pauls_branch after merging in your PR...

Note: I believe your changes may have busted the MSC integration into MTP... At least the volume I was working with no longer shows up.
I was testing with a modified version of a test sketch by @mjs513.

I think the zip file here is pretty up to date with what I was playing with.

Maybe we should coordinate so we don't get merge conflicts?
I think I might take a break from playing with MTP and play with some other things until the dust settles.
 

Attachments

  • mtp-test-integrityV2-241104a.zip
    11.8 KB · Views: 16
@KurtE
Played a bunch yesterday with you large-files branch. Ran 5gb file on the sd card and the USB Stick.

On the SD Card was able to successfully write 5gb to the sd card where as before it was an issue:
1730894722533.png


On the usb stick wrote 5gb with 512 and 4096 blocks:
Code:
Start write file: test5busb.txt length:5000000000 Write Size:512
.................................................................
Time to write: 8012085



Start write packet indexed file: test5gn2.txt length:5000000000 Write Size:4096
0000000000 - ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789AB

0000000002 - ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789AB

...........................................................................................
Time to write: 1476255
1730894925207.png


then tried a copy from the USB Stick to the laptop - still showing some issues:
1730894960462.png

Looking at the details of the diff - seems to be a couple of blocks:
1730895094494.png


Not going to worry about the media branch until MSC gets operational again.
 
Added the missing mediaPresent() function to USBFilesystem.
Thanks Paul.

I synced up the USBHost changes and I am still not seeing the USB drive show up in the MTP sketch I posted.

Will try to debug.

EDIT: Confirmed - So unless I am missing something, your changes removed any code, that could detect that a new USB drive was inserted and enumerate to find any volumes we could add to the list of storages...

Your change to USBHost:
Code:
bool mediaPresent() { return *this; }
Will I assume indirectly call:
Code:
operator bool() {
        // use of volatile prevents compiler from optimizing away
        // re-reading the pointer if program repeated checks bool()
        USBDrive *dev = *(USBDrive * volatile *)&device;
        return dev != nullptr;
    }

But as no one actually adds any volumes in... This would never be called.

Looks like we are back to the day where each of the sketches has to add their own code to detect when a
USB device is inserted and do the disk enumeration to find any volumes to the MTP list.

And/Or maybe adding code to USBHost and elsewhere, that manages active disks, with someway for a sketch and/or other
subsystem can register some form of callback to know that something changed.

Will be interesting to see what you come up with.
 
Last edited:
Looks like we are back to the day where each of the sketches has to add their own code to detect when a
USB device is inserted and do the disk enumeration to find any volumes to the MTP list.

And/Or maybe adding code to USBHost and elsewhere, that manages active disks, with someway for a sketch and/or other
subsystem can register some form of callback to know that something changed.

Will be interesting to see what you come up with.
As you said neuters mtp. Even more reason to stick with your large file branch for now. Spent a lot of time incorporating msc into mtp so definitely will be interesting to see any updates.
 
Back
Top