MTP file size limits.

M

mborgerson

Well-known member
I have developed a wildlife image-capture app for the FLIR Lepton thermal imaging camera. It is designed to capture about 8 38Kbyte frames /second when an object above a threshold temperature is in the frame. I must have set the threshold too low in the last 36-hour session, as I recorded about 7.4GB of data. The data was recorded on a 128GB micro-sd card formatted for EX-FAT.

The 7.4GB number is the size reported by the Teensy SD directory command. Time and date are good. I haven't opened the rain-resistant case to pull the card and read the data with a PC SD card reader, but have been using MTP to get the files.

Here's the issue: The PC MTP folder view shows the file to be only 3.4GB in length. A transfer to my PC resulted in a 3.4GB file.

I guess the question is: Can Teensy MTP handle files larger than 4GB?

If teensy MTP is indeed limited to 4GB files, I can add code to close a file and open a new file when the length approaches 4GB.
 
The MTP code is littered with places where 32-bit values are used for filesizes.
MTP does support larger files (at least the windows implementation supports it; whatever is used on macs might not) but it doesn't appear to be implemented in the Teensy library.
 
Sorry, I am really really rusty on the MTP stuff. Other than fixing some sketches that were not compiling, we have not done anything for about two years...

But as for large files, I believe it is a limitation of MTP. That is if you look at the specification of the ObjectInfo that gets sent or received with information about objects that we are about to send or about to receive...
1730035916142.png

The size field is only 32 bits.
 
It's not. If the size is >=4GB the compressed object size should be set to 0xFFFFFFFF and the actual size retrieved using SendObjectPropertyList (which uses two 32-bit size fields, upper and lower halves of a 64-bit value).
There's also an appendix in the specification specifically detailing how to transfer files larger than 4GB.
 
Last edited:
jmarsh said:
It's not. If the size is >=4GB the compressed object size should be set to 0xFFFFFFFF and the actual size retrieved using SendObjectPropertyList (which uses two 32-bit size fields, upper and lower halves of a 64-bit value).
There's also an appendix in the specification specifically detailing how to transfer files larger than 4GB.
Click to expand...
You are correct! Appendix H does show how to send and receive files > 4GB.
Which I am pretty sure we have not implemented.

@mborgerson - you might want to create an issue against the library. Hopefully at some point in the near future we will get back to fleshing out and releasing the library. As I mentioned all of us stopped working on the library a couple years ago, most of the file transfer stuff probably three years ago. At that time, Paul was planning to rewrite the transfer code, as to minimize any impact it might have on Audio... And then part shortages hit and ...
 
I thought I would mention, that I have started hacking on the MTP code to hopefully handle 64 bits. Not there yet, but making some
progress.

WIP branch - https://github.com/KurtE/MTP_Teensy/tree/large_files

For the fun of it, doing the code on RPI5, with IDE2, Sublimetext, github desktop, TyCommander, ...
Typing this in Chromium.

First problem I ran into, was the SDCard I was trying to put the file on was a 32gb card with Fat32... Fat32 does not handle files > 4GB...

20241028_14h47m49s_grim.png

But it is now at least reporting the file size with the full 64 bits.

Now to start working on transfers.
 
@KurtE
Downloaded your latest changes and loaded it up on a T3.6 using a USB stick (ps running on a Windows 7 maching running 1.8.19). Loaded up a 5GB file:
1730237789590.png


however noticed that when trying to look at the subdirectories only works when I click on a directory when the drive first connect - will have to try on a t4.1
 
Finally loaded up my MTP test sketch and didn't seem to have the issue with opening different directories. Did try to transfer the 5gb file but after about 1GB a MTP reset was issued and transfer stop. Tested with 2 drives - USB and SD card.
 
mjs513 said:
Finally loaded up my MTP test sketch and didn't seem to have the issue with opening different directories. Did try to transfer the 5gb file but after about 1GB a MTP reset was issued and transfer stop. Tested with 2 drives - USB and SD card.
Click to expand...
Try sync up again, I pushed up fix for the exiting at the early time. One last variable needed to change from uint32_t to uint64_t

I have had the file download successfully. Turned off debug And Windows and downloaded again, and looked the right size. But
windows complained it was not a valid zip file. But then I opened up terminal window, and compared the files and it looks like
they compared:
Code: 
D:\RaspberryPi_transfers>dir
 Volume in drive D is DATA
 Volume Serial Number is 5C92-8524

 Directory of D:\RaspberryPi_transfers

10/30/2024  10:21 AM    <DIR>          .
10/18/2024  10:38 AM       186,928,344 arduino-ide_2.3.3-snapshot-77136687_Linux_arm64.zip
07/28/2022  02:03 PM    <DIR>          Clare
10/28/2024  12:17 PM           893,123 favorites_10_28_24.html
10/28/2024  12:50 PM     4,970,478,511 HugeZipFile.zip
10/28/2024  12:50 PM     4,970,478,511 HugeZipFile_orig.zip
               4 File(s) 10,128,778,489 bytes
               2 Dir(s)  695,918,182,400 bytes free

D:\RaspberryPi_transfers>comp
Name of first file to compare: HugeZipFile.zip
Name of second file to compare: HugeZipFile_orig.zip
Option:
Comparing HugeZipFile.zip and HugeZipFile_orig.zip...
Files compare OK

Compare more files (Y/N) ? n

So looking better. I still have not done any tests or code fixing to go the other direction yet
 
Trying the latest code. From Linux, seems I can't write even a small file, using either main or large_files branch. :(

Will dig into the details soon...
 
@KurtE

Just tried downloading my 5.2GB file from a USB drive. It failed:
1730377488829.png


This was on my Win11 machine running 0.60.3. Last night I tried it on a Win 7 machine running 1.59 and same thing but was able to see that about 4.99Gb transferred before failing.

Just as a note I am able to transfer files back and forth between PC and USB Stick as well as the sd card
 
I was able to upload a photo again on W11 machine, and then double click and the picture came up.
1730380876321.png

Now to work on larger ones - which always were a PIA even before worrying about >4GB
Currently my Ubuntu machine is not hooked up. But was able to do similar a couple days agon on RPI5 running Raspberry OS 64 bits.
 
Quick update: Fixed one upload a file bug, where it did not compute free space correctly (32 bits) so depending on
size test would fail if the lower 32 bits of free space was < size of file to be uploaded.

Note: I left debug output turned on.
 
KurtE said:
What version of Ubuntu? Assuming TD60.3? Upload to or download from teensy.
Click to expand...
I have my ubuntu machine hooked up again. I am pretty sure I am running 22.04, should I update it to 24.04?

I did build the current stuff and was able to send some files to the teensy and retrieve some. Have not tried yet with large files.

Again I should mention, files > 4GB size are not allowed on FA32 volumes (I assume true for Fat16 as well), but they are supported
on ExFat. Question I have for self, can I detect which FS the Storage is?
 
KurtE said:
Question I have for self, can I detect which FS the Storage is?
Click to expand...
I have wondered for a while if the FS class should have some sort of FriendlyName() function that identifies the underlying filesystem type, or maybe a generic ioctl() function that would allow individual FS implementations to support additional functions (e.g. returning volume labels, setting file attributes like hidden or system, etc.).
It is possible to identify exactly what type of class is implementing FS using c++ features, but then you're stuck supporting only the types that your program knows about at compilation time.
 
jmarsh said:
It is possible to identify exactly what type of class is implementing FS using c++ features, but then you're stuck supporting only the types that your program knows about at compilation time.
Click to expand...
At one point we had code that we could pass in some additional information about an FS to MTP when we added an object. However, Paul wanted the code to be simpler, so the only thing we know is it is an FS... So probably the only way we will find out is if a write fails after we hit 4gb and we have space left...

Working on the large file upload, have it actually sending the whole data, but I am not processing it all properly yet.
Some debug output:
Code: 
loop:12606 CMD: 100c(SEND_OBJECT_INFO)l: 20 T:192 : 10001 2b
SendObjectInfo: 10001 2b Dataset len=182
File "HugeZipFile_orig.zip" size:4294967295 Created:6723d2b3 Modified:671f8895
 read consumed all data (TODO: how to check ZLP)
Size of object == 0xffffffff - Indicates >= 4GB file!
     ?TODO: query real size? FS supports this - FAT32 no?
loop:12625 CMD: 100d(SEND_OBJECT)l: 12 T:193
SendObject: 4294967295(0xffffffff) bytes, id=35
SO RC:1 CB:500 pos:0
SO RC:2 CB:512 pos:500
SO RC:9707966 CB:443 pos:4970478068
SO: receive failed pos:4970478511 size:18446744073709551615
SendObject complete
loop:301003 CMD: 1005(GET_STORAGE_INFO)l: 16 T:194 : 10001
65537 0 name:SD1

The file size is: 4970478511
I probably can detect it as the last packet I received length was < 512 bytes...
Will play more in morning
 
Uploaded large file :D debug is still on... Currently taking about 8.5 minutes from/to my 128gb SD card.
Appears to be a lot slower on some thumb drives. Wondering how much it would help to align the Reads and Writes.
That is the first packet we send out has 12 bytes of overhead so we output 500 bytes, all of the remaining ones
we output 512, until last one... May have to experiment some.


Code: 
loop:73169 CMD: 100c(SEND_OBJECT_INFO)l: 20 T:193 : 10001 2b
SendObjectInfo: 10001 2b Dataset len=182
File "HugeZipFile_orig.zip" size:4294967295 Created:67247c87 Modified:671f8895
 read consumed all data (TODO: how to check ZLP)
Size of object == 0xffffffff - Indicates >= 4GB file!
     ?TODO: query real size? FS supports this - FAT32 no?
loop:73180 CMD: 100d(SEND_OBJECT)l: 12 T:194
SendObject: 4294967295(0xffffffff) bytes, id=35
SO RC:1 CB:500 pos:0
SO RC:2 CB:512 pos:500
SO RC:9707966 CB:443 pos:4970478068
>4gb file EOF detected pos:4970478511
SendObject complete
loop:361740 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:195 : 35 dc02 (FORMAT)
loop:361740 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:196 : 35 dc01 (STORAGE_ID)
loop:361740 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:197 : 35 dc07 (OBJECT NAME)
loop:361741 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:198 : 35 dc0b (PARENT)
loop:361741 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:199 : 35 dc41 (PERSISTENT_UID)
loop:361741 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:19a : 35 dc44 (NAME)
loop:361742 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:19b : 35 dc03 (PROTECTION)
loop:361743 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:19c : 35 dc04 (SIZE)
    MTP_PROPERTY_OBJECT_SIZE: HugeZipFile_orig.zip 128437baf
loop:361743 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:19d : 35 dc08 (CREATED)
loop:361743 CMD: 9803(GET_OBJECT_PROP_VALUE)l: 20 T:19e : 35 dc09 (MODIFIED)
loop:361745 CMD: 1008(GET_OBJECT_INFO)l: 16 T:19f : 35
loop:361747 CMD: 1005(GET_STORAGE_INFO)l: 16 T:1a0 : 10001
65537 0 name:SD1
1730471385684.png
 
Tested the latest with Windows 11. Copying a WAV file works well. But playing it in Media Player has dropouts. Protocol analyzer shows 1 large GetObject transfer for the file copy, but when Media Player is used Windows sends several GetObject + CancelRequest.

Going to move back to Linux (Ubuntu 24.04) this morning and try to look into why it's not work.
 
Updating my Ubuntu machine to 24.04 may take a few hours...

PaulStoffregen said:
but when Media Player is used Windows sends several GetObject + CancelRequest.
Click to expand...
Would not surprise me if it does different things. Guessing it probably is downloading to a temporary file, that it plays. I know with some image files it comes up fine, other larger ones not always.
 
I have ubuntu 24.04 installed now. and MTP appears to somewhat work.
1730492118452.png

I was able to download this jpeg file and then open it... But it did not want to show the image if I double clicked on it in the
MTP window. Or maybe I did not wait long enough.
But it was able to directly open a smaller BMP file.
1730492406210.png

Maybe yesterday I should have used this photo, which I viewed from MTP disk on windows:
1730492809107.png
 
Last edited:
Sent a pull request to fix size handling in GetObjectPartial.

This may not be the only issue. I still hear weird artifacts in Windows Media Player. Playing on Linux is also not 100%, but it at least works for a while now.
 
PaulStoffregen said:
Sent a pull request to fix size handling in GetObjectPartial.

This may not be the only issue. I still hear weird artifacts in Windows Media Player. Playing on Linux is also not 100%, but it at least works for a while now.
Click to expand...
Merged,

Which apps/user actions call this? I see several more things that probably need fixing in it.
Like the size/pos fields probably need to be 64 bit. The size field argument is the maximum size to transfer and
we need to check the actual size returned from object and return that count... i.e. we may hit EOF.

Not sure in this case if 0xfffffffful for size means rest of file even when file > 4GB? The offset can not exceed 4GB...

Thanks
Kurt

EDIT: the size returned is probably taken care of by the code:
Code: 
if (NumBytes > size - offset) {
    NumBytes = size - offset;
  }

So far I changed those two fields to 64 bits. Not sure what to do in >4gb case. I will take a couple of guesses, along
the GetObject stuff... Where size of 0xfffffffful means the rest of file, so in this case will check for that and change
size to 64 bit version. And on return if count > 4gb will return the 0xfffffffful
 
Last edited:
Can GetObjectPartial support 64 bits?

I don't see it mentioned in Appendix H and I don't understand how the host could request the offset or size as 64 bits. Maybe I missed something?
 
You must log in or register to reply here.
Back
Top