Teensyduino File System Integration, including MTP and MSC

There are three open modes currently defined in FS.h
Code:
#define FILE_READ  0
#define FILE_WRITE 1
#define FILE_WRITE_BEGIN 2
And if you look at SD.h

Code:
		oflag_t flags = O_READ;
		if (mode == FILE_WRITE) flags = O_RDWR | O_CREAT | O_AT_END;
		else if (mode == FILE_WRITE_BEGIN) flags = O_RDWR | O_CREAT;

So as you can see FILE_WRITE opens at the end of file.

Note: SDFat does have other options:
Code:
#define O_RDONLY  0X00  ///< Open for reading only.
#define O_WRONLY  0X01  ///< Open for writing only.
#define O_RDWR    0X02  ///< Open for reading and writing.
#define O_AT_END  0X04  ///< Open at EOF.
#define O_APPEND  0X08  ///< Set append mode.
#define O_CREAT   0x10  ///< Create file if it does not exist.
#define O_TRUNC   0x20  ///< Truncate file to zero length.
#define O_EXCL    0x40  ///< Fail if the file exists.
#define O_SYNC    0x80  ///< Synchronized write I/O operations.
At times I wonder if FILE_WRITE_BEGIN should include O_TRUNC but currently does not.

I would assume for new created file or one with 0 length beginning and end are the same.
 
With FILE_WRITE_BEGIN having O_RDWR - some users might open and expect to read prior data before writing?

Adding O_TRUNC would remove that option?

But it doesn't seem a standard mode by name 'FILE_WRITE_BEGIN' anywhere and Reference/SDopen

So whatever makes sense - it is an addition ...

Python has this that would include O_TRUNC :
Code:
w+
Opens a file for both writing and reading. Overwrites the existing file if the file exists. If the file does not exist, creates a new file for reading and writing.
 
@mjs513 - Think I figured out what the trouble was. MY FAULT!! I was compiling ReadWrite.ino In Arduino 1.8.19 but was editing ReadWrite.ino in Arduino 1.8.18. You were right. @All - Thanks for the input. I know see that if FILE_WRITE does append to the end of the current file. FILE_WRITE_BEGIN always starts at the beginning of the file. So if the same file is reused with FILE_WRITE_BEGIN the write will start at position 0. What was confusing me was say the appended file size was 144 bytes and I opened it again with FILE_WRITE_BEGIN and wrote 72 bytes then closed the file it would read 144 bytes. So in theory I would need to truncate the file at 72 bytes then close the file. Or delete the file before opening it again. Truncating the file worked but it will have to keep track of how many bytes were written to the file before truncating and closing. Possibly use myFile.position() as an argument to truncate.

Thanks everybody for the help:)

Edit:
This works:
Code:
myFile.truncate(myFile.position());
 
Last edited:
@mjs513 - Think I figured out what the trouble was. MY FAULT!! I was compiling ReadWrite.ino In Arduino 1.8.19 but was editing ReadWrite.ino in Arduino 1.8.18. You were right. @All - Thanks for the input. I know see that if FILE_WRITE does append to the end of the current file. FILE_WRITE_BEGIN always starts at the beginning of the file. So if the same file is reused with FILE_WRITE_BEGIN the write will start at position 0. What was confusing me was say the appended file size was 144 bytes and I opened it again with FILE_WRITE_BEGIN and wrote 72 bytes then closed the file it would read 144 bytes. So in theory I would need to truncate the file at 72 bytes then close the file. Or delete the file before opening it again. Truncating the file worked but it will have to keep track of how many bytes were written to the file before truncating and closing. Possibly use myFile.position() as an argument to truncate.

Thanks everybody for the help:)

Wrong src edit is a pain ... :)

That seems a good and usable feature/process if behavior is known.
> User can read prior data and write beyond it - or seek, truncate as needed with a single open()
 
Wrong src edit is a pain ... :)

That seems a good and usable feature/process if behavior is known.
> User can read prior data and write beyond it - or seek, truncate as needed with a single open()

You are right. Keeping track of three computers always gives me teeth marks:) As per the EDIT: above myFile.truncate(myFile.posistion()) works well.

Edit:

As a side note, adding this to the POSIX close function works:
Code:
	fcbs[file & 0xff].truncate(fcbs[file & 0xff].position());
 
Last edited:
Out of curiosity I used @KurtE's memory breakout boards to test the following chips to make sure the still worked with the changes we made:
Code:
AdestoAtmel
W25Q64
W25Q256JW*M (new chip which I ordered by mistake)
W25Q256JV*Q
W25Q01

IMG-0562.png

Ran the a few quick benchmarks:
Code:
[B]Adesto/Atmel AT25SF041 (512K): [/B]
Writing 524288 byte file...  1321 ms, bandwidth = 396887 bytes/sec
Writing 524288 byte file...  8293 ms, bandwidth = 63220 bytes/sec
Writing 524288 byte file...  8294 ms, bandwidth = 63212 bytes/sec
Writing 524288 byte file...  8311 ms, bandwidth = 63083 bytes/sec
Writing 524288 byte file...  8305 ms, bandwidth = 63129 bytes/sec
Writing 524288 byte file...  8309 ms, bandwidth = 63098 bytes/sec
Writing 524288 byte file...  8311 ms, bandwidth = 63083 bytes/sec
Writing 524288 byte file...  8298 ms, bandwidth = 63182 bytes/sec
Writing 524288 byte file...  8285 ms, bandwidth = 63281 bytes/sec
Writing 524288 byte file...  8295 ms, bandwidth = 63205 bytes/sec
Bandwidth test finished

Start Big write of 217088 Bytes
Big write /0_bigfile.txt took  3.48 Sec for 217088 Bytes : file3.size()=215040
	Big write KBytes per second 62.45 

Bytes Used: 225280, Bytes Total:524288

[B]Winbond W25Q64JV*Q/W25Q64FV
[/B]Writing 524288 byte file...  1272 ms, bandwidth = 412176 bytes/sec
Writing 524288 byte file...  1271 ms, bandwidth = 412500 bytes/sec
Writing 524288 byte file...  1271 ms, bandwidth = 412500 bytes/sec
Writing 524288 byte file...  1735 ms, bandwidth = 302183 bytes/sec
Writing 524288 byte file...  1273 ms, bandwidth = 411852 bytes/sec
Writing 524288 byte file...  1272 ms, bandwidth = 412176 bytes/sec
Writing 524288 byte file...  1273 ms, bandwidth = 411852 bytes/sec
Writing 524288 byte file...  1273 ms, bandwidth = 411852 bytes/sec
Writing 524288 byte file...  1273 ms, bandwidth = 411852 bytes/sec
Writing 524288 byte file...  1273 ms, bandwidth = 411852 bytes/sec
Bandwidth test finished

Start Big write of 4087808 Bytes
Big write /0_bigfile.txt took 14.10 Sec for 4087808 Bytes : file3.size()=4085760
	Big write KBytes per second 289.96 

Bytes Used: 4259840, Bytes Total:8388608


[B]Winbond (W25Q256JW*M)
[/B]Writing 524288 byte file...  1604 ms, bandwidth = 326862 bytes/sec
Writing 524288 byte file...  1603 ms, bandwidth = 327066 bytes/sec
Writing 524288 byte file...  1603 ms, bandwidth = 327066 bytes/sec
Writing 524288 byte file...  1603 ms, bandwidth = 327066 bytes/sec
Writing 524288 byte file...  1603 ms, bandwidth = 327066 bytes/sec
Writing 524288 byte file...  1603 ms, bandwidth = 327066 bytes/sec
Writing 524288 byte file...  1603 ms, bandwidth = 327066 bytes/sec
Writing 524288 byte file...  1603 ms, bandwidth = 327066 bytes/sec
Writing 524288 byte file...  1603 ms, bandwidth = 327066 bytes/sec
Writing 524288 byte file...  1603 ms, bandwidth = 327066 bytes/sec
Bandwidth test finished

Start Big write of 25124864 Bytes
Big write /0_bigfile.txt took 77.38 Sec for 25124864 Bytes : file3.size()=25122816
	Big write KBytes per second 324.72 

Bytes Used: 27394048, Bytes Total:33554432


[B] Winbond W25Q512JV*Q
[/B] Writing 524288 byte file...  1340 ms, bandwidth = 391259 bytes/sec
Writing 524288 byte file...  1338 ms, bandwidth = 391844 bytes/sec
Writing 524288 byte file...  1339 ms, bandwidth = 391551 bytes/sec
Writing 524288 byte file...  1338 ms, bandwidth = 391844 bytes/sec
Writing 524288 byte file...  1338 ms, bandwidth = 391844 bytes/sec
Writing 524288 byte file...  1338 ms, bandwidth = 391844 bytes/sec
Writing 524288 byte file...  1339 ms, bandwidth = 391551 bytes/sec
Writing 524288 byte file...  1338 ms, bandwidth = 391844 bytes/sec
Writing 524288 byte file...  1339 ms, bandwidth = 391551 bytes/sec
Writing 524288 byte file...  1338 ms, bandwidth = 391844 bytes/sec
Bandwidth test finished

Start Big write of 25124864 Bytes............................................................
Big write /0_bigfile.txt took 63.06 Sec for 25124864 Bytes : file3.size()=25122816
	Big write KBytes per second 398.45 

Bytes Used: 25296896, Bytes Total:33554432
 
 
[B] Winbond W25Q01JV*Q
[/B]Writing 524288 byte file...  1586 ms, bandwidth = 330572 bytes/sec
Writing 524288 byte file...  1589 ms, bandwidth = 329948 bytes/sec
Writing 524288 byte file...  1586 ms, bandwidth = 330572 bytes/sec
Writing 524288 byte file...  1584 ms, bandwidth = 330989 bytes/sec
Writing 524288 byte file...  1583 ms, bandwidth = 331198 bytes/sec
Writing 524288 byte file...  1581 ms, bandwidth = 331617 bytes/sec
Writing 524288 byte file...  1590 ms, bandwidth = 329740 bytes/sec
Writing 524288 byte file...  1588 ms, bandwidth = 330156 bytes/sec
Writing 524288 byte file...  1587 ms, bandwidth = 330364 bytes/sec
Writing 524288 byte file...  1582 ms, bandwidth = 331408 bytes/sec
Bandwidth test finished

Start Big write of 25124864 Bytes............................................................
Big write /0_bigfile.txt took 111.41 Sec for 25124864 Bytes : file3.size()=25122816
	Big write KBytes per second 225.51 

Bytes Used: 25296896, Bytes Total:134217728
 
I started work on control transfers for MTP host initiated cancel. Here's the work-in-progress code so far on the core library side (Teensy 4.x only so far... will do 3.x later).

https://github.com/PaulStoffregen/cores/commit/6e6be0b3e6a9f976699ea737cbbf913fc0af7dd1

So far I've not touched MTP_Teensy. Wanted to first check in with where you're at, so I don't create any conflicts or undo your work in progress.

Short version: this adds a volatile "usb_mtp_status" byte on the core libtary side, which is normally 0x01. If the host wants to cancel (either because the user clicks the cancel button in the GUI, or we're going slower than 85.3 kbytes/sec and Windows gives up), usb_mtp_status gets set to 0x19.

Storage code in MTP_Teensy which actually copies the between media and MTP bulk transfer will also need to poll usb_mtp_status. If it is found to be 0x19, then the File needs to be closed. If the transfer was a write, maybe we need to delete the file? And of course the index file needs to be updated. When all cleanup is done, just set usb_mtp_status back to 0x01 to tell the USB host we've completed the cancel and we're ready for whatever it wants to do next.

Well, except what Windows will want soon after usb_mtp_status is 0x01 again is a GetStorageInfo command. Looks like we might have a bug where GetStorageInfo only works once when the host requests it at startup.

But first we need to put code into MTP_Teensy to poll usb_mtp_status and if it's seen at 0x19, actually abort the transfer & do cleanup. Any thoughts on that part? Should I try to dive into MTP_Teensy?
 
Last edited:
FWIW, there is also a way for the device side to initiate a cancel of whatever transfer is in progress, but it requires much more work involving stalling the bulk endpoints. I wasn't planning to implement cancel initiated from the Teensy side.
 
Paul sounds good.

Will try it out soon.

I have been playing more with the code to transfer (copy/move) files between different stores. Like copy form SD to Flash chip. We have been having issues with it failing with copy of folders and the like: I reworked one way still having issues, trying a different approach will try it out later today.


The checked in code in my debug branch has the version that would fail sometimes on writes... More if interested.

With the Host USB code there have been a couple of cases that we know of:

a) the case where we are doing a SendObject (host uploads to one of our storage). We are currently detecting issue when we get a Zero Length Packet. My guess is that this is the point that your code will indicate the new status and that we need to set the value to clear out the status which will be great.

As for removing the file... Can do... I assume only that file. That is if the user logically did an upload of a directory and the error happens on the 3rd of 10 files, I assume we leave the two files and I assuming the host probably aborted and wont send the additional 7 files. That is the SendObject works one object at a time...

b) Now in the case of the failures we have seen when we try to download a file or the like and we get some form of failure it will be interesting to see if something shows up in this new status...

c) CopyObject/MoveObject - The user for example selects a folder on SD storage and drops it on the storage for a flash chip. For the most part the host has no way of knowing how long this operation is going to take, especially if it is a folder to folder... It gives us a certain amount of time, and then it aborts.... Don't know anyway to give it feedback like I am now 10 percent done...

So current plan is to to try to get a rough idea how long both windows and linux give us before it aborts, then when we start up the operation, start an interval timer that at some point before the host will error out, return a status of OK... And then monitor additional messages come in and respond with I AM BUSY status... Sort of like we do at startup and I
think with format code...

GetStorageInfo - I would think should work at any time... But will check.

EDIT - on a) should mention that there were a couple issues with the SendObject failing. One was we were losing data in the middle of files which I believe was fully fixed by previous updates to USB code in the current release of Teensyduino.

The second part was - our writes to file system was taking too long (i.e we were not getting the prescribed throughput) and windows/linux gives up on us and aborts... Which this second part is what I believe the changes to core code in Paul's message should hopefully address.
 
Last edited:
@Paul @mjs513 ...

Sorry been distracted some this morning (and now afternoon),

Wondering with magic numbers: 1 and 0x19 and the like:
If I should go ahead and use the variable usb_mtp_status

Or should we wrap them, with functions like: if (usb_mtp_status_abort()) and like: usb_mtp_clear_abort();...
For now will hard code.

Also wondering if for example if SendObject detects this (guessing probably what also caused the 0 length packet...), that as you mentioned we could/should delete the
file that was in progress and remove it from index (I am torn) sometimes may want to simply abort as I know the data I wanted was early on...

But should it then clear the status then or should that be handled back in the main loop. Will also need to see in some cases if we are aborted do we send a response packet for that or not... Will have to experiment.

Will experiment probably with bogus fs, but could undo the large cluster size for Flash chip and see what it does.
 
Let's just use usb_mtp_status directly for now. Don't be shy to just check if it's 0x19 and write it to 0x01. We can always wrap stuff with abstractions later if there's really a need.

I'm going to look at the ZLP stuff next. Indeed BogusFS is really useful for testing. I was using it all last night for the host cancel testing. :)

Can you give me a quick update about the state of MTP_Teensy? I see you have 2 branches "debug" and "thowing_darts" with commits ahead of the main branch.
 
Let's just use usb_mtp_status directly for now. Don't be shy to just check if it's 0x19 and write it to 0x01. We can always wrap stuff with abstractions later if there's really a need.

I'm going to look at the ZLP stuff next. Indeed BogusFS is really useful for testing. I was using it all last night for the host cancel testing. :)

Can you give me a quick update about the state of MTP_Teensy? I see you have 2 branches "debug" and "thowing_darts" with commits ahead of the main branch.

@KurtE can probably give you a bit more details but the throwing_darts branch was what we were using to debug the send_object (PC to device transfers) issue that you fixed with the last update to USB for release 1.56. debug branch is what @KurtE has been working on to get copyObject and moveObject working so we can copy files between devices which didn't work in the original MTP_t4 library either. As for MTP_Teensy think its the latest version before the modifications started to test/debug copy/move object. So maybe thats the best to test with unless @KurtE has anything else to add.
 
Throwing darts - was I was trying a lot of different ways to see if I could isolate the SendObject... So probably don't need it.

The debug - is one that I have been working on trying to rework the Copy/Move Object code, it has a few other fixes in it.
I merged (git rebase -i ...) all of the smaller commits but still having issues with the CopyObject, was in process or redoing it...

As you mentioned I will use the stuff directly... Will work directly within the main branch for now. Once it appears to be
working better will then try a merge back into main.

Edit: alternative could go ahead and merge in the current stuff in debug and then continue working on the move/copy stuff as we work on this as well
 
Quick update on debug branch: Here is the commit message to show some of the stuff in the rebase:
Code:
This is a combine of multiple commits.

Move/copy check for 0 passed in which says use root of destination storage

Uses some of the functions used of SendObject to create the new destination.
This helps when for example the destination directory already exists.

Currently this code will try to overwrite the contents of any of the files
that already existed in the destination with the contents of the source
files

It will fail if destination exists and not of same type. That is suppose the
source directory contains a file named temp and the destination already has
a directory named temp.

The changes also tries to reduce how much stack space is used to complete
the operations. The previous code for example the top level operation may
have something like 6 Record objects and if it calls of to the move/copy
directory code each recursion may add similar number of these structures
plus a source and destination pathname object.

Also there is a start for detecting some errors like we ran out of disk
space and the writes fail. Also work to propagate these errors back
up the call chain.

Fixed CopyObjects for T3.x, we were not properly returning the newly
created object handle and the host would error out.

Cleaned up debug output in storage code, put in macro DBGPrintf that you did not have to put #if around...
 
Right now I am playing on the debug branch, I probably should just merge it back over as it did fix some issues.

I am trying to get the bogus fs to fail to transfer the file with your new core code. I have tried a few different versions of delays for each write and so far they all went through. Including taking over 6 minutes to send the file.
like delay(150); after each write...

Still playing. I think in the debug branch I will also remove the Simple copy code as that was an experiment to emulate the t3.x code to see back when we were failing if that would work... Made no difference other than really slowing it down.
 
Yes, please merge to the main branch, and then let's do everything on the main branch.

I would like to start really working on the ZLP handling today, but will need to mess with both core library and MTP_Teensy.
 
Yes, please merge to the main branch, and then let's do everything on the main branch.

I would like to start really working on the ZLP handling today, but will need to mess with both core library and MTP_Teensy.
debug has now been merged into main...
 
@Paul @mjs513 - Now debugging some issues with GetObject where we are trying to push packets and it is timing out... Probably related to status stuff...
Debug output was showing:
Code:
    >>w dp:2548 p:166388 len:512
push_packet: l:512 ret:0 loop:0
push_packet: l:512 ret:0 loop:1
push_packet: l:512 ret:0 loop:2
push_packet: l:512 ret:0 loop:3
push_packet: l:512 ret:0 loop:4
    >>w dp:3060 p:166900 len:512
push_packet: l:512 ret:0 loop:0
push_packet: l:512 ret:0 loop:1
push_packet: l:512 ret:0 loop:2
push_packet: l:512 ret:0 loop:3
push_packet: l:512 ret:0 loop:4
    >>w dp:3572 p:167412 len:512
...
Not checking status...
 
I pushed up some minor changes which look at the usb status and try to recover

Also changes to getobject that detects that writes to host times out trying to output and bails. Also in this case it sends status of incomplete transfer
 
I have been playing with file and directory names with SdFat. Some names containing certain characters cause the directory listing to to just keep scrolling endlessly. Working on what is causing this. Not sure yet. I ran into this several times now.
 
I have been playing with file and directory names with SdFat. Some names containing certain characters cause the directory listing to to just keep scrolling endlessly. Working on what is causing this. Not sure yet. I ran into this several times now.

What chars are problematic?

Using latest KurtE copy of CORES and MTP_Teensy. Not sure if there is other needed?

Did a trivial PR to comment in \simple_t41_qflash\simple_t41_qflash.ino.
> is the 'menu' portion supposed to exist/function?

Ref point: Simple copy of '111' to Flash failed to return usably :(
> and Win DlbClick on the a,b,c files may or may not even work without displaying 'Interrupted action ... unexpected error'
 
Simple SD sample creation below. WORKS on T_3.6 and T_4.1

Simple SD sample creation below. WORKS on T_3.6 and T_4.1 with BUILTIN_SDCARD , tested CS_SD==10 on T_4.1 Audio card.
Using this seems in sync with PJRC and nothing else new: KurtE copy of CORES and MTP_Teensy
<edit> ignore earlier 'user error' copy paste of wrong name for name change in sample

Code:
#include <SD.h>
#include <MTP_Teensy.h>

#define CS_SD BUILTIN_SDCARD  // Works on T_3.6 and T_4.1
// #define CS_SD 10  // Works on SPI with this CS pin

MTPStorage storage;     // TODO: storage inside MTP instance, not separate class
MTPD    MTP(&storage);  // TODO: MTP instance created by default

void setup()
{
  Serial.begin(9600);
  while (!Serial && millis() < 5000) {
    // wait for serial port to connect.
  }

  Serial.print(CrashReport);
  Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);
  delay(1000);

  // mandatory to begin the MTP session.
  MTP.begin();

  // Add SD Card
  if (SD.begin(CS_SD)) {
    storage.addFilesystem(SD, "SD Card");
    Serial.println("Added SD card using built in SDIO, or given SPI CS");
  } else {
    Serial.println("No SD Card");
  }
  Serial.println("\nSetup done");
}


void loop() {
  MTP.loop();  //This is mandatory to be placed in the loop code.
}
 
Last edited:
Back
Top