Teensyduino File System Integration, including MTP and MSC

defragster said:
Ref point: Simple copy of '111' to Flash failed to return usably
> and Win DlbClick on the a,b,c files may or may not even work without displaying 'Interrupted action ... unexpected error'

Noticed that as well. Found if you try and it doesn't open you can try it again about 10 seconds later and it will open. Bigger issue is that I found when I move off the disk it tends to loose MTP in explorer. Other issues with getobject and copying files are in work. TBD not sure.
 
Noticed that as well. Found if you try and it doesn't open you can try it again about 10 seconds later and it will open. Bigger issue is that I found when I move off the disk it tends to loose MTP in explorer. Other issues with getobject and copying files are in work. TBD not sure.

Not unknown - just doing a setpoint for MTP WIP ... and a super simple test for repro pass/fail.

Also should have noted:
- Here it would open - showing 'corrupted data' into notepad on some files off and on intermittently - though some almost seemed right at times, where when dir copied to PC - All are corrupted.
- never saw the explorer window lose the MTP device in my testing that simple '111' folder and files - something else to look for.
> thinking I should change the makefiles.ino for data set always end with known <EOF> signature so casual inspection on file open can detect pass/fail.
> can also add use of the 'chars that fail in filenames' to the file set when @wwatson indicates what they are.

Just found a windows console program: Portable Devices COM API sample
> it builds and runs! Not sure of the value as written or ability to extend ...
Talks to Teensy MTP: 15. List all functional categories supported by the device
Code:
7 Supported Events Found on the device

WPD_EVENT_DEVICE_RESET
Event Options:
        WPD_EVENT_OPTION_IS_BROADCAST_EVENT = TRUE

WPD_EVENT_STORAGE_FORMAT
Event Options:
        WPD_EVENT_OPTION_IS_BROADCAST_EVENT = TRUE

{400E0000-5738-4FF2-8445-BE3126691059}
Event Options:
        WPD_EVENT_OPTION_IS_BROADCAST_EVENT = TRUE

WPD_EVENT_OBJECT_REMOVED
Event Options:
        WPD_EVENT_OPTION_IS_BROADCAST_EVENT = TRUE

WPD_EVENT_OBJECT_UPDATED
Event Options:
        WPD_EVENT_OPTION_IS_BROADCAST_EVENT = TRUE

{40010000-5738-4FF2-8445-BE3126691059}
Event Options:
        WPD_EVENT_OPTION_IS_BROADCAST_EVENT = TRUE

WPD_EVENT_OBJECT_ADDED
Event Options:
        WPD_EVENT_OPTION_IS_BROADCAST_EVENT = TRUE
 
Thanks @defragster:

Simple SD sample creation below. WORKS on T_3.6 and T_4.1 with BUILTIN_SDCARD , tested CS_SD==10 on T_4.1 Audio card.
Using this seems in sync with PJRC and nothing else new: KurtE copy of CORES and MTP_Teensy
<edit> ignore earlier 'user error' copy paste of wrong name for name change in sample

Code:
#include <SD.h>
#include <MTP_Teensy.h>

#define CS_SD BUILTIN_SDCARD  // Works on T_3.6 and T_4.1
// #define CS_SD 10  // Works on SPI with this CS pin

MTPStorage storage;     // TODO: storage inside MTP instance, not separate class
MTPD    MTP(&storage);  // TODO: MTP instance created by default

void setup()
{
  Serial.begin(9600);
  while (!Serial && millis() < 5000) {
    // wait for serial port to connect.
  }

  Serial.print(CrashReport);
  Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);
  delay(1000);

  // mandatory to begin the MTP session.
  MTP.begin();

  // Add SD Card
  if (SD.begin(CS_SD)) {
    storage.addFilesystem(SD, "SD Card");
    Serial.println("Added SD card using built in SDIO, or given SPI CS");
  } else {
    Serial.println("No SD Card");
  }
  Serial.println("\nSetup done");
}


void loop() {
  MTP.loop();  //This is mandatory to be placed in the loop code.
}
Early Morning all (my time zone)

Personally I would not go too hog wild yet on simple examples. As several of these which are simple copy of some earlier ones, still include a few interesting comments like:
Code:
MTPStorage storage;     // TODO: storage inside MTP instance, not separate class
MTPD    MTP(&storage);  // TODO: MTP instance created by default
Simple questions like:
a) Is there any purpose in having two classes here? Unless maybe there is some parameter like Max storages that it has.... That is should storage either go away or be a contained object within MTP?
b) Does it make sense for the user to define the MTP object?
c) if USB type is defined to include MTP, then should it automatically include the logical <MTPD_teensy.h>
d) Will we include something like IntervalTimer as default option to when to run the MTP loop code? Would probably need ways to disable as some sketches like audio sampling do not want to be interrupted
e) the MTPD.begin() maybe we can somehow trigger itself which actually starts up an interval timer and anything else needed... But maybe some event on the USB mtp endpoint could trigger it.

If we do most of the above questions, then the simple example may be something like:

Code:
#include <LittleFS.h>
LittleFS_QSPI qspi;

void setup() 
{
    ...
  // Add one or more File System objects (anything derived from the FS class) to MTP
  // example one the that works only on t4.1 with bottom memory flash. ...
  if (qspi.begin()) {
   MTPD.addFS(qspi, qspi.displayName());
  }
...
}

void loop() {
...
}
 
@KurtE

Like your thoughts but maybe we could key some of that off the selection of MTP DISK(xxx) in the serial drop down. So when you select that automatically sets up the MTP object and the begin. Just not sure how to handle the MTPD.loop though.
 
@KurtE

Like your thoughts but maybe we could key some of that off the selection of MTP DISK(xxx) in the serial drop down. So when you select that automatically sets up the MTP object and the begin. Just not sure how to handle the MTPD.loop though.

Exactly!

That is for a short term work around that maybe in cores: usb_mtp.h would include <MTP_Teensy.h>

Later they would probable be combined...
 
Just not sure how to handle the MTPD.loop though.

For the most part I don't have a problem with MTPD.loop() call, although it is one extra thing to remember to do If you are simply adding MTPD to a sketch whose main purpose is something else. How much time and effort to make this seamless may depend on how well one can add MTP and maybe combine it with out USB types...

Again everything in MTP is geared around the host sends us one request and waits for us to respond. Which in some cases. sends lots of data back and forth. But the main thing is for the loop() is to get the next logical packet conversation going.

Today when we receive a packet from the host, the function rx_event in usb_mtp.c gets called by the low level usb code. This function simply puts the transfer object into a circular list (4 items). When loop() is called, it checkes this list sees that there is an item with data, and receives it, which copies the data into it's own buffer and then queues that transfer back to allow it to receive another packet... And then the code processes that message out of the buffer.

Another option would be for the rx_event code, to set an eventResponder object as triggered, probably be default would be setup for yield() then to call off to the logical loop() code.
Could also play with other options on the object as well..
 
Indeed - questions about questions. For Simple_SD.ino wanted to do one to see it work and have access to the SD for makefiles.ino testing. And I saw it hinted in email.

Not sure about the how it gets instantiated, including USB Type==MTP in build makes it easy - but then there is the proliferation of build options - there are already about 2 dozen of those (in TSET) and only one is MTP excluding the others except SEREMU

eventResponder came to mind before reading it in p#906. Like calls yield.c from: uint8_t _serialEventUSB2_default __attribute__((weak)) PROGMEM = 0 ;
Code:
#ifdef USB_TRIPLE_SERIAL
	if (yield_active_check_flags & YIELD_CHECK_USB_SERIALUSB2) {
		if (SerialUSB2.available()) serialEventUSB2();
		if (_serialEventUSB2_default) yield_active_check_flags &= ~YIELD_CHECK_USB_SERIALUSB2;
	}
#endif

Yield same as having it in loop() only more often.
 
MakeFiles Folders and Files update

Zip of resulting SD Disk image - with a folder holding makefile.ino:
Should offer a good base test for integrity and function using one or more of the folders

Changes running makefiles:
> USB send when complete will print DIR
> For Mike - disk can change from SD with:: #define DISK SD
> Adds folder 'Ascii' of 32/64 byte files of the filename char.
- with 'char'.txt for ascii chars 33 to 127 with a few skipped [=;,.><"|:*%?\/]
- upper/lower case are the same file.
Code:
 [U]// This is 54 files[/U]
 !.txt   #.txt   $.txt   &.txt   '.txt   (.txt   ).txt   +.txt   -.txt   0.txt   1.txt   2.txt   3.txt
4.txt   5.txt   6.txt   7.txt   8.txt   9.txt   @.txt   A.txt   B.txt   C.txt   D.txt   E.txt   F.txt   G.txt   H.txt
I.txt   J.txt   K.txt   L.txt   M.txt   N.txt   O.txt   P.txt   Q.txt   R.txt   S.txt   T.txt   U.txt   V.txt   W.txt
X.txt   Y.txt   Z.txt   [.txt   ].txt   ^.txt   _.txt   `.txt   {.txt   }.txt   ~.txt
> All other files still have File and HEX Block count indexing content up to the 'END'
> END of 16 chars plus 0-15 chars of *'s and the file path (truncated from start as needed).
When viewed files end like some of the following:
Code:
********/111/aaa.txt
*ManyD10/D0/2548.txt
D6/D7/D8/D9/2548.txt
****MoreFD/D0/D1/D2/10125.txt

The disk image (plus MakeFiles folder in the ZIP):
Code:
111
222
333
Ascii
[B]MakeFiles[/B] // source code
ManyD10
ManyD8
ManyF
MoreFD
 9 dirs 
 Total 3259 files of Size 3059276 Bytes

Code makes from:
Code:
  char szStart[][8] = { "", "ManyF", "ManyD10", "ManyD8", "MoreFD", "Ascii" };
  MakeData( szStart[0] );
  MakeDataFiles( szStart[1], 1024, 400, 1 );
  MakeDeepDirs( szStart[2], 10, 5, 500, 512 );
  MakeDeepDirs( szStart[3], 8, 6, 100, 400 );
  MakeDeepDirs( szStart[4], 4, 6, 125, 250, 8 );  
  MakeNames( szStart[5] );

EDIT: Tested T_4.1 with above SD image on MTP with simple_SD.
> Copy of each folder 1 by 1 - none look right - named file not holding that content when place on PC drive.
> ManyD8 and D10 copied only empty folder with no notice
> ManyF took a long time to est. time and said 35 minutes and worked for 5 and quit
-> Copy of All folders at once - estimated 7 minutes??? and didn't get far.

The ASCII files should be repeats of the title char and ones observed, like the other files have the file content disassociated from file name.
Casual browsing of .txt files now shows missing files, empty folders.
 
Last edited:
And now for something completely different along the lines of BogusFS ;)
Playing with HardwareFS..

Note this is a PIP (Play in Progress).

screenshot.jpg

The digital pins section does not do anything yet. The analog pins if you download them should be the current analog value for the pin when the file object was created.

Will play next with if I drop a file named 13 into digital and it has the value 1<cr> it will turn pin 13 to high ... so led would turn on...

Note: this is probably not overly useful with MTP. As for the most part you can not manipulate the objects using normal file system control on the PC, but instead have to go through MTP code.

Now if instead we had Mass Storage Interface, than I think something like this could be a lot of fun ;)
 

Attachments

  • simple_mtp_hardware.zip
    5.5 KB · Views: 34
I updated it slightly so now digital if you do a SendObject on it (copy down to PC) gives you the file with 0 or 1...

And if you do a GetObject (copy from PC to the store), it sets the digital value...
So I turned the led on the led debug board on and off for digital 0...
 

Attachments

  • simple_mtp_hardware.zip
    5.9 KB · Views: 34
Ok... Finally figured out what was going on in post #898. Again I had more then one version of the following that was causing problems with listing certain file names.

The problem code:
Code:
// List files only. Proccess wildcards if specified.
bool diskIO::lsFiles(void *dir, char *pattern, bool wc) {
#ifdef TalkToMe
  Serial.printf(F("lsFiles()...\r\n"));
#endif
    DateTimeFields tm;
	const char *months[12] = {
		"January","February","March","April","May","June",
		"July","August","September","October","November","December"
	};
	File fsDirEntry;
	File *fsDir = reinterpret_cast < File * > ( dir );
	fsDir->rewindDirectory(); // Start at beginning of directory.
	// Find next available object to display in the specified directory.
	while ((fsDirEntry = fsDir->openNextFile(O_RDONLY))) {
		if (wc && !wildcardMatch(pattern, fsDirEntry.name())) continue;
		if(!fsDirEntry.isDirectory()) {
			Serial.printf(F("%s "),fsDirEntry.name()); // Display filename.
			for(int i = 0; i < (50-strlen(fsDirEntry.name())); i++)
				[COLOR="#FF0000"]Serial.printf(" ");[/COLOR]
			Serial.printf(F("%10ld"),fsDirEntry.size()); // Then filesize.
			fsDirEntry.getModifyTime(tm);
			for(uint8_t i = 0; i <= 1; i++) Serial.printf(F(" "));
			Serial.printf(F("%9s %02d %02d %02d:%02d:%02d\r\n"), // Show date/time.
							months[tm.mon],tm.mday,tm.year + 1900, 
							tm.hour,tm.min,tm.sec);
		}
	}
	fsDirEntry.close(); // Done. Close sub-entry file.
	return true;
}

The code that works:
Code:
// List files only. Proccess wildcards if specified.
bool diskIO::lsFiles(void *dir, char *pattern, bool wc) {
#ifdef TalkToMe
  Serial.printf(F("lsFiles()...\r\n"));
#endif
    DateTimeFields tm;
	const char *months[12] = {
		"January","February","March","April","May","June",
		"July","August","September","October","November","December"
	};
	File fsDirEntry;
	File *fsDir = reinterpret_cast < File * > ( dir );
	fsDir->rewindDirectory(); // Start at beginning of directory.
	// Find next available object to display in the specified directory.
	while ((fsDirEntry = fsDir->openNextFile(O_RDONLY))) {
		if (wc && !wildcardMatch(pattern, fsDirEntry.name())) continue;
		if(!fsDirEntry.isDirectory()) {
			Serial.printf(F("%s "),fsDirEntry.name()); // Display filename.
			for(int i = 0; i < (50-strlen(fsDirEntry.name())); i++)
				[COLOR="#00FF00"]Serial.printf("%c",' ');[/COLOR]
			Serial.printf(F("%10ld"),fsDirEntry.size()); // Then filesize.
			fsDirEntry.getModifyTime(tm);
			for(uint8_t i = 0; i <= 1; i++) Serial.printf(F(" "));
			Serial.printf(F("%9s %02d %02d %02d:%02d:%02d\r\n"), // Show date/time.
							months[tm.mon],tm.mday,tm.year + 1900, 
							tm.hour,tm.min,tm.sec);
		}
	}
	fsDirEntry.close(); // Done. Close sub-entry file.
	return true;
}

Once again to much stale code hanging around. Need to archive:)
This was the filename causing the problem:
Code:
1983_Z8_Microcomputer_Technical_Manual.pdf
If I shortened up the filename by two characters it would list. Not really sure why...
 
Since I know that Paul is doing some massaging of the MTP code and also suggested it might be good to have some example sketches which are simplified...

Toward this end I am experimenting with some code rearranging like, standard MTPD object, no need to know about storage object, ... So I have a WIP branch:
https://github.com/KurtE/MTP_Teensy/tree/mtp_storage_child_object

Which was forked from the current main branch:
https://github.com/KurtE/MTP_Teensy/tree/mtp_storage_child_object

The first commit hopefully shows some of the changes:
Code:
 Make storage part of MTPD plus rename and have explicit one

A simple pass to maybe make example sketches a little cleaner.

Renamed MTPD class to MTPD_class like most of the usb classes are of names like:
usb_keyboard_class.

Defined  a standard object: MTPD_class MTPD

Removed Storage from being a top level object and instead an object contained withing the MTP object,

Added a few wrappers  to have top level MTPD.addFileSystem that calls through to the storage one.

added storage() to return pointer to it...

Need to decide how many should have top level wrappers.

Updated my ssimple_t41_qflash sketch to use this updated stuff.

created keywords.txt file

I updated 3 sketches... Actually updated 2 and pushed up changed Toy FS (Hardware) that I have playing with:
screenshot.jpg
Note the lines underlined in RED may over time be removed with some more updates in the code.

And it does run:
screenshot2.jpg

So far I have not tried to merge (PR) this back to main as wanting to see what Paul thinks, plus maybe some more tweaks like which of the storage class methods are exposed at the MTPD level.

But thought I might see what you all think.
 
Update to my message above:
With feedback received through email and the like, have made some additional changes:

MTPD is now MTP
In the Storage header, the structure Record is now defined inside of the Storage class.

The three examples I mentioned were updated. I will probably update a few more

Again currently I have the method storage() to return pointer to storage object. This may or may not continue to be there. Goal is to see what methods we have been calling
in storage and in probably most cases probably want a method at the top level MTP object. Maybe only for debug support, like dumping the list of index records or the like,
should we maybe go through access through the storage object....

Some of the items in the keywords.txt file will also hopefully be reduced. Like resetting MTP, which many of our sketches use with serial port command, when trying to recover from when MTP is not happy with us.

Likewise some of the calls to tell MTP we mucked with the files on the storage. Example we are running a logging program, which creates a log file, currently we have code in that we can then tell mtp that a file was updated. Or what if we format the storage locally... Maybe these might not be needed over time if the fs informed mtp that something changed.

Update: the storage object was updated to constexpr constructor... Will wait to update the main object, until some of the current major changes have been completed
 
Last edited:
Yeah, I'm making more major structural changes today, so everything is pretty much a moving target right now.

Just an FYI I have merged in all of the changes in the other branch to one commit:
Code:
D:\GitHub\MTP_Teensy>git rebase -i HEAD~5
[detached HEAD cfb6cba] Make storage part of MTPD plus rename and have explicit one
 Date: Thu Jan 6 10:16:11 2022 -0800
 10 files changed, 726 insertions(+), 242 deletions(-)
 create mode 100644 examples/simple_mtp_hardware/TeensyFS.h
 create mode 100644 examples/simple_mtp_hardware/simple_mtp_hardware.ino
 create mode 100644 keywords.txt
Successfully rebased and updated refs/heads/mtp_storage_child_object.

D:\GitHub\MTP_Teensy>push origin mtp_storage_child_object --force
'push' is not recognized as an internal or external command,
operable program or batch file.

D:\GitHub\MTP_Teensy>git push origin mtp_storage_child_object --force
Enumerating objects: 31, done.
Counting objects: 100% (31/31), done.
Delta compression using up to 8 threads
Compressing objects: 100% (18/18), done.
Writing objects: 100% (18/18), 7.82 KiB | 1.56 MiB/s, done.
Total 18 (delta 9), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (9/9), completed with 9 local objects.
To https://github.com/KurtE/MTP_Teensy.git
 + 684c6a9...cfb6cba mtp_storage_child_object -> mtp_storage_child_object (forced update)

D:\GitHub\MTP_Teensy>
So we now just need to rebase to the main branch, when you are close to being done with the major reorganizing changes.

Edit /Meaningless Comment: the git rebase... git push ... --force
stuff above was how I was asked to do when making contributions to some other projects. In particular with making contributions to Edison code as well as support libraries for it.
 
Right now would be a good moment to merge this, if you can. I have no uncommitted edits at this moment. If you merge this morning, then I'll sync up to get MTPD -> MTP_Class and all this other stuff, before I start the next round of changes.
 
Some parts Working well! Using #3_Simple_SD.ino Full SD2201 copy from Teensy SD to PC only had edits in ONE FILE : SD2201\MoreFD\D0\D1\D2\4125.txt

Four copies of that files - so it isn't the length (though each has diff signature data content) - was just some anomoly in Dir D2 of D0, D1, D2, D3 copies.
This is a tremendous improvement where before not ONE file would copy PC to Teensy without error!

Copy of PC to Teensy still WIP - dirs and files copy to some degree with all files ending of zero size when copied.
 
Pushed up changes to two other simple area example sketches that now build with updated mtp/storage stuff
 
Pushed up quick update to make constructor constexpr

This made a change ??? Test was the Simple_SD reading files to PC ....

I was copying the SD2201 from T_4.1 SD to PC folder and finding problems with file 4125.txt in one or the other of the 4 folders that hold it.

Pulled noted change and rebuilt and edited SD2201 to SD2202 where the 16 byte block terminator went from '<' to '\n' so the CodeCompare 'line based' diffs would show better what changed where.

Ran that 5 times from 2 diff SD cards to PC folder and NO DIFF errors with SD2202

Went back to SD with SD2201 and FOUR repeats and no errors.

First it seemed the content change - but no - the old that repro'd over 50% didn't happen?

Here is the SD2202 - newline after each 15 bytes to the end
View attachment SD2202.zip
 
MoreFD Copy Glitch

Yes, too early to expect perfection - but a testpoint to return to.
> Mike, Kurt: would be nice to have a repro on Windows (perhaps linux) - to know the repro is not from my PC.

Not Fixed yet - in spite of prior post: Since it was only the MoreFD folder causing trouble - I manually made copies of that folder in a folder and then copied those 4 copies to a folder.

First SD card at hand had the old '<' block end marker - did that and on second copy back to PC folder from Teensy SDIO SD one 4125.txt file was corrupted - as was always the case before.

Repeated on another SD card holding the '\n' newline block end marker. On the second try that just showed failure on D:\tmp\TD_1.56\MoreFD_cNL\D0 - Copy\10125.txt

Here is a ZIP of those two source folders one ...Clt for 'less than' the other _cNL for 'newline'.
View attachment MoreFD_COPIES.zip
Each folder holds: 192 File(s) 984,000 bytes

Here is what CodeCompare showed for the NewLine glitch version:
MoreFD_COPIES.jpg
Much more readable with the line breaks pointing out the dropped lines - each 16 bytes with \n
Dir entry of : 01/07/2022 10:31 PM 9,613 10125.txt
> Shows 512 Bytes missing on PC

Test steps: { simlar to other testing with SD2201 and SD2202
Unzip to PC folder and put SD card on PC
Copy both folders to SD card
Move SD card to T_4.1 running MTP_Teensy\examples\Simplified Examples\Example_3_simple_SD\Example_3_simple_SD.ino
Start Teensy
Make empty PC folder
Copy files MTP from Teensy to the PC - in same dir pattern
When copy completes do a folder based file compare
> if no repro - empty PC receive folder and repeat: Copy files ...
 
Last edited:
This made a change ??? Test was the Simple_SD reading files to PC ....
This was something Paul suggested be done in email/github comments or ...

If it built (I tried building of example on T4.1 and 3.6) constexpr on constructor - My simple understanding is if your object is defined with this, it implies,
that the whole object will be more or less initialized at build time, that is that there is no constructor code that has to run on sketch startup time. One nice thing of it
is that also implies you don't have issues, like suppose an object relies on another object being initialized before it is...

As for running tests:

My impression at this point, is Paul's major reworking to get better USB integration is still in progress. So I am holding off doing major testing.

Instead doing some more cleanup. Things like reduce/remove usage of MTP.storage()->

Example: uint32_t fsCount = MTP.storage()->getFSCount();

Maybe replaced with: MTP.getFilesystemCount();
or maybe: getStorageFilesystemCount();
or: queryFilesystemCount();
or: storageFileSystemCount();

I am leaning toward: getStorageFilesystemCount
I personally prefer: storageFileSystemCount

But I think the get/set is more common in our libraries...

Thoughts?
 
Code:
Quote Originally Posted by defragster :: This made a change ??? Test was the Simple_SD reading files to PC ....
This was something Paul suggested be done in email/github comments or ...

That was me wondering why several trials didn't get the prior repro just from making the class constexpr.


Anyhow did sync to github. I can still repro here this happened first time - missing 3 sets of 512:
Code:
 Directory of D:\tmp\TD_1.56\8Jan\a\MoreFD\D0\D1\D2
10125.txt
               1 File(s)          8,589 bytes

This took a couple tries - new file missing 512 bytes:
Code:
 Directory of D:\tmp\TD_1.56\8Jan\b\MoreFD - Copy\copies\D0\D1\D2
8125.txt
               1 File(s)          7,613 bytes

So not sure repro - and not clear or easy - doing combined posted file sets { 6,900 files in 138 folders and 7.72 MB src files }
 
Back
Top