Teensyduino File System Integration, including MTP and MSC

But maybe we should just support 255 byte filename length?

Any chance you could try some of Defragster's huge file sets with LittleFS formatted to allow 255 vs 39 bytes. Does it make any real difference in usage of the media? Or RAM usage?

If the impact is minimum, let's just get rid of the 39 byte limit and allow 255.

Might also be worthwhile to test filenames using UTF8 encoded international characters and symbols....

Will make the necessary adjustments and see what happens.
 
But maybe we should just support 255 byte filename length?

Any chance you could try some of Defragster's huge file sets with LittleFS formatted to allow 255 vs 39 bytes. Does it make any real difference in usage of the media? Or RAM usage?

If the impact is minimum, let's just get rid of the 39 byte limit and allow 255.

...........
I ran Defragster's huge file sets writing them directly to the N02 NAND Flash.

In terms of compile/memory there was a bit of a difference but not much between the 39/255 byte case:
39 Bytes
Code:
Memory Usage on Teensy MicroMod:
  FLASH: code:123008, data:17664, headers:8828   free for files:16365572
   RAM1: variables:19200, code:116152, padding:14920   free for local variables:374016
   RAM2: variables:17504  free for malloc/new:506784
255 bytes
Code:
Memory Usage on Teensy MicroMod:
  FLASH: code:122752, data:17664, headers:9084   free for files:16365572
   RAM1: variables:19200, code:115896, padding:15176   free for local variables:374016
   RAM2: variables:17504  free for malloc/new:506784

in terms of timing and disk usage = NO DIFFERENCE:
Code:
 39 Bytes: ================================ Media Size=265289728	Used Size=27787264	 us=40122043   KB/sec=1.85   Bytes written=76200
255 Bytes:  ================================ Media Size=265289728	Used Size=27787264	 us=40121824   KB/sec=1.85   Bytes written=76200

I also did a quick test of just coping the single level directories to the Q512 NOR flash and there was no difference in disk usage:
Code:
[B]FileNameLen = 255[/B]
 Space Used = 11796480
Filesystem Size = 67108864
Memory Usage on Teensy MicroMod:
  FLASH: code:159468, data:25928, headers:9160   free for files:16320516
   RAM1: variables:50848, code:150952, padding:12888   free for local variables:309600
   RAM2: variables:17504  free for malloc/new:506784

[B]FileNameLen = 39
[/B] Space Used = 11796480
Filesystem Size = 67108864
Memory Usage on Teensy MicroMod:
  FLASH: code:159468, data:25928, headers:9160   free for files:16320516
   RAM1: variables:50848, code:150952, padding:12888   free for local variables:309600
   RAM2: variables:17504  free for malloc/new:506784

And no more issues with File Name Lengths from what I see.

EDIT: If agreed will go ahead and do the PR
 
@Paul - With reference to post #1037 I was able to do a temporary workaround to LittleFS:
Code:
	const char * getPN() { 
//		PROGMEM static const char pn_name[] = "PROGRAM";
//		return pn_name; 
		return (const char *)F("PROGRAM");
	}

This eliminated the compile error. As far as I know the results should be the same?
 
It wasn't compiling after I synced to the latest changes, so I put those #if lines back in.

If it's causing you trouble now, I could just install MemoryHexDump here as a short-term solution. But for the long-term goal of bringing MTP_Teensy into the core library (hopefully for 1.57), we can't have it depend on including any other libraries.

Not a problem, I meant to do that... But is again more of a generic question, is there any way to do this Automatically in Arduino? More or less asking is there a way to say open this file if you can else please don't error out...
And yes I know the above probably would work with PlatformIO or other setups where you define the list of directories to include in the build...

This needs to be done inside LittleFS. It can't go into MTP_Storage, because MTP_Storage doesn't "know" these filesystem-specific limitations.

Sounds like maybe FS.h, the FS object should have some way to query the underlying fs for limitations or configuration information like this:
Like:
Max file name length
Max path length
Sector size
Cluster size
...


If this matters, we could also check for OpenSession in the timer.

Agreed, I've never seen Windows transmit CloseSession. We would also need to look at the something like usb_configuration.

CloseSession does get sent by Android File Transfer on MacOS.

So far I've not spent much time looking at what Linux actually does.

Note: the IntervalTimer does handle the OpenSession...
As for Mac sending CloseSession, how? yes maybe if I close their MTP app, or use something like eject... But if I unplug the usb cable...

Not sure if best to discuss some of it here or email... Will start here.

The IntervalTimer code here I added at the time as what I consider a big kludge to lessen some of the startup issues we were/are seeing, which include:
a) if we don't respond quick enough to some of the early message, all USB is toast... Not just the mtp part
b) SEREMU was not working most of the time, still loses the first about 5 seconds of the sketch startup outputs to Serial
c) Serial (USB) - while (!Serial) ; // would hang for ever... So hacks were added like while (!Serial && !Serial.available()) ; and the user had to enter something to get it to continue.

I have kludged around b) in some sketches by using a memory stream to try to capture all/most of the mtp output messages and when we finally get SEREMU, I dump the output from the memory stream to Serial... total kludge.

As for long term, I personally believe having the MTP startup code depending on the IntervalTimer is far from Ideal (sucks). Again example, with the image display sketch or many/most of the data capture programs, I could imagine that it could be running for several days or longer without the user ever plugging in the USB cable. So do you want that timer to run 20 times a second doing nothing... What about interrupting time critical tasks, or wanting to run in low power mode...

At least with current stuff, this sketch does not completely hang:
Code:
void setup()
{
  while (!Serial) ;
  Serial.begin(115200);
}
uint32_t loop_counter = 0;
void loop() {
  Serial.printf("%u %u\n", loop_counter++, millis());
  delay(250);
}
But again there is a long delay
Code:
0 5352
1 5602
2 5852
Where is if build for Serial, first line output: 0 532
Keyboard+Mouse+Joystick: 0 584

So to me, it would be great if we can somehow without user code involved be able to be like all of the other USB types and quickly satisfy whatever requests from the host that we need to to get the startup time, in
the same ballpark as the other USB Types.


Actually, running this stuff from interrupts makes me nervous. For now, I want to leave it as-is and focus on more important things. Long-term, I'm hoping to find a way to run this timer stuff and the loop function from yield() or MillisTimer() or something similar. And eventually I want more places in the core library to call yield(), like when the bool operator for Serial is about to return false, or when Serial.available() is about to return zero.

This actually scares me, as now there is assumption that the above operations are quick, I am calling these to see if there is some other tasks, I might need to handle and now the system could go off and do lots of stuff that could significantly change timings.

But right now, the absolute top priority is that we finally have reliable GetObject & SendObject. Some things in the code still concern me, like creating / opening the file inside SendObjectInfo, rather than SendObject.

Turning GetObject & SendObject into a state machine, so we don't stay in loop() for long times, is also pretty important.

SendObjectInfo - creating the file... To me that is the whole purpose of this message, it is supposed to give us enough information for us to check things like:
Is this object name valid for that storage, do we have enough space available, can I write to this device... If we pass these tests than we need to return to the host the new object ID... Yes we could could guess, and be wrong, like the SD card is write only, or the filename length exceeds what the store handles... But not sure what this gains us.

As for state machine and the like, Not sure how much we gain right now, if most of our time is spent maybe waiting for LittleFS to do a write, unless that can be done asynchronously....

Probably enough for this message.
 
@Paul and others - sorry about previous book ;)

I am just sort of wanting to get the startup code to not totally hang for 5 seconds if possible. I was going to mention that the interval timer or other startup message handling code should handle the GetStorageIDs
and in this case would probably return we have none...

Looks like you already did this. Then looked at other messages being asked for and tried to have them answered: Only one GetObjectPropsSupported
Added a few debug outputs...
Code:
+++ before MTP.begin() 352


*** Start Interval Timer ***
+++ before wait on !Serial 352
ERROR: intervaltimer received command with 65535 bytes
ERROR: intervaltimer received command with 65535 bytes
ERROR: intervaltimer received command with 65535 bytes
ERROR: intervaltimer received command with 65535 bytes
timer:652 CMD: 1002(OPEN_SESSION)l: 16 T:0 : 1
652 RESP:2001(RSP:OK)l: 12 T:0
timer:702 CMD: 1001(GET_DEVICE_INFO)l: 12 T:1
GetDeviceInfo size=211
702 RESP:2001(RSP:OK)l: 12 T:1
timer:752 CMD: 1014(GET_DEVICE_PROP_DESC)l: 16 T:2 : d402
752 RESP:2001(RSP:OK)l: 12 T:2
timer:1252 CMD: 1004(GET_STORAGE_IDS)l: 12 T:3
1252 RESP:2001(RSP:OK)l: 12 T:3
timer:1302 CMD: 9801(GET_OBJECT_PROPS_SUPPORTED)l: 16 T:4 : 3000
1302 RESP:2001(RSP:OK)l: 12 T:4
timer:1352 CMD: 9801(GET_OBJECT_PROPS_SUPPORTED)l: 16 T:5 : 3001
1352 RESP:2001(RSP:OK)l: 12 T:5
+++ after wait on !Serial 5000

C:\Users\kurte\Documents\Arduino\libraries\MTP_Teensy\examples\SD_Program_SPI_QSPI_MTP-logger\SD_Program_SPI_QSPI_MTP-logger.ino Jan 16 2022 11:20:19
5000 Initializing MTP Storage list ...Date: 16 Jan 2022 11:20:29
*MTP_class::send_Event(4004) 10001
timer:5052 CMD: 1005(GET_STORAGE_INFO)l: 16 T:6 : 10001
5052 RESP:2019(RSP:DEVICE_BUSY)l: 12 T:6
timer:5152 CMD: 1005(GET_STORAGE_INFO)l: 16 T:7 : 10001
5152 RESP:2019(RSP:DEVICE_BUSY)l: 12 T:7
timer:5252 CMD: 1005(GET_STORAGE_INFO)l: 16 T:8 : 10001
5252 RESP:2019(RSP:DEVICE_BUSY)l: 12 T:8
timer:5352 CMD: 1005(GET_STORAGE_INFO)l: 16 T:9 : 10001
5352 RESP:2019(RSP:DEVICE_BUSY)l: 12 T:9
timer:5452 CMD: 1005(GET_STORAGE_INFO)l: 16 T:a : 10001

Learned/reinforced a few things like: DONT call Serial.begin() especially if you have code like: while(!Serial && millis() < 5000) ;

As if you call this after it will still waste 2 seconds.

But the interesting thing is we are not receiving any more requests from USB code for MTP: last message/response is at 1353ms
But we don't exit the while(!Serial && millis() < 5000) ;
until the 5 seconds expires.

Now back to playing
 
Last edited:
But maybe we should just support 255 byte filename length?

Any chance you could try some of Defragster's huge file sets with LittleFS formatted to allow 255 vs 39 bytes. Does it make any real difference in usage of the media? Or RAM usage?

If the impact is minimum, let's just get rid of the 39 byte limit and allow 255.

Might also be worthwhile to test filenames using UTF8 encoded international characters and symbols. One corner case to consider is we get a long filename from MTP which uses Unicode16 and turns into longer than 255 bytes when encoded to UTF8. I'm pretty sure we are already properly truncating with UTF8 character boundary awareness (never creating invalid UTF8), but that hasn't actually been tested yet.

In MakeFiles is an ASCII test MakeNames() - perhaps that could easily be edited/cloned to make UTF8 names in some fashion. That was done when wwatson found some naming issue before it was understood.

I'm mostly offline again in daylight hours again today ... PR welcome.
 
Here with current MakeFiles.ino indeed, Serial connect to SerMon with SereMu at : Serial online : 5010

As Kurt notes - faster normal Serial would be better.

If yield() / eventResponder code after exiting setup() {where they handle MTP tasks when MTP included like serialEvent() with weak override} were good enough perhaps the intervalTime could be stopped at that time?

i.e. IntervalTimer (or like) is used during setup(), and then disabled when yield() polling begins?
 
And eventually I want more places in the core library to call yield(),
as well some of the infrastructure for event generation)
I was hoping events are deprectated :( So they will will keep influencing users programs, disable interrupts and slow everything down, making timing unpredictable - and this will even be extended. If there was at least something simple, easy to turn off completely, one could still live with it.
 
Maybe you're confusing 2 completely different things both named "event"?

MTP has "events" which are 12 to 24 byte messages sent to the USB host. These messages inform the USB host when changes (which the host did not make) have happened to the filesystems. They're completely separate from yield(), EventResponder, MillisTimer.

The main reason to use MTP rather than MSC, which is also why all modern Android phones only offer MTP, is to allow the software on the device side to still be able to use the filesystem while the host has access. MTP works as the file level (called "objects" in the MTP spec) and events which communicate feedback to the host when the local software makes any changes the host needs to know.

The "infrastructure for event generation" is to create the callbacks which trigger these 12-24 byte messages to be created and sent to the USB host.
 
Maybe yes. I'm not following this thread entirely. If the additional yield() code will not disable interrupts (edit: + and has no influence, timing-wise) , I've said nothing:)

Btw, yield() has a bug that needs to be fixed:
Code:
    if (running) return; // TODO: does this need to be atomic?
    running = 1;
does not work.
a) You need ldrex/strex here.
b) add data memory barrier.
 
UTF-8 Filename / Directory testing

First I hacked up the LFS usage example to use some UTF8 character filenames that I happen to get from a nice sketch that Bill G put together for the SDfat library (UnicodeFilenames.ino) which I ran for the SDIO card. Was also edited to use mtp for sdio/sflash6 on the memory card:
Code:
//#define UTF8_FOLDER u8"😀"
//const char* names[] = {u8"россиянин", u8"très élégant", u8"狗.txt", nullptr};
#include <MTP_Teensy.h>
#include <LittleFS.h>
#include <SD.h>


// Some variables for later use
uint64_t fTot, totSize1;

// To use SPI flash we need to create a instance of the library telling it to use SPI flash.
LittleFS_SPIFlash myfs;
#define CS_SD BUILTIN_SDCARD  // Works on T_3.6 and T_4.1

// Since we are using SPI we need to tell the library what the chip select pin
#define chipSelect 6  // use for access flash on audio or prop shield

// Specifies that the file, file1 and file3 are File types, same as you would do for creating files
// on a SD Card
File file, file1, file2;


void setup()
{
  // Open serial communications and wait for port to open:
  Serial.begin(115200);
  while (!Serial) {
    // wait for serial port to connect.
  }
  Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);

  Serial.print("Initializing LittleFS ...");

  // see if the Flash is present and can be initialized:
  // Note:  SPI is default so if you are using SPI and not SPI for instance
  //        you can just specify myfs.begin(chipSelect). 
  if (!myfs.begin(chipSelect, SPI)) {
    Serial.printf("Error starting %s\n", "SPI FLASH");
    while (1) {
      // Error, so don't do anything more - stay stuck here
    }
  }
  myfs.quickFormat();
  Serial.println("LittleFS initialized.");

  MTP.begin();

  // Add SD Card
  if (SD.begin(CS_SD)) {
    MTP.addFilesystem(SD, "SD Card");
    Serial.println("Added SD card using built in SDIO, or given SPI CS");
  } else {
    Serial.println("No SD Card");
  }
  
  MTP.addFilesystem(myfs, "sflash6");
  
  
  // To get the current space used and Filesystem size
  Serial.println("\n---------------");
  Serial.printf("Bytes Used: %llu, Bytes Total:%llu\n", myfs.usedSize(), myfs.totalSize());
  waitforInput();

  // Now lets create a file and write some data.  Note: basically the same usage for 
  // creating and writing to a file using SD library.
  Serial.println("\n---------------");
  Serial.println("Now lets create a file with some data in it");
  Serial.println("---------------");
  char someData[128];
  memset( someData, 'z', 128 );
  file = myfs.open("россиянин.txt", FILE_WRITE);
  file.write(someData, sizeof(someData));

  for (uint16_t j = 0; j < 100; j++)
    file.write(someData, sizeof(someData));
  file.close();
  
  // We can also get the size of the file just created.  Note we have to open and 
  // thes close the file unless we do file size before we close it in the previous step
  file = myfs.open("россиянин.txt", FILE_WRITE);
  Serial.printf("File Size of россиянин.txt (bytes): %u\n", file.size());
  file.close();

  // Now that we initialized the FS and created a file lets print the directory.
  // Note:  Since we are going to be doing print directory and getting disk usuage
  // lets make it a function which can be copied and used in your own sketches.
  listFiles();
  waitforInput();
  
  // Now lets rename the file
  Serial.println("\n---------------");
  Serial.println("Rename россиянин to très élégant");
  myfs.rename("россиянин.txt", "très élégant.txt");
  listFiles();
  waitforInput();
  
  // To delete the file
  Serial.println("\n---------------");
  Serial.println("Delete très élégant.txt");
  myfs.remove("très élégant.txt");
  listFiles();
  waitforInput();

  Serial.println("\n---------------");
  Serial.println("Create a directory and a subfile");
  myfs.mkdir("😀");

  file = myfs.open("😀/temp_test.txt", FILE_WRITE);
  file.println("SOME DATA TO TEST");
  file.close();
  listFiles();
  waitforInput();

  Serial.println("\n---------------");
  Serial.println("Rename directory");
  myfs.rename("😀", "😀1");
  listFiles();
  waitforInput();

  Serial.println("\n---------------");
  Serial.println("Lets remove them now...");
  //Note have to remove directories files first
  myfs.remove("😀1/temp_test.txt");
  myfs.rmdir("😀1");
  listFiles();
  waitforInput();

  Serial.println("\n---------------");
  Serial.println("Now lets create a file and read the data back...");
  
  // LittleFS also supports truncate function similar to SDFat. As shown in this
  // example, you can truncate files.
  //
  Serial.println();
  Serial.println("Writing to 狗.bin using LittleFS functions");
  file1 = myfs.open("狗.bin", FILE_WRITE);
  unsigned int len = file1.size();
  Serial.print("狗.bin started with ");
  Serial.print(len);
  Serial.println(" bytes");
  if (len > 0) {
    // reduce the file to zero if it already had data
    file1.truncate();
  }
  file1.print("Just some test data written to the file (by SdFat functions)");
  file1.write((uint8_t) 0);
  file1.close();

  // You can also use regular SD type functions, even to access the same file.  Just
  // remember to close the file before opening as a regular SD File.
  //
  Serial.println();
  Serial.println("Reading to 狗.bin using LittleFS functions");
  file2 = myfs.open("狗.bin");
  if (file2) {
    char mybuffer[100];
    int index = 0;
    while (file2.available()) {
      char c = file2.read();
      mybuffer[index] = c;
      if (c == 0) break;  // end of string
      index = index + 1;
      if (index == 99) break; // buffer full
    }
    mybuffer[index] = 0;
    Serial.print("  Read from file: ");
    Serial.println(mybuffer);
  } else {
    Serial.println("unable to open datalog.bin :(");
  }
  file2.close();


  Serial.println("\nBasic Usage Example Finished");
}

void loop() {
    MTP.loop();  //This is mandatory to be placed in the loop code.
}

void listFiles()
{
  Serial.println("---------------");
  printDirectory(myfs);
  Serial.printf("Bytes Used: %llu, Bytes Total:%llu\n", myfs.usedSize(), myfs.totalSize());
}

void printDirectory(FS &fs) {
  Serial.println("Directory\n---------");
  printDirectory(fs.open("/"), 0);
  Serial.println();
}

void printDirectory(File dir, int numSpaces) {
   while(true) {
     File entry = dir.openNextFile();
     if (! entry) {
       //Serial.println("** no more files **");
       break;
     }
     printSpaces(numSpaces);
     Serial.print(entry.name());
     if (entry.isDirectory()) {
       Serial.println("/");
       printDirectory(entry, numSpaces+2);
     } else {
       // files have sizes, directories do not
       printSpaces(36 - numSpaces - strlen(entry.name()));
       Serial.print("  ");
       Serial.println(entry.size(), DEC);
     }
     entry.close();
   }
}

void printSpaces(int num) {
  for (int i=0; i < num; i++) {
    Serial.print(" ");
  }
}

void waitforInput()
{
  Serial.println("Press anykey to continue");
  while (Serial.read() == -1) ;
  while (Serial.read() != -1) ;
}
Running this on the TMM with the PJRC memory board. The output just from using LitteFS commands:
Code:
---------------
Now lets create a file with some data in it
---------------
File Size of россиянин.txt (bytes): 12928
---------------
Directory
---------
россиянин.txt                12928

Bytes Used: 196608, Bytes Total:67108864
Press anykey to continue

---------------
Rename россиянин to très élégant
---------------
Directory
---------
très élégant.txt                   12928

Bytes Used: 196608, Bytes Total:67108864
Press anykey to continue

---------------
Delete très élégant.txt
---------------
Directory
---------

Bytes Used: 131072, Bytes Total:67108864
Press anykey to continue

---------------
Create a directory and a subfile
---------------
Directory
---------
😀/
  temp_test.txt                       19

Bytes Used: 262144, Bytes Total:67108864
Press anykey to continue

---------------
Rename directory
---------------
Directory
---------
😀1/
  temp_test.txt                       19

Bytes Used: 262144, Bytes Total:67108864
Press anykey to continue

---------------
Lets remove them now...
---------------
Directory
---------

Bytes Used: 131072, Bytes Total:67108864
Press anykey to continue

---------------
Now lets create a file and read the data back...

Writing to 狗.bin using LittleFS functions
狗.bin started with 0 bytes

Reading to 狗.bin using LittleFS functions
  Read from file: Just some test data written to the file (by SdFat functions)

Basic Usage Example Finished
So looks like it worked.

I also previously ran the SDfat sketch but forgot to enable long names in the config file so the directory name with the smiley face was empty but I was able to rename name it with some greek letters:
Capture.PNG
with the following contents
Capture1.PNG
and copying the directory from SD card to sflash6
Capture2.PNG
 
Did a double byte clone of MakeNames() in MakeFiles.ino:
Code:
dbMakeN File:россиянин/��.txt=32
dbMakeN File:россиянин/��.txt=32
dbMakeN File:россиянин/α.txt=32
dbMakeN File:россиянин/β.txt=32

Emoji's come out as 4 bytes as copies from the web.
<edit>: The Emoji's fail here on Forum - but work in SublimeText and TyCommanderSerMon:
dbMakeNames2.png

The Emojis names don't present well in Win Explorer:
dbMakeNames.png
Both .txt files are UNIQUE and open - but both just show as ".txt"

I see I didn't use the U8 string decoration that Mike presents ... I just plopped 'chars' in (with added space) and the code did what seems to be a usable thing.

Github updated as: T4LockBeta/tree/main/MakeFiles
 
No MakeFiles.ino Update, except a few minutes progress getting bearings in the file. A few minutes to complete left for after next daylight fades.

Code:
FILE    .++[U]zDDd. [B]151[/B]x[B]97071[/B]_ManyD10/D0.5/[B]500.txt[/B][/U] fid= [B]151[/B] bid=[B]97071[/B] N=[B]500.txt[/B]| 500.txt	500
...
FILE    .++zDDd. 200x982AF_5/D8.5/D9.5/2548.txt fid= 200 bid=982AF N=2548.txt| 2548.txt	2548

That shows the end of the file for now in the underlined portion, parsed values in bold, and the rest of the DirPrint for now
The BOLD fields as maintained in MakeFile.ino file creation extracted are:
> File ID: repeated in each block of the file, unique to each file
> Block ID: Incremented for each 16 Byte block written in all files.
-> this is 5 hex chars so each run of MakeFiles.ino limited to 16MB of working unique files, there is room to edit in *256 with 2 more chars
> File Base Name, excluding what is partial path on deeper files.

When file length matches the name the checks on each block will be: File ID match internally on all blocks and first block's Number increments with each block after to the last, then base name matches entry.name().

A few unchecked bytes of corruption could slip in undetected in any block - but at least this programmatically will verify most block content and ordering matching the file, and the point of failure shown. Such errors would be tough (except pinpoint media errors?), and have not been seen. Full file compare when arranged on PC still possible. Full CRC could be added for overall file, but that wouldn't pinpoint expected failures as this will.
 
Follow on to:
I am just sort of wanting to get the startup code to not totally hang for 5 seconds if possible. I was going to mention that the interval timer or other startup message handling code should handle the GetStorageIDs
and in this case would probably return we have none...

Looks like you already did this. Then looked at other messages being asked for and tried to have them answered: Only one GetObjectPropsSupported
Added a few debug outputs...

Good news.... Bad news...

Good news: I was able to get a faster startup time:
Code:
+++ before MTP.begin() 423


*** Start Interval Timer ***
+++ before wait on !Serial 425
...
timer:790 CMD: 1002(OPEN_SESSION)l: 16 T:0 : 1
790 RESP:2001(RSP:OK)l: 12 T:0
timer:827 CMD: 1001(GET_DEVICE_INFO)l: 12 T:1
GetDeviceInfo size=211
827 RESP:2001(RSP:OK)l: 12 T:1
timer:877 CMD: 1014(GET_DEVICE_PROP_DESC)l: 16 T:2 : d402
877 RESP:2001(RSP:OK)l: 12 T:2
+++ after wait on !Serial 964
...

Or looking at core debug output with a few additional printf ...
Code:
RX(3):run_callbacks
 still active
mtp:rx event i=2 t:839 st:01F08000 16 20203470 20204524
mtp:rx queue i=2 t:875
data
USB1_ENDPTSETUPSTAT=00000001
setup 03000921 00040000
hid set report 03000921 00040000
rx 4
USB1_ENDPTCOMPLETE=00080000
data
USB1_ENDPTSETUPSTAT=00000000
USB1_ENDPTCOMPLETE=00010001
complete 03000921 00040000 0000008B
seremu online
data
USB1_ENDPTSETUPSTAT=00000000
USB1_ENDPTCOMPLETE=00040008
RX(3):run_callbacks
 still active
mtp:rx event i=3 t:1264 st:01F48000 12 2020366C 202046CE
mtp:rx queue i=3 t:1275


Bad news: But this only works if, you are running the Arduino Serial Monitor... Which I typically only use if I am doing something real simple using Arduino IDE or have to...

Why? because, unless one sends a specific USB message to the Teensy, code like: if(Serial) will never be true...

Suggestion: we should as a bare minimum, have a USB type of MTP + Serial. In fact my guess would be that far more likely to be used than with Seremu...

Now back to more coffee.
 
KurtE said:
Suggestion: we should as a bare minimum, have a USB type of MTP + Serial. In fact my guess would be that far more likely to be used than with Seremu...

Now back to more coffee.
Nice work and yes think MTP+Serial would be a good idea - a lot of other USB types have this type of setup as well so would be right in line.
 
Nice work and yes think MTP+Serial would be a good idea - a lot of other USB types have this type of setup as well so would be right in line.

Quick update: :D
I opened up an issue up on github: https://github.com/Koromix/tytools/issues/86
Half hour later he posted an update, with a dev build... That works...

Code:
+++ before MTP.begin() 423


*** Start Interval Timer ***
+++ before wait on !Serial 425
ERROR: intervaltimer received command with 65535 bytes
ERROR: intervaltimer received command with 65535 bytes
ERROR: intervaltimer received command with 65535 bytes
ERROR: intervaltimer received command with 65535 bytes
timer:789 CMD: 1002(OPEN_SESSION)l: 16 T:0 : 1
789 RESP:2001(RSP:OK)l: 12 T:0
timer:827 CMD: 1001(GET_DEVICE_INFO)l: 12 T:1
GetDeviceInfo size=211
827 RESP:2001(RSP:OK)l: 12 T:1
timer:877 CMD: 1014(GET_DEVICE_PROP_DESC)l: 16 T:2 : d402
877 RESP:2001(RSP:OK)l: 12 T:2
+++ after wait on !Serial 1044

Still think we need to do the MTP + Serial...

Will try it out to see how much additional stuff is needed from the files in mtp_t4 for it...
 
Nice work and yes think MTP+Serial would be a good idea - a lot of other USB types have this type of setup as well so would be right in line.

I am playing with it some to add the Serial version...

I added it in to the cores branch: https://github.com/PaulStoffregen/cores/commit/6e6be0b3e6a9f976699ea737cbbf913fc0af7dd1

You would need to edit boards.txt to add it or:
I have a boards.local.txt file that does it...
Code:
teensy41.menu.usb.mtpserial=MTP Disk Serial (Experimental)
teensy41.menu.usb.mtpserial.build.usbtype=USB_MTPDISK_SERIAL

teensyMM.menu.usb.mtpserial=MTP Disk Serial (Experimental)
teensyMM.menu.usb.mtpserial.build.usbtype=USB_MTPDISK_SERIAL

teensy40.menu.usb.mtpserial=MTP Disk Serial (Experimental)
teensy40.menu.usb.mtpserial.build.usbtype=USB_MTPDISK_SERIAL

teensy36.menu.usb.mtpserial=MTP Disk SERIAL (Experimental)
teensy36.menu.usb.mtpserial.build.usbtype=USB_MTPDISK_SERIAL

teensy35.menu.usb.mtpserial=MTP Disk SERIAL (Experimental)
teensy35.menu.usb.mtpserial.build.usbtype=USB_MTPDISK_SERIAL

teensy31.menu.usb.mtpserial=MTP Disk SERIAL (Experimental)
teensy31.menu.usb.mtpserial.build.usbtype=USB_MTPDISK_SERIAL

It appears to run fine on Windows, but Ubuntu does not recognize it as an MTP setup...
@Paul - I know you needed to add the MTP string, which I think it still has... Although wonder if it needs something else... will debug some...

Will update with info off of Ubuntu...
Update:
Code:
kurte@kurte-XPS-8300:~/Arduino/libraries$ lsusb -v -d 16c0:0476

Bus 001 Device 020: ID 16c0:0476 Van Ooijen Technische Informatica 
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass          239 Miscellaneous Device
  bDeviceSubClass         2 
  bDeviceProtocol         1 Interface Association
  bMaxPacketSize0        64
  idVendor           0x16c0 Van Ooijen Technische Informatica
  idProduct          0x0476 
  bcdDevice            2.80
  iManufacturer           1 Teensyduino
  iProduct                2 Teensy MTP Disk/Serial
  iSerial                 3 7819620
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0069
    bNumInterfaces          3
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0xc0
      Self Powered
    MaxPower              100mA
    Interface Association:
      bLength                 8
      bDescriptorType        11
      bFirstInterface         0
      bInterfaceCount         2
      bFunctionClass          2 Communications
      bFunctionSubClass       2 Abstract (modem)
      bFunctionProtocol       1 AT-commands (v.25ter)
      iFunction               0 
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         2 Communications
      bInterfaceSubClass      2 Abstract (modem)
      bInterfaceProtocol      1 AT-commands (v.25ter)
      iInterface              0 
      CDC Header:
        bcdCDC               1.10
      CDC Call Management:
        bmCapabilities       0x01
          call management
        bDataInterface          1
      CDC ACM:
        bmCapabilities       0x06
          sends break
          line coding and serial state
      CDC Union:
        bMasterInterface        0
        bSlaveInterface         1 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x82  EP 2 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0010  1x 16 bytes
        bInterval               5
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        1
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass        10 CDC Data
      bInterfaceSubClass      0 
      bInterfaceProtocol      0 
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x03  EP 3 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x83  EP 3 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        2
      bAlternateSetting       0
      bNumEndpoints           3
      bInterfaceClass         6 Imaging
      bInterfaceSubClass      1 Still Image Capture
      bInterfaceProtocol      1 Picture Transfer Protocol (PIMA 15470)
      iInterface              4 MTP
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x84  EP 4 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x04  EP 4 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval               0
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x85  EP 5 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0020  1x 32 bytes
        bInterval               7
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass          239 Miscellaneous Device
  bDeviceSubClass         2 
  bDeviceProtocol         1 Interface Association
  bMaxPacketSize0        64
  bNumConfigurations      1
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0000
  (Bus Powered)

kurte@kurte-XPS-8300:~/Arduino/libraries$
Wondering if changing the MTP interface from 1 to 2, do I need to have different string id...

Update 2:
Code:
kurte@kurte-XPS-8300:~/Arduino/libraries$ mtp-detect
libmtp version: 1.1.17

Listing raw device(s)
Device 0 (VID=16c0 and PID=0476) is UNKNOWN in libmtp v1.1.17.
Please report this VID/PID and the device model to the libmtp development team
   Found 1 device(s):
   16c0:0476 @ bus 1, dev 51
Attempting to connect device(s)
USB low-level info:
   bcdUSB: 512
   bDeviceClass: 239
   bDeviceSubClass: 2
   bDeviceProtocol: 1
   idVendor: 16c0
   idProduct: 0476
   IN endpoint maxpacket: 512 bytes
   OUT endpoint maxpacket: 512 bytes
   Raw device info:
      Bus location: 1
      Device number: 51
      Device entry info:
         Vendor: (null)
         Vendor id: 0x16c0
         Product: (null)
         Vendor id: 0x0476
         Device flags: 0x00000000
Configuration 0, interface 2, altsetting 0:
   Interface description contains the string "MTP"
   Device recognized as MTP, no further probing.
Device info:
   Manufacturer: PJRC
   Model: Teensy
   Device version: 1.56 / MTP 1.0
   Serial number: 11146950
   Vendor extension ID: 0x00000006
   Vendor extension description: microsoft.com: 1.0;
   Detected object size: 64 bits
   Extensions:
        microsoft.com: 1.0
Supported operations:
   1001: Get device info
   1002: Open session
   1003: Close session
   1004: Get storage IDs
   1005: Get storage info
   1007: Get object handles
   1008: Get object info
   1009: Get object
   100b: Delete object
   100c: Send object info
   100d: Send object
   100f: Format storage
   1014: Get device property description
   1015: Get device property value
   1019: Move object
   101a: Copy object
   101b: Get partial object
   9801: Get object properties supported
   9802: Get object property description
   9803: Get object property value
   9804: Set object property value
Events supported:
   0x4001: CancelTransaction
   0x4002: ObjectAdded
   0x4003: ObjectRemoved
   0x4004: StoreAdded
   0x4005: StoreRemoved
   0x400b: DeviceReset
   0x400c: StorageInfoChanged
   0x400e: UnreportedStatus
   0xc801: ObjectPropChanged
Device Properties Supported:
   0xd402: Friendly Device Name
Playable File (Object) Types and Object Properties Supported:
   3000: Undefined Type
      dc01: Storage ID UINT32 data type ANY 32BIT VALUE form READ ONLY GROUP 0x0
      dc02: Object Format UINT16 data type ANY 16BIT VALUE form READ ONLY GROUP 0x0
      dc03: Protection Status UINT16 data type ANY 16BIT VALUE form READ ONLY GROUP 0x0
      dc04: Object Size UINT64 data type READ ONLY GROUP 0x0
      dc07: Object File Name STRING data type GET/SET GROUP 0x0
      dc08: Date Created STRING data type GET/SET GROUP 0x0
      dc09: Date Modified STRING data type GET/SET GROUP 0x0
      dc0b: Parent Object UINT32 data type ANY 32BIT VALUE form READ ONLY GROUP 0x0
      dc41: Persistant Unique Object Identifier UINT128 data type READ ONLY GROUP 0x0
      dc44: Name STRING data type READ ONLY GROUP 0x0
   3001: Association/Directory
      dc01: Storage ID UINT32 data type ANY 32BIT VALUE form READ ONLY GROUP 0x0
      dc02: Object Format UINT16 data type ANY 16BIT VALUE form READ ONLY GROUP 0x0
      dc03: Protection Status UINT16 data type ANY 16BIT VALUE form READ ONLY GROUP 0x0
      dc04: Object Size UINT64 data type READ ONLY GROUP 0x0
      dc07: Object File Name STRING data type GET/SET GROUP 0x0
      dc08: Date Created STRING data type GET/SET GROUP 0x0
      dc09: Date Modified STRING data type GET/SET GROUP 0x0
      dc0b: Parent Object UINT32 data type ANY 32BIT VALUE form READ ONLY GROUP 0x0
      dc41: Persistant Unique Object Identifier UINT128 data type READ ONLY GROUP 0x0
      dc44: Name STRING data type READ ONLY GROUP 0x0
Storage Devices:
   StorageID: 0x00010001
      StorageType: 0x0004 removable RAM storage
      FilesystemType: 0x0002 generic hierarchical
      AccessCapability: 0x0000 read/write
      MaxCapacity: 31902400512
      FreeSpaceInBytes: 31895748608
      FreeSpaceInObjects: 4294967295
      StorageDescription: SD Card
      VolumeIdentifier: (null)
Special directories:
   Default music folder: 0xffffffff
   Default playlist folder: 0xffffffff
   Default picture folder: 0xffffffff
   Default video folder: 0xffffffff
   Default organizer folder: 0xffffffff
   Default zencast folder: 0xffffffff
   Default album folder: 0xffffffff
   Default text folder: 0xffffffff
MTP-specific device properties:
   Friendly name: Teensy
   Synchronization partner: (NULL)
libmtp supported (playable) filetypes:
   Folder
OK.
kurte@kurte-XPS-8300:~/Arduino/libraries$
 
Last edited:
@Kurte

Just downloaded your Core updates for the T4 and updated my boards.local.txt. Wow comes up real fast as compared to before. Sounds like its a keeper.
 
I have not updated CORES for normal Serial.

As noted MakeFiles.INO was running as Serial and added #ifdef to ideally allow selective use of MTP.
> the MTP.H, and All MTP code is under: #ifdef USB_MTPDISK
- and it builds without error
<edit>: It is working as 'Serial USB' excluding MTP, it does stall on output but not hanging or resetting - on T_4.1

For some reason it won't run as Serial now - with no MTP? Not sure when I checked it last.
> On T_4.1 it comes up prints nothing but one line of the 'showMediaSpace()' - then restarts?

Also the same code that runs on T_4.1 acting Odd on T_MM?

Didn't run at all on SFun ATP? Moved to PJRC breakout and it is starting now.
> It could not write to the SD? All file/folder creates failed?
> there was a WINDOWS file this one? : System Volume Information
- when that existed on the SD card Makefiles.ino got stuck?
> With that removed - or using prior SD card that never had that 'magic' file it works to create
- but w/T_MM MCU the SerMon 'Enter' after files created would HANG on doing the 'VerifyDir' portion, this works as expected on T_4.1.

Also a new user to TyComm posted T_MM failure using DualSerial: Koromix/tytools/issues/87
Trying to repro that caused me to find this problem. And another posted on that Issue where once DualSerial is seen - going to MTP does not remove it.
> wondering if it has to do with the resent edit made? But not enough yet to try the prior version ... as I'm chasing the above ... and was hoping to fix Verify in MakeFiles.
 
Last edited:
defragster said:
Also a new user to TyComm posted T_MM failure using DualSerial: Koromix/tytools/issues/87
Trying to repro that caused me to find this problem. And another posted on that Issue where once DualSerial is seen - going to MTP does not remove it.
> wondering if it has to do with the resent edit made? But not enough yet to try the prior version ... as I'm chasing the above ... and was hoping to fix Verify in MakeFiles.

Tim - a bit confused on this one. I took the ATP Carrier board and compiled the tripple serial example using USB type Dual and Tripple serial and the latest version of TyCommander shows either two or three serial ports. I then took the Tripple Serial example without closing TyCom and initially TyComm shows Serial + Serial1 but Serial1 is grayed out and shows missing. Eventually it disappears completely as time goes by. So as far as I can see it works. Just have to remember to recompile with the USB Serial type you want.

Just downloaded and ran your latest Makefiles.ino on the PJRC TMM board and it is working fine for me on Serial, MTP_Disk, and MTP_DISK_SERIAL - no errors, files are created with no errors:
Capture.PNG

EDIT: Note the windows System Volume Information directory is still on the SD card as well

Reran the sketch with just Serial on the T4.1. Ran without stalling and no errors of file can not be opened. Heres the last bit of the Serial output:
Code:
Make DD File:ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/D7.5/D8.5/D9.5/500	File size=500=500	 us=11502433    end:zDDd. 196x980F7
Make DD File:ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/D7.5/D8.5/D9.5/1012	File size=1012=1012	 us=11550037    end:zDDd. 197x98135
Make DD File:ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/D7.5/D8.5/D9.5/1524	File size=1524=1524	 us=11596846    end:zDDd. 198x98193
Make DD File:ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/D7.5/D8.5/D9.5/2036	File size=2036=2036	 us=11646841    end:zDDd. 199x98211
Make DD File:ManyD10/D0.5/D1.5/D2.5/D3.5/D4.5/D5.5/D6.5/D7.5/D8.5/D9.5/2548	File size=2548=2548	 us=11694202    end:zDDd. 200x982AF

 ================================ Media Size=32014073856	Used Size=100401152	 us=2206747   KB/sec=33.72   Bytes written=76200
dbMakeN File:россиянин/��.txt=32
dbMakeN File:россиянин/��.txt=32
dbMakeN File:россиянин/α.txt=32
dbMakeN File:россиянин/β.txt=32

 ================================ Media Size=32014073856	Used Size=100564992	 us=114820   KB/sec=1.09   Bytes written=128
 
Morning all - Just pushed up a few simple updates:

updated MTP_Teensy.cpp - to handle the only message that we were sending we are busy too, before we were up at least far enough for Serial to come up without us having to respond we are busy...

Simple Example 2 compile issue
SD_Program_... - added the messages I showed yesterday about startup timing
 
Tim - a bit confused on this one. I took the ATP Carrier board and compiled the tripple serial example using USB type Dual and Tripple serial and the latest version of TyCommander shows either two or three serial ports. I then took the Tripple Serial example without closing TyCom and initially TyComm shows Serial + Serial1 but Serial1 is grayed out and shows missing. Eventually it disappears completely as time goes by. So as far as I can see it works. Just have to remember to recompile with the USB Serial type you want.

Just downloaded and ran your latest Makefiles.ino on the PJRC TMM board and it is working fine for me on Serial, MTP_Disk, and MTP_DISK_SERIAL - no errors, files are created with no errors:
View attachment 27254

EDIT: Note the windows System Volume Information directory is still on the SD card as well

Reran the sketch with just Serial on the T4.1. Ran without stalling and no errors of file can not be opened. Heres the last bit of the Serial output:
..

It was getting late here ... machine was calling for update restart ... the WIP MakeFiles was printing double/Quad chars it didn't need too ...

So machine state may have been an issue. But another sketch - a recent LUNI RTC example was set Dual Serial and TyComm showed two, then returning to MRP, neither went grey, and clicking on either instance might show SerMon Spew? It didn't go away until I closed TyComm ???

>> If you are seeing it right - adding a note to TyComm issue might help koromix.

Machine now rebooted - but another task at hand ...

I did make some cleanup headway on MakeFiles Verify - got some output cleaned up, but not all files parsing Verify properly where the '*' filler bytes pad out the EOF with the not yet completed parsing only last block that works on most files.

<edit>: Also some WIN file was removed - maybe there was something else on that SD card not used for some time ... though it was in the SFun Display T_MM when the Locked T_MM arrived?
 
It was getting late here ... machine was calling for update restart ... the WIP MakeFiles was printing double/Quad chars it didn't need too ...

So machine state may have been an issue. But another sketch - a recent LUNI RTC example was set Dual Serial and TyComm showed two, then returning to MRP, neither went grey, and clicking on either instance might show SerMon Spew? It didn't go away until I closed TyComm ???

>> If you are seeing it right - adding a note to TyComm issue might help koromix.

Machine now rebooted - but another task at hand ...

I did make some cleanup headway on MakeFiles Verify - got some output cleaned up, but not all files parsing Verify properly where the '*' filler bytes pad out the EOF with the not yet completed parsing only last block that works on most files.

<edit>: Also some WIN file was removed - maybe there was something else on that SD card not used for some time ... though it was in the SFun Display T_MM when the Locked T_MM arrived?

I did notice with TyCommander that if I left it open when I recompliled the sketched from dual to MTP the second serial Teensy showed Gray - but if I let it sit long enough the second teensy would disappear - so maybe it was a matter to timing on Tycom update - did notice there is an option in the drop down for a application reset that you might have to use. But it doesn't seem to be a Teensy issue but a TyCommander usage pattern.

I ran the sketches on the locked T_MM, unlocked T_MM and the T4.1 without a problem.
 
Quick update on trying to run MTP+Serial on Ubuntu...

As I mentioned a few posts back.

Sometimes it does show up, but not the same. It looks like then Shotwell ... With camera image instead of, not sure what the square one is...
Screenshot from 2022-01-18 11-22-00.png
Not sure if that is PTP instead of MTP?
 
Back
Top