Teensyduino File System Integration, including MTP and MSC

[mjs513]

But again this is an ExFat partition now so that is not right...
My guess is the code in bool SDClass::mediaPresent()
that does: ret = sdfs.restart();

That restart() may not be functioning like it was before SDFat update to latest...

Ok was playing a bit and think I found the solution for the exFAT formatting issue but it requires a minor change to the SD library:

	if (ret) {
		// TODO: Is begin() really necessary?  Is a quicker way possible?
		card->syncDevice();
		sdfs.restart(); // TODO: is sdfs.volumeBegin() enough??
	}

By adding in the syncDevice before the restart the SDIO exFAT will enumerate correctly.

[defragster]

@defragster - here is a snapshot of my flash after I ran the t option and it failed because disk was full. NOTE the many sub directories that were created:
...
kind of shows that LFS can create sub-directories. Yes I believe at one time there was an issue but it has since been fixed for quite a while.

My Bad for not checking enough jumping between LFS Flash and SD ...

> just off a 2+ hour telecon and got my outside project moved along ...

There must be something unique in my 'string' presentation in that and other 'function paths' - and the code responds differently ... maybe a missing or extra strcat( szPath, "/" ); somewhere ...

I'll get back and add string prints to the other DISK.mkdir( szPath ); places and find it.

Good to see the 'ABORT' code working - where before it would try and fail for the rest of that creation iteration when disk space ran out as in p#1297.

[defragster]

Found it: diff in SD versus LFS_Flash mkdir() UPDATED >> github.com/Defragster/T4LockBeta/tree/main/MakeFiles

When a folder in a folder is needed {and neither exists} the SD FS will create both "ManyF" and subfolder "111.128" with one :: DISK.mkdir( "ManyF/111.128" );

When 'DISK' is LittleFS_QSPIFlash myfs; doing DISK.mkdir( "ManyF/111.128" ); fails when ManyF is not present.

Given creation on SD FS this 'abuse' was tolerated and overlooked, needed to add for LFS to work:

void MakeDataFiles( char* szRoot, int numFiles, int startSize, int growSize ) {
// ...
    strcpy( szPath, szRoot );
    if (!DISK.exists(szPath)) {  // NEEDED to add this intermediate test and .mkdir()
      DISK.mkdir( szPath );
    }
    strcat( szPath, "/" );
    strcat( szPath, dirL[ii] );
    sprintf( szTmp, ".%d", numFiles);
    strcat( szPath, szTmp );
    DISK.mkdir( szPath );

When DISK {myfs} is SD: what does format look like? Adding this when SD: DISK.format(0, '.');

Doing that results in a restart showing 0 disk space until restart - probably part of known issue pending resolution?:

	========================= Media Size=0	Used Size=0	 us=7568006
 DISK FULL?

BTW: When Makefiles is running a command - don't 'Send' new Serial data as there is code in places of deep loops to terminate the process on input. Unless interrupt is desired, wait for complete.

[defragster]

Updated Defragster/T4LockBeta/tree/main/MakeFiles to include features and functions of SD_MTP_debug.ino

Edited 2nd SD to only ADD when this works: if (sdSPI.begin(CS_SD2))

All code from SD_MTP_debug.ino at bottom of file with rename of dSetup() and dLoop() called from existing code.

Execute the DEBUG commands by preceding with DOT / '.' i.e.: .c , .p , .s , .d , .r
> enable debug in MTP_Teensy.c
> not yet made to #ifdef out
> .r works but now redundant to 'U'

[KurtE]

Some updates on MTP and MSC (USBDrive/USBFilesystem):

I have been playing with some of the what I think of as missing pieces within the MTP and USB Filesystem code. In particular how to make it easier for Sketch to "Just use it"

Note: These changes are not in the main branches of USBHost_t36 (https://github.com/KurtE/USBHost_t36...format_cleanup)
or MTP_Teensy (https://github.com/KurtE/MTP_Teensy/...etect_sd_media)

I have not yet issued a Pull Request on these as still testing them out and also making sure they pass the smell test

USBHost_t36 changes:
a) Cleanup format code a little bit - what the branch was originally created for...

b) I did not like having the sketches have to call something, that when we detect a drive was plugged in or removed to then have the drive enumerate all of the partitions and if appropriate claim one or more
USBFilesystem objects to handle any Fat/ExFat partitions contained in the MBR, or GPT list...

So now the code is hooked up using the USBDrive.Task() method, which before was using the default {}... So this part is take care of by calls like: myUSB.Task();

c) I moved some of the stuff out of the USBFilesystem class to be within the base class, including being derived by the FS class. I also no longer link/unlink the objects depending on if they are claimed or not. I can teel if they are claimed as the device member variable is set or not...

d) Also cleaned up the code to know the connection state and the state of has the drive been formatted. Again part of the base class with a query and set function. Also a query/set function for has any device changed state since I last checked/cleared it...

e) probably a few other things, like rename some variables...


MTP_Teensy

a) Extended the MTP_class::loop() functionality... Actually MTPStorage.loop() which is already called by the MTP_class code. In here it allows one to register a callback function, to be called, either once per call for a logical type or once per instance of logical type... Currently just (unknown, SD (instance), USBFS(class))...

b) updated the MTP_class::addFilesystem to allow you to pass in optional parameter saying which one it is... default to unknown and callbacks won't be used... (except...)

c) Hacked MTP_Teensy.h header file in the class that if SD.h had been included in the sketch (before this header file was included), it added a simple method override for the SDClass object and if so, when
you add an SD it will automatically call off to the main one with the optional parameter...

Note: I wanted to simply use __has_include(<SD.h> but as I pointed out in another sketch if your code has something like:
if MTP_Teensy.h started off like:

...
#include "MTP_Storage.h"
#if defined(__has_include) && __has_include(<SD.h>)
#include <SD.h>
#endif
...

And your sketch had:

#include <SD.H>
#include <MTP_Teensy.h>

It build fine.

However if you had:

#include <MTP_Teensy.h>
#include <SD.H>

The code will fail to build as SD.h will be never be included... Other thread, but simply put Arduino builder will not include the directory that contains SD.h into the -I search list...
Turns out issue with GCC compilers...

d) I added two .cpp files, one each for SD and USBFilesystem...

Both start off something like:

#include "MTP_Teensy.h"
#include "MTP_Storage.h"

// This code should only be build and run if the SD library was included in the sketch
#if defined(__has_include) && __has_include(<SD.h>)
#include <SD.h>
// I am going to create a simple class with a constructor and maybe some simple data. 
// that can be associated with the SD...
// This is sort of a prototype, could expand to something more later.
class MTP_SD_Callback {
public:

	// constructor
	MTP_SD_Callback() {
		MTP.storage()->registerClassLoopCallback(MTP_FSTYPE_SD, &checkMediaPresent);   
	}
	
	// static callback for device check (media present)
	static bool checkMediaPresent(uint8_t storage_index, FS *pfs);

	static uint8_t media_present_prev_[MTPD_MAX_FILESYSTEMS];

};

uint8_t MTP_SD_Callback::media_present_prev_[MTPD_MAX_FILESYSTEMS] = {0};

// define one here so it's constructor will be called. 
static MTP_SD_Callback mtsdcb;

So in this case if the sketch includes SD.h somewhere, this code will be included and it defines a dummy class, whose constructor will call off to register the callback
function with Storage. So again this code will not be added unless SD was already going to be included.

And then the rest of the code in these files, take care of either checking for SD media present or not, and checks when a USBDrive is plugged in and the code went through and
enumerated and updated one or more USBFilesystems, this code will then see about adding/removing the drives from the MTP storage...

What does all of this mean
Here is a pretty simple sketch:

/*
  Simple MTP example that will add a RAM and Flash drive plus handles allowing you to insert one or more USB drives
  and shows those as well.

*/
#include <USBHost_t36.h>
#include <SD.h>
#include <MTP_Teensy.h>

// Setup USBHost_t36 and as many HUB ports as needed.
USBHost myusb;
USBHub hub1(myusb);

// Instances for the number of USB drives you are using.
USBDrive myDrive1(myusb);

// Instances for accessing the files on each drive
USBFilesystem pf1(myusb);
USBFilesystem pf2(myusb);
USBFilesystem pf3(myusb);
USBFilesystem pf4(myusb);


void setup() {
  // Open serial communications and wait for port to open:

  // setup to do quick and dirty ram stream until Serial or like is up...
  MTP.begin();

  SD.begin(BUILTIN_SDCARD);
  MTP.addFilesystem(SD, "SD Card");

  while (!Serial && millis() < 5000) { }
  if (CrashReport)  Serial.print(CrashReport);
  Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);

  myusb.begin();

}

void loop() {
  myusb.Task();
  MTP.loop();

}

With it, it will add the SD card to MTP... If the media is not present, if you plug in card it should detect it and update the MTP status.
Note: In this case the MTP window is not automatically refreshed. but if you hit F5 you will see the updated status.


And if a USB drive is detected, the File systems will be added automatically.



It ain't perfect, but...

[defragster]

@kurtE - Nice - will look at the usage and see if it helps MakeFiles - except that one wanted to pick a single 'DISK' for testing.

[mjs513]

@KurtE
Gave it a quick test run - nothing extensive and mostly with a hub attached more of stress test on the add/remove feature. It does work:

1.  Using my M2 drive inserted on boot up it recognizes and I can remove and replace.  If already started and I insert its hit or miss
2.  Normal Thumb drives seem to work
3.  on of my 2 SSD's work on the hub - one has a USB C cable the other has the 3.x cable and is not recogized (don't remember if this worked before though)

[wwatson]

@KurtE - Saw pretty much what @mjs513 saw.Serial monitor shows removal and insertion of SD card (built in) but the mount stayed mounted in the file explorer. This is using Ubuntu. As for USB it works flawlessly. Remove the SSD drive and MTP unmounts and the file explorer closes. Add it back and it mounts and the file explorer opens.This is the most stable version of MTP I seen yet

I was only able to use SSD drive. My other two thumb drives are formatted with ext4 and have test data on them right now for working with lwext on the T4.1.

Things seem to be coming along nicely

[KurtE]

Quick update:

I am trying an experiment with the SD drives and the notification event we send to host...

#if 1
		MTP.send_StoreRemovedEvent(storage_index);
		delay(5);
		MTP.send_StoreAddedEvent(storage_index);
#else
	    MTP.send_StorageInfoChangedEvent(storage_index);
#endif

So it sends two event messages to first remove it from list and add it back...

So far tested with simple sketch with just the 1 SD and the example with SD and USBFS I posted yesterday, and the SD card now does show updated state.
Note: It takes longer to detect when it is removed than when it is inserted as the SD code that detects media removed waits for timeout from device...

I pushed this stuff up in the MTP_Teensy branch mentioned in previous post

[mjs513]

Quick update:

........
I pushed this stuff up in the MTP_Teensy branch mentioned in previous post

Ok finally got a chance and downloaded your latest changes seems to be working as long as the cards are in the Builtin slot on Teensy start up. If its not and you insert a card it does recognize that you inserted the card. If it starts with the card in it works as you would expect. I think you mentioned this before.

However, if you use an external card reader it will recognize a card has been inserted whether or not the card was in the extern card reader or not.

[defragster]

Ok finally got a chance and downloaded your latest changes seems to be working as long as the cards are in the Builtin slot on Teensy start up. If its not and you insert a card it does recognize that you inserted the card. If it starts with the card in it works as you would expect. I think you mentioned this before.

However, if you use an external card reader it will recognize a card has been inserted whether or not the card was in the extern card reader or not.

Mike, are these (other) p#1299 edits in place for SD code: pjrc.com/threads/68139-Teensyduino-File-System-Integration-including-MTP-and-MSC?p=308206&viewfull=1#post308206

... I got Win_github to sync - but not compiled with the updates here yet

[mborgerson]

About 6 months ago I invested in an entry-level M1 Mac Mini (8GB RAM 256GB SSD). Overall, I'm very pleased with the system. With the latest version of Teensyduino/Arduino, I am able to compile my programs from source code transferred from my PC without problems. For an entry-level system, the performance during the compile stage is outstanding. It compiles and uploads to the Teensy about twice as fast as the same operations on my mid-level gaming PC (Ryzen 7, 8 cores, 3.6GHz, 16GB RAM, Samsung NVME EVO 970 Plus 500GB SSD). I suspect the speed increase, especially after recompiling where many object files are cached, is due to the very fast transfers to and from the Mac Mini internal SSD.

The one thing I have been missing until today was the ability to upload picture files from the T4.1 to the Mac using MTP. MacOS Monterey doesn't have native MTP drivers. I avoided that problem for the last few months by using a Dual Serial USB connection where one channel was used for controlling the Teensy App and the other is dedicated to file uploads. This required a custom dual-serial host app on the PC (written in C++ Builder) and a Python App on the Mac to handle the dual serial interface. This worked OK for single file transfers but was not suitable for transferring multiple files (such as pictures collected with my motion-detection app for the OV7670.)

I recently found the the Commander One MacOS app will mount MTP devices and allow file transfers. I installed the free version of the app and, voila, I can select and upload multiple files from the Teensy device! The Commander One app comes in a free version, and a Pro version---which costs $29.99. The Pro features are available for a 2-week trial period. The primary pro features are FTP file access and more ways to manipulate compressed files. Since I don't know whether MTP access is part of the Pro package, I went ahead and paid the $29.99.

I'm still using a fairly old MTP version that I customized a bit about a year ago. I look forward to the time that MTP is fully integrated into TeensyDuino. At this point, I have no particular use for LittleFS, MSC, and some of the other features in the current libraries under development, so there has been little incentive to try the new libraries as they are developed.

[mjs513]

Mike, are these (other) p#1299 edits in place for SD code: pjrc.com/threads/68139-Teensyduino-File-System-Integration-including-MTP-and-MSC?p=308206&viewfull=1#post308206

... I got Win_github to sync - but not compiled with the updates here yet

@defragster - sorry just saw the post. Yes @KurtE incorporated those changes into his fork of the SD library. Not sure if its been incorporated into Paul's master SD yet?

[KurtE]

Good morning all (at least my time zone)...

So far I have not seen the SD Pull Request (https://github.com/PaulStoffregen/SD/pull/43) has not been merged.

Ok finally got a chance and downloaded your latest changes seems to be working as long as the cards are in the Builtin slot on Teensy start up. If its not and you insert a card it does recognize that you inserted the card. If it starts with the card in it works as you would expect. I think you mentioned this before.

I am debugging this now. Note sometimes it works.... Sometimes it does not, but then if I pull it back out and push it back in it works?

And sometimes weird if I have TyCommander up and watching the Serial output, and I start up with SD not inserted, and then I try to insert, I see the Teensy tool, show a state change and say something like: attempting to erase... ??? Note: in this case I was testing with locked Micromod, not sure if that would make any difference... Maybe I should swap in a normal one?

[wwatson]

@KurtE @mjs513 @Paul - Here is an update on the progress with integrating lwext and FS. I finally have the ground work done on mounting USB and SD drives. That includes partitions on USB drives. The hardest part was figuring out how to open the block devices. There are two types of pointers used with the ext4_blockdev struct. One type is used for accessing the hardware info for a particular device that is used to initialize the block device interface (bdif). The other is for the mbr scan that gathers partition info for the ext4_mbr_bdevs struct. All of this info is put into two arrays of structs. One for the block device info and one for the mount info. Then it is just a matter of getting an index to the USB drives or the SD card by comparing the pointers from ext4 and the device list or mount list. That allows for identifying which partition on which drive ext4 is trying to access.
It's crude but works.

I was trying to figure out the best way to implement the thin wrapper for lwext. Looked at lfs driver portion of LittleFS and noticed that it and lwext have amazing similarities in structure and layout. So I decided to use LittleFS as a template for the lwext version. A lot of the ext4 methods are drop in replacements for lfs methods.

Anyway, having a lot of fun with this. Can't wait until it's ready for some of the classic SD sketches

[mjs513]

I was trying to figure out the best way to implement the thin wrapper for lwext. Looked at lfs driver portion of LittleFS and noticed that it and lwext have amazing similarities in structure and layout. So I decided to use LittleFS as a template for the lwext version. A lot of the ext4 methods are drop in replacements for lfs methods.

Anyway, having a lot of fun with this. Can't wait until it's ready for some of the classic SD sketches

Sounds like progress and you are having a lot of fun. Going to interesting when you get it completely integrated - then the fun begins to see about using it with MTP

Nice work on lext4 by the way. Didn't envy you getting it up and running.

[wwatson]

@All - I have the initial lwext4 FS wrapper working with ext4 formatted devices. It is far from finished but I have a usage sketch that is based on LittleFS_Usage.ino working:

// ext4Usage.ino  Teensy lwext4 testing 
// Based on lwext4 by: Grzegorz Kostka (kostka.grzegorz@gmail.com)
// Modified version of LittlFS_Usage.ino for use with LWextFS.

#include "ext4FS.h"

extern USBHost myusb;

LWextFS myext4fs;

File file, file1, file2;

#define sda1 0   // First partition on first USB device
#define sdd1 12  // First partition on SD device
char volume[32];

void setup() {
   while (!Serial) {
    yield(); // wait for serial port to connect.
  }
  if(CrashReport)
	Serial.print(CrashReport);

  Serial.printf("%cTeensy lwext file system test\n\n",12);

  Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);

  Serial.println("Initializing LWextFS ...");

  myusb.begin();

  if(!myext4fs.begin(sda1)) // Change this to sdd1 for SD card.
    Serial.printf("myext4fs.begin(sda1): Failed...\n");
  else
    Serial.printf("myext4fs.begin(sda1): passed...\n");
  Serial.printf("Volume Name: %s\n",myext4fs.getMediaName());
  strcpy(volume,"/mp/sda1/"); // Change this to /mp/sdd1/ for SD card.

  // Now lets create a file and write some data.  Note: basically the same usage for 
  // creating and writing to a file using SD library.
  Serial.println("\n---------------");
  Serial.println("Now lets create a file with some data in it");
  Serial.println("---------------");
  char someData[128];
  memset( someData, 'z', 128 );
  file = myext4fs.open("/mp/sda1/bigfile.txt", FILE_WRITE);
  file.write(someData, sizeof(someData));

  for (uint16_t j = 0; j < 100; j++)
    file.write(someData, sizeof(someData));
  file.close();
  
  // We can also get the size of the file just created.  Note we have to open and 
  // thes close the file unless we do file size before we close it in the previous step
  file = myext4fs.open("/mp/sda1/bigfile.txt", FILE_WRITE);
  Serial.printf("File Size of bigfile.txt (bytes): %u\n", file.size());
  file.close();

  // Now that we initialized the FS and created a file lets print the directory.
  // Note:  Since we are going to be doing print directory and getting disk usuage
  // lets make it a function which can be copied and used in your own sketches.
  listFiles();
  waitforInput();

  // Now lets rename the file
  Serial.println("\n---------------");
  Serial.println("Rename bigfile to file10");
  myext4fs.rename("/mp/sda1/bigfile.txt", "/mp/sda1/file10.txt");
  listFiles();
  waitforInput();

  // To delete the file
  Serial.println("\n---------------");
  Serial.println("Delete file10.txt");
  myext4fs.remove("/mp/sda1/file10.txt");
  listFiles();
  waitforInput();

  Serial.println("\n---------------");
  Serial.println("Create a directory and a subfile");
  myext4fs.mkdir("/mp/sda1/structureData1");

  file = myext4fs.open("/mp/sda1/structureData1/temp_test.txt", FILE_WRITE);
  file.println("SOME DATA TO TEST");
  file.close();
  listFiles();
  waitforInput();

  Serial.println("\n---------------");
  Serial.println("Rename directory");
  myext4fs.rename("/mp/sda1/structureData1", "/mp/sda1/structuredData");
  listFiles();
  waitforInput();

  Serial.println("\n---------------");
  Serial.println("Lets remove them now...");
  //Note have to remove directories files first
  myext4fs.remove("/mp/sda1/structuredData/temp_test.txt");
  myext4fs.rmdir("/mp/sda1/structuredData");
  listFiles();
  waitforInput();

  Serial.println("\n---------------");
  Serial.println("Now lets create a file and read the data back...");
  
  // LWextFS also supports truncate function similar to SDFat. As shown in this
  // example, you can truncate files.
  //
  Serial.println();
  Serial.println("Writing to datalog.bin using LWextFS functions");
  file1 = myext4fs.open("/mp/sda1/datalog.bin", FILE_WRITE);
  unsigned int len = file1.size();
  Serial.print("datalog.bin started with ");
  Serial.print(len);
  Serial.println(" bytes");
  if (len > 0) {
    // reduce the file to zero if it already had data
    file1.truncate();
  }
  file1.print("Just some test data written to the file (by SdFat functions)");
  file1.write((uint8_t) 0);
  file1.close();

  // You can also use regular SD type functions, even to access the same file.  Just
  // remember to close the file before opening as a regular SD File.
  //
  Serial.println();
  Serial.println("Reading to datalog.bin using LWextFS functions");
  file2 = myext4fs.open("/mp/sda1/datalog.bin");
  if (file2) {
    char mybuffer[100];
    int index = 0;
    while (file2.available()) {
      char c = file2.read();
      mybuffer[index] = c;
      if (c == 0) break;  // end of string
      index = index + 1;
      if (index == 99) break; // buffer full
    }
    mybuffer[index] = 0;
    Serial.print("  Read from file: ");
    Serial.println(mybuffer);
  } else {
    Serial.println("unable to open datalog.bin :(");
  }
  file2.close();

  Serial.println("\nBasic Usage Example Finished");
}

void loop() {

}

void listFiles()
{
  Serial.println("---------------");
  printDirectory(myext4fs);
  Serial.printf("Bytes Used: %llu, Bytes Total:%llu\n", myext4fs.usedSize(), myext4fs.totalSize());
}

void printDirectory(FS &fs) {
  Serial.println("Directory\n---------");
  printDirectory(fs.open(volume), 0);
  Serial.println();
}

void printDirectory(File dir, int numSpaces) {
   while(true) {
     File entry = dir.openNextFile();
     if (! entry) {
       //Serial.println("** no more files **");
       break;
     }
     printSpaces(numSpaces);
     Serial.print(entry.name());
     if (entry.isDirectory()) {
       Serial.println("/");
       printDirectory(entry, numSpaces+2);
     } else {
       // files have sizes, directories do not
       printSpaces(36 - numSpaces - strlen(entry.name()));
       Serial.print("  ");
       Serial.println(entry.size(), DEC);
     }
     entry.close();
   }
}

void printSpaces(int num) {
  for (int i=0; i < num; i++) {
    Serial.print(" ");
  }
}

void waitforInput()
{
  Serial.println("Press anykey to continue");
  while (Serial.read() == -1) ;
  while (Serial.read() != -1) ;
}

Results:

Teensy lwext file system test


/home/wwatson/Arduino/libraries/TeensyEXT4/examples/ext4Usage/ext4Usage.ino Jul  9 2022 14:11:42
Initializing LWextFS ...
myext4fs.begin(sda1): passed...
Volume Name: 126GEXT4

---------------
Now lets create a file with some data in it
---------------
File Size of bigfile.txt (bytes): 12928
---------------
Directory
---------
lost+found/
bigfile.txt                           12928

Bytes Used: 121186902016, Bytes Total:123728822272
Press anykey to continue

---------------
Rename bigfile to file10
---------------
Directory
---------
lost+found/
file10.txt                            12928

Bytes Used: 121186902016, Bytes Total:123728822272
Press anykey to continue

---------------
Delete file10.txt
---------------
Directory
---------
lost+found/

Bytes Used: 121186902016, Bytes Total:123728822272
Press anykey to continue

---------------
Create a directory and a subfile
---------------
Directory
---------
lost+found/
structureData1/
  temp_test.txt                       19

Bytes Used: 121186902016, Bytes Total:123728822272
Press anykey to continue

---------------
Rename directory
---------------
Directory
---------
lost+found/
structuredData/
  temp_test.txt                       19

Bytes Used: 121186902016, Bytes Total:123728822272
Press anykey to continue

---------------
Lets remove them now...
---------------
Directory
---------
lost+found/

Bytes Used: 121186902016, Bytes Total:123728822272
Press anykey to continue

---------------
Now lets create a file and read the data back...

Writing to datalog.bin using LWextFS functions
datalog.bin started with 0 bytes

Reading to datalog.bin using LWextFS functions
  Read from file: Just some test data written to the file (by SdFat functions)

Basic Usage Example Finished

There is still a lot of work left. Each mount point (sda1...sdd4) represents a partition so I have setup for 4 physical storage devices (sda..sdd) each having 4 partitions. That is a total of 16 mount points. Unfortunately ext4 only supports 4 mount points at a time. Haven't looked into whether this can be changed or not. Would there be any use for more than four? Also probably not good to use partitions on an SD card. Block device sdd is fixed mount point for the SD card.

Left to do:

1) A mount point like '/mp/sda1/' has to be prefixed to any file name of directory name access. This will have to be done in the wrapper.
2) Move all intermediate functions into the LWextFS class other than the ext4 driver itself.
3) Setup the time and date methods for directory listings.
4) Clean up the code and post on GitHub.

There are a lot of other features available with lwext4 such as extended attributes like ownership, permissions, unmounting (which should be done with ext4 as it keeps track of uncleanly unmounted devices) recovering unclean mounts and hard and symbolic links. Probably not all of this is needed or wanted. Not sure.

Ok back to it

[KurtE]

Paul and others, while waiting for 1.57 to be released and hopefully time to merge in more stuff, I am playing around with a few ideas:

For example with SD library:

There is a comment in the SDClass that says:

public: // allow access, so users can mix SD & SdFat APIs
	SDFAT_BASE sdfs;

Which gives us a little bit of ability to mix and match.... Mainly for things like how to use SPI1 instead of SPI.

But this does not allow us to do much with mix and match of the stuff associated with the underlying files. For example suppose I want to pre-allocate 40K or 400K. when I open the file.
I can do it, but then I can not use it as a File object...
That is we can do:

  FsFile fsfile = SD.sdfs.open("/prealloc.txt", O_RDWR | O_CREAT);
  if (fsfile) {
    fsfile.preAllocate(40*1024);
...

But currently there is no way then to convert this to a File object.

I have experimented with 4 different ways:

a) Punt (what we do now)

b) Duplicate the SDFile class to allow me to wrap it... This is more or less what the USBFilesystem class is doing today.. Just another wrapper of FsFile objects.

c) add a method to the SDClass that does this for us. For example I have this in experiment

	inline static File FileFromSDFile(SDFAT_FILE &file) {
			if (file) return File(new SDFile(file));
		return File();
	}

I changed the open method to use it:

File open(const char *filepath, uint8_t mode = FILE_READ) {
		oflag_t flags = O_READ;
		if (mode == FILE_WRITE) flags = O_RDWR | O_CREAT | O_AT_END;
		else if (mode == FILE_WRITE_BEGIN) flags = O_RDWR | O_CREAT;
		SDFAT_FILE file = sdfs.open(filepath, flags);
		return FileFromSDFile(file);
	}

Then in my sketch I had:

  File our_file;
  FsFile fsfile = SD.sdfs.open("/prealloc.txt", O_RDWR | O_CREAT);
  if (fsfile) {
    fsfile.preAllocate(40*1024);
    our_file = SD.FileFromSDFile(fsfile);
  }
  if (our_file) {
    our_file.write("Our preallocated file");
    our_file.close();
  }

And it worked.

d) Like c) except don't touch SDClass instead change SDFile to make it's constructor public instead of private.
Then example code would look like:

  File our_file;
  FsFile fsfile = SD.sdfs.open("/prealloc.txt", O_RDWR | O_CREAT);
  if (fsfile) {
    fsfile.preAllocate(40*1024);
    our_file = new SDFile(fsfile);
  }
  if (our_file) {
    our_file.write("Our preallocated file");
    our_file.close();
  }

Given c) or d) we could potentially eliminate the MSCFILE stuff within USBHost.


Second experiment:

Keep a linked list of SDClass objects:

	SDClass() {
		_nextSD = s_firstSD;
		s_firstSD = this;
	}
	~SDClass() {
		// unlink us... I expect rairly called.
		if (s_firstSD == this) s_firstSD = _nextSD;
		else {
			SDClass *psd = s_firstSD;
			while (psd && (psd->_nextSD != this)) psd = psd->_nextSD;
			if (psd) psd->_nextSD = _nextSD;
		}
	}

Why? Good question, just starting to play

But one goal with the MTP stuff is to hopefully be able to process SD insertion and removals, without user code needing to do anything special.

I have a hacked up version of MTP_Teensy, that tries to do it by: knowing we have SD.h included in our sketch and then add method override to object:

  #if defined(__SD_H__)
  uint32_t addFilesystem(SDClass &disk, const char *diskname) {
    printStream_->println("Add **SDClass** file system");
    return addFilesystem(disk, diskname, MTP_FSTYPE_SD);
  }
  #endif

That we then know that you are adding an SD object and then I have other code that uses this doing the mtp loop code... But it is error prone as SD.h has to be included before MTP_Teensy is included as
__has_include() has issues...

But another approach is for me to have a way to know if the FS you are adding is an SD object...
Another way would be then to walk this chain of SD objects and see if it is one of them...

For this experiment, I added method to SDClass:

	static bool isSDFS(FS *pfs) {
			SDClass *psd = s_firstSD;
			while (psd && (psd != pfs)) psd = psd->_nextSD;
			return (psd != nullptr);
		}

Overhead of this, I don't think will be much as most sketches maybe have one or two, maybe up to 4-5 SD objects. I doubt the destructor code will be called...

Again just trying a few experiments.

Thoughts?

[PaulStoffregen]

Thoughts?

Short term, I want to wrap up 1.57 next week, maybe even by this weekend. So right now is not the time to extend APIs.

After 1.57 release, I believe file preallocation belongs in a FS create() function. I believe we've talked about this before and the challenging part (the reason it's always been punted) is uncertainty about what should happen with filesystems that like LittleFS. We need to come up with an explanation (to become official documentation) of what the function does and how code using it is expected to deal with the resulting created file.

Likewise for media detection, long-term I want to do this in the FS class. We already have FS mediaPresent(), which probably isn't enough. After 1.57 is released, let's talk of how we can improve or extend FS so MTP can get the needed media detection features. I also want to make use of those APIs in the audio library, so we can gracefully handle media changes for media access like playing WAV files from SD cards.

One direction I do not want to take is special functions like isSDFS(). Yes, it is the quickest and easiest path to solving a specific problem, but the long term cost is we surely end up with issues like Defragster hit with Arduino IDE 2.0 not working on Windows when the temporary folder is on a ramdisk. Not that microcontrollers will ever run the Arduino IDE! And maybe some of those types of problems are inevitable. But let's plan on at least try to extend the FS class in a way where we're not needing to detect the underlying filesystem class and build libraries that will later have issues when used with other filesystems.

[KurtE]

Short term, I want to wrap up 1.57 next week, maybe even by this weekend. So right now is not the time to extend APIs.

Totally understand... Sorry I may have not been clear that I was assuming nothing else would be merged for 1.57!

But was trying to jump start for post 1.57 before I forget what we were trying to do.

After 1.57 release, I believe file preallocation belongs in a FS create() function. I believe we've talked about this before and the challenging part (the reason it's always been punted) is uncertainty about what should happen with filesystems that like LittleFS. We need to come up with an explanation (to become official documentation) of what the function does and how code using it is expected to deal with the resulting created file.

Yes the one specific case (pre-allocate) I showed here has been mentioned before. And yes LittleFS may not support it. But I would hope we don't always go for least common denominator.
At least not without some way to make use of the FS specific capabilities. when needed.

In this specific case: suppose a preAllocate method was added to File Interface, we have several options:

If we have it return as a bool... So LittleFS would always fail... But this may not be good thing as it would not tell us if it failed because of not supported, or not enough space available, or ???

Or it could could return some enumeration value....

Or could have another method, like supportsPreAllocate which returns false. Or coulbe part of some capabilities method or...

But in this case I am only using the half open Barn door. That is the SDFat object is public in the SD object, so the open and the like is fully available. thing I was adding, was an ability to wrap the File object.

And as mentioned in b) if I needed this functionality, I would just make my own copy of it, like we did in USBHost_t36.
It also gives you the ability to for example use different open file options like: O_EXCL or O_SYNC

Likewise for media detection, long-term I want to do this in the FS class. We already have FS mediaPresent(), which probably isn't enough. After 1.57 is released, let's talk of how we can improve or extend FS so MTP can get the needed media detection features. I also want to make use of those APIs in the audio library, so we can gracefully handle media changes for media access like playing WAV files from SD cards.

Yes mediaPresent is a good first step - BUT as the state of the system is, it can get a bit tricky.
Obviously it only helps if someone calls it and knows what to do with the change of status.

So for example if you add a couple file systems to MTP and lets say 2 of them are SD drives
(Internal and Audio). And you are in a basic simple example, you would like the system to be able to handle the insertions and removals automatically. And in most cases you don't want to have to do anything special.

However as some checks for SD cards take over a second or two to complete, you want to have some control of when this is called. In current setup I have it called only after some time period since the last time we checked and if the user calls the MTP loop function. But even with that would still maybe want an MTP method to say, don't do this while I am doing something time critical...

Likewise if a new SSD is plugged into USBHost that has 3 filesystems on it. When should this be checked? So this created 3 new disk objects, who is responsible for then deciding to add these File Systems to MTP. How much control does the MTP code want to filter which of these drives should be added...

More on this later. Will probably hold off doing anything else on this, until some of the fundamental things are decided.

One direction I do not want to take is special functions like isSDFS(). Yes, it is the quickest and easiest path to solving a specific problem, but the long term cost is we surely end up with issues like Defragster hit with Arduino IDE 2.0 not working on Windows when the temporary folder is on a ramdisk. Not that microcontrollers will ever run the Arduino IDE! And maybe some of those types of problems are inevitable. But let's plan on at least try to extend the FS class in a way where we're not needing to detect the underlying filesystem class and build libraries that will later have issues when used with other filesystems.

I hear you. That was a quick and dirty and more for a specific purpose, toward trying to automate being able to call the mediaPresent... i.e. some process/thread/piece of code - may want to have a list of SD objects that it can then monitor changes of state.

This is what some code in other branch of MTP_Teensy is doing, but it uses an override of the addFileSystem (uint32_t addFilesystem(SDClass &disk, const char *diskname)
To know the user passed in SD File system... But was just experimenting with other approaches.

As for Computer systems being able to identify additional information about a Filesystem object or Handle... They probably all do. For example on Linux, doing Serial, I found earlier when talking to servos, I really needed to know if I am talking to FTDI type objects or not to handle latency issues. On FTDI TC_DRAIN really helps, on others, it really hurts... In some of these cases would find out where the object was in the device tree.

Again lots of interesting things to discuss when you have the time.

So hopefully we can get 1.57 out quickly!

[wwatson]

@All - Now that TD1.57B4 has been released I tested the latest version of TeensyEXT4 with it. It is working I was waiting to see if any of the changes to USBHost_t36 would affect it. I did not run across any issues at all. The thin wrapper for ext4 is working. I based this on LittleFS as it is very much like lwext4 in structure. There is so much more that can be done with ext4 but I kept it within the bounds of FS for testing.

There is still a lot of work to be done with it and I know my coding stinks so it's still WIP

Here is the latest working version:
https://github.com/wwatson4506/Teens...e/TeensyEXT4V2

I have included the following working examples:

- ext4Usage.ino - This sketch contains information on the general usage of TeensyEXT4 and is similar to LittleFS_Usage and SdFat_Usage.
- ext4DeviceInfo.ino - Gives various stats about a mounted block device and it's partitions.
- ListFiles.ino - Gives a directory listing.
- Files.ino - Tests file existance.
- Datalogger.ino - Logs ADC data to a block device.
- Dump.ino - Dumps the data from Datalogger.ino.
- ReadWrite.ino - Demonstrates reading and writing test strings to a block device (append mode).
- wavePlayerExt4.ino - Plays wav files.

Any directory or file access must include the mount point like "/mp/sda1/xxxxx.xxx" or "/mp/sdc3/xxxxx.xxx". I created a function to do this so it would be transparent to the user but I think it can be done better:

	File open(const char *filepath, uint8_t mode = FILE_READ) {
		if(strlen(filepath) == 1) filepath = ""; // Remove leading "/" for root dir.
		strcpy(tmp, filepath);
		lwext_get_mp(tmp, id);

And:

void lwext_get_mp(const char *fn, uint8_t id) {
	char tbuf[256];
	sprintf(tbuf,"%s%s",mount_list[id].pname, fn);
	strcpy((char *)fn,tbuf);
}

Formatting a ext4 volume is not working yet. Been working on this for the last week or so. When I turn on the debug in the options file it will sometimes format the USB drive without an error and I can plug it into my Linux box and it works. But if I plug it into the T4.1 it fails. Think it has something to do with needing delays. Not sure.

Again, just format an SD card (/mp/sdd1 on the T4.1) or a USB drive and add a volume name. You should be able to run the example sketches.

Please test if you can

[KurtE]

@wwatson - Sorry I missed this update.

I will try to play soon, looks like fun

[KurtE]

Notes to self and others:

Today I was playing around with trying to figure out how to transfers files using USB from RPI to T4.1, to try to help out on the thread: https://forum.pjrc.com/threads/70719...-to-Teensy-4-1

Already we can manipulate the files with GUI Files. But wondered if we can do it as well with CLI, as maybe in case of RPI, I will be running headless.

So I found that under Ubuntu at least the later ones. I am running 22.04, it is setup using gvfs..

When the teensy is detected and the like it is added into the name space at: /run/user/$USER/gvfs/*

In my case right now: /run/user/1000/gvfs/mtp:host=Teensyduino_Teensy_MTP_Disk_12158030

where I have CD down to the SD so my current directory is:/run/user/1000/gvfs/mtp:host=Teensyduino_Teensy_MTP_Disk_12158030/SD Card

From the command line I can do ls and it appears to show the files. I think I can also copy a text file from the Teensy to ubuntu. But doing a normal copy to teensy was failing.

I did some searching around and it looks like the command gio can handle it

Example:

kurte@kurte-XPS-8300:/run/user/1000/gvfs/mtp:host=Teensyduino_Teensy_MTP_Disk_12158030/SD Card/from_rpi/mtp_teensy$ cp ~/Arduino/key.pem .
cp: cannot create regular file './key.pem': Operation not supported
kurte@kurte-XPS-8300:/run/user/1000/gvfs/mtp:host=Teensyduino_Teensy_MTP_Disk_12158030/SD Card/from_rpi/mtp_teensy$ gio copy  ~/Arduino/key.pem .
kurte@kurte-XPS-8300:/run/user/1000/gvfs/mtp:host=Teensyduino_Teensy_MTP_Disk_12158030/SD Card/from_rpi/mtp_teensy$ ls
key.pem  MTP_Teensy.cpp
kurte@kurte-XPS-8300:/run/user/1000/gvfs/mtp:host=Teensyduino_Teensy_MTP_Disk_12158030/SD Card/from_rpi/mtp_teensy$

The command has several sub-commands:

kurte@kurte-XPS-8300:/run/user/1000/gvfs/mtp:host=Teensyduino_Teensy_MTP_Disk_12158030/SD Card/from_rpi/mtp_teensy$ gio
Usage:
  gio COMMAND [ARGS…]

Commands:
  help     Print help
  version  Print version
  cat      Concatenate files to standard output
  copy     Copy one or more files
  info     Show information about locations
  launch   Launch an application from a desktop file
  list     List the contents of locations
  mime     Get or set the handler for a mimetype
  mkdir    Create directories
  monitor  Monitor files and directories for changes
  mount    Mount or unmount the locations
  move     Move one or more files
  open     Open files with the default application
  rename   Rename a file
  remove   Delete one or more files
  save     Read from standard input and save
  set      Set a file attribute
  trash    Move files or directories to the trash
  tree     Lists the contents of locations in a tree

Use “gio help COMMAND” to get detailed help.

Now back to playing

[wwatson]

@wwatson - Sorry I missed this update.

I will try to play soon, looks like fun

That's alright I know you guy's have been busy Been using my big fly swatter. Lots of bugs. I do have the ext4 formatter working now but it is SLOW!
There is a lot more going on when formatting to ext4. I thought it was slow on a Linux box It's not to bad formatting a SD card but a 128Gig USB drive took 4 hours!!! Something might not be right yet. Block sizes are 4096 bytes and when I monitor the sector count it is showing 8. So that's 32768 bytes.

Anyway I has been fun and a learning experience. Hope to update my GitHub repo soon. I haven't tested copying yet. Curious about the performance of cached reads and writes.

If you get time to test that would be great. I know it's in need of professional eyes

[KurtE]

That's alright I know you guy's have been busy Been using my big fly swatter. Lots of bugs. I do have the ext4 formatter working now but it is SLOW!
There is a lot more going on when formatting to ext4. I thought it was slow on a Linux box It's not to bad formatting a SD card but a 128Gig USB drive took 4 hours!!! Something might not be right yet. Block sizes are 4096 bytes and when I monitor the sector count it is showing 8. So that's 32768 bytes.

Anyway I has been fun and a learning experience. Hope to update my GitHub repo soon. I haven't tested copying yet. Curious about the performance of cached reads and writes.

If you get time to test that would be great. I know it's in need of professional eyes

Let us know when you have swatted enough bugs and pushed up your current stuff.

Interesting I found with Windows 11 and the wsl 2, that indirectly windows can now do some stuff on Linux file systems. That is for example my two wsl installs on the machine now have their filesystems show up in the File explorer. And there are ways to mount other Linux drives.