Teensyduino File System Integration, including MTP and MSC

That is what I am thinking about caching...

I noticed in the SDFat code there is stuff like:
Code:
class SdFat32 : public SdBase<FatVolume> {
 public:
  /** Format a SD card FAT32/FAT16.
   *
   * \param[in] pr Optional Print information.
   * \return true for success or false for failure.
   */
  bool format(print_t* pr = nullptr) {
    FatFormatter fmt;
    uint8_t* cache = cacheClear();
    if (!cache) {
      return false;
    }
    return fmt.format(card(), cache, pr);
  }
Which clears the cache...

Likewise some of their sketches clear the cache for other reasons:
Code:
  // Use SdFat's internal buffer.
  emptyStack[0] = (block_t*)sd.vol()->cacheClear();
  if (emptyStack[0] == 0) {
    error("cacheClear failed");
  }
But so far I see this in FatLib and ExFatLib but not in FsFatLib.

So far not found an easy way to update the format call to something like: SD.sdfs.vol()->cacheClear();
It will say that it is not a member of
So have not found a clean fix yet...
 
Last time I checked these functions were all private, so could not call them from external things like our callback.

Well I have spent the last few weeks working with this. Trying to tunnel into SdFat to get access to the private functions:
Code:
#if MAINTAIN_FREE_CLUSTER_COUNT
  int32_t  m_freeClusterCount;     // Count of free clusters in volume.
  void setFreeClusterCount(int32_t value) {
    m_freeClusterCount = value;
  }
  void updateFreeClusterCount(int32_t change) {
    if (m_freeClusterCount >= 0) {
      m_freeClusterCount += change;
    }
  }
#else  // MAINTAIN_FREE_CLUSTER_COUNT
  void setFreeClusterCount(int32_t value) {
    (void)value;
  }
  void updateFreeClusterCount(int32_t change) {
    (void)change;
  }
#endif  // MAINTAIN_FREE_CLUSTER_COUNT
As well as:
Code:
  int32_t  m_freeClusterCount;     // Count of free clusters in volume.
Not going to happen without moving these to public. The first thing that shoots it down is this:
Code:
  // Indicate unknown number of free clusters.
  setFreeClusterCount(-1);
This is located in FatPartition::init(). Every time the Teensy is powered or reset it will see this:
Code:
#if MAINTAIN_FREE_CLUSTER_COUNT
[COLOR="#FF0000"] if (m_freeClusterCount >= 0) {[/COLOR] 
    return m_freeClusterCount;
  }
#endif  // MAINTAIN_FREE_CLUSTER_COUNT
This forces a complete scan of free clusters. No way around it causing the up to 30 second or more delay.

As a test I moved the private code into public code for testing. Not the above code in the ::init function. I used Linux to format a Fat32 SDHC card and read in the FsInfo sector and reinterpreted sector buffer to the FsInfo struct. Saw this after copying arduino-1.8.16 to the the SD card:

Code:
Initializing External SD card...External SD card is present.
getLe32(pfsi->leadSignature) = 0x41615252
getLe32(pfsi->structSignature) = 0x61417272
getLe32(pfsi->trailSignature) = 0xaa550000
getLe32(pfsi->freeCount) = 0x1841950
getLe32(pfsi->nextFree) = 0x66175

Have a test sketch modified to test getting free cluster count. If I init the the SD card and do something like this:
Code:
sdVol.setFreeClusterCount(getLe32(pfsi->freeCount));
There is no delay as we have faked out SdFat but given the current free cluster count based on the last file/directory operations with Linux. If not done SdFat will will force a rescan of free clusters. Which happens for every power cycle or reset.
The nextFree term is a "HINT" as to the next free cluster if the device has been properly unmounted as well as freeCount. This gives way to an un-mount function which is not present in Sdfat. Let's say we open a file and write to it but do not close the file before we unplug the device. This would leave an invalid freeCount and nextFree value. Unmounting should sync and close the file as well as syncing and closing any other open files then update freeCount and nextFree. This is not done and should be done before hot plugging to update the FsInfo sector. After reading in the FsInfo sector during initialization the freeCount and nextFree terms should be initialized to -1 (0xFFFFFFFF) and written to the FsInfo sector. This would indicate that the device was not cleanly unmounted and would trigger a rescan of free clusters if not unmounted before unplugging the device. If cleanly unmounting before unplugging the device the -1 for freeCount and nextFree would be overwritten with the correct values. The current value of m_freeClusterCount could be written to freeCount and nextFree (which is not maintained in SdFat) could be found and updated with the FatPartition::allocateCluster() function in FatPartition.cpp.

This may be a lot of pointless rambling but to implement the FsInfo sector SdFat would have to be modified or as @KurtE and @mjs513 did with PFsLib, re-create the rest of SdFat our way. YIKES... Not going to happen:)

To get to the point I was able to eliminate the Fat32 freeCount delay with the first access to freeClusterCount(). That's as far as I went with it so far. I'm not really sure if it is still an issue?
 
Added a first attempt at SD.mediaPresent()

https://github.com/PaulStoffregen/SD/commit/049ada702a6aa41de2d6d20bc816c2518a976877



EDIT: simple demo program

Code:
#include <SD.h>
//const int chipSelect = BUILTIN_SDCARD;
const int chipSelect = 10;


void setup() {
  Serial.begin(9600);
  while (!Serial) ; // wait for serial port to connect.

  Serial.println("Initializing SD card...");
  SD.begin(chipSelect);
}

void loop() {
  bool p = SD.mediaPresent();
  Serial.print("SD card is ");
  Serial.println(p ? "inserted" : "missing");
  delay(1000);
}
 
Last edited:
@wwatson yep still an issue. will hopefully ply with this more soon!

Hopefully I did not confuse the issue. I am waiting for my first MicoMod ATP and Teensy boards. FedEx could not scan the shipping label and had to replace it so delivery is delayed:(
I see the progress with MTP and Linux. I want to join in with the testing. I think I know whose/what libraries I should pull in. But...
 
I think I know whose/what libraries I should pull in. But...

Yeah, a lot of files have changed. I was planning to package up 1.56-beta2 installers today, but then got into the USB serial thing again last night, and filling in the SD functions today.

I really should make a new set of installers soon so it's easy to get everything updated.
 
Yeah, a lot of files have changed. I was planning to package up 1.56-beta2 installers today, but then got into the USB serial thing again last night, and filling in the SD functions today.

I really should make a new set of installers soon so it's easy to get everything updated.

I saw that post. I can wait till things settle down. In the mean time will continue to with learning the ways of MicroMod... If and when it gets here:) Should be tomorrow.
 

Good morning all
Just updated SD library and tried the media present demo sketch.

Doesn't seem to work well with the Builtin sd card. Keep seeing alternating between missing and inserted:
Code:
Initializing SD card...
SD card is missing
SD card is inserted
SD card is missing
SD card is inserted
SD card is missing
SD card is inserted
SD card is missing
SD card is inserted
With that said it seems to work great on SPI (using the audio shield sd card). I did a slightly modified sketch to list files when card is inserted (just once so no spew):
Code:
#include <SD.h>
//const int chipSelect = BUILTIN_SDCARD;
const int chipSelect = 10;


void setup() {
  Serial.begin(9600);
  while (!Serial) ; // wait for serial port to connect.

  Serial.println("Initializing SD card...");
  SD.begin(chipSelect);
}

bool p_prev = false;
void loop() {
  bool p = SD.mediaPresent();
  Serial.print("SD card is ");
  Serial.println(p ? "inserted" : "missing");
  if(p == true && p_prev == false){
    File root = SD.open("/");
    printDirectory(root, 0);
    Serial.println("done!");
  }
  p_prev = p;
  delay(1000);
}

void printDirectory(File dir, int numSpaces) {
   while(true) {
     File entry = dir.openNextFile();
     if (! entry) {
       //Serial.println("** no more files **");
       break;
     }
     printSpaces(numSpaces);
     Serial.print(entry.name());
     if (entry.isDirectory()) {
       Serial.println("/");
       printDirectory(entry, numSpaces+2);
     } else {
       // files have sizes, directories do not
       unsigned int n = log10(entry.size());
       if (n > 10) n = 10;
       printSpaces(50 - numSpaces - strlen(entry.name()) - n);
       Serial.print("  ");
       Serial.print(entry.size(), DEC);
       DateTimeFields datetime;
       if (entry.getModifyTime(datetime)) {
         printSpaces(4);
         printTime(datetime);
       }
       Serial.println();
     }
     entry.close();
   }
}

void printSpaces(int num) {
  for (int i=0; i < num; i++) {
    Serial.print(" ");
  }
}

void printTime(const DateTimeFields tm) {
  const char *months[12] = {
    "January","February","March","April","May","June",
    "July","August","September","October","November","December"
  };
  if (tm.hour < 10) Serial.print('0');
  Serial.print(tm.hour);
  Serial.print(':');
  if (tm.min < 10) Serial.print('0');
  Serial.print(tm.min);
  Serial.print("  ");
  Serial.print(months[tm.mon]);
  Serial.print(" ");
  Serial.print(tm.mday);
  Serial.print(", ");
  Serial.print(tm.year + 1900);
}
and starting with the card inserted - pulling it out - re-inserting on the audio shield.
Code:
Initializing SD card...
SD card is inserted
System Volume Information/
  WPSettings.dat                                   12    13:40  October 5, 2021
  IndexerVolumeGuid                                76    13:40  October 5, 2021
done!
SD card is inserted
SD card is inserted
SD card is inserted
SD card is inserted
SD card is inserted
SD card is inserted
SD card is inserted
SD card is missing
SD card is missing
SD card is missing
SD card is missing
SD card is missing
SD card is inserted
System Volume Information/
  WPSettings.dat                                   12    13:40  October 5, 2021
  IndexerVolumeGuid                                76    13:40  October 5, 2021
done!
SD card is inserted
SD card is inserted
 
Continuation of demo results for T4.1

Ran the original demo sketch first for the Builtin_SDCard and on the T4,1 it worked - no alternating missing/inserted. However, if I use my modified sketch to list the files on card inserted it crashes the sketch:
Code:
D:\Users\Merli\Documents\Arduino\SD_MediaPresent\SD_MediaPresent.ino Oct  6 2021 06:11:42
Initializing SD card...
SD card is missing
SD card is missing
SD card is missing
SD card is missing
SD card is missing
SD card is missing
SD card is inserted
mtpindex.dat    <---- CRASH HERE

-----------------------------------------
CrashReport:
  A problem occurred at (system time) 6:12:17
  Code was executing from address 0xA16
  CFSR: 8200
	(PRECISERR) Data bus error(address in BFAR)
	(BFARVALID) Accessed Address: 0x2007029C
  Temperature inside the chip was 45.00 °C
  Startup CPU clock speed is 600MHz
  Reboot was caused by auto reboot after fault or bad interrupt detected

Using the SD Card on pin 10 (audio shield) no problems are shown - runs the same way as described on the T3.6.
 
Doesn't seem to work well with the Builtin sd card. Keep seeing alternating between missing and inserted:

Oh, that's not good.

Bill's code returns 0 for the case of error reading status, rather than an actual true/false whether the card responded, like he does for all the others.

I was a bit worried some cards might not follow the spec with their 32 bit status register, which seems to say 3 of the bits encode the card state that should be non-zero after the card is initialized. Seems we can't depend on zero status as an indication of hardware removal. :(
 
Oh, that's not good.

Bill's code returns 0 for the case of error reading status, rather than an actual true/false whether the card responded, like he does for all the others.

I was a bit worried some cards might not follow the spec with their 32 bit status register, which seems to say 3 of the bits encode the card state that should be non-zero after the card is initialized. Seems we can't depend on zero status as an indication of hardware removal. :(

Actually based on your comment thought about something - wondered if the SD Card I had in the Builtin card reader may have gotten corrupted some how so I reformated the SD Card and reinserted into the slot and guess what it worked fine to detect the card and list the files.

Inserted 3 other sd cards 1 512GB ExFat, and 2 Fat32 32GB cards an it works so you can :) again

But probably need something to test the error code in case the card is bad - so guess wasn't a total loss to test.
 
Good Morning (at least for me 5am+) everyone...

Looks like lots of things happen during that time :D
Added a first attempt at SD.mediaPresent()
Will play with that some this morning.

This may not be the most elegant solution, but it works.
Maybe in the future we'll find a more efficient way to just reinitialize the filesystem part of SdFat?

Maybe, although in some case it probably needs to do this. For example suppose that real long Fat32 time to get free space I say heck with it and take the SD to Windows and format it as exfat. Which has the benefit that it gives you free space nice and quick. Bad side is allocations are much larger... So I take it back to using with SDFat... Then format it, either through SD or directly through SDFat, and its code will format it back to Fat32. And I don't see anything in SDFat (FSLib stuff), that FSVolume/partition code will validate that the handles that it is holding onto are still the right ones...

The current hack is probably close to what is needed and worked in my test case :D But probably need to see if we can detect if the SD object was started earlier by one of the alternative begins like:
Code:
SD.sdfs.begin(SdSpiConfig(chipSelect, SHARED_SPI, SD_SCK_MHZ(24)));

Yeah, a lot of files have changed. I was planning to package up 1.56-beta2 installers today, but then got into the USB serial thing again last night, and filling in the SD functions today.

I really should make a new set of installers soon so it's easy to get everything updated.

For playing with this stuff, I believe the current stuff, to be reasonably up to date with changes for testing this stuff include:

Libraries:
SD - Paul's Juse_Use_SdFat branch (always wondered if it was supposed to be: Just_Use_SdFat :D
SDFat - Paul's master branch
LittleFS - Paul's main branch

Libraries not in builds:
MTP_Teensy - My main branch
UsbMscFat - Mike's FS_Integration branch - Note I have not tried this in a bit...

Cores - I am not sure how many changes went in since Beta 1
For sure: FS.h files changes (all three cores)
usb_desc.c in T3 and T4 for MTP descriptor to work better with Linux
usb_mtp.c (and .h) for T4 for events.

Maybe missing a few things.

Also not specific to this, but I know I have installed things like Serial monitor stuff in the tools directory...

So yes we can do it by manually updating

Just ask Bill instead experimenting and guessing?

I will probably issue an issue with SDFat project on this. As I believe the main library code has similar issue. I may or may not update my test code go directly to SDFat instead of starting from SD.
The simple issues are: From an FSVolume there is no way to tell the underlying FatVolume or ExFatVolume to invalidate it's internal state. And our code is calling his Format functions, where you pass in the BlockDevice and not the volume, so this code can not in itself invalidate the state of the actual volume objects...

And there is also the issue that I mentioned about kludge or not, about updating the FSVolume information. That is most of its methods are setup like:
Code:
  bool ls(print_t* pr) {
    return m_fVol ? m_fVol->ls(pr) :
           m_xVol ? m_xVol->ls(pr) : false;
  }
Where before the format is done, m_xVol had pointer to the underlying exVolume object which was the right one, but after if it changed volume types it should then be m_fVol which is valid... An in these cases, my guess is the change of which underlying object is needed would probably facilitate the need to run through the at least logical begin code...

Actually based on your comment thought about something - wondered if the SD Card I had in the Builtin card reader may have gotten corrupted some how so I reformated the SD Card and reinserted into the slot and guess what it worked fine to detect the card and list the files.

Inserted 3 other sd cards 1 512GB ExFat, and 2 Fat32 32GB cards an it works so you can :) again

But probably need something to test the error code in case the card is bad - so guess wasn't a total loss to test.

Glad it is working now. Will try it out soon.

Just finished verifying that formatting of my flash through the MTP_Teensy Simple sketch actually no longer shows the cached list of files.... :D

Now back to more coffee and playing!
 
Glad it is working now. Will try it out soon.

Just finished verifying that formatting of my flash through the MTP_Teensy Simple sketch actually no longer shows the cached list of files....

Now back to more coffee and playing!
Me too. Didn't realize there was an issue with SD card until Paul mentioned that
Bill's code returns 0 for the case of error reading status, rather than an actual true/false whether the card responded, like he does for all the others.

Thats when I decided to reformat the SD Card. But that brings up a point - normally when there is an issue with the SD Card it just doesn't start and gives an error.

Yes checked the MTP_Teensy_Simple as well and seemed to format but have a strange issue using MTP_test started explaining it earlier but think I am isolating it down. Normally we have if using the SD Cards:
EDIT: This is on the T4.1.

Code:
 #if USE_SD==1
    #if defined SD_SCK
      SPI.setMOSI(SD_MOSI);
      SPI.setMISO(SD_MISO);
      SPI.setSCK(SD_SCK);
    #endif

    for(int ii=0; ii<nsd; ii++)
    { 
      #if defined(BUILTIN_SDCARD)
        if(cs[ii] == BUILTIN_SDCARD)
        {
          if(!sdx[ii].begin(cs[ii]))
          { Serial.printf("SDIO Storage %d %d %s failed or missing",ii,cs[ii],sd_str[ii]);  Serial.println();
          }
          else
          {
            storage.addFilesystem(sdx[ii], sd_str[ii]);
            uint64_t totalSize = sdx[ii].totalSize();
            uint64_t usedSize  = sdx[ii].usedSize();
            Serial.printf("SDIO Storage %d %d %s ",ii,cs[ii],sd_str[ii]); 
            Serial.print(totalSize); Serial.print(" "); Serial.println(usedSize);
          }
        }
        else if(cs[ii]<BUILTIN_SDCARD)
      #endif
      {
        pinMode(cs[ii],OUTPUT); digitalWriteFast(cs[ii],HIGH);
        if(!sdx[ii].begin(cs[ii])) 
        { Serial.printf("SD Storage %d %d %s failed or missing",ii,cs[ii],sd_str[ii]);  Serial.println();
        }
        else
        {
          storage.addFilesystem(sdx[ii], sd_str[ii]);
          uint64_t totalSize = sdx[ii].totalSize();
          uint64_t usedSize  = sdx[ii].usedSize();
          Serial.printf("SD Storage %d %d %s ",ii,cs[ii],sd_str[ii]); 
          Serial.print(totalSize); Serial.print(" "); Serial.println(usedSize);
        }
      }
    }
    #endif
where sdx[ii] is defined as:
Code:
SDClass sdx[nsd];

It seems now that for the Builtin SDCard it wants:
Code:
      #if defined(BUILTIN_SDCARD)
        if(cs[ii] == BUILTIN_SDCARD)
        {
          if(!SD.begin(cs[ii]))
          { Serial.printf("SDIO Storage %d %d %s failed or missing",ii,cs[ii],sd_str[ii]);  Serial.println();
          }
          else
          {
            storage.addFilesystem(SD, sd_str[ii]);
            uint64_t totalSize = sdx[ii].totalSize();
            uint64_t usedSize  = sdx[ii].usedSize();
            Serial.printf("SDIO Storage %d %d %s ",ii,cs[ii],sd_str[ii]); 
            Serial.print(totalSize); Serial.print(" "); Serial.println(usedSize);
          }
        }
where I have to specifically use SD instead of sdx[ii] if I use builtin and a external card. In this case I can tell it to format and copy files no issues

Next scenario - is I only specify using an external card I can still copy files but it will fail formating saying it can not format.

So not sure what the real problem this is pointing too. Think going to modify the Simple sketch to use 2 sd cards and see what happens

EDIT: Seeing the same issue using MTP_Simple sketch as well modified as:
Code:
// Add the SD Card
  if (sd0.begin(SD_ChipSelect)) {
    storage.addFilesystem(sd0, "SD_Card");
    DBGSerial.println("SD Card initialized");
  }

  // Add the SD Card
  if (sd1.begin(extSD_ChipSelect)) {
    storage.addFilesystem(sd1, "extSD_Card");
    DBGSerial.println("SD Card initialized");
  }

where:
SDClass sd0;
SDClass sd1;
if I change the SD_Card to use SD and leave the second alone it works as it should. Again this is on a T4.1
 
Last edited:
Paul and Mike ... Just pushed up PR to SD library format code.

Problem was the code for format started of by using predefined SD object instead of the instance that was passed in...

Still need to work on MTP code, With the FS only changes and removal of the Callback class, I sort of neutered the Format code that if we think the format may take awhile, then I go ahead and send back to MTP a status saying OK everything is fine..... And then do the format, and have IntervalTimer running responding to any MTP message with Sorry we are busy.

I will probably just always do that for now. Yes it will give the user the false impression that the format completed early, but then better than totally hung out to dry.

Update: I am going to try a combination way for this. That is to always start up the Interval timer, then start the format, then the IT code will BUSY out most messages, and if a certain time elapses for the format, like lets say 5 seconds, then it will respond to the host, that format is complete... It will continue to BUSY out all other messages until the format completes. If format completes before timeout will cancel IT and send the normal ACK response...
 
Last edited:
Actually based on your comment thought about something - wondered if the SD Card I had in the Builtin card reader may have gotten corrupted some how so I reformated the SD Card and reinserted into the slot and guess what it worked fine to detect the card and list the files.

Inserted 3 other sd cards 1 512GB ExFat, and 2 Fat32 32GB cards an it works so you can :) again

But probably need something to test the error code in case the card is bad - so guess wasn't a total loss to test.

I tested with 18 different SD cards this morning. I couldn't find any which return status 0 when they had been detected in the SDIO socket. I'm hoping the current mediaPresent() will be good enough to move forward for now?

Turns out most of my very old cards don't work in the SDIO socket on Teensy 4.1, but they do work over SPI on the audio shield. Some them (all full size cards I'm using with an adaptor) cause Teensy 4.1 to reboot on insertion and removal.

sdcards.jpg
 
I tested with 18 different SD cards this morning. I couldn't find any which return status 0 when they had been detected in the SDIO socket. I'm hoping the current mediaPresent() will be good enough to move forward for now?

Turns out most of my very old cards don't work in the SDIO socket on Teensy 4.1, but they do work over SPI on the audio shield. Some them (all full size cards I'm using with an adaptor) cause Teensy 4.1 to reboot on insertion and removal.

Sorry - had to close my eyes for a little bit - up too early. The micro SD I was using has been abused with all the testing we did. Once reformated the few SD cards I had all seemed to work without an issue. Don't even know where my old full size cards are but looks like I am going to be ordering one for the heck of it.

Unless Kurt has found any issues I would say that current mediaPresent() is working.
 
Much has happened recently, so where are we at and what to do next?

I'm guessing we need some sort of timer-based function to call mediaPresent() on all the connected filesystems and send events and do whatever else is needed to manage media. I'm hoping we can use MillisTimer rather than risking running stuff from interrupts. Some parts of the core library might need more yield() calls added....

So far I've been focusing on SD and LittleFS but haven't spent much time with UsbMscFat. It probably needs work to implement the new FS format() and mediaPresent() functions.

For the USB host connected storage, rather than use the callback interface, I want to eventually see a model similar to how we do generic HID, where the user creates 1 or more controller instances which are similar HIDParser, and the user also creates several filesystem instances, which are similar to JoystickController, DigitizerController, RawHIDController, etc. When a USB mass storage device is enumerated by ehci, one of the controller instances claims it, and then it for each partition which is found on the disk, those filesystem instances claim the partitions. Each filesystem instance gets polled by FS mediaPresent(), so MTP can handle whatever combination of partitions are found as FS interface.

MPT_Teensy still has a lot of media-specific code which (hopefully) isn't used anymore. Any thoughts on when we should delete it? Or maybe make a branch so it's still around for easy viewing, but delete it from main? Perhaps after UsbMscFat is accessed only by FS?

Eventually we need to generate events to the MTP initiator on the PC when code running on Teensy writes or changes files and folders. I still don't have a perfectly clear idea on this, but I'm imagining extending File and FileImpl, so MTP can get callbacks as important operations happen. This won't be a generic API - it'll be special for MTP.

Even farther out is some sort of API for code on Teensy to be notified of when the MTP initiator changes file. No idea what that will look like yet...

Maybe other stuff???
 
Oh... just remembered, we have the matter of PFsLib, and the changes Bill wasn't willing to merge into SdFat.

Kurt, can you remind me about those changes again? Or point me to the old messages? If possible, I'd like to just merge that stuff into Teensy's fork of SdFat. So far Teensy's copy in 1.56-beta2 is only config changes and minor edits to examples. But if we're going to have a fork of SdFat, let's try to have it all in 1 copy rather than duplicating lots of code just for USB storage.
 
Right now having issues with format timing out in MTP which Kurt alluded to in the Beta2 thread. As a example, from my tests with a T4.1:
1. 32GB FAT SDIO and 32GB FAT in Audio. The SDIO format seems to work but will fail on the Audio card - timeout and wants to format it twice. Format on the Audio shield seems to take forever to be honest....
Capture.PNG
and its still formatting - waiting long enough
Code:
33921 RESP:2001(RSP:OK)l: 16 T:31 : 10001
41797 CMD: 100f(FORMAT_STORE)l: 20 T:32 : 20001 0
 MTPD::formatStore called post:0
Writing FAT ................................
Format Done
113633 RESP:2001(RSP:OK)l: 20 T:32 : 20001 0
 MTPD::formatStore called post:1
*** Start Interval Timer ***
Writing FAT ................................
Format Done
*** end Interval Timer ***
At this point you will see it does the format but then you can't copy files and if you do a 'r'estart you loose the drives in explorer

2. A 512 GB fails on SDIO (formatted as exFAT) and same for the 32GB on audio shield

Once they fail I tend to loose all the drives - but format does seem to work after I power cycle.

Was waiting on Kurt to finish his magic before coming to a conclusion.

So far I've been focusing on SD and LittleFS but haven't spent much time with UsbMscFat. It probably needs work to implement the new FS format() and mediaPresent() functions.
Current version of format and mediaPresent - don't think will work for MSC drives. Maybe for the drive present but also needs to know the partitions which we don't deal with for the SD Cards. Similar but different structure. Maybe Kurt can explain better.

For the USB host connected storage, rather than use the callback interface,
Yep remember you mentioning that one :)

MPT_Teensy still has a lot of media-specific code which (hopefully) isn't used anymore. Any thoughts on when we should delete it? Or maybe make a branch so it's still around for easy viewing, but delete it from main? Perhaps after UsbMscFat is accessed only by FS?
Think alot can be deleted - mtpstorage_SD especially. Don't remember if we use anything from CB have to double check.

Ok confusing myself now and getting late here

EDIT: forgot to mention PFsLib as well - take it for granted at this point
 
Sorry, my brain is sort of toast for today... So will try to give some more complete answers on some of this stuff tomorrow.

Format timing... I was playing around with some of the support for this today. I was trying to get to the point where in most cases the Forrmat starts, and we wait until completion and then we send the response to the host saying yep we are done... On most drives this is working OK.

Issue however is if it takes greater than time X, MTP will fail and nothing else works... So code is trying by starting up an Interval Timer, than start the format.
Then the IT checks to see if any pending messages are received, it will then try to respond with I am busy... Mentioned this before.

Addition however is that it then checks to see if Time X has occurred since I started the format, in which case it tries to send the response of I am done (More specific MTP_RESPONSE_OK)
That I tried to get to post from the IT... I am hoping that I can get it to send this, at which time the Host is free to send new messages, which I again will try to say I am busy.

Problem is it does not appear to be working fully. I do get the timeout, and I do look like I send a response back to host, that appears like the normal response like if the format had ended on time...

But in this case the host does end the Format busy dialog and say that the format failed... So something wrong... And even after the format completes which may be awhile, nothing works....

Note: put the MTP_Teensy changes in a WIP branch... In case anyone wishes to look before I get back to it in the morning.

One of the main things with MTP is everything is transactional. That is if for example if the user drops a 3mb image onto the storage, it will send a SendObjectInfo message, then followed by the whole set of bits... So there is no chance of any additional status or messages from the host until the whole 3MB has been sent and we have replied to the message.

Side note: I am thinking I may need to try an IntervalTImer approach to the SendObject as to hopefully allow us to pace the reading in of data to keep the host happy, especially when we do writes to some LittleFS storages which can take a long long time.

USB stuff more tomorrow.

PFSLib stuff versus FSLib stuff - Will need to refresh on all of it. But If I remember correctly for example with your current SD.format() code, it is using the SDFat FSLib (actually then to Fatlib and ExFatLib) format functions.

Will suppose you pop in a nice shiny SDCard setup to use with an RPI. There are multiple partitions on this, including a small Fat partition and then most a Linux Format. With SDFat code, if I remember correctly and my few minutes of checking out the SD library usage, I believe that we simply pass in the BlockDevice and this format code will wipe the entire SDCard clean, ie. it cares less about what is on the MBR...

I believe the code in PFSLib is setup to be able to understand the MBR record and only format within an actual Partition. This is obviously needed for MSC like drives, but maybe needed for SDcards for the reasons I just mentioned. Sort of the difference on windows from using Windows Format a drive which only formats that specific partition versus SDFormater app, which blows away everything... Both have their purposes.

Other things we have played with and would really like to address is Fat32 used/Free space. Several postings about this by Mike, and wwatson and myself on this.
What I would like to see is some ability to get this number without taking seconds...

Couple of different partial approaches:
a) Read and update the Information sector: Which has a guess of how much space is free on disk... (https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system#FS_Information_Sector)
This is completely ignored by Fatlib, except to write out the default one when formatted. Would be good if we read this and at different points updated it, with at times maybe doing some version of background read of fat and generate...

b) I get the impression that ESP32 has an option (default turned off I think), that says as an estimate return back simply the last cluster allocated as an estimate. It will be off as probably some earlier custers were freed, but maybe good enough up till a point.

c) When we do want a better update would be nice if we could somehow read each cluster in the background when the SD was not busy, and generate the count. Probably some issues with suppose we have read half it in and then sketch decides to allocate 50 clusters. What now? do we know or want to know if the new allocations were below or above how far we processed? And how accurate in most cases do we need. That is if my 32gb card is 95% empty do I care if actually it is 96%...

I know there is probably other stuff but ...
 
Back
Top