Teensyduino File System Integration, including MTP and MSC

Merged.

Is the current worst case freeClusterCount() ~1.5 second lag acceptable performance? Or do we need to also work on using the info sector?
 
Merged.

Is the current worst case freeClusterCount() ~1.5 second lag acceptable performance? Or do we need to also work on using the info sector?

I would say at this point, hopefully good enough. If we run into issues later could always revisit..
 
I started looking at alternatives to restarting SdFat when the card has been removed and then the same or a different card inserted. Sadly, it seems Bill just didn't design the library to begin again with the same config. The hardware init and card init are in the same functions. SDIO is better. SPI uses some of the config near the end to switch to the final speed.

So far I see 3 ways we could handle media change.

1: Add SD.begin(Config) and advise users not to use SD.sdfs.begin(Config), as we're currently documenting in the SdFat_Usage example.

2: Require SD.mediaPresent() to be called with the same input as SD.begin(), so it can restart SdFat when a new card is detected

3: Significant edits to SdFat to separate hardware init from card init, and add a restart API to re-detect and initialize the card.

Thoughts?
 
Ok found the issue with all the problems with the ext Card so it now works. Will push the change to SD in a few minutes.

In SD.h begin for csPin should be:
Code:
#ifdef BUILTIN_SDCARD
		if (csPin == BUILTIN_SDCARD) {
			bool ret = sdfs.begin(SdioConfig(FIFO_SDIO));
			cardPreviouslyPresent = ret;
			return ret;
		}
#endif
		if(csPin < 254) {
			bool ret = sdfs.begin(SdSpiConfig(csPin, SHARED_SPI, SD_SCK_MHZ(25)));
			cardPreviouslyPresent = ret;
			return ret;
		}

and not:
Code:
#ifdef BUILTIN_SDCARD
		if (csPin == BUILTIN_SDCARD) {
			bool ret = sdfs.begin(SdioConfig(FIFO_SDIO));
			cardPreviouslyPresent = ret;
			return ret;
		}
#endif
		bool ret = sdfs.begin(SdSpiConfig(csPin, SHARED_SPI, SD_SCK_MHZ(16)));
		cardPreviouslyPresent = ret;
		return ret;
 
I started looking at alternatives to restarting SdFat when the card has been removed and then the same or a different card inserted. Sadly, it seems Bill just didn't design the library to begin again with the same config. The hardware init and card init are in the same functions. SDIO is better. SPI uses some of the config near the end to switch to the final speed.

So far I see 3 ways we could handle media change.

1: Add SD.begin(Config) and advise users not to use SD.sdfs.begin(Config), as we're currently documenting in the SdFat_Usage example.

2: Require SD.mediaPresent() to be called with the same input as SD.begin(), so it can restart SdFat when a new card is detected

3: Significant edits to SdFat to separate hardware init from card init, and add a restart API to re-detect and initialize the card.

Thoughts?

Think it would really be between (2) and (3) at least in my mind. Guess from a easier perspective would be item (2) unless we can find a way to get at the SpiConfig settings Can is you define them instead of calling them direct. So guess I always opt for the easiest to implement. Thats my 2 cents at least.
 
@mjs513... @Paul ... Pushed up MTP changes for format.

Works better now with LittleFS and smaller SD drives... Still more testing on that.

Also wondering at the SD.format()...

Currently I experimenting with mine just being:
Code:
bool SDClass::format(int type, char progressChar, Print& pr)
{
	return sdfs.format(&pr);
}

It does not do a new begin on the device, but I think the fact it is reusing the fscache buffer as it's buffer is maybe removing some of the issue of file names bleeding through after format.

Does not do anything about what if the format changes the FS type...
 
I started looking at alternatives to restarting SdFat when the card has been removed and then the same or a different card inserted. Sadly, it seems Bill just didn't design the library to begin again with the same config. The hardware init and card init are in the same functions. SDIO is better. SPI uses some of the config near the end to switch to the final speed.

So far I see 3 ways we could handle media change.

1: Add SD.begin(Config) and advise users not to use SD.sdfs.begin(Config), as we're currently documenting in the SdFat_Usage example.

2: Require SD.mediaPresent() to be called with the same input as SD.begin(), so it can restart SdFat when a new card is detected

3: Significant edits to SdFat to separate hardware init from card init, and add a restart API to re-detect and initialize the card.

Thoughts?

4. As I mentioned in a different post, we could extend our begin like:
Code:
bool begin(uint8_t csPin = 10, uint32_t maxSpeed=SD_SCK_MHZ(16), uint8_t opt=0, SPIClass & spi) {
From this we can generate the SD.sdfs begin of the two different configs... And hold onto the information for later...
Would want to update the example sdfs sketch to use this format of begin.

5. Hack SDFat, that their begin methods, stash a copy of config data, and allow you to query it back...

Note: I like 3: but I can imagine a reasonable amount of work to get this correct.
 
@mjs513... @Paul ... Pushed up MTP changes for format.

Works better now with LittleFS and smaller SD drives... Still more testing on that.

Also wondering at the SD.format()...

Currently I experimenting with mine just being:
Code:
bool SDClass::format(int type, char progressChar, Print& pr)
{
	return sdfs.format(&pr);
}

It does not do a new begin on the device, but I think the fact it is reusing the fscache buffer as it's buffer is maybe removing some of the issue of file names bleeding through after format.

Does not do anything about what if the format changes the FS type...

Pulled in the changes and that seems to have got it working without the need for the devicereset with was bothering the heck out me. Testes on 32GB SD put 2gb NAND and flash on the memory board.

As mentioned, just tested on a T3.6 with just the Builtin SD card and the Ext card - format does work, here are the results so far.:
Code:
1. Can play wav files for SD1
2. Dates are correct for new directories
3. Dates are correct when I run datalogger option - does read and write
4. Formated SD1 32GB card.
5. FIRST PROBLEM _ can not copy files to sd card.

EDIT: Really have to solder a flash on to the Audio shield

UPDATE:
6. Delete files work
7 copy files does not work to sd cards or LFS Flash (NOR or NAND) as well. Moved the memory board
 
Last edited:
5. Hack SDFat, that their begin methods, stash a copy of config data, and allow you to query it back...

Note: I like 3: but I can imagine a reasonable amount of work to get this correct.

#5 sounds like a good option, as that will let us restart the library even if the user didn't call SD.begin(), but not require really deep changes inside SdFat.
 
Ok, here's option #5.

I added a restart() function to SdFat:

https://github.com/PaulStoffregen/SdFat/commit/f034ace5eca1e1afa17a7a85977de36b04e160fc

Then in the SD wrapper, I removed storage of the pin number and call restart() when we detect a new card or after format.

https://github.com/PaulStoffregen/SD/commit/dbc2eabefe2e71766abceda800f019f8f0adcfbe

I'm pretty sure this will solve the problem where we weren't able to work if the library was started by direct SD.sdfs.begin() rather than SD.begin().
 
Also wondering at the SD.format()...

Me too. Hoping you or Mike might look at this...

Is just calling sdfs.format(&pr) is enough? I'm happy to remove all that extra code if it's not needed. (I mostly based it from SdFormatter.ino...) If it's unnecessary, don't be shy to send a pull request to just delete it.

Do we need to call sdfs.restart() after formatting? Or maybe only sdfs.volumeBegin() is plenty?
 
Me too. Hoping you or Mike might look at this...

Is just calling sdfs.format(&pr) is enough? I'm happy to remove all that extra code if it's not needed. (I mostly based it from SdFormatter.ino...) If it's unnecessary, don't be shy to send a pull request to just delete it.

Do we need to call sdfs.restart() after formatting? Or maybe only sdfs.volumeBegin() is plenty?

I am not sure... As I mentioned there are a few cases where formtting with change from Fat to ExFat for ExFat to Fat...

Although not sure how much sleep to lose over some cases like this.

Will play.
 
Good Morning all (at least for a little more time):

I have been doing some testing on T3.6 with MTP code. Setup with T3.6 and Rev D Audio board (yes wrong one), but only using it for SPI and memory chip... And RAM drive... Which is probably the issue...
Funny how describing problem may lead to issue.

But problem is that:
Instrumented SendObject code:
Code:
bool MTPD::SendObject() {
  uint32_t len = ReadMTPHeader();
  printf("MTPD::SendObject %u\n", len);
  while (len) {
    receive_buffer();
    uint32_t to_copy = data_buffer_->len - data_buffer_->index;
    to_copy = min(to_copy, len);
    elapsedMicros emw = 0;
    bool write_status = storage_->write((char *)(data_buffer_->buf + data_buffer_->index),
                         to_copy);
    printf("    %u %u %u %u\n", len, to_copy, (uint32_t)emw, write_status);

    if (!write_status) return false;
    data_buffer_->index += to_copy;
    len -= to_copy;
    if (data_buffer_->index == data_buffer_->len) {
      usb_free(data_buffer_);
      data_buffer_ = NULL;
    }
  }
  // lets see if we should update the date and time stamps.
  storage_->updateDateTimeStamps(object_id_, dtCreated_, dtModified_);
  storage_->close();
  return true;
}

And debug output:
Code:
405392 CMD: 100d(SEND_OBJECT)l: 12 T:58
MTPD::SendObject 8732
    8732 52     808 64 70 1
    744 64 70 1
    680 64 70 1
    616 64 71 1
    552 64 80 1
    488 64 72 1
    424 64 70 1
    360 64 70 1
    296 64 70 1
    232 64 70 1
    168 64 70 1
    104 40 44 1
    64 16 19 1
    48 24 37 1
    24 24 28 1
1155532 RESP:2001(RSP:OK)l: 12 T:58
Did not make sense... How did going from 8732 and subtract 52, get me down to 744?

Will remove Ramdrive in this case and see if it is happier

EDIT: Edited and now was able to copy 8k file :D
Code:
   8732 52 87 1
    8680 64 70 1
    8616 64 70 1
    8552 64 71 1
    8488 64 75 1
    8424 64 82 1
    8360 64 72 1
    8296 64 82 1
    82    808 64 71 1
    744 64 74 1
    680 64 74 1
    616 64 70 1
    552 64 81 1
    488 64 70 1
    424 64 70 1
    360 64 71 1
    296 64 70 1
    232 64 70 1
    168 64 76 1
    104 64 70 1
    40 40 638 1
You will notice the writes of not sector size writes causes discrepancies in write timing...
Actually ARGH still bad. The first write appeared to again screw up the count. ...

EDIT2: Did not look like memory was issue in build:
Code:
"C:\\arduino-1.8.16\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-size" -A "C:\\Users\\kurte\\AppData\\Local\\Temp\\arduino_build_27941/simple_t4_audio_sd.ino.elf"
Sketch uses 119028 bytes (11%) of program storage space. Maximum is 1048576 bytes.
Global variables use 9100 bytes (3%) of dynamic memory, leaving 253044 bytes for local variables. Maximum is 262144 bytes.
C:\arduino-1.8.16\hardware\teensy/../tools/teensy_post_compile -file=simple_t4_audio_sd.ino -path=C:\Users\kurte\AppData\Local\Temp\arduino_build_27941 -tools=C:\arduino-1.8.16\hardware\teensy/../tools -board=TEENSY36 -reboot -port=usb:0/140000/0/1/1/4 -portlabel=hid#vid_16c0&pid_0478 Bootloader -portprotocol=Teensy
 
Last edited:
Actually I am pretty sure there is some form of corruption going on in the T3.6 when calling off to a file (SD/SDFat), where the write actually does a write.
Up till then I think it has a data buffer to copy the data into and then it does the actual sector write. Somewhere in that stage it appears like some memory may be changed...

I commented out the writes to the file and it went closer to processing all of the data.

May be issue still of the code not properly counting bytes to know how many of the length is for the file versus including the header...
Code:
21005 CMD: 100d(SEND_OBJECT)l: 12 T:57
MTPD::SendObject 8732
    8732 52 1 1
    8680 64 0 1
    8616 64 0 1
    8552 64 0 1
    8488 64 0 1
    8424 64 0 1
    8360 64 0 1
    8296 64 0 1
    8232 64 0 1
    8168 64 0 1
    8104 64 0 1
    8040 64 0 1
    7976 64 0 1
    7912 64 0 1
    7848 64 1 1
    7784 64 0 1
    7720 64 0 1
    7656 64 1 1
    7592 64 0 1
    7528 64 0 1
    7464 64 0 1
    7400 64 0 1
    7336 64 0 1
    7272 64 0 1
    7208 64 0 1
    7144 64 1 1
    7080 64 1 1
    7016 64 0 1
    6952 64 0 1
    6888 64 0 1
    6824 64 0 1
    6760 64 0 1
    6696 64 0 1
    6632 64 0 1
    6568 64 1 1
    6504 64 1 1
    6440 64 0     6312 64 0 1
    6248 64 1 1
    6184 64 1 1
    6120 64 0 1
    6056 64 0 1
    5992 64 0 1
    5928 64 1 1
    5864 64 1 1
    5800 64 0 1
    5736 64 0 1
    5672 64 0 1
    5608 64 0 1
    5544 64 0 1
    5480 64 0 1
    5416 64 0 1
   ...
    872 64 1 1
      744 64 0 1
    680 64 0 1
    616 64 1 1
    552 64 0 1
    488 64 0 1
    424 64 0 1
    360 64 1 1
    296 64 0 1
    232 64 0 1
    168 64 1 1
    104 40 1 1
    64 16 0 1
It sat in this state for a long time... And then error...
Guessing not taking header size into account or the like.

Will debug some more.
Then try to flash memory...
 
@Paul It would be interesting to see if you have any ideas on the T3.x MTP USB code.

As we mentioned when we do a send object to the save a file out... It appears like we may lose a packet or two. Also once I figure that part out, need to see why things may get corrupted when it tries to save buffer to SD (in my case internal).

Sorry I have not done much at this end of the USB stuff.

So I wonder things like on T3.x can RX and TX have same end points. I look at some of the other descriptors like Serial and it looks different for t3.x vs 4.x...

Also wondering on how or if it makes sense to have more than one transfer like object ready for RX in cases like this?

Sorry not sure if this makes sense or not?
 
Paul, back to MTP on T3.x and transfer of files.
For the heck of it I tried updating the descriptor to have TX and RX on different end points... Did not help...

Right now the code for T3.x for send object is really simple.
For the transfer there are two messages:
First: 1191302 CMD: 100c(SEND_OBJECT_INFO)l: 20 T:34 : 10001 ffffffff
Which describes the file including how big it is.
If we say that looks good, it then sends the file to us as one large message.
Obviously if the message > 64 bytes long on T3.6) this breaks into as many messages as it takes to send that many bytes.
So this end point can not send any additional messages until this whole thing is transferred.
In the last case:
I printed out:
Code:
1191321 CMD: 100d(SEND_OBJECT)l: 12 T:35
MTPD::SendObject 8732

Note: Some writes take a long while and if I try to print everything like each partial message I think I am losing a lot of the messages (SEREMU).


But on a run that I tried, for special file I created... Each line is 64 bytes long. Again the counts came out like it dropped one 64 byte message from host.

I added code to my copy of SendObject which does a timeout of 1/4 second instead of hanging for next message that won't come...

I sent the 32kb file again ended like it missed message...

I restated the sketch then renamed file and copied the file back and yes looks like it is missing a chunk.

screenshot.jpg

The only other difference was the ending of the file...
 
15 days after ordering I finally got my MM and ATP carrier board. (CO to WA). Many complications. Fed... Low and behold my existing USB C cable was charge only. Tonight after getting the proper cable I was able to program and use the MM. Looking forward to testing with it:)
 
15 days after ordering I finally got my MM and ATP carrier board. (CO to WA). Many complications. Fed... Low and behold my existing USB C cable was charge only. Tonight after getting the proper cable I was able to program and use the MM. Looking forward to testing with it:)

Have fun. It is great for USB Host with great 5V power.
 
Fixed the T3.x copy issue. Can post additional information if interested.

Just gave it a quick test by transferring a 75MB directory of wav files from the PC to both the external card reader (audio shield) and the Builtin card reader - both transferred successfully. Was able to play files from the cards without issue.

Yeah.......
 
15 days after ordering I finally got my MM and ATP carrier board. (CO to WA). Many complications. Fed... Low and behold my existing USB C cable was charge only. Tonight after getting the proper cable I was able to program and use the MM. Looking forward to testing with it:)

:D - Mine normally only takes a week to get here from CO :) Wonder why yours took so long. And when I purchased my first MMod found I did not have any of the right cables... Only had ones for IPAD, but they were C on both ends and my computer does not have any of those type USB connections...

Good luck. Should be fun!
 
Good Morning all including Paul, mjs513, wwatson ... : (maybe) Probably afternoon by the time I type this ;)
Sorry in advance it probably will be reasonably long.

Now that a lot of the FS only stuff. I was thinking about how best to integrate the MSC functionality into the system and how to then if desired to expose disks or volumes through MTP.

this is integrated and interpreting some previous comments, that were either on forum or email...

@Paul @wwatson @mjs513 - Hopefully I am not too far off and understanding and maybe direction to go.

Currently in the USBHost library there is a class: msController that is a "normal" USBHost object, that is setup to handle when a Storage device like hard disk, or Memory Stick is plugged in. So the normal USB enumeration stuff happens here and for example if your code is setup to include:
Code:
#include "USBHost_t36.h"

USBHost myusb;
USBHub hub1(myusb);
USBHub hub2(myusb);

msController msDrive1(myusb);
msController msDrive2(myusb);
When the first drive is plugged in, then likely the msDrive1 one will bool function will return true... And the code does all of the necessary stuff to talk to the drive at a low level (SCSI)...

Then there is the code in the library UsbMscFat (MSC), which deals with the actual storages on the disk, which I subdivide into two (maybe more) pieces

There is the PFsLib - portion, which is very much like the FsLib in SDFat. The main difference is that PFsLib works with Partitions, so for example the formatting code you pass in a partition number and it only formats that partition. Whereas SDFat format code will format the entire drive as one partition regardless of if the drive was previously setup with partitions.
Will be interesting to see where this code should reside.

Then the rest of the library has to do with then taking the one or more logical Drive objects and using them.
In the mscFS.h file There is the MSCClass which is derived from FS.

The begin method of the MSCClass you pass it in one of the msController objects as well as optionally which partition this object should work with... And from here you have the normal FS capabilities like file open...
So current MTP integration, we do something like define an array of 3 "Drive" objects and lets say 8 of the Volume objects and then we have code we run at loop time, to scan which drives have changed state (connected or removed) and check in the connected state, if this drive has partitions, then we iterate through calling the begin for the next free volume object... If drive went away we check for any volumes that were using that drive and close them...

Another option might be: Sorry if not completely fleshed out:
Suppose we have the sketch setup as:

Code:
USBHost myusb;
USBHub hub1(myusb);
USBHub hub2(myusb);

msController msDrive1(myusb);
msController msDrive2(myusb);

msVolume msVol1(myusb);
msVolume msVol2(myusb);
msVolume msVol3(myusb);
msVolume msVol4(myusb);
msVolume msVol5(myusb);
Each of these drives and volumes at creation time link themselves into lists...
Then when USBHost code detects the new drive, it works like current stuff and finds an unused msController object, and claims it like today. Then potentially this code could in theory, check for MBR and if it has one, it could iterate over the partitions, and if they are types we can handle and we have a free volume object, then volume object would be claimed by that controller object...
And in the end user code can detect if drives or volumes are there by boolean test...

Does this make sense? Should we instead of having to declare Volume object simply have them magically be created? Or should each Controller object have setup lets say 4 such objects...

Then there is the other half of the issue associated with MTP.
So MSC just found a new drive with two partitions. How does MTP get notified of this? Of course this is also similar to how should we handle SD drives which did not have a card in it and we now detected that now there is...

Should the user code call addStorage for all 5 of these volumes at setup time, and have them show up as not valid? Or should the appear when a drive is plugged in...

Lots more to this, but wondering what directions we should aim toward?
 
Back
Top