Many TLAs: MTP MSC FS SD SDFat LittleFS UsbMSCFat to work with each other 8)

I used the original repository's default branch. Haven't looked at any other branches (yet). I tested against all the example programs on the main branch of the original UsbMscFat.

The 1 and only branch I've created was meant to preserve a copy before I deleted lots of stuff.

Looks like I missed a bunch of stuff on branches. Don't panic. It can all get merged, though I do want to carefully consider how to do so. Even if it takes a while longer to restructure somewhat, I want to limit how far our SdFat fork diverges from Bill's original code.

Another question is how much of this stuff really must get merged before we start 1.57-beta1? Some major improvements have happened since 1.56, like Wire library slave mode, Audio ADC usable performance, and of course the SdFat 2.1.0 -> 2.1.2 to fix large file append. More can always be merged for 1.57-beta2. Getting this to a usable point is currently the thing holding up beginning 1.57 beta testing.
 
Another question is how much of this stuff really must get merged before we start 1.57-beta1? Some major improvements have happened since 1.56, like Wire library slave mode, Audio ADC usable performance, and of course the SdFat 2.1.0 -> 2.1.2 to fix large file append. More can always be merged for 1.57-beta2. Getting this to a usable point is currently the thing holding up beginning 1.57 beta testing.

Obviously it is your decision on when you wish to release a new beta. If There is lots of stuff that others are needing, one option would be to simply release the beta without any of the MSC stuff.

Looks like I missed a bunch of stuff on branches. Don't panic. It can all get merged, though I do want to carefully consider how to do so. Even if it takes a while longer to restructure somewhat, I want to limit how far our SdFat fork diverges from Bill's original code.

Understood. I am not panicking, but for now I will hold off and work on some different stuff.
 
Another question is how much of this stuff really must get merged before we start 1.57-beta1? Some major improvements have happened since 1.56, like Wire library slave mode, Audio ADC usable performance, and of course the SdFat 2.1.0 -> 2.1.2 to fix large file append. More can always be merged for 1.57-beta2. Getting this to a usable point is currently the thing holding up beginning 1.57 beta testing.

As to the question of releasing 1.57 Beta-1. As Kurt said the final decision is yours but right now you probably have a base for MSC that you could go ahead and release but am concerned that some of the MTP_Teensy (our version) and MTP_T4 will probably be broken or limited in use. Haven't really checked yet but just a gut feeling.

EDIT: On a second thought maybe should leave the MSC stuff out for Beta-1. Without all the other pieces may cause more problems than its worth.

Looks like I missed a bunch of stuff on branches. Don't panic. It can all get merged, though I do want to carefully consider how to do so. Even if it takes a while longer to restructure somewhat, I want to limit how far our SdFat fork diverges from Bill's original code.

Again as Kurt said not panicking but kind of stuck for bit with using MTP which is really where MSC was coming in handy and we do use.
 
Last edited:
It's kinda late to turn back now, kind of like swimming half way across a river.

I'll plan on waiting a few more days to package up 1.57-beta1.
 
Ok, after much thought and research and experimenting yesterday, I decided to modify SdFat FsVolume begin() so we can explicitly tell SdFat the location of the partition to use.

https://github.com/PaulStoffregen/SdFat/commit/1dd7b5436e1589caa8f51f47ecc8a14c5a683e87

The idea is instead of telling FsVolume begin() which partition, we tell it the starting sector and number of sectors to use. So far I've only done FsVolume, but I'm planning to do the same for format.

This gives USBHost_t36 the responsibility to determine where the partitions are located. For now, I committed a very simple MBR-only function.

https://github.com/PaulStoffregen/USBHost_t36/commit/246a1a63d7a23fdd52d8dab7092d573eee9b3c31

Of course, the plan is to bring all the GPT stuff into these functions in USBHost_t36. At this moment all of my USB drives have only MBR with only 1 partition. Later today I'll set up a couple drives with complex partitioning.

While this is a structural change from how everything has been developed so far inside PFsLib, the idea is to leave SdFat basically as Bill intended. It should (hopefully) result in a "cleaner" patch where we add alternate begin() & init() functions, and except for the 2TB capacity limit, we shouldn't need to keep changing stuff inside SdFat.
 
A lot of work Paul but sounds like you got the final structure you want to use which is good. Let us know when you want us to test.

Of course, the plan is to bring all the GPT stuff into these functions in USBHost_t36. At this moment all of my USB drives have only MBR with only 1 partition. Later today I'll set up a couple drives with complex partitioning.
You mean like this:
IMG-0637.jpg
 
Quick update...

Things are coming along :D Probably a few more pieces to be put into place, which I am guessing as it has been progressing that Paul has been filtering in the changes.


Right not playing around with Test sketch where I plug in a CircuitPython board into the USBHost board that has MTP setup to forward the MSC stuff, plus stuff in place to also talk to the USBSerial object.

I am mostly now testing using the not official MTP+Serial as a I am also glutton for punishment and using IDE2 some of the time and currently still does not support SEREMU.

For anyone else who might try this on Linux... One thing I ran into was MTP would work fine with MTP_DISK but not MTP_SERIAL... Ubuntu would not open up MTP window.
Yet the MTP library code said it was a valid MTP drive...

Long Story Short:
Found out, that for whatever reason the udev rules (00-teensy.rules) was setup to stop it from working...
In particular this line: ATTRS{idVendor}=="16c0", ATTRS{idProduct}=="04*", ENV{ID_MM_DEVICE_IGNORE}="1", ENV{ID_MM_PORT_IGNORE}="1"

After I modified that line, like magic it worked.

Now back to figuring out why the USBHOst code and MPy code is not getting all of the Python Serial output... Not that the Serial code is losing it... We are not getting any USB traffic for these messages.
Yet others come through fine... Not sure which side it is...
 
Ditto here on playing with the latest changes from this morning.

Mainly been playing with the MTP Integrity sketch (kitchen sink version) and it seems to be working with some caveats. As Kurt said been using the unoffocial MTP+Serial (hopefully eventually official).

The only modification made to the sketch, besides the obvious change to class names was to use Kurt's updated checkMSCChanges() function which provides a hack for now of only setting up the first volume for each USB drive. Reason: the bool operator on the USBFileSystem object has not been added yet nor way to delete/release them so this is just a preliminary test of MTP + USB changes. Yes I can add and delete files - formatting test is tbd as we know so didnt try....

Code:
@@@@@@@@@@@@@@@ NEW Drives @@@@@@@@@@@

##USB Drive: 0 connected

Partition Table
	part,boot,bgnCHS[3],type,endCHS[3],start,length
*** GPT Disk WIP ***
GPT guard:	1,0,0x0,0x2,0x0,0xEE,0xFE,0xBF,0xDB,1,4294967295
pt_#0:	2,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
pt_#0:	3,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
pt_#0:	4,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
	 < unused area starting at: 0 length 242075647 >

GPT partition header revision: 10000
LBAs current:1 backup:242075647 first:34 last:242075614
Disk GUID:494E6C4B-0928-4119-6EED-8BD052C8A7BAStart LBA Array: 2 Count: 128 size:128
Part	 Type Guid, Unique Guid, First, last, attr, name
0	EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, 466E5E58-6453-84BB-6CDD-3B813E5167A3, 2048, 60520447, 0, 
>>> Microsoft Basic Data Partition
1	EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, 08F18C4C-41DD-48E2-2784-EFD893305C7C, 60520448, 242073599, 0, 
>>> Microsoft Basic Data Partition
    EXFAT:
3	00000000-0000-0000-0000-000000000000, 00000000-0000-0000-0000-000000000000, 0, 0, 0, 
Found new Volume:0

@@@@@@@@@@@@@@@ NEW Drives @@@@@@@@@@@

##USB Drive: 1 connected

Partition Table
	part,boot,bgnCHS[3],type,endCHS[3],start,length
FAT32:	1,0,0x20,0x21,0x0,0xC,0xFE,0xFF,0xFF,2048,47298560
Extend:	2,0,0xFE,0xFF,0xFF,0xF,0xFE,0xFF,0xFF,47300608,73844736
exFAT:	2:1,0,0x20,0x21,0x0,0x7,0xFE,0xFF,0xFF,47300672(64),73844672 (0)
pt_#0:	3,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
pt_#0:	4,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
Found new Volume:1

Dump Storage list(7)
.....
store:5 storage:60001 name:USB0-GPTxFat fs:20011b0c pn:
store:6 storage:70001 name:USB1-FAT32-P fs:20011fa0 pn:

I also attached a SSD drive and that worked as well.
 
I'm leaning towards packaging up 1.57-beta1 this afternoon. We're far from finished for a final 1.57 release, but I hope we're far enough along to make this and so many other non-filesystem improvements available to everyone willing to run a beta installer (but probably not cobble together many github downloads not knowing which versions & branches are meant to be dependencies).

Any change to the USB Type list involve a bunch of thorny issues, especially with Linux and udev rules. I do intend to add a MTP+Serial before final 1.57 release. But I'm not eager to do all that testing right now, holding up a 1.57-beta1 installer.

The main point a 1.57-beta1 is to give people who want to join in testing & using all this new stuff an accessible way to get it installed with all dependencies correct.
 
I'm leaning towards packaging up 1.57-beta1 this afternoon. We're far from finished for a final 1.57 release, but I hope we're far enough along to make this and so many other non-filesystem improvements available to everyone willing to run a beta installer (but probably not cobble together many github downloads not knowing which versions & branches are meant to be dependencies).

Any change to the USB Type list involve a bunch of thorny issues, especially with Linux and udev rules. I do intend to add a MTP+Serial before final 1.57 release. But I'm not eager to do all that testing right now, holding up a 1.57-beta1 installer.

The main point a 1.57-beta1 is to give people who want to join in testing & using all this new stuff an accessible way to get it installed with all dependencies correct.

MTP+Serial: Understood and understandable. Will be nice when 1.57 is released and to be honest didn't think it would be in for beta1 testing.

Guess for initial beta1 release its a great start. A lot of work put in to getting it all incoporated. Think for the beta2 release probably should try and get in formatting stuff and get the partition enumeration (did I say that right). Maybe good idea to add a note on limitation/capability for the beta release so people know what to expect. Without those 2 things may break some things people have already implemented in MTP if they are using partitioned drives or with USBMscFat.
 
Last edited:
One issue with MTP+Serial is that it uses for the moment the PID of Everything
Even if MTP+Serual is not yet ready to be included for full, I continue to suggest that Paul defines a dedicated PID, so we can run the test with out own implementations.
I ONLY use MTP+Serial, but I'm on windows using Putty to communicate with Teensy and to log the Serial.print to disk.
 
I'm leaning towards packaging up 1.57-beta1 this afternoon. We're far from finished for a final 1.57 release, but I hope we're far enough along to make this and so many other non-filesystem improvements available to everyone willing to run a beta installer (but probably not cobble together many github downloads not knowing which versions & branches are meant to be dependencies).

Any change to the USB Type list involve a bunch of thorny issues, especially with Linux and udev rules. I do intend to add a MTP+Serial before final 1.57 release. But I'm not eager to do all that testing right now, holding up a 1.57-beta1 installer.

The main point a 1.57-beta1 is to give people who want to join in testing & using all this new stuff an accessible way to get it installed with all dependencies correct.

Likewise understood. Although my guess is that many of us have done more testing using +Serial than with +Seremu...

The issue I mentioned, sort of stumped me for a bit, but then by chance looked at the udev rules.

I don't know what ranges of PID's you have available, but with:
Code:
#elif defined(USB_MTPDISK)
  #define VENDOR_ID		0x16C0
  #define PRODUCT_ID		0x04D1
...
And if it was available:
If instead of current 0x476 which was the everything PID
We could use 0x4D2 I am pretty sure your udev rules would not have to change.

like:
Code:
#elif defined(USB_MTPDISK_SERIAL)
  #define VENDOR_ID             0x16C0
  #define PRODUCT_ID            0x04D2  // New MTP + Serial
  #define RAWHID_USAGE_PAGE 0xFFAB  // recommended: 0xFF00 to 0xFFFF
  #define RAWHID_USAGE    0x0200  // recommended: 0x0100 to 0xFFFF
  #define DEVICE_CLASS    0xEF
  #define DEVICE_SUBCLASS 0x02
  #define DEVICE_PROTOCOL 0x01

  #define MANUFACTURER_NAME     {'T','e','e','n','s','y','d','u','i','n','o'}
  #define MANUFACTURER_NAME_LEN 11
  #define PRODUCT_NAME          {'T','e','e','n','s','y',' ','M','T','P',' ','D','i','s','k','/','S','e','r','i','a','l'}
  #define PRODUCT_NAME_LEN      22
  #define EP0_SIZE    64

  #define NUM_ENDPOINTS         5
  #define NUM_INTERFACE   3
  #define CDC_IAD_DESCRIPTOR  1
  #define CDC_STATUS_INTERFACE  0
  #define CDC_DATA_INTERFACE  1 // Serial
  #define CDC_ACM_ENDPOINT  2
  #define CDC_RX_ENDPOINT       3
  #define CDC_TX_ENDPOINT       3
  #define CDC_ACM_SIZE          16
  #define CDC_RX_SIZE_480       512
  #define CDC_TX_SIZE_480       512
  #define CDC_RX_SIZE_12        64
  #define CDC_TX_SIZE_12        64
  
  #define MTP_INTERFACE   2 // MTP Disk
  #define MTP_TX_ENDPOINT 4
  #define MTP_TX_SIZE_12  64
  #define MTP_TX_SIZE_480 512
  #define MTP_RX_ENDPOINT 4
  #define MTP_RX_SIZE_12  64
  #define MTP_RX_SIZE_480 512
  #define MTP_EVENT_ENDPOINT  5
  #define MTP_EVENT_SIZE  32
  #define MTP_EVENT_INTERVAL_12 10  // 10 = 10 ms
  #define MTP_EVENT_INTERVAL_480 7  // 7 = 8 ms

  #define ENDPOINT2_CONFIG  ENDPOINT_RECEIVE_UNUSED + ENDPOINT_TRANSMIT_INTERRUPT
  #define ENDPOINT3_CONFIG  ENDPOINT_RECEIVE_BULK + ENDPOINT_TRANSMIT_BULK
  #define ENDPOINT4_CONFIG  ENDPOINT_RECEIVE_BULK + ENDPOINT_TRANSMIT_BULK
  #define ENDPOINT5_CONFIG  ENDPOINT_RECEIVE_UNUSED + ENDPOINT_TRANSMIT_INTERRUPT

But as you said it should not hold up Beta1...
 
I added some comments in the 3 storage examples about the begin() function being a temporary API.

Planning to package up 1.57-beta1 in a couple hours....
 
Cool ... Looking forward to 1.57b1 and getting back to this. Keeping up was a chore with variety of peripheral adds ...

Seems I found the code in progress ...
 
If instead of current 0x476 which was the everything PID
We could use 0x4D2 I am pretty sure your udev rules would not have to change.

like:
Code:
#elif defined(USB_MTPDISK_SERIAL)
  #define VENDOR_ID             0x16C0
  #define PRODUCT_ID            0x04D2  // New MTP + Serial
  #define RAWHID_USAGE_PAGE 0xFFAB  // recommended: 0xFF00 to 0xFFFF
  #define RAWHID_USAGE    0x0200  // recommended: 0x0100 to 0xFFFF
  #define DEVICE_CLASS    0xEF
  #define DEVICE_SUBCLASS 0x02
  #define DEVICE_PROTOCOL 0x01
....

Not sure if the RAWHID_* and DEVICE_ * are needed and what they are doing
they com from USB_EVERYTHING

they are used in part for
USB_SERIAL_HID (DEVICE_*)
USB_RAW_HID (RAWHID_*)
but
USB_MIDI4,16 ,USB_FLIGHTSIM_JOYSTICK,USB_MIDI16_AUDIO_SERIAL use also BCD_DEVICE

We can easily use 0x04D2 (I used a fake PID initially), but then we must convince Paul to add ii to the Teensy loader environment (I guess it is Teensy_reboot.exe) so that we can download without button press.
 
TD 1.57b1 available - if IDE 2.0 tested/used check the build dir the IDE uses. After 1.57b1 installed IDE 1.8.19 still used %appdata%\...\1.56.1 install, not where installed in the IDE unZip folder.
> noted on Beta 1.57b1 announce

>> Found T_4.1 makefiles.ino that was creating uniform test files to verify.
SD card from Feb sitting in that T_4.1 is oddly full and large from a world of stuff with 107K files:
Code:
Total 107,999 files of Size 1,088,487,493 Bytes
Bytes Used: 28,699,394,048, Bytes Total:128,144,375,808
> not exactly sure why the 27 extra GB between Size and Used, maybe a var overflow - folders and small files and large block alloc size?
Code:
Windows shows:
107,991 files and 12,724 folders
12.4GB Size and 24.6GB used on disk
So only 8 files diff - some hidden Windows may not count.

And doing the 'v' verify - built without MTP so just doing 'Serial' with MTP #ifdef out the code does a Teensy Restart - so need to discover that...

Is there working MTP sketch in the install? Need to find the working changes for what was there before ...
> total fail now without : #include <MTP_Teensy.h>
 
Good morning @defragster -

Not sure what you are asking? If you are asking about sizes of files versus bytes used...

I am guessing it all depends on the cluster size of your storage. So for example if you have 10 files that 3 bytes long,

that adds up to 30 bytes, but if your cluster size is 32K, those 10 files take up 320K on disk...

I don't believe MTP_Teensy is included in this beta release. Many of the sketches in the current library may have issues working if they deal with MSC.
Many of these you can turn this off, but the MSC interfaces are evolving right now. And as such Paul noted that within a few examples.

@mjs513 and myself have updated a couple of them to semi-function with current version. Like right now my CircuitPython test code only handles the first partition
of a disk plugged in. Why the partition code should soon be reintroduced in some form, where the disk code or like will do the enumeration and claim a partition FileSystem object for them...

And likewise the Filesystem object does not have any method on it, to be released... again will be fixed.
 
Good morning @defragster -

Not sure what you are asking? If you are asking about sizes of files versus bytes used...

I am guessing it all depends on the cluster size of your storage. So for example if you have 10 files that 3 bytes long,

that adds up to 30 bytes, but if your cluster size is 32K, those 10 files take up 320K on disk...

I don't believe MTP_Teensy is included in this beta release. Many of the sketches in the current library may have issues working if they deal with MSC.
Many of these you can turn this off, but the MSC interfaces are evolving right now. And as such Paul noted that within a few examples.

@mjs513 and myself have updated a couple of them to semi-function with current version. Like right now my CircuitPython test code only handles the first partition
of a disk plugged in. Why the partition code should soon be reintroduced in some form, where the disk code or like will do the enumeration and claim a partition FileSystem object for them...

And likewise the Filesystem object does not have any method on it, to be released... again will be fixed.

Good morning @KurtE, ...
Only question was what is needed to get back to MTP current in sync with 1.57b1 ... wrongly assumed it was merged.
> i.e. current examples for MTP and MSC, and any external sources.

File size issue was just noted because the SD card is comically large and an issue HERE to fix.
 
Back
Top