MTP Responder Contribution

[KurtE]

@all - Sometimes hard to know which thread to discuss things like integration of MTP amd MSC... and the like.

I am going to try doing some additional changes to my branch of the code to add some more support for adding MSC drives/partitions also wonder if they should apply to SD drives as well.

Examples of changes I am thinking about:

a) Something simple: (@WMXZ) - In MTP class I think that the Loop() method should instead use the Task method... That is if you look at all of our examples you will notice that in their loops they all do:

void loop()
{
  myusb.Task();

Where it is setup the main USBHost object will loop through and call each devices Task method to allow them to do any function like what you do in your Loop function...

b) Lots of stuff with MSC - First pass startup timing: Example maybe have some drives NOT at startup and/or when MSC Drive Insertion not at that time try to get the free/used space of the device, but wait to do that until after MTP is happy.. On at least a few thumb drives and maybe others this may take several seconds...

b1) I may first try a hack. Currently when I have an LittleFS drive do a format operation, I have an IntervalTimer looking for messages from the host, and this code always returns I am busy status... Wonder if I can get away with that now? That is maybe while we are looking at all of our devices at startup, have the background code try to stall the host, until we are ready for it... Currently if our startup is too slow both Windows and especially Linux won't show anything nor recover.

c) Partitions - Especially with USB Drives, but maybe also SDCards? At startup and/or insertion detection. Detect that there are multiple volumes on that device and if so maybe add storage for each one. Have some pieces in place with SDFat and MSC stuff, but not in the integration. Currently SDFat code has FatPartion and EXPartion classes, but no unifier in the FsLib (i.e no FsPartition.h/cpp files) that work with either in the same way as FsVolume works with FatVolume and ExFatVolume...

d) Again more MSC specific although again could apply to SDCards, like cards you use for RPI have multiple partitions, although so far we don't handle any of the linux file formats...
d1) Wondering about MSC object(s), How should the functionality be factored. More later in MSC thread... Or again wondering if makes sense to talk about all of this in some Integration thread...

[mjs513]

Morning @all

@KurtE - ambitious and good luck.

But to your point on SDCards - think you are going to have to implement something similar if it has more than one partition. But I think if you get it to work with USB devices the same methods can be used for SDCards.

Thinking that we should have a second MTP_test sketch that just addresses SD and USB devices. The one we have has everything in it and getting really big - maybe include prog ram as well just as a sanity check.

[KurtE]

@mjs513 - Yep I am also thinking a smaller test with fewer Storage types would be nice and/or split some of them into multiple tabs...

And yes right not I am still playing with the MSC multi partition sketch first... Again hard to know which thread to talk on...

Right now I am hacking up a new file FsPartition.h which I am part way through... Currently will be a tab in the VolumeName sketch until I get it working... Actually I am doing it in a sublime text project, but...

It is sort of a hack:

class FsPartition {
 public:
  /** Create an instance of FsPartition
   */
  FsPartition() {}

  /** \return The shift count required to multiply by bytesPerCluster. */
  uint8_t bytesPerClusterShift() const {
    if (m_fatType==FAT_TYPE_EXFAT) return m_partEx.bytesPerClusterShift();
    return m_part.bytesPerClusterShift();
  }
  /** \return Number of bytes in a cluster. */
  uint16_t bytesPerCluster() const {
    if (m_fatType==FAT_TYPE_EXFAT) return m_partEx.bytesPerCluster();
    return m_part.bytesPerCluster();
  }
  /** \return Number of bytes per sector. */
  uint16_t bytesPerSector() const {
    if (m_fatType==FAT_TYPE_EXFAT) return m_partEx.bytesPerSector();
    return m_part.bytesPerSector();
  }
...

It has 3 member variables.
The only interesting method so far is:

  bool init(BlockDevice* dev, uint8_t part = 1)
  {
    if (m_partEx.init(dev, part)) {
      m_fatType=FAT_TYPE_EXFAT;
      return true;
    }

    if (m_part.init(dev, part)) {
      m_fatType=m_part.fatType();
      return true;
    }
    return false;
  }

Still making my way through. Started off with copy of FatPartition.h and just hacking on it. Then will include in sketch and see how many screw ups... Like not sure if I can use const on these or not... Will find out soon.

EDIT: I probably should start off just with the members we call out for but... And then there is the issue of having this be a subclass of FS? Will figure that part out after I finish this part

[wwatson]

@KurtE - I am always amazed at solutions you guys come up with. What would an integration thread be? Would that just be discussions that pertain to the MTP, FS, MSC and littleFS on a new thread? If so it would be good to not have to bounce between threads. My forum savy is showing isn't it

Today I think I will on lwext4 some more.

[KurtE]

@KurtE - I am always amazed at solutions you guys come up with. What would an integration thread be? Would that just be discussions that pertain to the MTP, FS, MSC and littleFS on a new thread? If so it would be good to not have to bounce between threads. My forum savy is showing isn't it

Today I think I will on lwext4 some more.

I sort of started a thread a little bit ago: https://forum.pjrc.com/threads/66338...h-each-other-8)
Where I was maybe setting up for some integration stuff... But only myself and @mj513 posted anything...

Right now playing with your USBMscFat code: sort of relating to what I posted on the MSC thread. about having FsLib support partitions. It would be easy to get it to maybe work by changing the begin Method... But not sure if anyone would take it...

So may duplicate that class in your project with it... May also add in a volumeName method which wraps up the functionality in the thread you just added to that project.
Plus a little cleanup...

My advanced c++ skills are a bit rusty: But I think the FsVolume object leaks other objects:

class FsVolume {
 public:
  FsVolume() {}

  ~FsVolume() {end();}
,,,
  /** free dynamic memory and end access to volume */
  void end() {
    m_fVol = nullptr;
    m_xVol = nullptr;
  }
...
bool FsVolume::begin(BlockDevice* blockDev, bool setCwv, uint8_t part) {
  Serial.printf("FsVolume::begin(%x)\n", (uint32_t)blockDev);
  m_blockDev = blockDev;
  m_fVol = nullptr;
  m_xVol = new (m_volMem) ExFatVolume;
  if (m_xVol && m_xVol->begin(m_blockDev, setCwv, part)) {
    goto done;
  }
  m_xVol = nullptr;
  m_fVol = new (m_volMem) FatVolume;
  if (m_fVol && m_fVol->begin(m_blockDev, setCwv, part)) {
    goto done;
  }
  m_cwv = nullptr;
  m_fVol = nullptr;
  return false;

 done:
  m_cwv = this;
  return true;
}

I believe that the end function setting the pointers to null frees the object that was created by new? Could be wrong.
Also dito in the begin method, that if the begin for exFatVolume fails... then it should delete the exFatVolume before it sets nullptr...

[wwatson]

I sort of started a thread a little bit ago: https://forum.pjrc.com/threads/66338...h-each-other-8)
Where I was maybe setting up for some integration stuff... But only myself and @mj513 posted anything...

Right now playing with your USBMscFat code: sort of relating to what I posted on the MSC thread. about having FsLib support partitions. It would be easy to get it to maybe work by changing the begin Method... But not sure if anyone would take it...

So may duplicate that class in your project with it... May also add in a volumeName method which wraps up the functionality in the thread you just added to that project.
Plus a little cleanup...

My advanced c++ skills are a bit rusty: But I think the FsVolume object leaks other objects:

class FsVolume {
 public:
  FsVolume() {}

  ~FsVolume() {end();}
,,,
  /** free dynamic memory and end access to volume */
  void end() {
    m_fVol = nullptr;
    m_xVol = nullptr;
  }
...
bool FsVolume::begin(BlockDevice* blockDev, bool setCwv, uint8_t part) {
  Serial.printf("FsVolume::begin(%x)\n", (uint32_t)blockDev);
  m_blockDev = blockDev;
  m_fVol = nullptr;
  m_xVol = new (m_volMem) ExFatVolume;
  if (m_xVol && m_xVol->begin(m_blockDev, setCwv, part)) {
    goto done;
  }
  m_xVol = nullptr;
  m_fVol = new (m_volMem) FatVolume;
  if (m_fVol && m_fVol->begin(m_blockDev, setCwv, part)) {
    goto done;
  }
  m_cwv = nullptr;
  m_fVol = nullptr;
  return false;

 done:
  m_cwv = this;
  return true;
}

I believe that the end function setting the pointers to null frees the object that was created by new? Could be wrong.
Also dito in the begin method, that if the begin for exFatVolume fails... then it should delete the exFatVolume before it sets nullptr...

I had seen that thread and of course forgot about it. That would be a good thread I think IMHO.

[mjs513]

@KurtE
My advanced c++ skills are limited to looking things up as I go

Anyway, what you say seems to make sense, at least to me. Now just to decided where to put all this. At this point maybe stick to the integration thread since we are jumping between libraries.

[KurtE]

@KurtE
My advanced c++ skills are limited to looking things up as I go

Anyway, what you say seems to make sense, at least to me. Now just to decided where to put all this. At this point maybe stick to the integration thread since we are jumping between libraries.

Actually I received a response on github from Bill... Now there is not a leak as the new is not actually allocating memory...
As the code like: m_fVol = new (m_volMem) FatVolume;

is actually just assigning the memory in a memory storage of the object to the pointer... More or a less a cast...

No I don't really use dynamic memory. For embedded systems in my world it was forbidden. SdFat never uses the heap.

I define my own new operator to share RAM between FAT and exFAT since only one can be active.
Here is the magic:
FsNew.h
/** 32-bit alignment */
typedef uint32_t newalign_t;

/** Custom new placement operator */
void* operator new(size_t size, newalign_t* ptr);
FsNew.cpp
#include "FsNew.h"
void* operator new(size_t size, newalign_t* ptr) {
(void)size;
return ptr;
}
It's allocated in FsVolume.h here. FS_ALIGN_DIM is the max size required.
newalign_t m_volMem[FS_ALIGN_DIM(ExFatVolume, FatVolume)];

[mjs513]

Actually I received a response on github from Bill... Now there is not a leak as the new is not actually allocating memory...
As the code like: m_fVol = new (m_volMem) FatVolume;

is actually just assigning the memory in a memory storage of the object to the pointer... More or a less a ca

Guess that answers that question

[mborgerson]

I just transferred a solar energy evaluation program from the T3.6 that I used last summer to the T3.5. I ran into some compilation issues in MTP.h and MTP.cpp in the tests for processor type:

#if defined(__MK66FX1M0__)..... code

There was no path to compile the code for the T3.5. I solved the problem by changing the #define:

#if defined(__MK66FX1M0__) || defined(__MK64FX512__)

The compilation proceeded and the program seems to work fine.

If you have made modifications to the MTP code over the last six months, please check to see that you have adjusted the #define statements to allow for the T3.5. I was using code from about 6 months ago which I had modified to allow passing in an initialized file system when setting up the storage. This saves the data space that would result if you had a file system in your foreground code and initialized a new file system inside the MTP startup. This may become even more important on the T3.5 with its smaller memory and the proliferating number of storage devices that are being added to the MTP system.

[Hystrix]

I have successfully got MTP working with a Teensy 3.6 using the code from here: https://github.com/WMXZ-EU/MTP_t4)...but only with Windows 10.

With Linux (Ubuntu 20.04), the Teensy is recognised with lsusb:

Bus 002 Device 002: ID 0781:5583 SanDisk Corp. Ultra Fit
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 0c45:671e Microdia Integrated_Webcam_HD
Bus 001 Device 022: ID 16c0:04d1 Van Ooijen Technische Informatica Teensy MTP Disk
Bus 001 Device 003: ID 0cf3:e009 Qualcomm Atheros Communications 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

...and dmesg:

2590.640519] usb 1-2: new full-speed USB device number 22 using xhci_hcd
[12590.789140] usb 1-2: New USB device found, idVendor=16c0, idProduct=04d1, bcdDevice= 2.77
[12590.789142] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[12590.789144] usb 1-2: Product: Teensy MTP Disk
[12590.789145] usb 1-2: Manufacturer: Teensyduino
[12590.789146] usb 1-2: SerialNumber: 9053380
[12590.792081] hid-generic 0003:16C0:04D1.0013: hidraw1: USB HID v1.11 Device [Teensyduino Teensy MTP Disk] on usb-0000:00:14.0-2/input1

...but the Teensy does not mount.

I have compiled using in PlatformIO using TeensyDuino 1.54b7, and the following build parameters (platformio.ini):

[env:teensy36]
platform = teensy
board = teensy36
framework = arduino
build_flags = -D USB_MTPDISK
platform_packages =
    framework-arduinoteensy@https://github.com/maxgerhardt/teensy-core-pio-package.git

Is there a specific udev entry I need to make in /lib/udev/rules.d/69-libmtp.rules ?

I have installed gvfs-backends, jmtpfs and mtp-tools. I have read through 37 pages of this thread for ideas/solutions.

What have I missed? Or is getting Teensy 3.6 MTP working with linux not yet possible?

It's a bit frustrating that it works flawlessly under Windows 10, showing up just like a USB stick would, but under Linux...recognised, but will not mount.

Any help will be appreciated.

[WMXZ]

It's a bit frustrating that it works flawlessly under Windows 10, showing up just like a USB stick would, but under Linux...recognised, but will not mount.

Colleagues of mine have used MTP successfully with linux (ubuntu)
please note: MTP responder answers to a proper implemented MTP initiator. It seems that not all Linux implementations behave the same.
I have no Linux machine, so I cannot test. Volunteers are welcome.

[KurtE]

I have been able to get it to sort of work with Ubuntu, at least with the set of binaries that several of us are working with. BUT with many caveats.

Note: Linux is not my main machines, so I only know enough to be dangerous.

But some of the things I have seen include: You need to install libmtp, something like: sudo apt-get install libmtp

I don't remember what else I may or may not have had to install. The GUI
Shows all of the storages with the same name (the MTP device name).

If anything takes long to setup, like getting the count of free sectors/clusters of a disk, it won't work... Example larger SD card running SDFAT32, with default stuff won't work well.
It was not handling events like: add this storage to the list...

Again I don't use it much, but will try it out again soon.

[Hystrix]

I have had some success getting USB_MTPDISK to work with a Teensy 4.1 and TeensyDuino v154b9 using the code from here https://github.com/WMXZ-EU/MTP_t4. Under Ubuntu (20.04) it mounts as "Teensy MTP Disk" and all folders and files are visible.

However, any file operation, for example copying a file and pasting to local folder, results in an error message: 'Error while copying "filename.txt" There was an error copying the file into /home/user/Downloads. Error getting file: -7: I/O problem'

The file is actually copied, but sometimes the resulting file size is slightly smaller than the original file. A small file - 166 bytes - sometimes copies ok despite the error message. Sometimes the resulting file size is 0 bytes.

It seems that I am a lot closer to getting this working, but obviously something is not quite right.

[Fluxanode]

I installed MTP_T4 from https://github.com/WMXZ-EU/MTP_t4 on the Teensy 3.5 and made the modifications to the usb_desc.h as indicated in the readme.
I am using arduino IDE with Teensyduino version 1.54-beta 9 and a 16G SD in the card slot. Everything compiles and loads fine.

Issues:
1. The device shows up in the device manager as MTP USB Device not Teensy MTP device. Is there a driver issue?
2. It is not showing up in Windows explorer no matter what i try.
3. After making the changes to USB_DESC should there be a Serial device in device manager for the Teensy 3.5? Actually I found a serial port in the arduino IDE and also in Tera Term and used it to program the 3.5 but it is not in device manager. How can this happen? Seems to me it should be in device manager.
4. Using mtp-test.ino example program, is there something needed to be changed for the 3.5?

Thanks for any help...

[mjs513]

@Fluxanode
Did you remember to change the USB Type to MTP Disk (Experimental) from the Tools -> USB Type dropdown when did compile and upload?

[Fluxanode]

@Fluxanode
Did you remember to change the USB Type to MTP Disk (Experimental) from the Tools -> USB Type dropdown when did compile and upload?

Yes is did that, at compile time it throws an error if MTP Disk (Experimental) is not set up.

I'm sure i must have missed something in the install for the 3.5. Can anyone guide me on getting this going?
I set it up straight out of the box except for the serial port mods.

[Fluxanode]

@Hysrix - can you describe what you did to set up on 3.6?

[defragster]

@Hysrix - can you describe what you did to set up on 3.6?

From the 'verbose console' - perhaps showing the list of "libraries Used" might point to the problem.
Have not done this lately - and never with a T_3.x IIRC - but know having 'local' sketchbook libraries can cause trouble if not the right ones being used.

[Fluxanode]

@defragster

Libraries used:

Multiple libraries were found for "SD.h"
Used: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD
Not used: C:\Program Files (x86)\Arduino\libraries\SD
Multiple libraries were found for "SdFat.h"
Used: D:\ChuckW\Arduino\libraries\SdFat
Not used: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SdFat
Using library SD at version 2.0.0 in folder: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD
Using library SdFat at version 2.0.6 in folder: D:\ChuckW\Arduino\libraries\SdFat
Using library SPI at version 1.0 in folder: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SPI
Using library MTP_t4-master at version 1.0.0-beta.1 in folder: D:\ChuckW\Arduino\libraries\MTP_t4-master
Using library Time at version 1.6 in folder: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\Time
"C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-size" -A "C:\\Users\\ChuckW\\AppData\\Local\\Temp\\arduino_ build_187955/mtp-test.ino.elf"
Sketch uses 87160 bytes (16%) of program storage space. Maximum is 524288 bytes.
Global variables use 9304 bytes (3%) of dynamic memory, leaving 252832 bytes for local variables. Maximum is 262136 bytes.


It looks like I have some duplicate Libs. How do I get rid of them? Anything else you see?
Where else can i look?

Thanks for your help as always!

[defragster]

TD 1.54 b9 includes the current SdFat so this should be removed :: Used: D:\ChuckW\Arduino\libraries\SdFat

Not sure if that solves anything. And not sure about the rest ... but now posted others can review.

[Fluxanode]

I cleaned up the duplicate libs (deleted them) and no real change except now there is a mystery drive F: under portable devices and there is another MTP USB Device. I do not normally have an F drive in device manager and it doesn't show up in explorer. EDIT - found the F: drive is my sd card reader when it does not have a card in it...

[Fluxanode]

Maybe my teensy 3.5 is broken?

[defragster]

I cleaned up the duplicate libs (deleted them) and no real change except now there is a mystery drive F: under portable devices and there is another MTP USB Device. I do not normally have an F drive in device manager and it doesn't show up in explorer.

So that is unchanged from p#915 where explorer doesn't present the drive?
Assuming the Windows is version 10?
When it was done here on Win 10 with T_4.x's no 'other' Windows drivers were needed.

Seems I saw a note :: It may be the drive info is not coming out of Teensy fast enough to suit Explorer recognition? But the system in the USB icon saw something?

Not sure of the format on the SD card matters? Is the drive empty or full or just a couple of files on the root? With the SD card in the PC does it show as healthy and what is the File System format indicated in properties?

With the SD card back in T_3.5 what do the SD/Examples show running the DiskInfo and other examples that list files out the SerMon?

... again, sorry I'm not in a position to test this myself ... just trying to probe for info one of the others here might recognize.

Running those samples will show if the T_3.5 and SD card are working well from that point of view.

[Fluxanode]

Windows 10 and just two 500k-ish text files in the root of the SD.

Yes, explorer is not showing the drive.
The SD is formatted fat 32 and i am using the recommended SD card formatter a program called SD card formatter by Tuxera.
I haven't tried the SD examples, just recently got around to playing with the 3.5. I had issues on the 3.2 where the drive did show up and you could see the files but I could only move one file (slowly) then it would not transfer any more files. So I bought a 3.5 to try it. I like the 3.5 features and am attempting to build a commercial product (multi instrument transmitter and data logger) using the 3.5. The MTP would save me from having to make a standalone windows app to move the data files.

I suspect it has something to do with the device showing up as MTP usb device in device manager and not teensy mtp like it did on the 3.2