MTP status

I've developed a Teensy 4.1-based system which records GNS and inertial data to files on a microSD card in response to a pushbutton operation. The final piece required is the ability to retrieve files from the SD card via the USB port when connected to a host computer. It's not necessary to do so while data are being recorded. I'm struggling a bit with incorporation of MTP support in my application. Perhaps someone would be so kind as to provde a bit of guidance.

Background: Arduino IDE 2.3.0 in a Linux environment, with libraries cloned fron GitHub into "$HOME/Arduino/libraries/".

The first issue concerns choice of library. There are at least two on GitHub which appear to relate to both MTP and Teensy 4, MTP_Teensy and MTP_t4. MTP_Teensy does not appear to have been updated in the past several years, while changes have been committed to the MTP_t4 src directory since December. Based upon the descriptions accompanying each repository, either ought to meet my rather simple needs. In both cases, however, none of the examples I've tried to compile does so without errors. I've made some attempts to address those, but so far without success. As an example, I see this message when attempting to build $HOME/Arduino/libraries/MTP_t4/examples/mtp-test/mtp-test.ino:

Code:
In file included from /home/mcollins/Arduino/libraries/MTP_t4/examples/mtp-test/mtp-test.ino:1:
/home/mcollins/Arduino/libraries/MTP_t4/src/MTP.h:36:6: error: #error "You need to select USB Type: 'MTP Disk (Experimental)'"
   36 |     #error "You need to select USB Type: 'MTP Disk (Experimental)'"
      |      ^~~~~
exit status 1

Compilation error: exit status 1

It's not clear to me how or where I select USB type. Attempts to compile MTP_Teensy examples generate far more errors.

If MTP_t4 is a preferred library for my needs, what steps are required in order to compile the examples for Teensy 4.1? If there is a better choice of library, please advise.

Thanks,
-- Mike --
 
If you have installed the latest Teensyduino package (1.60) you do not need any MTP library (and should remove any you have added to avoid chances of a conflict), it is included in the core code. Change the USB type in the tools menu of the Arduino IDE.
 
Ah, neglected to include the Teensyduino package revision, which is in fact 1.60. Can you point me to some documentation or examples which will help me understand how to incorporate MTP support into my application code?

Thanks for the prompt response.
 
Well, that was certainly much easier than expected. Adding the required updates to my code to incorporate MTP support took all of maybe 3 minutes, and the results are exactly what I was hoping to achieve.

Thanks for the assistance, and to all who did the work to make this feature available. It is most appreciated.

-- Mike --
 
Oh, this is cool. I got the example running, but I also need a serial USB interface. I wasn't sure they could co-exist, but if I do:

Code:
void loop()
{
  MTP.loop(); // Perform media access when PC wants access
  if( Serial.available())
  {
    Serial.read();
    Serial.println("hello");
  }
}

it seems to work.

Now, my problem is I am basically doing a data logger application that uses sdfat. The MTP example won't compile if I substitute 'sdfat.h' for 'SD.h'.

Is MTP not compatible with sdfat?

Also, if that is fixable, I'd like to not have MTP.loop(); running all the time, just when I want PC access to the SD card . Can I just run that loop based on the logger status (say, by sending a serial command)?
 
Is MTP not compatible with sdfat?

MTP requires media presence and auto-restart functionality which isn't officially part of SdFat.

So directly, MTP is not compatible with SdFat. It is only possible to use SdFat indirectly through SD.h (and FS.h) because MTP need that stuff which SdFat doesn't normally provide.
 
Ah, ok., thanks Paul. Too bad.
My backup plan was to just read the file and send it over USB-serial and capture that to a file. Not elegant , but functional.
In any case, kudos for all the clever additions to Teensyduino. It’s humbling when I explore the core and library code.
 
Why is it "Too bad"?

Maybe you're thinking of the old days where SD library was very limited (8.3 filenames, only up to 32 GB size, etc) and SdFat was technically superior? Since version 1.54, we removed that ancient SD library. It has since then been just a thin wrapper for SdFat. You can get access to the SdFat APIs, if you really want. See the SD example SdFat_Usage for details.

From a design perspective, there are 2 reasons why MTP (and many other things) should not be tightly tied to SdFat.

1: If we use SdFat APIs directly, then other libraries like MTP that want to access filesystems become tied to only FAT filesystems SdFat support. That closes the door to supporting LittleFS and other non-FAT filesystems, or in the future things like network filesystems. This is the reason for FS.h, so libraries and programs needing access to files and filesystems don't need to be hard coded to 1 specific filesystem library.

2: SdFat simply lacks media management that MTP requires. I did communicate with Bill about this. At least at the time, he wasn't interested in adding that capability to SdFat.
 
I appreciate the feedback, thanks!

I was thinking, 'too bad' in that it doesn't integrate directly into my code. I'm probably out of touch on this. I based my logger on the 'bench.ino' example, as I need large files and fast SD writes (I'm logging at 18Mbytes/sec for an hour). That code did that. I'm more than happy to use something else if it's better. Should I be using something different?
 
@slash2, the program below is the SdFat bench example modified to use #include <SD.h>. I think you should be able to make the same changes ot your program. Here's a summary:
  • replace #include "sdfat.h" with #include <SD.h>
  • delete all of the SdFat macros and associated logic
  • delete stuff related to SPI, which you don't need with built-in SD on T4.1
  • add definition of FsFile file before SD.begin()
  • replace sd.begin(SD_CONFIG) with SD.begin(BUILTIN_SDCARD)
  • replace all other instances of "sd" with "SD.sdfs", as @jmarsh said
I get exactly the same results with this program as with the original SdFat bench example.

Code:
/* bench_SD.ino
 * This program is a simple binary write/read benchmark.
 * It has been modified to use SD rather than SdFat
 */
#include <SD.h>  // "SdFat.h"
#include "sdios.h"
#include "FreeStack.h"

// Set PRE_ALLOCATE true to pre-allocate file clusters.
const bool PRE_ALLOCATE = true;

// Set SKIP_FIRST_LATENCY true if the first read/write to the SD can
// be avoid by writing a file header or reading the first record.
const bool SKIP_FIRST_LATENCY = true;

// Size of read/write.
const size_t BUF_SIZE = 512;

// File size in MB where MB = 1,000,000 bytes.
const uint32_t FILE_SIZE_MB = 5;

// Write pass count.
const uint8_t WRITE_COUNT = 2;

// Read pass count.
const uint8_t READ_COUNT = 2;
//==============================================================================
// End of configuration constants.
//------------------------------------------------------------------------------
// File size in bytes.
const uint32_t FILE_SIZE = 1000000UL*FILE_SIZE_MB;

// Insure 4-byte alignment.
uint32_t buf32[(BUF_SIZE + 3)/4];
uint8_t* buf = (uint8_t*)buf32;

// Serial output stream
ArduinoOutStream cout(Serial);
//------------------------------------------------------------------------------
// Store error strings in flash to save RAM.
#define error(s) SD.sdfs.errorHalt(&Serial, F(s))
//------------------------------------------------------------------------------
void cidDmp() {
  cid_t cid;
  if (!SD.sdfs.card()->readCID(&cid)) {

    error("readCID failed");
  }
  cout << F("\nManufacturer ID: ");
  cout << hex << int(cid.mid) << dec << endl;
  cout << F("OEM ID: ") << cid.oid[0] << cid.oid[1] << endl;
  cout << F("Product: ");
  for (uint8_t i = 0; i < 5; i++) {
    cout << cid.pnm[i];
  }
  cout << F("\nVersion: ");
  cout << int(cid.prv_n) << '.' << int(cid.prv_m) << endl;
  cout << F("Serial number: ") << hex << cid.psn << dec << endl;
  cout << F("Manufacturing date: ");
  cout << int(cid.mdt_month) << '/';
  cout << (2000 + cid.mdt_year_low + 10 * cid.mdt_year_high) << endl;
  cout << endl;
}
//------------------------------------------------------------------------------
void clearSerialInput() {
  uint32_t m = micros();
  do {
    if (Serial.read() >= 0) {
      m = micros();
    }
  } while (micros() - m < 10000);
}
//------------------------------------------------------------------------------
void setup() {
  Serial.begin(9600);

  // Wait for USB Serial
  while (!Serial) {
    yield();
  }
  delay(1000);
  cout << F("\nUse a freshly formatted SD for best performance.\n");
  // use uppercase in hex and use 0X base prefix
  cout << uppercase << showbase << endl;
}
//------------------------------------------------------------------------------
void loop() {
  float s;
  uint32_t t;
  uint32_t maxLatency;
  uint32_t minLatency;
  uint32_t totalLatency;
  bool skipLatency;

  // Discard any input.
  clearSerialInput();

  // F() stores strings in flash to save RAM
  cout << F("Type any character to start\n");
  while (!Serial.available()) {
    yield();
  }
#if HAS_UNUSED_STACK
  cout << F("FreeStack: ") << FreeStack() << endl;
#endif  // HAS_UNUSED_STACK

  if (!SD.begin(BUILTIN_SDCARD)) {
    SD.sdfs.initErrorHalt(&Serial);
  }
  if (SD.sdfs.fatType() == FAT_TYPE_EXFAT) {
    cout << F("Type is exFAT") << endl;
  } else {
    cout << F("Type is FAT") << int(SD.sdfs.fatType()) << endl;
  }

  cout << F("Card size: ") << SD.sdfs.card()->sectorCount()*512E-9;
  cout << F(" GB (GB = 1E9 bytes)") << endl;

  cidDmp();

  // open or create file - truncate existing file.
  FsFile file;
  if (!file.open("bench.dat", O_RDWR | O_CREAT | O_TRUNC)) {
    error("open failed");
  }

  // fill buf with known data
  if (BUF_SIZE > 1) {
    for (size_t i = 0; i < (BUF_SIZE - 2); i++) {
      buf[i] = 'A' + (i % 26);
    }
    buf[BUF_SIZE-2] = '\r';
  }
  buf[BUF_SIZE-1] = '\n';

  cout << F("FILE_SIZE_MB = ") << FILE_SIZE_MB << endl;
  cout << F("BUF_SIZE = ") << BUF_SIZE << F(" bytes\n");
  cout << F("Starting write test, please wait.") << endl << endl;

  // do write test
  uint32_t n = FILE_SIZE/BUF_SIZE;
  cout <<F("write speed and latency") << endl;
  cout << F("speed,max,min,avg") << endl;
  cout << F("KB/Sec,usec,usec,usec") << endl;
  for (uint8_t nTest = 0; nTest < WRITE_COUNT; nTest++) {
    file.truncate(0);
    if (PRE_ALLOCATE) {
      if (!file.preAllocate(FILE_SIZE)) {
        error("preAllocate failed");
      }
    }
    maxLatency = 0;
    minLatency = 9999999;
    totalLatency = 0;
    skipLatency = SKIP_FIRST_LATENCY;
    t = millis();
    for (uint32_t i = 0; i < n; i++) {
      uint32_t m = micros();
      if (file.write(buf, BUF_SIZE) != BUF_SIZE) {
        error("write failed");
      }
      m = micros() - m;
      totalLatency += m;
      if (skipLatency) {
        // Wait until first write to SD, not just a copy to the cache.
        skipLatency = file.curPosition() < 512;
      } else {
        if (maxLatency < m) {
          maxLatency = m;
        }
        if (minLatency > m) {
          minLatency = m;
        }
      }
    }
    file.sync();
    t = millis() - t;
    s = file.fileSize();
    cout << s/t <<',' << maxLatency << ',' << minLatency;
    cout << ',' << totalLatency/n << endl;
  }
  cout << endl << F("Starting read test, please wait.") << endl;
  cout << endl <<F("read speed and latency") << endl;
  cout << F("speed,max,min,avg") << endl;
  cout << F("KB/Sec,usec,usec,usec") << endl;

  // do read test
  for (uint8_t nTest = 0; nTest < READ_COUNT; nTest++) {
    file.rewind();
    maxLatency = 0;
    minLatency = 9999999;
    totalLatency = 0;
    skipLatency = SKIP_FIRST_LATENCY;
    t = millis();
    for (uint32_t i = 0; i < n; i++) {
      buf[BUF_SIZE-1] = 0;
      uint32_t m = micros();
      int32_t nr = file.read(buf, BUF_SIZE);
      if (nr != BUF_SIZE) {
        error("read failed");
      }
      m = micros() - m;
      totalLatency += m;
      if (buf[BUF_SIZE-1] != '\n') {

        error("data check error");
      }
      if (skipLatency) {
        skipLatency = false;
      } else {
        if (maxLatency < m) {
          maxLatency = m;
        }
        if (minLatency > m) {
          minLatency = m;
        }
      }
    }
    s = file.fileSize();
    t = millis() - t;
    cout << s/t <<',' << maxLatency << ',' << minLatency;
    cout << ',' << totalLatency/n << endl;
  }
  cout << endl << F("Done") << endl;
  file.close();
}
 
Thanks Joe,
Looks real straightforward. I was totally misunderstanding that SDFat was the best to use. I’ll try it out.
Today I just loaded MTP after each run to get the data file. Clumsy, but so much better than moving the SD card.
 
You're welcome. When Paul says that SD is a "thin wrapper" for SdFat, what that means is that SD fully contains SdFat. In other words, when you #include <SD.h>, you get all of SdFat, but it's inside the SD class, and you access it through the SD class interface. That's what you're doing when you replace "sd", which in your program is an instance of the SdFat class, with "SD.sdfs", which is the SdFat file system that is inside the SD class. It's actually a lot easier to use SD with the T4.1 built-in SD than it is to use SdFat.

You asked previously whether you can or should avoid calling MTP.loop() while you're logging, and the answer is yes. When you're done with logging, you can call MTP.reset(), which will update MTP so it knows about the new file you just created, and then enter a loop that calls MTP.loop() so that you can transfer files.
 
I made the changes and added the MTP stuff in. I selected 'Serial + MTP' as I need both. Had a small glitch on compiling as it looks like MTP grabs some of RAM2, which I use as a secondary buffer when logging. I just reduced my buffer size a bit and it compiled and ran. I only have a class 1 SD card at home so I could only clock data at 10Mbytes/sec (normally it's 18). but it seemed to handle that. I'll need to verify with the actual system and fast card on Monday. This is so much better than trying to upload over USB-serial.
Thanks!
 
Back
Top