Teensyduino File System Integration, including MTP and MSC

KurtE · Nov 2, 2021

Morning all,

@Paul, @mjs313 @defragster - I have been testing mostly with our released version of LittleFS and see the issue, so as @mjs513 said it is unclear where the issues are. As I mentioned here or elsewhere, maybe not bad idea to update to more recent ones, especially if it maybe fixes some other things.

As for the issues of: partial file copy, and at times random reboot, or crash, trying to do some things that maybe try to isolate the problem(s), some which include:

a) Look again through the code to make sure I am not doing something stupid when we buffer the data.

b) I am hacking up a dummy FS, which does nothing that I can play with delays on writes and see if it shows up anything...

c) maybe take a break and do something else...

As we all know that some of these type problems are pain to isolate as maybe there is some simple off by one bug, which is maybe some code writes to wrong memory, but this does not hurt anything 99+% of the time... And only shows up in certain conditions. And this could be anywhere...

Now back to pulling out hair...

KurtE · Nov 2, 2021

Follow on to previous message b)

I now have a completely Bogus FS:

Code:

#ifndef __BogusFS_H__
#define __BogusFS_H__

#include <Arduino.h>
#include <FS.h>

class BogusFile : public  FileImpl {
public:
	BogusFile(const char *filename, uint8_t mode) {
		strlcpy(_filename, filename, sizeof(_filename));
		_fOpen = true;};
	~BogusFile() {close(); }
	virtual size_t read(void *buf, size_t nbyte) { return 0; }
	virtual size_t write(const void *buf, size_t size) {
		delay(5);
		return size;
	}
	virtual int available() { return 0;}
	virtual int peek() {return -1;}
	virtual void flush() {}
	virtual bool truncate(uint64_t size=0) {return true;};
	virtual bool seek(uint64_t pos, int mode) {return false;}
	virtual uint64_t position()  { return 0;}
	virtual uint64_t size()  { return 0;}
	virtual void close() {_fOpen = false;};
	virtual bool isOpen() {return _fOpen;}
	virtual const char* name() {return _filename;}
	virtual bool isDirectory() { return (strcmp(_filename, "/") == 0);}
	virtual File openNextFile(uint8_t mode=0) {return File();}
	virtual void rewindDirectory(void) {};
	char _filename[256];
	bool _fOpen;
};


class BogusFS : public  FS
{
public:
	BogusFS() {};
	virtual File open(const char *filename, uint8_t mode = FILE_READ) {
		Serial.printf("BogusFS::open(%s, %x)\n", filename, mode);
		return File(new BogusFile(filename, mode));
	}
	virtual bool exists(const char *filepath) {Serial.printf("BogusFS::exists(%s)\n", filepath); return true;}
	virtual bool mkdir(const char *filepath)  {Serial.printf("BogusFS::mkdir(%s)\n", filepath); return true;}
	virtual bool rename(const char *oldfilepath, const char *newfilepath) {Serial.printf("BogusFS::rename(%s, %s)\n", oldfilepath, newfilepath); return true;}
	virtual bool remove(const char *filepath)  {Serial.printf("BogusFS::remove(%s)\n", filepath); return true;}
	virtual bool rmdir(const char *filepath)  { Serial.printf("BogusFS::rmdir(%s)\n", filepath); return true;}
	virtual uint64_t usedSize()  { return 16000000000ul;} 
	virtual uint64_t totalSize() { return 32000000000ul;}
};

#endif // __BogusFS_H__

Which I hacked up USB_MTP-logger to include:

Code:

#include "BogusFS.h"
BogusFS bogusfs;

and in setup:

Code:

 storage.addFilesystem(bogusfs, "Bogus");

Right now writes just have a 5ms delay... And it copied almost all of the data for that 22mb file, and then the program reset:

Now back to ...

defragster · Nov 2, 2021

Most of the few runs I've done were posted ... but indeed some starts after an upload failed before the posted results.

Will get back to it soonish - and see what shows up with some different tests.

Is focus on MTP or MSC? Any proven best sketches to work with?

KurtE · Nov 2, 2021

The good news is for me I have a 100% crash case... Now to debug...

I put the BogusFS.h as part of the USB_MTP-logger sketch...

I posted the sketch yesterday I was using to create the large USB Packet aligned serial numbers file...
I believe it is up in posting #421

As I showed up above it gets to the end and then crashes...

At least for this issue, it sort of rules out the file systems like littlefs and SD...

MSC stuff - Also good to test, but currently not doing much there until we decide if current path is the right one or if we need to migrate everything back to the MSC Github project.

KurtE · Nov 2, 2021

Another quick update:

To make sure that our memcpy functions did not point to bogus areas, where we might over run stuff, I put the calls to memcpy into a helper function:

Code:

void MTPD::check_memcpy(uint8_t *pdest, const uint8_t *psrc, size_t size, const uint8_t *pb, size_t pb_size) {
  static uint32_t call_count = 0;
  call_count++;
  if (((uint32_t)pdest < (uint32_t)pb) || (((uint32_t)pdest + size) > ((uint32_t)pb + pb_size)) ) {
    printf("\n####### Check_memcpy ******(%u): %u %u %u (%u %u)\n", call_count, (uint32_t)pdest, (uint32_t)psrc, size, (uint32_t)pb, pb_size);
  }
  memcpy(pdest, psrc, size);  
}

Did not get any outputs and still crashed...

I then tried sending a different smaller file to it and did not fault:

Code:

Modified: 6159df44
BogusFS::open(/FS.h, 2)
73410 RESP:2001(RSP:OK)l: 24 T:d : 20001 ffffffff 14
73410 CMD: 100d(SEND_OBJECT)l: 12 T:e
MTPD::SendObject: len:8732
BF Sequence error 0 0
BF Sequence error 0 0
BF Sequence error 0 0
BF Sequence error 0 0
BF Sequence error 0 0
BF Sequence error 0 0
BF Sequence error 0 0
BF Sequence error 0 0
BF Sequence error 0 0
BF Sequence error 0 0
BogusFS::open(/FS.h, 0)
 # USB Packets: 17 total: 0 avg ms: 0 max: 0
 # Write: 5 total:16 avg ms: 3 max: 4
>>>Total Time: 15490
73426 RESP:2001(RSP:OK)l: 12 T:e

It also showed my sequence number checking is being hit...

So what is left to try?

Maybe see about the code for update date/time... I don't think so as calls to bogusFS just return...

So again wondering if maybe hitting USB code issue? I am probably missing some other obvious thing, but...

mjs513 · Nov 2, 2021

Took a slightly different but similar approach. Using @KurtE's indexFile created on the SD card I did a transfer of the file to Flash5 then downloaded and used CodeCompare and zero differences so this is just another confirmation that the issue is not with SD and LittleFS as Kurt said.

Didn't have a crash but after file copy to the PC MTP froze in explorer and could not do anything anything else in Explorer

The sketch below allows you to create the index file on the SD Card and then transfer it to flash. Since its linked to MTP the sketch is self contained.

Code:

#include <SD.h>
#include <SPI.h>
#include <LittleFS.h>
#include <MTP_Teensy.h>

// Add in MTPD objects
MTPStorage storage;
MTPD mtpd(&storage);

LittleFS_SPIFlash myfs;

File myFile, dataFile;
int iRecord = 0;
int line = 0;

const int csFlash = 5;
const int chipSelect = BUILTIN_SDCARD;
#define BUFFER_SIZE_INDEX 128
uint8_t write_buffer[BUFFER_SIZE_INDEX];
#define buffer_mult 16
uint8_t buffer_temp[buffer_mult*BUFFER_SIZE_INDEX];

int bytesRead;

void setup()
{
  while (!Serial && millis() < 5000);
  Serial.begin(9600);

  if (CrashReport) {
    Serial.print(CrashReport);
  }
  Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);
  delay(3000);
  Serial.print("Initializing SD card...");

  if (!SD.begin(chipSelect)) {
    Serial.println("SD initialization failed!");
    return;
  }
  Serial.println("SD initialization done.");

  if (!myfs.begin(csFlash)) {
    Serial.println("Flash initialization failed!");
    return;
  }
  Serial.println("Flash initialization done.");
  
  mtpd.begin();
  storage.addFilesystem(myfs, "sflash");
  
  Serial.println("MTP FS added.");



  
  menu();
  
}

void loop() {
  mtpd.loop();
  
  if (Serial.available()) {
    uint8_t command = Serial.read();
    switch (command) {
    case 'w':
      writeIndexFile();
      break;
    case 't':
      copySD2Flash();
      break;
    case '\r':
    case '\n':
    case 'h':
      menu();
      break;
    default:
      menu();
      break;
    }
    while (Serial.read() != -1)
      ; // remove rest of characters.
  }
  
}

void menu() {
  Serial.println();
  Serial.println("Menu Options:");
  Serial.println("\tw - Write Index File to SD card");
  Serial.println("\tt - Transfer Index File from SD to Flash");
  Serial.println("\th - Menu");
  Serial.println();
}

void writeIndexFile() 
{
  // open the file.
  Serial.println("Write Large Index File");
  myFile = SD.open("LargeIndexedTestfile.txt", FILE_WRITE_BEGIN);
  if (myFile) {
    myFile.truncate(); // Make sure we wipe out whatever was written earlier
    for (uint32_t i = 0; i < 43000*4; i++) {
      memset(write_buffer, 'A'+ (i & 0xf), sizeof(write_buffer));
      myFile.printf("%06u ", i >> 2);  // 4 per physical buffer
      myFile.write(write_buffer, i? 120 : 120-12); // first buffer has other data...
      myFile.printf("\n");
    }
    myFile.close();
    Serial.println("\ndone.");
  }
}

void copySD2Flash()
{
  // open the file on SD Card.
  Serial.println("Copy Large Index File from SD to SPI Flash");
  myFile = SD.open("LargeIndexedTestfile.txt", FILE_READ);

  //open file on flash
  dataFile = myfs.open("LITF.txt", FILE_WRITE_BEGIN);

  dataFile.truncate();
  if (myFile) {
    Serial.println("Files Writing");
    while (myFile.available()) {
        memset(write_buffer, 0, sizeof(write_buffer));
        memset(buffer_temp, 0, sizeof(buffer_temp));
    uint16_t byteCount = 0;
        for(uint8_t i=0; i < buffer_mult; i++) {
          bytesRead = myFile.read(write_buffer, line? BUFFER_SIZE_INDEX : BUFFER_SIZE_INDEX-12); 
          if(bytesRead > 0) {
            for(int j = 0; j < (line? BUFFER_SIZE_INDEX : BUFFER_SIZE_INDEX-12); j++) {
              buffer_temp[byteCount+j] = write_buffer[j];
            }
          }
      byteCount += (line? BUFFER_SIZE_INDEX : BUFFER_SIZE_INDEX-12);
      line += 1;
        }
        dataFile.write((void *)buffer_temp, byteCount);
        
        iRecord += 1;
        Serial.printf("Record Number: %d, BytesWritten: %d\n", iRecord, byteCount);
        if(bytesRead < 0) break;
      }  // if the file isn't open, pop up an error:
  }
  myFile.close();
  dataFile.close();

  Serial.println("Files Closed");
}

KurtE · Nov 2, 2021

Still not making much progress on figuring out where things are getting corrupted... When you boil it down, the send object of this one file more or less logically does:

Read in 43000 512 byte USB packets. It then stuffs more or less 4 of them at a time into an output buffer,
that we then do a write out to the destination file so about 10750 2k writes) Note: My bogus FS puts in 3ms delay per write, next try 2 and 1...

So far I have created a dummy FS, that more or less does nothing, so does not feel like that part is the issue. The copy of 4 reads into one buffer to do a write, I put in sandbox code to make sure that the memcpy calls do not extend outside of the buffer... So that sort of leaves us with the USB code...

The usb_mtp.c code is more or less an exact copy of the usb_rawhid.c (in cores/teensy4). Minor differences on it allows for the 512 byte transfers.

I simplified some of the code calling it, but the code simply calls:
usb_mtp_recv code with fixed buffer and timeout.

And again this code is more or less identical to rawhid: except it is possible that the size of buffer we allocate per transfer (512), may or may not be the same size we receive. Depends on which endpoint ... is connected (512 or 64)... Note: The MTP code does not check to see which one is actually connected...

Wondering now how to best test.

Note: I did test the code I put up last, on both Windows 10 and Ubuntu... both either rebooted or sometime incomplete transfers. Sometimes at the very end, sometimes not very far into the transfer.

Feels like some minor time window where things can get corrupted.

Edit what happens if I try to send that many packets over RAWHID

wwatson · Nov 2, 2021

@KurtE - All I had time for tonight was to try increasing the size of the buffer in MTP_Teensy.h:

Code:

  static const uint32_t DISK_BUFFER_SIZE = 32 * 1024;

After that I had consistent transfers from Linux to two of my older Kingston USB sticks and my 120G SSD drive. Copying between USB devices is still a little sketchy. I was copying a 103Meg wav file. In the past I experimenting with different USB storage devices I learned that the size of the internal buffering for the device really determines the size of the external buffering needed. Did not have time to play with with different size buffers. Also, there is the matter of internal overhead within the device. With that you can get a USB pipe stall until the device is finished it's internal work.

Nothing is ever simple I guess

mjs513 · Nov 2, 2021

@Paul, @KurtE, @wwatson, @defragster

Spent the day on and off playing around with timeouts and rx recv timeouts in usb_mtp.c to no avail. As well as changing the tx_ and rx_ size in usb_desc.h. Always dies at the end. In addition when comparing the file on the Jump Drive and the file from the PC I am seeing missing lines (usually 3 or 4) and the last lines repeating. In addition after the transfer to the PC which is successful tend to loose ability to do anything else in MTP.

@wwatson did try changing the disk buffer size for the jump drive I am using but that did not work either still resulted in missing packets. Oh and before I forget again the 2x1024 for buffer size works for flash - anything higher causes issues

Moving to the T3.6 which does not use HS USB and send_object is handled a bit differently but the transfer completes with issue or hanging MTP.

EDIT: The missing packets always seem to be towards the beginning of the transfer then nothing at the larger indicies. Kind of reminds me of the issues with the Serial issues at 480 but in reverse.

defragster · Nov 3, 2021

Another slow day here ... there is an update to TyComm to allow push or not push date on upload.

mjs513 said:
...

Moving to the T3.6 which does not use HS USB and send_object is handled a bit differently but the transfer completes with issue or hanging MTP.

EDIT: The missing packets always seem to be towards the beginning of the transfer then nothing at the larger indicies. Kind of reminds me of the issues with the Serial issues at 480 but in reverse.

@mjs513 - is this the desired english from p#434? : completes with issue or hanging MTP
> expected to read 'without' ???

-> Also I understood the T_3.6 USB Host port was 480Mbs HS USB hardware, unlike the 12Mbs device port? Is it purposely being run slower, perhaps from shared USB core device code?

And the 'EDIT':
> in looking at the 'Serial issues' : if that refers to notes on the Beta 1.56 threads?

The last I saw/posted was if the Teensy starts before SerMon goes online - the Windows already has 10 or 100 MB buffered before SerMon can fully connect.
> and it seemed almost WORSE IIRC when SerMon is Ready at the start as it start pulling/processing data right away. It takes a long time for the .EXE to get a valid set up transmissions.
Not sure which is the worst/cause {early or late start} - but screen garbage in first few seconds and the lines/sec count bounces around really low at the start as well.
> and the serialTest__.exe code IIRC shows more confusion on starting when there isn't a uniform Windows backlog buffer.
-> I could test/confirm/observe this in a short time ... but not just now ... Zzzz's.

> It is almost like Windows has to 'change gears' to prepare for a data deluge and it 'drops the ball' when it also starts piping it out at the quickly before resources are ready?

This is anecdotal and 'recollection based' since I haven't played with it in a week since the last posts (TD 1.56 beta 1 thread) when those things became apparent - and 'anecdotal' from 'observing garbage' after the fact: >> First discard 115353600 Bytes. Discarded 230,752,256 total Bytes. << 230MB DISCARDED before the EXE enters the processing loop()!
-> that ino makes 3.8MB about 6 or 7 times per second when 'connected'/reading, and even starting reading ASAP on the Teensy start sometimes results in unparse-able data for some time.
>> b#1post #266 shows piping the data to a file resulted in the first 108 blocks of data - there were loses until it got synced and then was loss free.

There is some general indication that 'losses early are typical and true' when a HS link is established to Windows.

@KurtE: re p#432 - if this is a related real thing ... perhaps start the first few packets with larger delay {long enough to not quite time out} ... then reduce the delay once the process is up to speed?
> what is the max 'keep alive' timeout on Windows before it decides the device is non responsive?
> Does linux do this without the errors that Windows shows?

>> As noted on Windows 11 I am seeing 5 or more seconds for File Explorer to refresh the 'folder view'!
- there 'should be' no hardware reason for this as I'm on a fast SSD on a fast machine
- Today copying contents of updated TyComm over the old goes normally asking to replace existing files - then an 'interminably' long wait for refresh to click 'tycommander.exe' to properties/"unblock" the exe so it will run after pulling from the zip file.
-> This may be only Win 11 doing double duty if I read correctly - they scan the folder for the Win 11 updated view and also again for the legacy Win 10 elements
-> but the common element may be the overhead of the file create/allocate process ...

mjs513 · Nov 3, 2021

@defragster

@mjs513 - is this the desired english from p#434? : completes with issue or hanging MTP
> expected to read 'without' ???

You are correct - T3.6 transfers complete without issues or hanging MTP.

In regards to the edit, that was just an observation and a comparison to similar issues that we saw before. Not necessarily the same issue.

Out of curiosity I modified that copSD2flash sketch to copy the indexBlockFile to the USB Drive as opposed to the flash drive. In this case no missed blocks were seen until the end when the last about 127K bytes failed to be written to the Jump Drive:

Note: the file on the left is the master file which has 42,999 blocks of index lines and the file on the left is from the jump drive after sketch completes

Seems to be real as the file on the jump drive is 20.8megs and the master file is 20.9megs. Since its really early for me not sure if I messed up the sketch:

Code:

#include <SD.h>
#include <SPI.h>
#include <MTP_Teensy.h>
#include <USBHost_t36.h>
#include <USBHost_ms.h>


// Add in MTPD objects
MTPStorage storage;
MTPD mtpd(&storage);

// Add USBHost objectsUsbFs
USBHost myusb;
USBHub hub1(myusb);
USBHub hub2(myusb);
USBHub hub(myusb);

// MSC objects.
msController drive1(myusb);
msController drive2(myusb);
msController drive3(myusb);

msFilesystem msFS1(myusb);
msFilesystem msFS2(myusb);
msFilesystem msFS3(myusb);
msFilesystem msFS4(myusb);
msFilesystem msFS5(myusb);

// Quick and dirty
msFilesystem *pmsFS[] = {&msFS1, &msFS2, &msFS3, &msFS4, &msFS5};
#define CNT_MSC  (sizeof(pmsFS)/sizeof(pmsFS[0]))
uint32_t pmsfs_store_ids[CNT_MSC] = {0xFFFFFFFFUL, 0xFFFFFFFFUL, 0xFFFFFFFFUL, 0xFFFFFFFFUL, 0xFFFFFFFFUL};
char  pmsFS_display_name[CNT_MSC][20];

msController *pdrives[] {&drive1, &drive2, &drive3};
#define CNT_DRIVES  (sizeof(pdrives)/sizeof(pdrives[0]))
bool drive_previous_connected[CNT_DRIVES] = {false, false, false};

FS *mscDisk;

File myFile, dataFile;
int iRecord = 0;
int line = 0;

const int chipSelect = BUILTIN_SDCARD;
#define BUFFER_SIZE_INDEX 128
uint8_t write_buffer[BUFFER_SIZE_INDEX];
#define buffer_mult 4
uint8_t buffer_temp[buffer_mult*BUFFER_SIZE_INDEX];

int bytesRead;

void setup()
{
  while (!Serial && millis() < 5000);
  Serial.begin(9600);

  if (CrashReport) {
    Serial.print(CrashReport);
  }
  Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);
  delay(3000);
  Serial.print("Initializing SD card...");

  if (!SD.begin(chipSelect)) {
    Serial.println("SD initialization failed!");
    return;
  }
  Serial.println("SD initialization done.");

  mtpd.begin();
  
  myusb.begin();
  Serial.print("Initializing MSC Drives ...");
  Serial.println("\nInitializing USB MSC drives...");
  Serial.println("MSC and MTP initialized.");
  checkMSCChanges();

  menu();
  
}

void loop() {
  checkMSCChanges();
  mtpd.loop();
  
  if (Serial.available()) {
    uint8_t command = Serial.read();
    switch (command) {
    case 'w':
      writeIndexFile();
      break;
    case 't':
      copySD2Flash();
      break;
    case '\r':
    case '\n':
    case 'h':
      menu();
      break;
    default:
      menu();
      break;
    }
    while (Serial.read() != -1)
      ; // remove rest of characters.
  }
  
}

void menu() {
  Serial.println();
  Serial.println("Menu Options:");
  Serial.println("\tw - Write Index File to SD card");
  Serial.println("\tt - Transfer Index File from SD to Flash");
  Serial.println("\th - Menu");
  Serial.println();
}

void writeIndexFile() 
{
  // open the file.
  Serial.println("Write Large Index File");
  myFile = SD.open("LargeIndexedTestfile.txt", FILE_WRITE_BEGIN);
  if (myFile) {
    myFile.truncate(); // Make sure we wipe out whatever was written earlier
    for (uint32_t i = 0; i < 43000*4; i++) {
      memset(write_buffer, 'A'+ (i & 0xf), sizeof(write_buffer));
      myFile.printf("%06u ", i >> 2);  // 4 per physical buffer
      myFile.write(write_buffer, i? 120 : 120-12); // first buffer has other data...
      myFile.printf("\n");
    }
    myFile.close();
    Serial.println("\ndone.");
  }
}

void copySD2Flash()
{
  // open the file on SD Card.
  Serial.println("Copy Large Index File from SD to SPI Flash");
  myFile = SD.open("LargeIndexedTestfile.txt", FILE_READ);

  //open file on flash
  mscDisk = storage.getStoreFS(0);
  dataFile = mscDisk->open("LITF.txt", FILE_WRITE_BEGIN);

  dataFile.truncate();
  if (myFile) {
    Serial.println("Files Writing");
    while (myFile.available()) {
        memset(write_buffer, 0, sizeof(write_buffer));
        memset(buffer_temp, 0, sizeof(buffer_temp));
    uint16_t byteCount = 0;
        for(uint8_t i=0; i < buffer_mult; i++) {
          bytesRead = myFile.read(write_buffer, line? BUFFER_SIZE_INDEX : BUFFER_SIZE_INDEX-12); 
          if(bytesRead > 0) {
            for(int j = 0; j < (line? BUFFER_SIZE_INDEX : BUFFER_SIZE_INDEX-12); j++) {
              buffer_temp[byteCount+j] = write_buffer[j];
            }
          }
      byteCount += (line? BUFFER_SIZE_INDEX : BUFFER_SIZE_INDEX-12);
      line += 1;
        }
        dataFile.write((void *)buffer_temp, byteCount);
        
        iRecord += 1;
        Serial.printf("Record Number: %d, BytesWritten: %d\n", iRecord, byteCount);
        if(bytesRead < 0) break;
      }  // if the file isn't open, pop up an error:
  }
  myFile.close();
  dataFile.flush();
  dataFile.close();

  Serial.println("Files Closed");
}


void checkMSCChanges() {
  myusb.Task();

  USBMSCDevice mscDrive;
  PFsLib pfsLIB;
  for (uint8_t i=0; i < CNT_DRIVES; i++) {
    if (*pdrives[i]) {
      if (!drive_previous_connected[i]) {
        if (mscDrive.begin(pdrives[i])) {
          Serial.printf("\nUSB Drive: %u connected\n", i);
          pfsLIB.mbrDmp(&mscDrive, (uint32_t)-1, Serial);
          Serial.printf("\nTry Partition list");
          pfsLIB.listPartitions(&mscDrive, Serial);
          drive_previous_connected[i] = true;
        }
      }
    } else {
      drive_previous_connected[i] = false;
    }
  }
  bool send_device_reset = false;
  for (uint8_t i = 0; i < CNT_MSC; i++) {
    if (*pmsFS[i] && (pmsfs_store_ids[i] == 0xFFFFFFFFUL)) {
      Serial.printf("Found new Volume:%u\n", i); Serial.flush();
      // Lets see if we can get the volume label:
      char volName[20];
      if (pmsFS[i]->mscfs.getVolumeLabel(volName, sizeof(volName)))
        snprintf(pmsFS_display_name[i], sizeof(pmsFS_display_name[i]), "MSC%d-%s", i, volName);
      else
        snprintf(pmsFS_display_name[i], sizeof(pmsFS_display_name[i]), "MSC%d", i);
      pmsfs_store_ids[i] = storage.addFilesystem(*pmsFS[i], pmsFS_display_name[i]);

      // Try to send store added. if > 0 it went through = 0 stores have not been enumerated
      if (mtpd.send_StoreAddedEvent(pmsfs_store_ids[i]) < 0) send_device_reset = true;
    }
    // Or did volume go away?
    else if ((pmsfs_store_ids[i] != 0xFFFFFFFFUL) && !*pmsFS[i] ) {
      if (mtpd.send_StoreRemovedEvent(pmsfs_store_ids[i]) < 0) send_device_reset = true;
      storage.removeFilesystem(pmsfs_store_ids[i]);
      // Try to send store added. if > 0 it went through = 0 stores have not been enumerated
      pmsfs_store_ids[i] = 0xFFFFFFFFUL;
    }
  }
  if (send_device_reset) mtpd.send_DeviceResetEvent();
}

KurtE · Nov 3, 2021

And now for something completely different...

As I am thinking there are some underlying USB issues and the usb_mtp code is based off of the usb_rawhid code.

Wondering if it is time to take a diversion and play with RawHid?

But I wonder does anyone know of some simple RAWHID code for a PC, that hopefully works on windows, with hopefully simple support within Visual Studio?

Or am I currently better off for now seeing if example code works on Ubuntu?

That is would like to see if we can replicate issue with existing released pieces... Note: AFAIK the current RawHid is limited to 64 bytes... Wonder how hard to extend to have
something like RAWHID512?

The test should be pretty simple: Try to send enough packets for something like 22mb of data... Have each record contain a sequence number and have simple sketch on Teensy that receives the packets and checks the sequence numbers, and see if it can run long enough without errors. And starting adding delays on receiving some/all of the packets and see if that makes a difference...

If this fails to reliably receive the data, then it probably shows that the some/all of the MTP problems are related...

Make Sense?

KurtE · Nov 3, 2021

KurtE said:
And now for something completely different...

As I am thinking there are some underlying USB issues and the usb_mtp code is based off of the usb_rawhid code.

Wondering if it is time to take a diversion and play with RawHid?

Make Sense?

Ok been playing on Ubuntu... hacked up a mtp host example sketch that sends like the equivalent of a 22mb file...

Code:

#include <stdio.h>
#include <stdlib.h>
#include <stdarg.h>
#include <stdint.h>
#include <string.h>

#if defined(OS_LINUX) || defined(OS_MACOSX)
#include <sys/ioctl.h>
#include <termios.h>
#elif defined(OS_WINDOWS)
#include <conio.h>
#endif

#include "hid.h"


static char get_keystroke(void);

const uint32_t COUNT_PACKETS = 350000;

int main()
{
	int r;
	char buf[64];

	// C-based example is 16C0:0480:FFAB:0200
	r = rawhid_open(1, 0x16C0, 0x0480, 0xFFAB, 0x0200);
	if (r <= 0) {
		// Arduino-based example is 16C0:0486:FFAB:0200
		r = rawhid_open(1, 0x16C0, 0x0486, 0xFFAB, 0x0200);
		if (r <= 0) {
			printf("no rawhid device found\n");
			return -1;
		}
	}
	printf("found rawhid device\n");

	printf("Starting output\n");

	for (uint32_t packet_num = 0; packet_num < COUNT_PACKETS; packet_num++){
		memset(buf, 'A' + (packet_num & 0xf), 64) ;
		sprintf(buf, "%07u", packet_num);
		buf[7] = (packet_num == (COUNT_PACKETS-1))? '$' : ' ';
		buf[63] = '\n';
		rawhid_send(0, buf, 64, 100);
		if ((packet_num & 0x1ff) == 0) printf(".");
		if ((packet_num & 0xffff) == 0) printf("\n");
	}
	printf("\nDone...\n");
	return 0;
}

Changed Makefile to build this one instead...

Have Arduino sketch to talk to it. That checks the serial numbers... So far it crawls without crashing on new lockable T4RC that has not been locked... And has not shown errors... But runs REAL SLOW... And I have not put in any delays to emulate the time writing the data to files would take.

Code:

/* Basic Raw HID Example
  Teensy can send/receive 64 byte packets with a
  dedicated program running on a PC or Mac.

  You must select Raw HID from the "Tools > USB Type" menu

  Optional: LEDs should be connected to pins 0-7,
  and analog signals to the analog inputs.

  This example code is in the public domain.
*/

// RawHID packets are always 64 bytes
byte buffer[64];
unsigned int packetCount = 0;
uint32_t last_packet_number = (uint32_t) - 1;
uint32_t packet_count = 0;
elapsedMillis em;
elapsedMillis emTotal;
bool run_active = false;

void setup() {
  while (!Serial);
  pinMode(13, OUTPUT);
  Serial.begin(9600);
  Serial.println(F("RawHID lots of input test"));
  em = 0;
  Serial.println(F("Waiting for packets"));
}


void loop() {
  int n;
  n = RawHID.recv(buffer, 0); // 0 timeout = do not wait
  if (n > 0) {
    packet_count++;
    digitalToggleFast(13);
    // check Serial numbers
    uint32_t packet_number = 0;
    for (int i = 0; buffer[i] >= '0' && buffer[i] <= '9'; i++) {
      packet_number = packet_number * 10 + buffer[i] - '0';
    }
    if (packet_number == 0) {
      Serial.println("Looks like new run started");
      last_packet_number = 0;
      packet_count = 1;
      emTotal = 0;
    } else if (packet_number != (last_packet_number + 1)) {
      Serial.printf("Missing? cnt: %u, Last: %u cur:%u\n", packet_count, packet_number, last_packet_number);
    }
    if (buffer[7] == '$') {
      Serial.printf("Received end marker: %u %u Time:%u\n", packet_count, packet_number,
        (uint32_t)emTotal);
    }
    last_packet_number = packet_number;
    run_active = true;
    em = 0;
    if ((packet_count & 0x3ff) == 0) Serial.print(".");
    if ((packet_count & 0xffff) == 0) Serial.println();
  } else if (run_active && (em > 1000)) {
    Serial.printf("\nTimeout: %u %u\n", packet_count, last_packet_number);
    run_active = false;
    
  }
}

Here shows running the linux app a couple times:

Code:

Looks like new run started
................................................................
................................................................
................................................................
................................................................
................................................................
.....................Received end marker: 350000 349999 Time:88042

Timeout: 350000 349999
Looks like new run started
................................................................
................................................................
................................................................
................................................................
................................................................
.....................Received end marker: 350000 349999 Time:87917

Timeout: 350000 349999

So it took around 88 seconds to send the data...

Question to self and others... Wonder if possible to make a RAWHID512 and see what it would do?

Or maybe barking up wrong tree...

defragster · Nov 3, 2021

There were notes/questions about RAWHID512 some month back with some links ...

The TeensyTransfer used RAWHID reliably for PC transfers - in the T_3.x timeframe ... was that recently updated for T_4.x's or perhaps not? - Frank B made some note ...

(one) POST :: Fast-and-reliable-data-exchange-between-Raspberry-pi-and-teensy-4-1

KurtE · Nov 3, 2021

defragster said:
There were notes/questions about RAWHID512 some month back with some links ...

The TeensyTransfer used RAWHID reliably for PC transfers - in the T_3.x timeframe ... was that recently updated for T_4.x's or perhaps not? - Frank B made some note ...

(one) POST :: Fast-and-reliable-data-exchange-between-Raspberry-pi-and-teensy-4-1

Thanks @defragster -I don't see anything in that thread about using RawHid except for a post by me wondering about it...

at this point it is more of an exploration, to say wonder where things are getting hung in MTP... Will see how hard it will be to add 512 support (to play with...) And see what that does.

defragster · Nov 3, 2021

KurtE said:
Thanks @defragster -I don't see anything in that thread about using RawHid except for a post by me wondering about it...

at this point it is more of an exploration, to say wonder where things are getting hung in MTP... Will see how hard it will be to add 512 support (to play with...) And see what that does.

Not sure if that was the only thread - just remembered there was one ... then saw you tagged it too ... but I only looked at first result - seems that may be the only one ...

KurtE · Nov 3, 2021

defragster said:
Not sure if that was the only thread - just remembered there was one ... then saw you tagged it too ... but I only looked at first result - seems that may be the only one ...

Thanks,

Question to self and others... The T4 MTP code sets ups
that if the USB is running at speed(480) it uses RX/TX size of 512, but if running USB at lower rate... 64... Yet the code assumes 512...

So how does user or library code detect at what speed it is running...

EDIT: Looks like code uses the magic variable: extern volatile uint8_t usb_high_speed;

EDIT2: In teensy4 cores: MTP does look at this:

Code:

void usb_mtp_configure(void)
{
	printf("usb_mtp_configure\n");
    	if (usb_high_speed) {
		tx_packet_size = MTP_TX_SIZE_480;
		rx_packet_size = MTP_RX_SIZE_480;
	} else {
		tx_packet_size = MTP_TX_SIZE_12;
		rx_packet_size = MTP_RX_SIZE_12;
	}

However there is nothing exported from this that the MTP_teensy can look at...
Will probably add function/ methods to usb_mtp.h

like usb_mtp_rxSize()
...

Right now all code for t4s assume 512...

mjs513 · Nov 3, 2021

KurtE said:
Thanks,

Question to self and others... The T4 MTP code sets ups
that if the USB is running at speed(480) it uses RX/TX size of 512, but if running USB at lower rate... 64... Yet the code assumes 512...

So how does user or library code detect at what speed it is running...

EDIT: Looks like code uses the magic variable: extern volatile uint8_t usb_high_speed;

EDIT2: In teensy4 cores: MTP does look at this:

Code:

void usb_mtp_configure(void) { printf("usb_mtp_configure\n"); if (usb_high_speed) { tx_packet_size = MTP_TX_SIZE_480; rx_packet_size = MTP_RX_SIZE_480; } else { tx_packet_size = MTP_TX_SIZE_12; rx_packet_size = MTP_RX_SIZE_12; }

However there is nothing exported from this that the MTP_teensy can look at...
Will probably add function/ methods to usb_mtp.h

like usb_mtp_rxSize()
...

Right now all code for t4s assume 512...

That was pretty much what I was looking at yesterday - did try to change the RX/TX size in usb_desc.h for MTP with no real luck. Tried 128 and 64 but same thing occurred. Know it changed the sizes cause I saw it when dumping the data from send_object.

EDIT: Oh - never could find where usb_high_speed is actually set.

defragster · Nov 3, 2021

@KurtE - Interesting speed/buffer detail on MTP. Only been using 1062's here ...

@KurtE & @mjs513 : have you been using the updated TSET with the (final question) "T" option to build and upload from SublimeText using Teensy.exe loader when TyComm is active? It sets up to prompt for the desired Teensy then does '--delegate' to have Teensy.exe handle the upload. I just tried it again here and it works with a locked Beta T_4.1 where TyComm is NOT integrated to the IDE. Now I can close 8 open IDE windows ...

KurtE · Nov 3, 2021

Playing a little with a version of RAWHID512... Hard coded in the usb_desc.h with #define for 480 mode...

WIP up at https://github.com/KurtE/cores/tree/rawhid_512

Added methods to RawHid class to get the rx and tx sizes...

Updated test sketch and linux app..

As expected 8 times faster:

Code:

RawHID lots of input test
RawHid RX Size: 512
Waiting for packets
Looks like new run started
..........................................Received end marker: 43750 43749 Time:10974

Timeout: 43750 43749
Looks like new run started
..........................................Received end marker: 43750 43749 Time:10946

Timeout: 43750 43749

Will now insert to delays...

Not sure if this will go anywhere except for exercise. But may hack boards.txt to add USB type=RawHid512... which will be same except a couple of defines...

Again if goes anywhere will add the query for rx size and tx size to teensy3 although they will always return 64...

KurtE · Nov 3, 2021

defragster said:
@KurtE - Interesting speed/buffer detail on MTP. Only been using 1062's here ...

@KurtE & @mjs513 : have you been using the updated TSET with the (final question) "T" option to build and upload from SublimeText using Teensy.exe loader when TyComm is active? It sets up to prompt for the desired Teensy then does '--delegate' to have Teensy.exe handle the upload. I just tried it again here and it works with a locked Beta T_4.1 where TyComm is NOT integrated to the IDE. Now I can close 8 open IDE windows ...

I forgot to mention yes, I do use the updted TSET with the T option and working pretty well... I also have the updated release of Tycommander with slightly different icon...

wwatson · Nov 3, 2021

@KurtE - Here's where I step off and watch

defragster · Nov 3, 2021

So bumping RawHid 8x packets transfers without issue. How does that speed compare to 'other' USB?

Kurt - great to know TSET 't' is working there too.

Yesterday's TyComm updated today - not sure when that icon changed - but it is different.
> Timed the PASTE of files from the ZIP - 15 seconds until Win11 finished the redraw after the copy to do the properties/Unblock
> Also 15 seconds to cut ZIP from HDD downloads and paste onto SSD TyComm folder

mjs513 · Nov 4, 2021

@KurtE
Nice job on getting Rawhid512 working. May come in handy for other projects.

@defragster - not using TSET right now - trying to stick with the IDE until beta is over for locked Teensies and MTP stuff.

Was playing some more this morning with the USB/MTP and did a minor hack on send_object just to prevent the incomplete transfer messages and see what happens with the transfer.

Bottom line is that its loosing at 3 packets towards the beginning of the transfer (this time around it was within the first 1000 blocks of 512byte transfers). Think if maybe the 480 transfer on the usb host port is slowed a bit maybe we won't loose those missing packets.

Nothing else I tried seemed to make a difference - either got more missing packet or hung MTP.

So at this point just going to do some other testing for now.

KurtE · Nov 5, 2021

Morning all - Having fun with Rawhid... So far my Rawhid tests have not died/hung the same way... Will play more.

In the Mean time, will try a few things with the:
Again now compare exactly the configuration differences for RAWHID and MTP...

For example: In the usb_desc.c file:
For RAWHID - the rx section:

Code:

        // endpoint descriptor, USB spec 9.6.6, page 269-271, Table 9-13
        7,                                      // bLength
        5,                                      // bDescriptorType
        RAWHID_RX_ENDPOINT,                     // bEndpointAddress
        0x03,                                   // bmAttributes (0x03=intr)
        LSB(RAWHID_RX_SIZE_480),                // wMaxPacketSize 
        MSB(RAWHID_RX_SIZE_480),
        RAWHID_RX_INTERVAL,			// bInterval

But for MTP we have:

Code:

        0,                                      // bInterval
        // endpoint descriptor, USB spec 9.6.6, page 269-271, Table 9-13
        7,                                      // bLength
        5,                                      // bDescriptorType
        MTP_RX_ENDPOINT,                        // bEndpointAddress
        0x02,                                   // bmAttributes (0x02=bulk)
        LSB(MTP_RX_SIZE_480),MSB(MTP_RX_SIZE_480), // wMaxPacketSize
        0,                                      // bInterval

Note MTP is configured for Bulk and rawhid is configured for interrupt... Wonder if I change to interrupt... See what happens.

Also I am going to try a version of SendObject which goes back to basics, like the T3 version, receive a packet write the contents... And see if that makes any differences.
I doubt it as been running into this off and on from starting to play with MTP and I know I tried this earlier.

Edit - Actually the differences I showed for MTP versus rawhid are T3 version.

T4 are both setup as Interrupt on RX... Also wondering though on T4 RX/TX of MTP are one endpoint... On Rawhid it uses different endpoint for RX and TX... Is there is there a difference in behavior?

Teensyduino File System Integration, including MTP and MSC

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Well-known member

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Well-known member

Senior Member+

Senior Member+

Senior Member+