MTP Responder Contribution

demonstrating that the relevant snippet in usb.c is not working as expected.

You need a new copy of teensy_serialmon, which I haven't published yet. The current copy isn't transmitting the HID feature report when it opens the device. The new teensy_serialmon will be in 1.54-beta6.

I was planning to wait another week, but if you're needing to use this now, I could package up beta6 later today. Would that help?
 
You need a new copy of teensy_serialmon, which I haven't published yet. The current copy isn't transmitting the HID feature report when it opens the device. The new teensy_serialmon will be in 1.54-beta6.

I was planning to wait another week, but if you're needing to use this now, I could package up beta6 later today. Would that help?

Sure, we can wait. I will then remove the ugly hack from mtp-test.

Any suggestion on the MTPDISK_SERIAL discussion (dedicated PID)?
 
You need a new copy of teensy_serialmon, which I haven't published yet. The current copy isn't transmitting the HID feature report when it opens the device. The new teensy_serialmon will be in 1.54-beta6.

I was planning to wait another week, but if you're needing to use this now, I could package up beta6 later today. Would that help?

FYI it also breaks other programs like TyCommander... Which just recently had a fix to allow the default Serial only objects to work again with the Special Interface...

I will kill off the branch that used the interface numbers in the same order as the interfaces are defined in the USB header, appeared to work. What actually makes it output is a call to something like Serial.read();
Example in sketch:
Code:
void setup() {
  while (!Serial && millis() < 5000) ;
  Serial.println("MTP_test");
  Serial.println(millis());
  Serial.println(usb_configuration);
  Serial.println(usb_seremu_online);
  pinMode(13, OUTPUT);
}
void loop() {
  digitalToggleFast(13);
  delay(250);
  Serial.read();
  Serial.printf("%d %d\n", usb_configuration, usb_seremu_online);
  
}
The first set of lines does not output, but the ones in loop do...
Code:
1 0
1 0
1 0
 
@all - Hoping everyone is doing well...

Been bouncing back and forth between MTP and LittleFS stuff(https://forum.pjrc.com/threads/58033-LittleFS-port-to-Teensy-SPIFlash?p=262697#post262697)

Hopefully trying to understand and maybe help resolve a couple of issues. Like why LittleFS on my QSPI chips is maybe a couple hundred times slower than SDIO...

But also why when copying larger files, to them on a couple of my boards something like 300+KB files ends up not completing the write of the file, and then you here the windows Sound of Drive going away and it no longer works, until restarted...

I believe these are maybe two distinct issues. What I have been seeing is that I believe windows has a timeout when it is trying to send data and the system does not read in any of the USB buffers for some period of time. So I am thinking of doing an experiment to see what happens and see if it might help solve the issue and keep the data flowing.

Currently the code sort of works like:

Currently have one disk_buffer (8KB), that as long as we still have bytes to transfer, we wait for a RX buffer to be received. We copy it into our disk_buffer. When the disk buffer is full we call off
to the storage (File System) to do a write of the 8kb buffer and we wait for that write to complete before it returns. And on one of these writes we timeout...

What I am thinking about trying includes:

Instead of one disk_buffer we have two.

Before we start the size_t bytes_written = storage_->write((const char *)disk_buffer, DISK_BUFFER_SIZE);
call we setup a callback. For now probably an EventResponder yield callback, where currently littleFS calls yield when it is waiting for an operation like a write or a erase operation to complete.

Have this yield function, either at some interval of time or the like, check to see if it has any pending packets:!usb_mtp_available()
And read in one of these packets into the second buffer... When the write operation completes we disable this callback and allow the main loop to continue the filling of the next buffer.
Hopefully we can call it fast enough to avoid the timeouts, but slow enough like every 100ms or 200ms such that we should not fully complete filling it during the callback and have to figure out what to do then...

Does this make sense?
 
@KurtE
Not sure it is a simple timeout issue
Just copied 1.8 MB to 16 MB-SPI chip, it was transferring fine.
same file to 16 MB-QSPI chip failed after initial tranfer.
To me it seems a QSPI implementation issue, as QSPI should be faster than SPI, right?
 
It may or may not depend on which memory chip including which "build run" of it... Don't know the proper name, but the 3rd line number. The original T4.1 beta chips have a run number of the same product number and for copy of files and the like, the earlier chips are maybe a third faster?

Others have experimented more with these chips, but for example I have one of the newer WinBond larger chips PDF open and I believe the QSPI gains us very very little in these cases.
Why I believe this, is that the QSPI only speeds up the read and write operations, but in these cases, any speed gain is overshadowed by the time it takes to do Sector Erases.

Again I need to look at for example when is Write versus Page Program.

But Page program takes typical .7 to max 3.5ms to complete
As compared to Sector erase time of 4KB that takes typical 50ms and Max of 400ms to complete. And this is where I believe our time is being eaten...
 
Yes this is an inherent property of flash chips.
But the erasing can be paused or stopped when it runs, and in can run in the background when no flash-writes (and reads? not sure..) are happening.
SD-Cards have a little CPU on it that does this (and other things, like wear-leveling).
Maybe some kind of "task" that erases unused sectors?
But i guess this needs an other filesystem - or to fork/modifiy littlefs and stay with the version we now have - which means it is in 3 year too old..

A trim-like function (as the PC does it for SSDs) would be good, too.
 
So, I did another test,
what I have is 1 QSPI and 2 SPI chips
QSPI: EF-40-18 transfer fail
SPI: EF-70-18 transfer OK
SPI: EF-40-18 transfer failed

So it seems it is also chip dependent the (the *IM (DTR)) version worked while the *IQ version failed
 
Another observation I made (by chance)
It seems that LittleFS reads the data back after have written them to flash.
 
So, I did another test,
what I have is 1 QSPI and 2 SPI chips
QSPI: EF-40-18 transfer fail
SPI: EF-70-18 transfer OK
SPI: EF-40-18 transfer failed

So it seems it is also chip dependent the (the *IM (DTR)) version worked while the *IQ version failed
WHat are the exact Partnumbers of these chips? I can search for the Datasheets and compare.. :)
 
Just did another run on LittleFS - going to post after KurtE's link above.

All the FLASH chips are WINBOND - diff sizes and manufacture dates ... it seems the date code on them is YYWW :: Year and Week number of production. ANd alternate dates even the same part have diff perf.

Indeed as noted the QSPI { not } always { seeming } as much faster as it should be - when formatting comes in it hides the difference - but maybe more. I have code avoiding the formatting - when preFormatted that reduces that - but haven't done side by side SPI .vs. QSPI with that in mind - but tests run show the diff wasn't as significant as it might be.

Anyhow over to the LittleFS thread ... pjrc.com/threads/58033-LittleFS-port-to-Teensy-SPIFlash
 
Last edited:
MTP failure on Teensy3.2

MTP failure on Teensy3.2
Using the latest WMXZ-EU/MTP_t4 MTP Lib.
Using the mtp_test example.
I am having what I think is a disconnection issue. In Win Explorer I can see the SD and the files on it. I can copy one file at a time a couple of times then Explorer seems to lose the connection, the files are still the in the list but the file preview cannot show the preview any longer. The windows error message is:
"An unexpected error is keeping you from copying the file...
Error 0X8007065D: Data supplied is of wrong type."

Any ideas?
 
I have been hacking away, and have a version of the sendObject that reads in packets in a yield EventResponder object.

But still not working properly on my slow chip and larger files.

So added more detail information. It is hanging waiting for USB packet to be returned... So added more debug information and a quick and dirty timeout...
Where:
Code:
    int MTPD::pull_packet(uint8_t *data_buffer)
    {
      elapsedMillis em = 0; 
      while(!usb_mtp_available() && (em < 5000));
      if (em >= 5000) {
        printf("*** MTPD::pull_packet timeout ***\n");
        return -1; 
      }
      return usb_mtp_recv(data_buffer,60);
    }
before it would hang forever waiting on usb_mtp_available

Also in send object:
Code:
          if (pull_packet(cur_disk_buffer+disk_pos) != MTP_RX_SIZE) {
            printf("sendObject pullpacket fail\n");
            return false;
          }                 // read directly in.
Debug output. I have my yield callback output debug information.
A '.' every time called where time since last process > timeout (last one 5ms).
a '+' if there is a USB packet available for me to read.
and a '*' if my output buffer does not have enough room left to read in whole RX packet...

Debug output:
Code:
MTP_test
SDIO Storage 0 254 sdio 264289280 55229952
SD Storage 1 10 sd1 failed or missing
RAM Storage 0 RAM1 1999872 512
RAM Storage 1 RAM2 4000000 512
Program Storage 0 PROGM 983040 8192
QSPI Storage 0 QSPI 16777216 778240
SPIFlash Storage 0 3 nand1 failed or missing
SPIFlash Storage 1 4 nand2 failed or missing
SPIFlash Storage 2 5 nand3 failed or missing
SPIFlash Storage 3 6 nand4 8388608 651264

**** dir of sd[0] ****
overlays/
kernel7.img
mtpindex.dat
kernel.img
bcm2710-rpi-3-b-plus.dtb
test1.txt
kernel7l.img
kernel8.img
bcm2708-rpi-b-plus.dtb
zzz.jpg
issue.txt
IMG_1240.jpg
bcm2708-rpi-b.dtb
bcm2708-rpi-cm.dtb
bcm2708-rpi-zero-w.dtb
bcm2708-rpi-zero.dtb
bcm2709-rpi-2-b.dtb
bcm2710-rpi-2-b.dtb
start_cd.elf
start.elf
cmdline.txt
config.txt
bcm2710-rpi-3-b.dtb
bcm2710-rpi-cm3.dtb
bcm2711-rpi-4-b.dtb
COPYING.linux
T4.1-Cardlike.jpg
start_db.elf
start_x.elf
fixup.dat
fixup_cd.dat
fixup_db.dat
fixup_x.dat
bootcode.bin
start4.elf
start4cd.elf
start4db.elf
start4x.elf
fixup4.dat
T4-Cardlike.jpg
fixup4cd.dat
fixup4db.dat
fixup4x.dat
LICENCE.broadcom
Teensy-TTL-Level_converters.jpg

Setup done
1002 16 1 0:  1
1001 12 1 1: 
1014 16 1 2:  d402
1004 12 1 3: 
1005 16 1 4:  1
1005 16 1 5:  2
1005 16 1 6:  3
1005 16 1 7:  4
1005 16 1 8:  5
1005 16 1 9:  6
9801 16 1 10:  3000
9801 16 1 11:  3001
1007 24 1 12:  5 0 ffffffff
9803 20 1 13:  e dc02
9802 20 1 14:  dc03 3000
9803 20 1 15:  e dc03
9802 20 1 16:  dc04 3000
9803 20 1 17:  e dc04
1008 16 1 18:  e
9802 20 1 19:  dc01 3000
9803 20 1 20:  e dc01
9802 20 1 21:  dc07 3000
9803 20 1 22:  e dc07
9802 20 1 23:  dc41 3000
9803 20 1 24:  e dc41
9802 20 1 25:  dc44 3000
9803 20 1 26:  e dc44
9803 20 1 27:  d dc02
9803 20 1 28:  d dc03
9803 20 1 29:  d dc04
1008 16 1 30:  d
9803 20 1 31:  d dc01
9803 20 1 32:  d dc07
9803 20 1 33:  d dc41
9803 20 1 34:  d dc44
9803 20 1 35:  c dc02
9803 20 1 36:  c dc03
9803 20 1 37:  c dc04
1008 16 1 38:  c
9803 20 1 39:  c dc01
9803 20 1 40:  c dc07
9803 20 1 41:  c dc41
9803 20 1 42:  c dc44
9803 20 1 43:  b dc02
9803 20 1 44:  b dc03
9803 20 1 45:  b dc04
1008 16 1 46:  b
9803 20 1 47:  b dc01
9803 20 1 48:  b dc07
9803 20 1 49:  b dc41
9803 20 1 50:  b dc44
9803 20 1 51:  a dc02
9803 20 1 52:  a dc03
9803 20 1 53:  a dc04
1008 16 1 54:  a
9803 20 1 55:  a dc01
9803 20 1 56:  a dc07
9803 20 1 57:  a dc41
9803 20 1 58:  a dc44
9803 20 1 59:  9 dc02
9803 20 1 60:  9 dc03
9803 20 1 61:  9 dc04
1008 16 1 62:  9
9803 20 1 63:  9 dc01
9803 20 1 64:  9 dc07
9803 20 1 65:  9 dc41
9803 20 1 66:  9 dc44
9803 20 1 67:  8 dc02
9803 20 1 68:  8 dc03
9803 20 1 69:  8 dc04
1008 16 1 70:  8
9803 20 1 71:  8 dc01
9803 20 1 72:  8 dc07
9803 20 1 73:  8 dc41
9803 20 1 74:  8 dc44
9803 20 1 75:  7 dc02
9803 20 1 76:  7 dc03
9803 20 1 77:  7 dc04
1008 16 1 78:  7
9803 20 1 79:  7 dc01
9803 20 1 80:  7 dc07
9803 20 1 81:  7 dc41
9803 20 1 82:  7 dc44
9803 20 1 83:  6 dc02
9803 20 1 84:  6 dc03
9803 20 1 85:  6 dc04
1008 16 1 86:  6
9803 20 1 87:  6 dc01
9803 20 1 88:  6 dc07
9803 20 1 89:  6 dc41
9803 20 1 90:  6 dc44
100b 20 1 91:  a 0
100c 20 1 92:  5 ffffffff
100d 12 1 93: 
MTPD::SendObject: len:434010
EV-At.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 417638 142972
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 409446 101497
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 401254 101023
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 393062 99530
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 384870 102117
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 376678 101920
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 368486 99056
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 360294 101594
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 352102 99906
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 343910 101809
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 335718 101714
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 327526 96993
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 319334 99931
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 311142 99009
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 302950 98957
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 294758 102638
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 286566 100345
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 278374 99092
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 270182 101693
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 261990 99895
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 253798 99997
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 245606 99088
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 237414 99132
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 229222 102672
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 221030 99914
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 212838 99131
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 204646 99211
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 196454 97159
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+*WR 8192 8192 188262 99897
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.....WR 8192 8192 180582 96586
*** MTPD::pull_packet timeout ***
sendObject pullpacket fail
The WR parts are after each write to storage, with size passed in size written, how many bytes left to process and elapsedMicros around the call...

Still playing, current stuff in a my own fork/branch: https://github.com/KurtE/MTP_t4/tree/SendObject_yield

Edit Note: I am not then hung but I don't think it will allow me to do much else here. Not sure in case like this if we should close the file or the like.
 
I did some more hacking in the above version and I was able to get the 300+KB file copy to my slow QSPI flash... But a 3mb file still fails...

I made the buffer a large buffer 8 *8KB, and have my yield function do all of the reading... I have played with how many ms to wait between reads. I have it however not wait when I do not have a write operation to FS in progress as to try to fill up one buffer as quick as possible.

If anyone is again interested code is updated on my fork/branch.

Debug output slightly different: I have a run of three files. A 60+kb file, the 300+file that failed before and a 3+mb file that still fails.
If you look at the debug info, edited to remove some of the many many output lines and the like:
Code:
MTP_test
SDIO Storage 0 254 sdio 264289280 58487808
SD Storage 1 10 sd1 failed or missing
RAM Storage 0 RAM1 1999872 512
RAM Storage 1 RAM2 4000000 512
Program Storage 0 PROGM 983040 8192
QSPI Storage 0 QSPI 16777216 1867776
SPIFlash Storage 0 3 nand1 failed or missing
SPIFlash Storage 1 4 nand2 failed or missing
SPIFlash Storage 2 5 nand3 failed or missing
SPIFlash Storage 3 6 nand4 8388608 651264

100c 20 1 337:  5 ffffffff
100d 12 1 338: 
MTPD::SendObject: len:69245
EV-At..+..+..+..+..+..+..+..+..+..+..+..+..+..+..+..+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 48265 0 147060
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 40073 8192 98651
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 31881 16384 100853
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 23689 24576 101522
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 15497 32768 99252
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 7305 40960 98658
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+$WR 8192 8192 0 49152 98353
WR 8192 8192 0 57344 98929
WR 3709 0 65536 5668
CL 69245
9803 20 1 339:  44 dc02
9803 20 1 340:  44 dc01
9803 20 1 341:  44 dc07
9803 20 1 342:  44 dc0b
9803 20 1 343:  44 dc41
9803 20 1 344:  44 dc44
9803 20 1 345:  44 dc03
9803 20 1 346:  44 dc04
1008 16 1 347:  44
1005 16 1 348:  5
100b 20 1 349:  3b 0
100c 20 1 350:  5 ffffffff
100d 12 1 351: 
MTPD::SendObject: len:341024
EV-At..+..+..+..+..+..+..+..+..+..+..+..+..+..+..+..+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 320556 0 138461
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 312364 8192 98444
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 304172 16384 101169
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 295980 24576 100978
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 287788 32768 96884
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 279596 40960 101210
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 270892 49152 101382
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 262188 57344 102203
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 253996 65536 99729
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 245804 73728 101269
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 237612 81920 98689
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 228908 90112 102132
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 220716 98304 99486
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 212012 106496 102953
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 203308 114688 104152
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 195116 122880 99545
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 186924 131072 99994
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 178732 139264 101327
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 170028 147456 101392
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 161324 155648 102369
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 153132 163840 99619
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 144940 172032 98760
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 136236 180224 101253
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 127532 188416 102228
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 118828 196608 102140
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 110636 204800 98200
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 101932 212992 101697
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 93228 221184 101723
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 84524 229376 102504
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 75820 237568 102664
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 67628 245760 100509
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 59436 253952 100273
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 51244 262144 101088
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 43052 270336 101200
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 34860 278528 98694
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 26668 286720 101062
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 18476 294912 96831
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 9772 303104 102925
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 1580 311296 101350
.+.+.+.+$WR 8192 8192 0 319488 99734
WR 8192 8192 0 327680 102427
WR 5152 0 335872 52189
CL 341024
9803 20 1 352:  45 dc02
9803 20 1 353:  45 dc01
9803 20 1 354:  45 dc07
9803 20 1 355:  45 dc0b
9803 20 1 356:  45 dc41
9803 20 1 357:  45 dc44
9803 20 1 358:  45 dc03
9803 20 1 359:  45 dc04
1008 16 1 360:  45
1005 16 1 361:  5
100c 20 1 362:  5 ffffffff
100d 12 1 363: 
MTPD::SendObject: len:3280204
EV-At..+..+..+..+..+..+..+..+..+..+..+..+..+..+..+..+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3259736 0 139948
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3251032 8192 102352
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3242840 16384 101173
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3234648 24576 98677
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3226456 32768 95942
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3218264 40960 99595
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3209560 49152 102267
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3201368 57344 98594
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3192664 65536 101311
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3183960 73728 102223
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3175256 81920 103836
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3167064 90112 98685
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3158872 98304 100378
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3150168 106496 102432
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3141464 114688 102203
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3133272 122880 98782
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3124568 131072 101769
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3116376 139264 99761
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3108184 147456 99715
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3099480 155648 101487
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3091288 163840 101465
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3083096 172032 97135
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3074392 180224 102442
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3066200 188416 101495
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3057496 196608 101626
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3049304 204800 99818
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3040600 212992 102575
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3032408 221184 96254
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3024216 229376 101538
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3016024 237568 97176
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 3007320 245760 102216
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2998616 253952 103262
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2990424 262144 100888
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2982232 270336 97020
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2973528 278528 104006
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2965336 286720 96123
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2957144 294912 101380
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2948440 303104 102391
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2940248 311296 99828
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2932056 319488 101392
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2923864 327680 98775
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2915672 335872 99643
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2907480 344064 99804
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2898776 352256 101304
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2890584 360448 98585
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2881880 368640 102204
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2873688 376832 99466
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2865496 385024 100751
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2857304 393216 100002
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2848600 401408 101879
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2839896 409600 102353
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2831704 417792 101307
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2823512 425984 100710
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2815320 434176 100028
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2807128 442368 98985
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2798936 450560 98550
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2790744 458752 101452
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2782552 466944 99293
.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+.+WR 8192 8192 2774360 475136 99473
.+.+.+.+.+.+.+.+.+.+.+.+.+...WR 8192 8192 2767704 483328 101494
..WR 8192 8192 2767704 491520 100619
..WR 8192 8192 2767704 499712 102476

I edited out most of the repeated dots (more than 5 in row that starts at start of file and of failure when I wait and timeout after 5 or 10 seconds...

I also set an IO pin HIGH when start write and set it low when complete. I toggle another IO pin when I have USB packet to receive...
Capture of the second file (341024) bytes...
screenshot.jpg

And here is the one that failed.
screenshot2.jpg
As you can see it was reading in data packets all through this, but then it just stopped receiving any data... Then my code detected this I bailed out... But program/USB connection did not recover
 
Quick update: I wanted to see if I could reproduce the issue of longer files or the like causing Windows/MTP to stop communicating... with something simple that hopefully is more reproducible...

Also to eliminated the reliance on my slower version of the QSPI Flash.

So in my version of the mtp-test.ino I created a slow version of the littleFS ram...

So added #define at start...

Then if Defined:
Code:
#if USE_LFS_SLOW_RAM == 1 // like RAM but made to be SLOOOWWW
  const char *slfs_ram_str[]={"SLOW_RAM"};     // edit to reflect your configuration
  const int slfs_ram_size[] = {2'000'000}; // edit to reflect your configuration
  const int snfs_ram = sizeof(slfs_ram_str)/sizeof(const char *);

  class LittleFS_SLOW_RAM : public LittleFS_RAM
  {
  public:
    LittleFS_SLOW_RAM() : LittleFS_RAM() { }
    static uint32_t    WRITE_TIME_MS;  // time for writes to take. 

    bool begin(uint32_t size) {
      bool fRet = LittleFS_RAM::begin(size);

      // Lets overwrite the parent classes prog function
      parent_static_prog = config.prog; // remember the prevoius one. 
      config.prog = &static_prog;
      return fRet;
    }

    private:
      static int (*parent_static_prog)(const struct lfs_config *c, lfs_block_t block, lfs_off_t offset, const void *buffer, lfs_size_t size);
      static int static_prog(const struct lfs_config *c, lfs_block_t block,
            lfs_off_t offset, const void *buffer, lfs_size_t size) {
        elapsedMillis em = 0;
        int ret = (*parent_static_prog)(c, block, offset, buffer, size);
        int delay_time = (int)(WRITE_TIME_MS - em);
        if (delay_time > 0) delay(delay_time);
        return ret;
      }
  };

  uint32_t LittleFS_SLOW_RAM::WRITE_TIME_MS = 2;
  int (*LittleFS_SLOW_RAM::parent_static_prog)(const struct lfs_config *c, lfs_block_t block, lfs_off_t offset, const void *buffer, lfs_size_t size) = nullptr;
  LittleFS_SLOW_RAM sramfs[snfs_ram]; 
#endif
It makes sure that all PROG calls take at least 2ms to complete...

And section to add the FS...
Code:
    #if USE_LFS_SLOW_RAM==1
    for(int ii=0; ii<snfs_ram;ii++)
    {
      if(!sramfs[ii].begin(slfs_ram_size[ii])) 
      { Serial.printf("Slow Ram Storage %d %s failed or missing",ii,slfs_ram_str[ii]); Serial.println();
      }
      else
      {
        storage.addFilesystem(sramfs[ii], slfs_ram_str[ii]);
        uint64_t totalSize = sramfs[ii].totalSize();
        uint64_t usedSize  = sramfs[ii].usedSize();
        Serial.printf("Slow RAM Storage %d %s ",ii,slfs_ram_str[ii]); Serial.print(totalSize); Serial.print(" "); Serial.println(usedSize);
      }
    }
    #endif
I was able to do the 60+KB file with it. The 330+KB file failed...
 
@WMXZ - Now that some of the LittleFS stuff is starting to work reasonably well (with @defragster) changes to not always format sectors,

it will be interesting to see if some of the experiments I have done to for example use an idle callback to read in the next logical block of data while the previous one is being written out. My current stuff sets up 3 buffers for TX (with DMAMEM) Still lots to experiment with.

But then I was wondering about if/how I might use MTP in real programs. And for example I could see my never to be complete Well Monitor program, where I was logging data, it would be nice if I could simply plug in my portable and grab log files.

So then wondered about what a program should do to create a log file. So I started looking at your mtp-logger sketch which looks like fun and I started asking myself questions. And wondering your ideas and plans for having MTP and FS(s) know about each other.

Currently your sketch when logging does:
Code:
int16_t file_open(uint16_t store)
{ char filename[80];
  if(!makeFilename(filename)) return 0;
  if(!checkPath(store, filename)) return 0;
  mfile = sdx[store].open(filename,FILE_WRITE);
  return !(!mfile);
}
So in your case you are opening a file for write in your first SD drive... Where you may have several different logical drives associated with your MTP data...

I notice in your mSD_Base class you have public methods like:
Code:
    uint32_t sd_getFSCount(void) {return fsCount;}
    const char *sd_getFSName(uint32_t store) { return sd_name[store];}

    File sd_open(uint32_t store, const char *filename, uint32_t mode) { return sdx[store]->open(filename,mode);  }
    bool sd_mkdir(uint32_t store, char *filename) {  return sdx[store]->mkdir(filename);  }
    bool sd_rename(uint32_t store, char *oldfilename, char *newfilename) { return sdx[store]->rename(oldfilename,newfilename);  }
Which as we can see maps to sdx[store]->open(filename,mode);

And your class MTPStorage_SD : public MTPStorageInterface, mSD_Base

So a couple of questions: How does one know what store value to pass in? Logically I can imagine remembering how many Storage items I added (as long as some other library did not add their own)...

So wondering if: void addFilesystem(FS &fs, const char *name) { sd_addFilesystem(fs, name);}
should maybe not be void but uint32_t which returns the store (index into an array)...

And/OR - Add method to MTPStorage_SD to something like: uint32_t mapFSToStorage(FS *pfs);
And/Or - method to lookup by name: uint32_t mapNameToStorage(const char *name);

And/Or - add logical names to FS, like: FILE *open("sdio:/foo.txt", FILE_WRITE);

Side Note: Wondering if this class name and method names should be a little less SD related? As it now can be RAM/FLASH/PROGRAM/...

Also for the fun of it, I might play around a little with the functions like sd_open, and see what happens if I wrap them into a helper class:
like:
Code:
class MTPFile : public FILE {
    MTPFile(FILE &f) : _f = f {};
	virtual ~LittleFSFile() {close();}
	virtual size_t write(const void *buf, size_t size) {return _f.write(buf, size);}
    ...
      	virtual void close() {
            _f.close();
            <Notifiy MTP system about file>
        }
...
Again so far I don't know enough on how much additional information the class needs to hold onto, like the name of the file? Or maybe the open actually generates the appropriate index and storage index, such that can create an event or??? But could be interesting.

Make Sense?

Kurt
 
@KurtE,
A lot of suggestions.
Not sure if it is useful to map FS with MTP (or I misundertand).
The main program must know the disks attached to system for logging, and the MTP is not talking to main program but to PC.
The logging program has not to know how MTP enumerates the disks. At the same time the MTP is not a FS and does not need to know the underlaying FS.
Were to put the index file is a matter of convention. Maybe it is useful to have a dedicated call to assign a store for index file. I'm happy with the convention, that the first disk added will contain the index file.

Not sure how logic disk name can be implemented in SdFat. It is easy with uSdFs, because "Elm Chan" has setup the FS this way.
Anyhow, MTP does not need logic names, only the logger program.

That the naming of classes and some functions are sd centric, well that is history and could be changes.
Both Storage and MTP should not know the underlaying filesystem only FS.h

Obviously, we still lack the functionality to tell the PC to reset the MTP session without disconnecting the USB.
using the device manager works most of the time for me, but I hear others had problems
 
Note that it is not necessary to physically unplug the USB or use the Windows device manager to get the PC to update the MTP status. A call to usb_init() will reset the usb link and windows will treat this like a USB disconnect. The downside is that any program connecting to Serial also has to reconnect.
 
@mborgerson is there a way to detect the need to call usb_init()? Can the communication loss be trapped and then call the usb_init or is there an there another method of determining when to reinitialize?
 
Note that it is not necessary to physically unplug the USB or use the Windows device manager to get the PC to update the MTP status. A call to usb_init() will reset the usb link and windows will treat this like a USB disconnect. The downside is that any program connecting to Serial also has to reconnect.

This is an operation from within the Teensy. IMO, what is needed an operation on PC to reset session.
I have good results with using the device manager, but as I said, I hear that others have problems.
using Serial (via some sort of menu) to kill itself seems a weird approach. then I can also restart the whole program and have a better defined entry program.
 
Yes lots of questions/and wondering :D

Note: I can see several usage patterns, that may or not be useful to some or easy/doable.


a) MTP is just there as a bolt on. Sort of as you mentioned. That is it would be nice to get files to it and from it... Again very useful.

b) Some of the things I was asking about were ways for the users changes to underlying FS system show up in MTP... As you mentioned currently you have to more or less restart the Teensy to get new files created on the Teensy to show up.

So in your case with logger, you call: mfile = sdx[store].open(filename,FILE_WRITE);
Again if underlying FS had hooks that MTP could latch onto for system changes... Maybe you dont need to do anything. But until then if you could do something like:
mfile = storage.sd_open(store, filename,FILE_WRITE);
And the storage could then pass this data off to the underlying fs, and if it succeeds, wrap it and then do MTP specific things like add it to your node list send an event, and hold on to whatever extra data needed and when completes maybe send an event about change... Then maybe the file would automatically show up on the PCs if the user was watching that folder. Or maybe worst case you might have to do a refresh of the window.

c) I can also see cases where maybe I want to open up a logical configuration file on my PC and make changes, and when I do a save, my sketch on the Teensy would somehow get some form of FileChange notification that I might then reconfigure what my sketch is doing.

Example maybe my sketch is a python interpreter and the user just downloaded an updated python file. Note: I used to do things like back a few years ago when I was playing with Edison processors. Again I don't know at what level, you might get some notifications, like: by File? By Directory? By FS?

Again I see possibilities for people wanting a little functionality or a lot of functionality depending on the sketch.
 
This is an operation from within the Teensy. IMO, what is needed an operation on PC to reset session.
I have good results with using the device manager, but as I said, I hear that others have problems.
using Serial (via some sort of menu) to kill itself seems a weird approach. then I can also restart the whole program and have a better defined entry program.

I wondered what having a set of #define files presented with MTP might do?

They would be files or non files that the PC sees - or creates?

>> ResetMe.txt or UnusedFormatMe.txt or USBinitMe.txt

I thought of that in regard to the formatUnused space for LittleFS Flash media where writes to unformatted blocks slow the process down by a factor of 5.

If the Teensy knew the user was okay with a long wait to prepare for a series of writes but wanted to have the media prepared for a writes - either FROM the PC, or because the PC just offloaded some megabytes of logged data leaving the Flash blocks generally Dirty but not used where writes incur a ~25ms penalty per block (4K)

This is part of the LFSintegrity test sketch - but requires SerMon.

So if these 'special marker files' were presented and the PC deleted them or some overt recognizable action the Teensy could:
> format the media ( ONLY the UNUSED SPACE that has unformatted blocks ) which ( can take over 10 minutes on a dirty 64MB Flash - which would add 10 minutes to a disk filling write ) - then be ready for use - perhaps followed by a Teensy RESET or usb_init() to make sure the PC saw the media directory fresh and ready in its current state.
> RESET the Teensy for fun - just to be sure all media changes are in sync between PC and Teensy
> usb_init : drop and restore USB connect - just to be sure all media changes are in sync between PC and Teensy

>> The Teensy could present these files by default on ROOT or in a subdir? And maybe they wouldn't even be present on the media if the Teensy could support ghost files?
>> Or the Teensy could watch for the PC to attempt to create these files and respond accordingly - but then the 'PC USER' would need to know the #define names.
 
Back
Top