Teensyduino File System Integration, including MTP and MSC

Well thank you guys for the confirmation and thanks for the hell of a job you guys are doing here ;).
I am using this for easy file transfer between PC and my own build ATARI hard drive emulation based on Teensy 4.1 for Atari ST/e ACSI port this peripheral bus is very similiar to the industry standard SCSI (Small Computer Systems Interface) bus.

Sounds like fun...

@mjs513 got the fix in this morning, which I merged.


SCSI - The @wwatson stuff MSC stuff we are integrating into USBHost_t36 does use at least a subset of the SCSI command interface.

Now back to playing
 
Also mentioned on the github issue, these wrapper classes need to inherit either the LittleFS class or at the very least FS class.

This morning I'm working on an issue that came up with the pogo pins on the Teensy 4.0 test fixture, so I can't look at this right now. But if needed, I could dive into it this afternoon or evening.
 
<edit> Updated to latest PJRC/git LittleFS: copy of files from SD indicated below still shows 10 of 111 files 'different'.
> given the folder ZIP - if it repros for Kurt/Mike then it isn't a missing WIP issue.
Code:
Multiple libraries were found for "LittleFS.h"
 Used: C:\T_Drive\tCode\libraries\LittleFS
 Not used: C:\T_Drive\arduino-1.8.18\hardware\teensy\avr\libraries\LittleFS

Didn't pull latest LittleFS changes - but TD 1.56 b4 updated from KurtE github with:
- USBHost_t36
- MTP_Teensy
- SD
>> If this isn't the complete set of WIP libs needed please post.
Code:
Multiple libraries were found for "USBHost_t36.h"
 Used: C:\T_Drive\tCode\libraries\USBHost_t36
 Not used: C:\T_Drive\arduino-1.8.18\hardware\teensy\avr\libraries\USBHost_t36
Multiple libraries were found for "SD.h"
 Used: C:\T_Drive\tCode\libraries\SD
 Not used: C:\T_Drive\arduino-1.8.18\libraries\SD
 Not used: C:\T_Drive\arduino-1.8.18\hardware\teensy\avr\libraries\SD
[B]Using library SD at version 2.0.0 in folder: C:\T_Drive\tCode\libraries\SD[/B] 
Using library SdFat at version 2.1.0 in folder: C:\T_Drive\arduino-1.8.18\hardware\teensy\avr\libraries\SdFat 
Using library SPI at version 1.0 in folder: C:\T_Drive\arduino-1.8.18\hardware\teensy\avr\libraries\SPI 
[B]Using library MTP_Teensy at version 1.0.0 in folder: C:\T_Drive\tCode\libraries\MTP_Teensy 
Using library USBHost_t36 at version 0.1 in folder: C:\T_Drive\tCode\libraries\USBHost_t36[/B] 
Using library LittleFS at version 1.0.0 in folder: C:\T_Drive\arduino-1.8.18\hardware\teensy\avr\libraries\LittleFS

MTP on T_MM (lockable) - disabled SPI, FRAM and other unused:
Code:
Dump Storage list(2)
STORE:0 storage:10001 name:sd1 fs:2000d420
store:1 storage:20001 name:PROGM fs:2000ce28
NOTE: When unsupported/unpopulated FS items are included, it does not fail gracefully - the Teensy device simply fails to appear.

Code:
<edit> [B]Ref folder for indicated MemoryHexDump[/B] : [ATTACH]26934._xfImport[/ATTACH]

Default PROG was 1MB, SD 128GB. Copied github folder of "MemoryHexDump": 111 files in 79 folders reported size of 77.6 KB on Windows ( 136KB size on disk )
> Folder copy to SD - completed okay
> Folder copy to 1MB Prog - failed : assume it is a granularity issue with larger format blocks?

Increased T_MM PROG to 10MB:
> Copy completed - a ringing set of Win PC chimes suggesting errors in process
> Win Explorer shows 0 bytes free of 9.56 MB

> many things missing

COPY SD folder MemoryHexDump back to PC disk :: BAD : not read on SD to see if they are good there
> The copy on PC with CodeCompare shows a ten files that are truncated and/or corrupted, i.e. different
> on PC copy back folder:: 111 files in 79 folders reported size of 133 KB on Windows ( 468KB size on disk )

<edit>: Moved T_MM SD card to PC and copied files from SD to PC
> CodeCompare of files PC copied from the SD are 100% identical: MTP PC to Teensy worked properly
-> PC properties on that copied folder are the same: 111 files in 79 folders reported size of 77.6 KB on Windows ( 136KB size on disk )


COPY PROG folder MemoryHexDump back to PC disk :: "error copying file or folder // The library, drive, or media pool is empty."
> pc made folder MemoryHexDump and ONE empty subfolder ".git"

<edit> on the 'error copying' case:
> restarted with updated LittleFS and FORMAT from PC on the 10MB PROG drive worked.
> It copies much of the .git subfolder first before the error chimes when it runs out of space.
In the .git folder on PROG disk is "COMMIT_EDITMSG", and copying that file alone to PC folder gives this debug on SerMon:
Code:
LP:730092 CMD: 1009(GET_OBJECT)l: 16 T:11c2 : e2
730093 RESP:2001(RSP:OK)l: 16 T:11c2 : e2
LP:730094 CMD: 1009(GET_OBJECT)l: 16 T:11c3 : e2
730096 RESP:2001(RSP:OK)l: 16 T:11c3 : e2
 
Last edited:
Wrapper update: Github was updated to have classes be derived from FS...
Also SPI version now has ability to specify which SPI buss...

On a different subject, I remember wondering about what happens if you run USB at full speed: USB1_PORTSC1 |= USB_PORTSC1_PFSC; // force 12 Mbit/sec

It does bring up them MTP sketch...

Brings up the MTP folder... But trying to upload a file to a littleFS fails...

Code:
LP:40324 CMD: 100c(SEND_OBJECT_INFO)l: 20 T:2d : 40001 ffffffff
40324 DATA:100c(SEND_OBJECT_INFO)l: 174 T:2d : 0 3000 127ee6 3000 0
SendObjectInfo: 262145 4294967295 20004d60: ST:0 F:3000 0 SZ:1212134 3000 0 0 0 22c 221 20 0 0 0 0 : 
Created: 0
Modified: 0
OpenFileByIndex failed to open (22):/ mode: 2
MTPStorage::Create  failed to create file
40424 RESP:2020(RSP:SPECIFICATION_OF_DESTINATION_UNSUPPORTED)l: 24 T:2d : 40001 ffffffff ffffffff
LP:40424  UNKWN: 6300
 UNKWN: 6300
6900l: 1929400081 T:6c006c00 : 73006f00 6f006300 65007000 62002e00 70006d00
!!! unexpected/unknown message type:25344 len:1929400081 op:26880
LP:40424  UNKWN: 30
 UNKWN: 30
0l: 3014708 T:30003212 : 30003100 35003000 32003000 31005400 31003200
!!! unexpected/unknown message type:48 len:3014708 op:0
Time to debug...
 
Ok, I see at least one issue...

Suppose we to sendObject to a teensy store... Host will send us a SendObectInfo packet with some data...

Code:
uint32_t MTPD::SendObjectInfo(uint32_t storage, uint32_t parent,
                              int &object_id) {
  pull_packet(rx_data_buffer);
  read(0, 0); // resync read
  printContainer();
  printf("SendObjectInfo: %u %u %x: ", storage, parent,
         (uint32_t)rx_data_buffer);
  uint32_t store = Storage2Store(storage);

  int len = ReadMTPHeader();
  char filename[MAX_FILENAME_LEN];
  dtCreated_ = 0;
  dtModified_ = 0;

  printf("Len:%d ST:%x ", len, read32());
  len -= 4; // storage
  uint16_t oformat = read16();
  len -= 2; // format
  printf("F:%x ", oformat);
  bool dir = oformat == 0x3001;
Well suppose slightly modified output from previous post:
Code:
945998 RESP:2001(RSP:OK)l: 24 T:10 : 40001 0 ffffffff
LP:952735 CMD: 100c(SEND_OBJECT_INFO)l: 20 T:11 : 40001 ffffffff
952735 DATA:100c(SEND_OBJECT_INFO)l: 166 T:11 : 0 3000 3378c7 3000 0
SendObjectInfo: 262145 4294967295 20004d60: Len:154 ST:0 F:3000 0 SZ:3373255 3000 0 0 0 c00 900 18 0 0 0 0 : 
Created: 0
Modified: 0
OpenFileByIndex failed to open (20):/ mode: 2
MTPStorage::Create  failed to create file
952742 RESP:2020(RSP:SPECIFICATION_OF_DESTINATION_UNSUPPORTED)l: 24 T:11 : 40001 ffffffff ffffffff
So the length field at start says this command packet is 154 bytes long.

Now at High speed the packet we received with this data is 512 bytes long so no problem... But at full speed the packet is 64 bytes long. And the current T4 code is not setup to handle
this and simply walks into garbage... Actual buffer is 512 bytes long...

So in this cases like this, the code need to know it needs to fetch additional usb packets to complete the command.
 
@Paul - fixed the LittleFS wrapper classes for used/total sizes, issued new PR

Also issued PR for SD - mediaPresent - allow external pin to be specified for some SD Card readers, to speed them up. Tested with SD_MTP_Logger sketch
 
Just pushed up changes to MTP_Teensy for T4.x to know the actual TX and RX sizes for the end point that the PC connected to, such that
if we connect at HIGH speed (majority of cases) the RX and TX sizes are 512 bytes
But if for some reason we connect at full speed or lower than the sizes are 64 bytes...

So forced the T4.1 to be a 12mbs speed and now was able to send an object to the teensy (image file), then download it again to different folder and verified that the image file would open up properly...
 
Just pushed up changes to MTP_Teensy for T4.x to know the actual TX and RX sizes for the end point that the PC connected to, such that
if we connect at HIGH speed (majority of cases) the RX and TX sizes are 512 bytes
But if for some reason we connect at full speed or lower than the sizes are 64 bytes...

So forced the T4.1 to be a 12mbs speed and now was able to send an object to the teensy (image file), then download it again to different folder and verified that the image file would open up properly...

Nice job and fix Kurt.
 
Nice job and fix Kurt.

Indeed - lots of good work!

I was half surprised the folder copy worked through MTP - but apparently - of course it does!
>> 111 files in 79 folders reported size of 77.6 KB on Windows ( 136KB size on disk )

per post #804 that GIT_ZIP (when unzipped) goes to SD perfectly! - but it won't come back usably :(

Did anyone confirm it was a real problem and not just my WIP setup? See post #804

So far just tested on T_MM_lockable since it was new, and put it on a new logger carrier, so no PSI or QSPI to test.

That is some 190 disk dir entries with files/dirs - not a large file set - but it blows away even 10MB of LFS_PROG space with the large alloc units? Are they now 64KB? At 64KB that would be only 160 usable disk blocks?
> but when it fails there is a general failure on some files
 
@defragster
Yep at least partly. If you ditch copy th objects folder in .hit works perfectly. Watching what’s going on looks for every fir created and small file created it’s using 64k regardless of file size. So you will never get it all transferred. Also stopped it part way through and noticed that it was eating up half the disk and only part way through the objects dir. so it’s running out of disk space.

Haven’t looked at issue with sd yet. Later will look more but off to do a few things
 
@defragster
Yep at least partly. If you ditch copy th objects folder in .hit works perfectly. Watching what’s going on looks for every fir created and small file created it’s using 64k regardless of file size. So you will never get it all transferred. Also stopped it part way through and noticed that it was eating up half the disk and only part way through the objects dir. so it’s running out of disk space.

Haven’t looked at issue with sd yet. Later will look more but off to do a few things

The 64KB PROG blocks running out of space not surprising - well it was a bit given it is only 80KB of 'data' it should fit on 1MB or 10MB and both fail - but disks are what they are and larger block format determines that, so it will have good use for few and larger files, but not a ton of chaff. Just odd and wrong it fails on reading after space abused on Write to Teensy.

The SD copy was the real problem indicator - files truncated and corrupted on READ back from SD where they got a perfect copy to the Teensy.

Once I know that 'SD READ' from Teensy isn't my WIP file setup I can move to a unit with SPI/QSPI and see if that works. But if libs/sources here are out of sync with WIP - then failure is just chasing my tail ...
 
As for WIP setup - What I am more or less running:

a) Arduino 1.8.18
b) Teensyduino 1.56 beta 4

cores - the above
MTP_Teensy4 - my main branch - changes pushed up earlier.

USBHost_t36 - My FS_Integration_MSC branch. https://forum.pjrc.com/threads/6813...ntegration-including-MTP-and-MSC?p=295714#top

SD - I am running my card_detect_pin branch (which has additional support media detection, where member to optionally specify SPI external card detect pin) otherwise only that commit ahead of Paul's and beta4..
SDFat - I believe is in sync with Beta4

LittleFS - I am running my main branch which is now in sync with Paul's main branch


Trying to decide to decide what to work on next:

Like maybe better support for Copy/Move file for not having the host fail when you do something like copy a 3MB bitmap from SD card on Teensy to slow SPI Flash drive, which might take something like 30 seconds or more. Could do by interval timer... And/or could add support within the storage, to call back with progress...
 
@KurtE - @defragster

Think I am all synched up again - I hope - am using your setmediadetect branch, hopefully that gets pulled in. Anyway trying to figure out what going on with file transfer from SD to PC using the directories in @defragsters zip file.

Here we go. I am using a T4.1 with the following config:
Code:
Dump Storage list(11)
store:0 storage:10001 name:RAMindex fs:2000ccd8
store:1 storage:20001 name:sdio fs:2000d594
store:2 storage:30001 name:sd1 fs:2000da64
store:3 storage:40001 name:RAM0 fs:2000d39c
store:4 storage:50001 name:RAM1 fs:2000d468
store:5 storage:60001 name:QSPI fs:2000d2c4
store:6 storage:70001 name:sflash5 fs:2000df34
store:7 storage:80001 name:sflash6 fs:2000e010
store:8 storage:90001 name:WINBOND1G fs:2000cdac
store:9 storage:a0001 name:WINBOND2G fs:2000cea0
store:10 storage:b0001 name:MSC0-exFAT-128 fs:2000ab70

Step 1. Copy memorydump directory to sdio card.
Step 2. Copy memorydump directory to a newfolder directory on the PC.
Step 3. Open memorydump.h in 3 locations, SDIO, PC and in copy from Drive to pc.

Results
Directory size comparisons:
Code:
[B]Original[/B]: 
Size: 77.6 KB (79,552 bytes)
Size on disk: 164 KB (167,936 bytes)
111 Files, 79 Folders

On SDIO:
Size: 77.6 KB (79,552.00 bytes)
111 Files, 79 Folders

Copy on PC from SDIO:
Size: 133 KB (136,384 bytes)
Size on Disk: 468 KB (479,232 bytes)
111 Files, 79 Folders

Implies the more files on messed up when downloading using getObject? Looking at memoryDump.h you will see what is happening:
on SDIO:
Code:
// https://github.com/kurte/MemoryHexDump
//
// Warning this is Kurt's hacked up Memory dump library
//    probably like 50 other dump libraries...
//
//THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
//IMPLIED
#ifndef _memoryhexdump_h_
#define _memoryhexdump_h_

#include <Arduino.h>

void MemoryHexDump(Print& out, void const* address, size_t count, bool remove_duplicate_lines, 
	const char *szTitle=NULL, uint32_t max_output_lines=(uint32_t)-1, 
	uint32_t starting_display_addr = (uint32_t)-1 );

#endif
and on the file copied to the PC from the SDIO card:
Code:
// https://github.com/kurte/MemoryHexDump
//
// Warning this is Kurt's hacked up Memory dump library
//    probably like 50 other dump libraries...
//
//THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
//IMPLIED
#ifndef _memoryhexdump_h_
#define _memoryhexdump_h_

#include <Arduino.h>

void MemoryHexDump(Print& out, void const* address, size_t count, bool remove_duplicate_lines, 
	const char *szTitle=NULL, uint32_t max_output_lines=(uint32_t)-1, 
	uint32_t starting_display_addr = (uint32_t)-1 );

#endifump
[COLOR="#FF0000"]//
// Warning this is Kurt's hacked up Memory dump library
//    probably like 50 other dump libraries...
//
//THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
//IMPLIED
#ifndef _memoryhexdump_h_
#define _memoryhexdump_h_

#include <Arduino.h>

void MemoryHexDump(Print& out, void const* address, size_t count, bool remove_duplicate_lines, 
	const char *szTitle=NULL, uint32_t max_output_lines=(uint32_t)-1, 
	uint32_t starting_display_addr = (uint32_t)-1 );

#endif[/COLOR]
Looks like it duplicated the file contents to fill the buffer? Just guess here.

And in memoryDump.cpp the following were added lines to the file on pc after copying:
Code:
*** Max Count of Output Lines Reached ***");
			return;
		}
	}
	if (0 != duplicate_line_cached) {
		// last line was a duplicate and 16 bytes long
		if ((p - 16) != last_line_output && 1 < duplicate_line_cached) {
			out.print("...\t ");
			out.print(duplicate_line_cached - 1);
			out.print(" duplicate line(s) removed.\n");
		}
		display_addr = starting_display_addr + ((uint32_t)(p-16) - (uint32_t)mem);
		output_one_line(out, p - 16, output_count, display_addr);
		count_output_lines++;
	}
}

NOTE: If I xfer memorydump.h by itself - no other files - no issues with file size or contents.
 
Last edited:
@KurtE - @defragster

Think I am all synched up again - I hope - am using your setmediadetect branch, hopefully that gets pulled in. Anyway trying to figure out what going on with file transfer from SD to PC using the directories in @defragsters zip file.
...
NOTE: If I xfer memorydump.h by itself - no other files - no issues with file size or contents.

NEAR OPPS - was doing EDIT not reply and wondered why it didn't end with [/quote]


Mike, Thanks for confirming :) - your results seem similar :( . I just did a full folder 'CodeCompare' - some files truncated - some extended/corrupted.

So it isn't just a Windows 11 issue or a missing WIP_FIX on my end.

I wasn't sure if it was a 'small file case' that @wwatson ran into and maybe I didn't have the fix for.

Good it is on SD because it can be removed to PC to confirm it is valid there - not possible with other chip based media.

Again that folder was just handy - there may be a simpler repro case of nested files and folders.
 
Folder copy from MTP Teensy shows errors

Here is a purpose made simpler sample: shows failure reading files off SD card.

MakeFiles.zip: has a sketch that made the files on SD
> three folders: 111,222,333
> three files in each folder with unique content of name:size: aaa:256B, bbb:512B, ccc:768

Code:
[ATTACH]26937._xfImport[/ATTACH]

In the zip is the sketch
> the SD_src as copied after the sketch ran
> the SD_mtp showing what I got with the MTP copy : each erroneous file may have unique errors showing parts of one file in another.

Wrote the MakeFiles as sketch assuming it may need to evolve in complexity - but first try was all it took.
This is simple and small enough it should work on LFS_PROG as well as others for testing

I didn't try copy of the files over to Teensy as that worked before.
 
defragster said:
Folder copy from MTP Teensy shows errors
Yep single files seem to be working but copying directories are definitely an issue. Tomorrow will probably dig into it more. Found the one fix but something else is going on. Not sure where its broken.
 
Yep single files seem to be working but copying directories are definitely an issue. Tomorrow will probably dig into it more. Found the one fix but something else is going on. Not sure where its broken.

Yes, singles files had a good set of tests. With the simple MakeFiles it should help see the crossover issue. Then see it work on the earlier MemoryHexDump and then Makefiles can get arbitrarily complexified: Deeper and larger files.
 
Folder copy from MTP Teensy 3.6 WORKS!

Here is a purpose made simpler sample: shows failure reading files off SD card.
...

Built and ran my mtp-test.ino for T_3.6 and these MakeFiles file are identical after


ALSO identical with copy of MemoryHexDump_git to Teesny MTP SD then copy back to PC

So the code is generally right - except the High Speed buffering part versus the 12 Mbps?

<edit> quick edit in hand to add same 3 dirs to prior 3 dirs and allow start and grow of the files sizes to change. Optional in code can use it when current case is seen to work.
 
Last edited:
@Paul, @defragster and @mjs513...

I was playing some this morning with the 3 directory transfer and can reproduce the issue.
I instrumented the code some:
Code:
int MTPD::push_packet(uint8_t *data_buffer, uint32_t len) {
  int count_sent;
  while ((count_sent = usb_mtp_send(data_buffer, len, 60)) <= 0) {
    printf("push_packet: l:%u ret:%d\n", len, count_sent);
  }
  return 1;
}

Code:
void MTPD::GetObject(uint32_t object_id) {
  uint32_t size = storage_->GetSize(object_id);
  printf("\nGetObject(%u) size: %u\n", object_id, size);

  if (write_get_length_) {
    write_length_ += size;
  } else {
    uint32_t pos = 0; // into data
    uint32_t len = sizeof(MTPHeader);

    disk_pos = DISK_BUFFER_SIZE;
    while (pos < size) {
      if (disk_pos == DISK_BUFFER_SIZE) {
        uint32_t nread = min(size - pos, (uint32_t)DISK_BUFFER_SIZE);
        storage_->read(object_id, pos, (char *)disk_buffer_, nread);
        printf("  >>r p:%u l:%u\n", pos, nread);
        disk_pos = 0;
      }

      uint32_t to_copy = min(size - pos, mtp_tx_size_ - len);
      to_copy = min(to_copy, DISK_BUFFER_SIZE - disk_pos);

      memcpy(tx_data_buffer + len, disk_buffer_ + disk_pos, to_copy);
      disk_pos += to_copy;
      pos += to_copy;
      len += to_copy;

      if (len == (uint32_t)mtp_tx_size_) {
        push_packet(tx_data_buffer, mtp_tx_size_);
        printf("    >>w dp:%u p:%u len:%u\n", disk_pos, pos, len);
        len = 0;
      }
    }
    printf("    >>>>w len:%u\n", len);
    if (len > 0) {
      push_packet(tx_data_buffer, len);
      len = 0;
    }
    printf("    >><< Done\n");
  }
}

I also changed the last push to : push_packet(tx_data_buffer, len)
instead of full packet size...

So some debug output when I select the 3 directories and do a copy to directory on PC: Note I am using built-in SD:

Code:
GetObject(33) size: 256

GetObject(33) size: 256
  >>r p:0 l:256
    >>>>w len:268
    >><< Done
76470 RESP:2001(RSP:OK)l: 16 T:10b : 21
LP:76488 CMD: 1009(GET_OBJECT)l: 16 T:10c : 22

GetObject(34) size: 512

GetObject(34) size: 512
  >>r p:0 l:512
    >>w dp:500 p:500 len:512
    >>>>w len:12
    >><< Done
76489 RESP:2001(RSP:OK)l: 16 T:10c : 22
LP:76490 CMD: 1009(GET_OBJECT)l: 16 T:10d : 22

GetObject(34) size: 512

GetObject(34) size: 512
  >>r p:0 l:512
    >>w dp:500 p:500 len:512
    >>>>w len:12
    >><< Done
76497 RESP:2001(RSP:OK)l: 16 T:10d : 22
LP:76508 CMD: 1009(GET_OBJECT)l: 16 T:10e : 23

GetObject(35) size: 768

GetObject(35) size: 768
  >>r p:0 l:768
    >>w dp:500 p:500 len:512
    >>>>w len:268
    >><< Done
76516 RESP:2001(RSP:OK)l: 16 T:10e : 23
push_packet: l:16 ret:0
LP:76604 CMD: 1009(GET_OBJECT)l: 16 T:10f : 23

GetObject(35) size: 768

GetObject(35) size: 768
  >>r p:0 l:768
    >>w dp:500 p:500 len:512
    >>>>w len:268
    >><< Done
76613 RESP:2001(RSP:OK)l: 16 T:10f : 23
LP:76623 CMD: 1009(GET_OBJECT)l: 16 T:110 : 24

GetObject(36) size: 256

GetObject(36) size: 256
  >>r p:0 l:256
    >>>>w len:268
    >><< Done
76625 RESP:2001(RSP:OK)l: 16 T:110 : 24
LP:76633 CMD: 1009(GET_OBJECT)l: 16 T:111 : 24

GetObject(36) size: 256

GetObject(36) size: 256
  >>r p:0 l:256
    >>>>w len:268
    >><< Done
76642 RESP:2001(RSP:OK)l: 16 T:111 : 24
LP:76649 CMD: 1009(GET_OBJECT)l: 16 T:112 : 25

GetObject(37) size: 512

GetObject(37) size: 512
  >>r p:0 l:512
push_packet: l:512 ret:0
    >>w dp:500 p:500 len:512
    >>>>w len:12
push_packet: l:12 ret:0
push_packet: l:12 ret:0
push_packet: l:12 ret:0
push_packet: l:12 ret:0
push_packet: l:12 ret:0
push_packet: l:12 ret:0
push_packet: l:12 ret:0
push_packet: l:12 ret:0

Now on the cores side we have
Code:
int usb_mtp_send(const void *buffer, uint32_t len, uint32_t timeout)
{
	transfer_t *xfer = tx_transfer + tx_head;
	uint32_t wait_begin_at = systick_millis_count;

	while (1) {
		if (!usb_configuration) return -1; // usb not enumerated by host
		uint32_t status = usb_transfer_status(xfer);
		if (!(status & 0x80)) break; // transfer descriptor ready
		if (systick_millis_count - wait_begin_at > timeout) return 0;
		yield();
	}
	uint8_t *txdata = txbuffer + (tx_head * MTP_TX_SIZE_480);
	memcpy(txdata, buffer, len);
	arm_dcache_flush_delete(txdata, tx_packet_size );
	usb_prepare_transfer(xfer, txdata, len, 0);
	usb_transmit(MTP_TX_ENDPOINT, xfer);
	if (++tx_head >= TX_NUM) tx_head = 0;
	return len;
}
So it is returning 0 so the transfer descriptor is not getting into a ready state...

Need to debug... Paul any suggestions? I think we are only using one tx... Should I try to extend to 2 or 3 and see what happens? I might...
Probably should not iterate forever calling this if it fails... So will probably build in some form of timeout...

Other suggestions?

Edit: We are setup with 4 rx and 4 tx buffers/transfers

Edit2: turned on debug printing in cores also turned it on to output to USB and added debug on timeout...
Code:
$$$usb_mtp_send timeout head:1 status:010C0080
push_packet: l:12 ret:0

Edit 3: should mention, that it is not always a smaller packet that hangs...
Code:
GetObject(36) size: 256
  >>r p:0 l:256
    >>>>w len:268
    >><< Done
65391 RESP:2001(RSP:OK)l: 16 T:ef : 24
LP:65398 CMD: 1009(GET_OBJECT)l: 16 T:f0 : 25

GetObject(37) size: 512

GetObject(37) size: 512
  >>r p:0 l:512

$$$usb_mtp_send timeout head:0 status:00100080
push_packet: l:512 ret:0

$$$usb_mtp_send timeout head:0 status:00100080
push_packet: l:512 ret:0

$$$usb_mtp_send timeout head:0 status:00100080
push_packet: l:512 ret:0
 
Last edited:
@Paul, @defragster and @mjs513...

I was playing some this morning with the 3 directory transfer and can reproduce the issue.
...

Glad you could get a repro with the 3 dirs, yes it was seen here on builtin SD of T_MM and T_4.1. Didn't try with just one or two of the 1,2,3 dirs.

CodeCompare DIFF on folders shows all 9 of the files in the three dirs are 'corrupted' - where one third are 256B, another at 512B and the last at 768B - so it isn't a partial buffer issue.

And as noted no repro on T_3.6 builtin with either DIR ZIP above, but you saw that and know the code - so it is 480Mbps path.

Hopefully the file content is unique enough to help see what is going where? As noted start was simple path toward a concept to dupe the initial git folder - and it was enough as coded.


SUPER WINDY here again - Lights just got a couple second outage.
 
Note: I updated the code to send the last actual packet as actual size instead of bytes left...
When I tried sending with full size, all 9 files went through but were corrupted. Not sure why other times downloading works with the extra padding, but not this one...

Still investigating... Almost like when it is doing the bulk receive and it gets an packet < full size it does not reset it...
 
Note: I updated the code to send the last actual packet as actual size instead of bytes left...
When I tried sending with full size, all 9 files went through but were corrupted. Not sure why other times downloading works with the extra padding, but not this one...

Still investigating... Almost like when it is doing the bulk receive and it gets an packet < full size it does not reset it...

Quite a puzzle! Given that one third of the files were chosen as 'packet size' 512B - but perhaps some 'packets' contain just the file data but also other DIR or transfer info?

Not knowing what the BULK transfer results in - perhaps it just goes into a STREAM of bytes sharing multiple files and indicators and not 'per file'?

Gives an idea for alternate files names/sizes variation and more numerous files test data set.
 
Quite a puzzle! Given that one third of the files were chosen as 'packet size' 512B - but perhaps some 'packets' contain just the file data but also other DIR or transfer info?

Not knowing what the BULK transfer results in - perhaps it just goes into a STREAM of bytes sharing multiple files and indicators and not 'per file'?

Gives an idea for alternate files names/sizes variation and more numerous files test data set.
Note when you send a file of 512 bytes, the actual stream is 524 bytes long as the file data is preceded by a 12 byte header.

That is why my Bogus test code was setup that the first record was reduced by 12 bytes, such that each received 512 byte buffer would start off with a sequence number.

Now back to investigating..

Again wondering if anything interesting shows up in USB data... Can try wireshark or would be interesting to see if USB scope shows something like some packets or information we are not processing.
 
Note when you send a file of 512 bytes, the actual stream is 524 bytes long as the file data is preceded by a 12 byte header.

That is why my Bogus test code was setup that the first record was reduced by 12 bytes, such that each received 512 byte buffer would start off with a sequence number.

Now back to investigating..

Again wondering if anything interesting shows up in USB data... Can try wireshark or would be interesting to see if USB scope shows something like some packets or information we are not processing.

Ah ... THOSE 12 bytes ... got it. Can alter the base case to shift 12 bytes so that one third should work: 244, 500, 756.

This has the 'bbb's as 500 bytes so they should work:
Code:
[ATTACH]26939._xfImport[/ATTACH]
Make File:/111/aaa.txt File size=244
Make File:/111/bbb.txt File size=500
Make File:/111/ccc.txt File size=756
Make File:/222/aaa.txt File size=244
Make File:/222/bbb.txt File size=500
Make File:/222/ccc.txt File size=756
Make File:/333/aaa.txt File size=244
Make File:/333/bbb.txt File size=500
Make File:/333/ccc.txt File size=756
 
Back
Top