Teensyduino File System Integration, including MTP and MSC

I'm still working on the USB received packet loss issue. It's a tough one.

But if you'd like an ugly & slow workaround to at least get reliable communication now, try adding delayMicroseconds(10) at the end of schedule_transfer() in usb.c.

It's also becoming apparent we could do quite a lot of improve performance in the future. But right now I just want to get it to a stable & usable point which we could include in release 1.56. Performance optimization can always come later.

I just tried it:
Code:
...
	endpoint->next = (uint32_t)transfer;
	endpoint->status = 0;
	USB1_ENDPTPRIME |= epmask;
	endpoint->first_transfer = transfer;
end:
	endpoint->last_transfer = transfer;
	__enable_irq();
	//digitalWriteFast(4, LOW);
	//digitalWriteFast(3, LOW);
	//digitalWriteFast(2, LOW);
	//digitalWriteFast(1, LOW);
	//if (transfer_log_head > LOG_SIZE) transfer_log_head = 0;
	//transfer_log[transfer_log_head++] = ret;
	//transfer_log_count++;
	 delayMicroseconds(10);
}
Sent the large file to SDCard, looks like it is still failing:
Code:
LP:51172 CMD: 100d(SEND_OBJECT)l: 12 T:90
MTPD::SendObject: len:22015988

MTPD::SendObject *** USB Read 0 bytes ***

MTPD::SendObject *** USB Read 0 bytes ***
len 1524
 # USB Packets: 42999 total: 2651 avg ms: 0 max: 1
 # Write: 10750 total:1332 avg ms: 0 max: 92(1233)
  >> >=100ms: 0 50:2 40:0 30:8 20:1 10:6
  >> Last write times
  0 0 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>Total Time: 5062954
56235 RESP:2007(RSP:INCOMPLETE_TRANSFER)l: 12 T:90
LP:56271 CMD: 1005(GET_STORAGE_INFO)l: 16 T:91 : 80001
524289 7 name:SD_Builtin
524289 7 name:SD_Builtin
56271 RESP:2001(RSP:OK)l: 16 T:91 : 80001
Looks like it lost 3 packets. Although I downloaded it back and did a compare and it looked like it just missed one:
screenshot.jpg

EDIT: forgot I did not update the len: line in this version of SendObject.. Updated it to show the len left plus how much is in the output buffer.

Also rebuilt using sketch with bogus.fs... Updated that version slightly as well to make it easier to pick out which packet was dropped:
Code:
BogusFS::open(/T4LargeIndexedTestfile.txt, 2)
46713 RESP:2001(RSP:OK)l: 24 T:2f : 20001 ffffffff 16
LP:46713 CMD: 100d(SEND_OBJECT)l: 12 T:30
MTPD::SendObject: len:22015988
$$$BF Sequence error 29699 29697

MTPD::SendObject *** USB Read 0 bytes ***

MTPD::SendObject *** USB Read 0 bytes ***
len 512 diskpos: 1524
Bogus::close(/T4LargeIndexedTestfile.txt) LPN:42999
Bogus::close(/T4LargeIndexedTestfile.txt) LPN:42999
Bogus::close(/T4LargeIndexedTestfile.txt) LPN:42999
BogusFS::open(/T4LargeIndexedTestfile.txt, 0)
Bogus::setModifyTime: /T4LargeIndexedTestfile.txt 11/05/2021 07:20
Bogus::setCreateTime: /T4LargeIndexedTestfile.txt 12/07/2021 14:21
Bogus::close(/T4LargeIndexedTestfile.txt) LPN:-1
Bogus::close(/T4LargeIndexedTestfile.txt) LPN:-1
Bogus::close(/T4LargeIndexedTestfile.txt) LPN:-1
 # USB Packets: 42999 total: 2663 avg ms: 0 max: 1
 # Write: 10750 total:221 avg ms: 0 max: 1(12)
  >> >=100ms: 0 50:0 40:0 30:0 20:0 10:0
  >> Last write times
  0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
>>>Total Time: 3902707
50616 RESP:2007(RSP:INCOMPLETE_TRANSFER)l: 12 T:30
LP:50658 CMD: 1005(GET_STORAGE_INFO)l: 16 T:31 : 20001

So lost packet: $$BF Sequence error 29699 29697
 
Last edited:
Here's a proper fix.

https://github.com/PaulStoffregen/cores/commit/e04c6eace5cf11999481b5fe12ccb454127d3f0a

So far I've run only a couple quick tests on a relatively slow Windows 10 laptop. But I'm feeling pretty good this should really fix the problem.

FWIW, the bug was this 4th case of schedule_transfer() which NXP's documentation describes as "If status bit read in (3) is 0 then Goto Case 1: Step 1" was falling through to the code for case 1, but writing to endpoint->first_transfer transfer is incorrect for case #4. This fix duplicates the case 1 code for case 4, but without the write to endpoint->first_transfer.
 
:D

I Did 10 writes of the big file to the bogus FS... All went through.

I also then rebuilt other sketch and copied several large audio files and they went through fine to the SD Card...

So looks like a great fix!

Obviously still does not fix the MTP timeout issue when writing to FLASH drive (6 on the memory board) ;)

Code:
Created: 61af7593
Read DateTime string: 20211201T062452.0
>> date/Time: 61a71534 12/1/2021 6:24:52
Modified: 61a71534
484911 RESP:2001(RSP:OK)l: 24 T:f5 : 70001 ffffffff 27
LP:484912 CMD: 100d(SEND_OBJECT)l: 12 T:f6
MTPD::SendObject: len:3135327

MTPD::SendObject *** USB Read 0 bytes ***

MTPD::SendObject *** USB Read 0 bytes ***
len 2887019 diskpos: 500
 # USB Packets: 485 total: 31 avg ms: 0 max: 1
 # Write: 122 total:3018 avg ms: 24 max: 49(86)
  >> >=100ms: 0 50:0 40:61 30:0 20:0 10:0
  >> Last write times
  1 6 45 6 41 6 47 6 45 6 43 6 43 6 42 6 48 6 42 6 43 6 43 6 43 6 47 6 49 6 43 6
>>>Total Time: 5065390
489978 RESP:2007(RSP:INCOMPLETE_TRANSFER)l: 12 T:f6
 
I probably never would have been able to solve this without Kurt's work on BogusFS ;)
Glad it worked!

Mike and I were trying just about everything we could think of to localize the issue. Finally created an indexed file where the first buffer
would be 500 bytes long and all other 512 bytes long (first one has 12 byte header) . Actually for both issues. The first one was to see if changing delays on how long each write might take, would mater or not and then for the dropped packets...

Started off sending the big file up and have it fail, then copy it back down or move SD back down to PC and then compare the received file to the one we sent...
It worked but was a pain...
 
:D

I Did 10 writes of the big file to the bogus FS... All went through.

I also then rebuilt other sketch and copied several large audio files and they went through fine to the SD Card...

So looks like a great fix!

Obviously still does not fix the MTP timeout issue when writing to FLASH drive (6 on the memory board) ;)

Code:
Created: 61af7593
Read DateTime string: 20211201T062452.0
>> date/Time: 61a71534 12/1/2021 6:24:52
Modified: 61a71534
484911 RESP:2001(RSP:OK)l: 24 T:f5 : 70001 ffffffff 27
LP:484912 CMD: 100d(SEND_OBJECT)l: 12 T:f6
MTPD::SendObject: len:3135327

MTPD::SendObject *** USB Read 0 bytes ***

MTPD::SendObject *** USB Read 0 bytes ***
len 2887019 diskpos: 500
 # USB Packets: 485 total: 31 avg ms: 0 max: 1
 # Write: 122 total:3018 avg ms: 24 max: 49(86)
  >> >=100ms: 0 50:0 40:61 30:0 20:0 10:0
  >> Last write times
  1 6 45 6 41 6 47 6 45 6 43 6 43 6 42 6 48 6 42 6 43 6 43 6 43 6 47 6 49 6 43 6
>>>Total Time: 5065390
489978 RESP:2007(RSP:INCOMPLETE_TRANSFER)l: 12 T:f6

@PaulStoffregen - @KurtE
Wow I disappear for awhile today and when I come back all heck breaks loose. I downloaded the updated Teensy 4 core just in case I tinkered with any other changes and put MTP_teensy back to the master branch without all the extra debug stuff.

Transfer multiple files to SDIO in one copy/paste and all worked. Also transfered our indexed file multiple times without an issue. Heres a quick screen shot of the directory. As Kurt said doesn't help the issue with flash6 (the 512mb winbond flash):
Capture.PNG

EDIT: Did large file transfer to SD10, USB drive, Winbond1G Nand and a 8mb to QSPI (16mb flash). Had to double check :)
 
Good Morning all:

Yes, I believe a function for pre-allocating space is the major piece we're still missing in FS.h.

We could add some method like: bool preAllocate(uint64_t length)
To the File and FileImpl classes. Which would be easy to know what to do with, for SD and MSC classes.

Not sure how it would work with LittleFs, I have not looked at internals much, like would you chain all of them? Would you (if needed) erase all of the blocks?
If so, then this might be a long operation, on the SPI Flash (like 5 or 6 on memory board), I think we have seen an erase of a 4K block takes something like 50-60ms to complete.
Which matches the Spec:
screenshot.jpg

So for example If I am going to upload a 3mb bitmap, we are going to need something like 820+ sectors, which if we need to erase all of them will take > 40 seconds to complete.
Which for sure MTP would timeout if we used this in processing the SendObjectInfo call.

Note posting #628, where I tried uploading the large indexed file and it timed out...
Code:
 # USB Packets: 485 total: 31 avg ms: 0 max: 1
 # Write: 122 total:3018 avg ms: 24 max: 49(86)
  >> >=100ms: 0 50:0 40:61 30:0 20:0 10:0
  >> Last write times
  1 6 45 6 41 6 47 6 45 6 43 6 43 6 42 6 48 6 42 6 43 6 43 6 43 6 47 6 49 6 43 6
You can see in the last line here, the last several calls to write to the FS where we pass 2K at a time, The last line is shown in MS.
So you can see a lot of the alternating long/short times, which shows it is having to erase a block and then can do it's write and the next write on that newly erased block...

Note: if we go through and use the Low Level format command in some of our sketches, we can usually upload those larger filers.

Again I wonder if the FS class should give us some hints on this? I know with some other devices talking to Ubuntu (like Serial) my user code needs to deduce things like, am I talking to FTDI or not). If FTDI then I need to use the tcdrain call to minimize latency... But calling tcdrain on other drivers just inserts a long delay into the code...

At times I wonder if it would make sense to add some method to FS, something like: idle, or loop, or houseCleaning
that we could call when we are not doing anything, to allow some background stuff to go on. Like maybe SD checking state of inserted.
And maybe LittleFs could erase a sector if needed... Not sure if this makes sense?

Also for FS.h not sure if we want to yet tackle adding some FS Events, like MTP register an event to say notify me if files are created or deleted on this FS?

Now for some more Coffee!
 
The main missing part are concurrent accesses from interrupts (like audio lib). Everything else is nice to have.

And what happens when the PC accesses a file that is opened by the user program. A log file for example. These basics should should be done first.
 
As @KurtE said earlier sflash5 or the W25Q512 chip seems to be an issue on the memory board. The Winbond Nand chips don't have a problem with transferring large files. Getting my new memory board so I can test some other chips on Friday so will have to wait to see their performance. Any on sflash5 and sflash6 still reporting 0 length packets and MTP is timing out. So decided to take a look at what LittleFS is doing on sflash5.

Looks like when it sees the send_object command its doing reads and some writes of the Dir pair (blocks 0/1) when thats completed it starts the actual writing of the data to the disk:
Code:
  flash wr: block=4808, offset=0, size=256
Write:     12 01 2C 80 00 
    30 30 30 30 30 30 20 41 41 41 41 41 41 41 41 41 41 41 41 41 
  flash rd: block=4808, offset=0, size=256
Read: 
  addrbits=32
    13 01 2C 80 00 
    30 30 30 30 30 30 20 41 41 41 41 41 41 41 41 41 41 41 41 41 
Size: 256
20201F00 - 30 30 30 30 30 30 20 41  41 41 41 41 41 41 41 41  : 000000 A AAAAAAAA
20201F10 - 41 41 41 41 41 41 41 41  41 41 41 41 41 41 41 41  : AAAAAAAA AAAAAAAA
...	 4 duplicate line(s) removed.
20201F60 - 41 41 41 41 41 41 41 41  41 41 41 41 41 41 41 41  : AAAAAAAA AAAAAAAA
20201F70 - 41 41 41 0A 30 30 30 30  30 30 20 42 42 42 42 42  : AAA.0000 00 BBBBB
20201F80 - 42 42 42 42 42 42 42 42  42 42 42 42 42 42 42 42  : BBBBBBBB BBBBBBBB
...	 5 duplicate line(s) removed.
20201FE0 - 42 42 42 42 42 42 42 42  42 42 42 42 42 42 42 42  : BBBBBBBB BBBBBBBB
20201FF0 - 42 42 42 0A 30 30 30 30  30 30 20 43 43 43 43 43  : BBB.0000 00 CCCCC
  flash wr: block=4808, offset=256, size=256
Write:     12 01 2C 81 00 
    43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 
  flash rd: block=4808, offset=256, size=256
When it goes to the next block, which is 4k for the 512 chip it does a flash erase:
Code:
  flash er: block=4809
  flash rd: block=4809, offset=0, size=256
Read: 
  addrbits=32
    13 01 2C 90 00 
    FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 
Size: 256
20203FA8 - FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  : ........ ........
...	 14 duplicate line(s) removed.
20204098 - FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF  : ........ ........
when the block is erased it writes the next block:
Code:
 flash wr: block=4809, offset=0, size=256
Write:     12 01 2C 90 00 
    C8 12 00 00 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 
  flash rd: block=4809, offset=0, size=256
Read: 
  addrbits=32
    13 01 2C 90 00 
    C8 12 00 00 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 41 
Size: 256
20201F00 - C8 12 00 00 41 41 41 41  41 41 41 41 41 41 41 41  : ....AAAA AAAAAAAA
20201F10 - 41 41 41 41 41 41 41 41  41 41 41 41 41 41 41 41  : AAAAAAAA AAAAAAAA
...	 4 duplicate line(s) removed.
20201F60 - 41 41 41 41 41 41 41 41  41 41 41 41 41 41 41 41  : AAAAAAAA AAAAAAAA
20201F70 - 41 41 41 41 41 41 41 0A  30 30 30 30 30 38 20 42  : AAAAAAA. 000008 B
20201F80 - 42 42 42 42 42 42 42 42  42 42 42 42 42 42 42 42  : BBBBBBBB BBBBBBBB
...	 5 duplicate line(s) removed.
20201FE0 - 42 42 42 42 42 42 42 42  42 42 42 42 42 42 42 42  : BBBBBBBB BBBBBBBB
20201FF0 - 42 42 42 42 42 42 42 0A  30 30 30 30 30 38 20 43  : BBBBBBB. 000008 C
  flash wr: block=4809, offset=256, size=256
Write:     12 01 2C 91 00 
    43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 43 
  flash rd: block=4809, offset=256, size=256
....
Write:     12 01 2C 9F 00 
    4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 
  flash rd: block=4809, offset=3840, size=256
Read: 
  addrbits=32
    13 01 2C 9F 00 
    4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 4F 
Size: 256
20201F00 - 4F 4F 4F 4F 4F 4F 4F 4F  4F 4F 4F 4F 4F 4F 4F 4F  : OOOOOOOO OOOOOOOO
...	 5 duplicate line(s) removed.
20201F60 - 4F 4F 4F 4F 4F 4F 4F 4F  4F 4F 4F 4F 4F 4F 4F 4F  : OOOOOOOO OOOOOOOO
20201F70 - 4F 4F 4F 4F 4F 4F 4F 0A  30 30 30 30 31 35 20 50  : OOOOOOO. 000015 P
20201F80 - 50 50 50 50 50 50 50 50  50 50 50 50 50 50 50 50  : PPPPPPPP PPPPPPPP
...	 5 duplicate line(s) removed.
20201FE0 - 50 50 50 50 50 50 50 50  50 50 50 50 50 50 50 50  : PPPPPPPP PPPPPPPP
20201FF0 - 50 50 50 50 50 50 50 0A  30 30 30 30 31 36 20 41  : PPPPPPP. 000016 A
  flash er: block=4810
  flash rd: block=4810, offset=0, size=256
Read: 
  addrbits=32
    13 01 2C A0 00 
    FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
and that cycle will continue until either the file completes transfer or we get the incomplete transfer.

Now for the problem of 0 length packets and when it hits - same as seen before (based on wireshark - of on 260K packet):
Code:
 flash wr: block=4852, offset=2048, size=256
Write:     12 01 2F 48 00 
    4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 
  flash rd: block=4852, offset=2048, size=256
Read: 
  addrbits=32
    13 01 2F 48 00 
    4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 4E 
Size: 256
20201F00 - 4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E  : NNNNNNNN NNNNNNNN
...	 2 duplicate line(s) removed.
20201F30 - 4E 4E 4E 4E 4E 4E 4E 4E  4E 4E 4E 4E 4E 4E 4E 4E  : NNNNNNNN NNNNNNNN
20201F40 - 4E 4E 4E 4E 4E 4E 4E 0A  30 30 30 33 35 35 20 4F  : NNNNNNN. 000355 O
20201F50 - 4F 4F 4F 4F 4F 4F 4F 4F  4F 4F 4F 4F 4F 4F 4F 4F  : OOOOOOOO OOOOOOOO
...	 5 duplicate line(s) removed.
20201FB0 - 4F 4F 4F 4F 4F 4F 4F 4F  4F 4F 4F 4F 4F 4F 4F 4F  : OOOOOOOO OOOOOOOO
20201FC0 - 4F 4F 4F 4F 4F 4F 4F 0A  30 30 30 33 35 35 20 50  : OOOOOOO. 000355 P
20201FD0 - 50 50 50 50 50 50 50 50  50 50 50 50 50 50 50 50  : PPPPPPPP PPPPPPPP
...	 1 duplicate line(s) removed.
20201FF0 - 50 50 50 50 50 50 50 50  50 50 50 50 50 50 50 50  : PPPPPPPP PPPPPPPP

MTPD::SendObject *** USB Read 0 bytes ***

MTPD::SendObject *** USB Read 0 bytes ***
>> Left over write. len 500
then it finishes up with the last 500 left over bytes so we wind up with a total of 356 packets transfered:
Code:
>> File Close
 # USB Packets: 357 total: 2 avg ms: 0 max: 1
 # Write: 90 total:3034 avg ms: 33 max: 70(32)
  >> >=100ms: 0 50:29 40:1 30:14 20:13 10:32
  >> Last write times
  3 19 61 21 56 19 65 21 57 21 61 19 54 17 56 18 53 18 57 18 53 18 54 16 58 19 65 17 56 18 60 20
>>>Total Time: 5090839
38644 RESP:2007(RSP:INCOMPLETE_TRANSFER)l: 12 T:11
Note the write times are off since I am doing a ton of prints from LFS
 
The reason we don't already have preallocation is because it's difficult to define...

Maybe this makes more sense to implement as part of FS.open(), or maybe as a dedicated function specific to creating a file?

However it's done, I believe we need to clearly document the contents of the preallocated file can be expected to be random leftover data from the underlying media. The SdFat documentation seems to imply that's what it does. I'm not sure how we can possibly implement this on LittleFS. If the filesystem can't support preallocation or sparce allocation, we might just have to return a normal empty file.
 
Preallocation is normally not needed and in case of SD only there to speed up writing. Preallocating blindly, independend of the medium would be worse and not helpful. It should be used for this usecase only,
and only if it makes sense (- SD with the right filesystem)

User libraries just must have a way to ask which FS it is.

Really writing an empty file anywhere is NOT helpful. Don't do that.
 
Before taking more about the sflash5 I agree with Frank - for LittleFS I don't think will support pre-allocation like you do with SD Cards. Ok back to sflash5. Normally for testing I have been using the MTP_test_integrity sketch so I can get access to all devices. But just did a hack to the LFS_MTP_SPI_SIMPLE_DATALOGGER sketch to just focus on sflash5.
Code:
/*
  LittleFS  datalogger

  This example shows how to log data from three analog sensors
  to an storage device such as a FLASH.

  This example code is in the public domain.
*/
#include <LittleFS.h>
#include <MTP_Teensy.h>

// LittleFS supports creating file systems (FS) in multiple memory types.
// Depending on the
// memory type you want to use you would uncomment one of the following
// constructors
LittleFS_RAM lfsram;
LittleFS_SPIFlash sflash5; // https://github.com/PaulStoffregen/LittleFS#nand-flash
const int chipSelect = 5;

#ifdef ARDUINO_TEENSY41
extern "C" uint8_t external_psram_size;
uint32_t MEMFS_SIZE = 65536; // probably more than enough...

#endif

File dataFile; // Specifes that dataFile is of File type

int record_count = 0;
bool write_data = false;
uint32_t diskSize;

// Add in MTPD objects
MTPStorage storage;
MTPD mtpd(&storage);

FS *mtpDisk;

void setup() {
  // Open serial communications and wait for port to open:
  Serial.begin(115200);
//  Serial4.begin(115200);  // Use to echo stuff out for debug core.
  while (!Serial && millis() < 5000) {
    // wait for serial port to connect.
  }
  Serial.print(CrashReport);
  Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);

  Serial.print("Initializing LittleFS ...");

  // checks that the LittFS program has started with the disk size specified
  if (!sflash5.begin(chipSelect, SPI)) {
    Serial.printf("Error starting %s\n", "PROGRAM FLASH DISK");
    while (1) {
      // Error, so don't do anything more - stay stuck here
    }
  }

  mtpd.begin();
// lets initialize a RAM drive.
#if defined ARDUINO_TEENSY41
  if (external_psram_size)
    MEMFS_SIZE = 4 * 1024 * 1024;
#endif

  if (lfsram.begin(MEMFS_SIZE)) {
    Serial.printf("Ram Drive of size: %u initialized\n", MEMFS_SIZE);
    uint32_t istore = storage.addFilesystem(lfsram, "RAM");
    Serial.printf("Set Storage Index drive to %u\n", istore);
  }
  mtpDisk = &lfsram;  // so we don't start of with NULL pointer
  
  storage.addFilesystem(sflash5, "sflash5");

  Serial.println("LittleFS initialized.");

  menu();
}

void loop() {
  mtpd.loop();

  if (Serial.available()) {
    uint8_t command = Serial.read();
    int ch = Serial.read();
    uint32_t drive_index = CommandLineReadNextNumber(ch, 0);
    while (ch == ' ')
      ch = Serial.read();

    switch (command) {
    case 'l':
      listFiles();
      break;
    case 'e':
      eraseFiles();
      break;
    case 's': {
      Serial.println("\nLogging Data!!!");
      write_data = true; // sets flag to continue to write data until new
      // command is received
      // opens a file or creates a file if not present,  FILE_WRITE will append
      // data to
      // to the file created.
      dataFile = mtpDisk->open("datalog.txt", FILE_WRITE);
      logData();
    } break;
    case 'x':
      stopLogging();
      break;
    case '1': {
      // first dump list of storages:
      uint32_t fsCount = storage.getFSCount();
      Serial.printf("\nDump Storage list(%u)\n", fsCount);
      for (uint32_t ii = 0; ii < fsCount; ii++) {
        Serial.printf("store:%u storage:%x name:%s fs:%x\n", ii,
                      mtpd.Store2Storage(ii), storage.getStoreName(ii),
                      (uint32_t)storage.getStoreFS(ii));
      }
      Serial.println("\nDump Index List");
      storage.dumpIndexList();
    } break;
    case '2':
      Serial.printf("Drive # %d Selected\n", drive_index);
      mtpDisk = storage.getStoreFS(drive_index);
      break;
    case 'd':
      dumpLog();
      break;
    case 'r':
      Serial.println("Send Device Reset Event");
      mtpd.send_DeviceResetEvent();
      break;
    case 'R':
      Serial.print(" RESTART Teensy ...");
      delay(100);
      SCB_AIRCR = 0x05FA0004;
      break;
    case '\r':
    case '\n':
    case 'h':
      menu();
      break;
    default:
      menu();
      break;
    }
    while (Serial.read() != -1)
      ; // remove rest of characters.
  }

  if (write_data)
    logData();
}

void logData() {
  // make a string for assembling the data to log:
  String dataString = "";

  // read three sensors and append to the string:
  for (int analogPin = 0; analogPin < 3; analogPin++) {
    int sensor = analogRead(analogPin);
    dataString += String(sensor);
    if (analogPin < 2) {
      dataString += ",";
    }
  }

  // if the file is available, write to it:
  if (dataFile) {
    dataFile.println(dataString);
    // print to the serial port too:
    Serial.println(dataString);
    record_count += 1;
  } else {
    // if the file isn't open, pop up an error:
    Serial.println("error opening datalog.txt");
  }
  delay(100); // run at a reasonable not-too-fast speed for testing
}

void stopLogging() {
  Serial.println("\nStopped Logging Data!!!");
  write_data = false;
  // Closes the data file.
  dataFile.close();
  Serial.printf("Records written = %d\n", record_count);
  mtpd.send_DeviceResetEvent();
}

void dumpLog() {
  Serial.println("\nDumping Log!!!");
  // open the file.
  dataFile = mtpDisk->open("datalog.txt");

  // if the file is available, write to it:
  if (dataFile) {
    while (dataFile.available()) {
      Serial.write(dataFile.read());
    }
    dataFile.close();
  }
  // if the file isn't open, pop up an error:
  else {
    Serial.println("error opening datalog.txt");
  }
}

void menu() {
  Serial.println();
  Serial.println("Menu Options:");
  Serial.println("\t1 - List USB Drives (Step 1)");
  Serial.println("\t2 - Select USB Drive for Logging (Step 2)");
  Serial.println("\tl - List files on disk");
  Serial.println("\te - Erase files on disk");
  Serial.println("\ts - Start Logging data (Restarting logger will append "
                 "records to existing log)");
  Serial.println("\tx - Stop Logging data");
  Serial.println("\td - Dump Log");
  Serial.println("\tr - reset MTP");
  Serial.println("\tR - restart Teensy");
  Serial.println("\th - Menu");
  Serial.println();
}

void listFiles() {
  Serial.print("\n Space Used = ");
  Serial.println(mtpDisk->usedSize());
  Serial.print("Filesystem Size = ");
  Serial.println(mtpDisk->totalSize());

  File root = mtpDisk->open("/");
  printDirectory(root, 0);
  root.close();
}

void eraseFiles() {
  //DBGSerial.println("Formating not supported at this time");
  Serial.println("\n*** Erase/Format started ***");
  mtpDisk->format(1, '.', Serial);
  Serial.println("Completed, sending device reset event");
  mtpd.send_DeviceResetEvent();
}

void printDirectory(File dir, int numSpaces) {
  DateTimeFields dtf;
  while (true) {
    File entry = dir.openNextFile();
    if (!entry) {
      // Serial.println("** no more files **");
      break;
    }
    printSpaces(numSpaces);
    Serial.print(entry.name());
    printSpaces(36 - numSpaces - strlen(entry.name()));

    if (entry.getCreateTime(dtf)) {
      Serial.printf(" C: %02u/%02u/%04u %02u:%02u", dtf.mon + 1, dtf.mday, dtf.year + 1900, dtf.hour, dtf.min );
    }

    if (entry.getModifyTime(dtf)) {
      Serial.printf(" M: %02u/%02u/%04u %02u:%02u", dtf.mon + 1, dtf.mday, dtf.year + 1900, dtf.hour, dtf.min );
    }
    if (entry.isDirectory()) {
      Serial.println("  /");
      printDirectory(entry, numSpaces + 2);
    } else {
      // files have sizes, directories do not
      Serial.print("  ");
      Serial.println(entry.size(), DEC);
    }
    entry.close();
  }
}

void printSpaces(int num) {
  for (int i = 0; i < num; i++) {
    Serial.print(" ");
  }
}

uint32_t CommandLineReadNextNumber(int &ch, uint32_t default_num) {
  while (ch == ' ')
    ch = Serial.read();
  if ((ch < '0') || (ch > '9'))
    return default_num;

  uint32_t return_value = 0;
  while ((ch >= '0') && (ch <= '9')) {
    return_value = return_value * 10 + ch - '0';
    ch = Serial.read();
  }
  return return_value;
}
 
As I mentioned earlier (maybe Github Issue),

In the old days like with DOS or the like, I don't remember there being a pre-allocate method, but instead a program might do something like:
Open the file, use the Seek operation to large size, (Maybe have to do write to make it take), seek to start of file, do all of their write operations, and then truncate and close the file.

And I do see cases where it would be nice to be able to take an FS handle and/or a File Handle and be able to get back to the underlying File System. In majority of cases don't need it that often, but again I have done that before on both Windows and Linux.

Been a long time since I tried it on Wndows, and I think it has become easier with: https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfinalpathnamebyhandlea

In the case I mentioned earlier about needing to to handle Serial port (Dynamixel Servos latency), I would detect if the File Descriptor was FTDI, probably in a convoluted way, by:
using:
Code:
		sprintf(szProcFD, "/proc/self/fd/%d", gSocket_fd);
		ich = readlink(szProcFD, szPath, sizeof(szPath));
And look for ttyUSB

So again question is do we provide something like this?
Something like:
Code:
File myFile;
FS *pfs = myFile.getFS();

While walking the dogs, I had another completely random idea, that might be interesting.

Suppose we add an extra uint32_t data item to both FS and File: maybe choose a name like in Events like Context.
Add two member functions to both: getContext, setContext.

Now suppose by default when we create a File we default set the Context to the FS...

And maybe by Default for FS, we set the context of each to some Special values: FS::SD_FILESYSTEM maybe defined like: 0xFFFF0001 or some such invalid address and also
High word not 0... We could define ones for each of the LITTLEFS files we open. Ditto for MSC volume.

Now sketches Like MTP we could set them different if that makes sense.
But for example with the above we might be able to do something like:

Code:
void myFunction(FS* pfs) {
    if (pfs->getContext() == FS::SD_FILESYSTEM) {
      SDClass*psd = (SDClass*)pfs;
      uint32_t  blocks_per_cluster = psd->blocksPerCluster();
    }
...

Again I know completely random
 
In the old days like with DOS or the like, I don't remember there being a pre-allocate method, but instead a program might do something like:
Open the file, use the Seek operation to large size, (Maybe have to do write to make it take), seek to start of file, do all of their write operations, and then truncate and close the file.
This might work if you do the write to new position unless LittleFS has something that forbids going to a file area that hasn't been written to yet. That we can try now.

And I do see cases where it would be nice to be able to take an FS handle and/or a File Handle and be able to get back to the underlying File System. In majority of cases don't need it that often, but again I have done that before on both Windows and Linux.
Well might be possible to set another attr on each file to know where its is - SPIflash, NAND etc but you would still need a link from the storageID to media type. Now this might be possible by having each class in LittleFS have another function like whoAmI that returns whatever we want to designate and then append it to the file with setattr then its just a matter of getting that info.

Note these are just me talking out loud.
 
This might work if you do the write to new position unless LittleFS has something that forbids going to a file area that hasn't been written to yet. That we can try now.

Well might be possible to set another attr on each file to know where its is - SPIflash, NAND etc but you would still need a link from the storageID to media type. Now this might be possible by having each class in LittleFS have another function like whoAmI that returns whatever we want to designate and then append it to the file with setattr then its just a matter of getting that info.

Note these are just me talking out loud.
Me too ;)

I was thinking that if by chance we went with something like the get/set Context like stuff we could define values anyway we want so could do something like:
Code:
enum {
	CONTEXT_FS_SD 				0xFCFC0100,

	CONTEXT_F_LFS_RAM 			0xFCFC0200,
	CONTEXT_F_LFS_SPI_FLASH 	0xFCFC0210,
	CONTEXT_F_LFS_SPI_NAND 		0xFCFC0211,
	CONTEXT_F_LFS_SPI_FRAM 		0xFCFC0212,
	CONTEXT_F_LFS_QSPI_FLASH 	0xFCFC0220,
	CONTEXT_F_LFS_QSPI_NAND 	0xFCFC0221,
	CONTEXT_F_LFS_PROGRAM 		0xFCFC0230,
}

As I think I mentioned recently, was trying to assemble another of the memory boards, and at times have troubles soldering those 8WSON chips, with my 10 left thumbs.
So I did another quick and dirty memory board which only holds 3 chips but with the 16 soic parts. The order arrived yesterday at our mailbox, picked up this morning and assembled one.

screenshot.jpg
So tried it with the example sketch SD_Program_SPI_QSPI_MTP-logger, which use some wrappers for the SPI/QSPI littleFS classes and tries to work with which ever LittleFS chips are on the pins...

screenshot2.jpg

Now to play
 
Me too ;)
...
As I think I mentioned recently, was trying to assemble another of the memory boards, and at times have troubles soldering those 8WSON chips, with my 10 left thumbs.
...
Now to play

Have fun playing ...

I tried a WSON on the KurtE 4x Flash and one side worked but the other side exposed too little of the pad to get solder on 2 pads with the fat new tip on my iron :(

The old tip was okay - and smaller - not sure where I put it, so that got stalled.

It is a few year old SparkFun decent temp control unit - but not sure about tips - this extra was one they carried at some point - but larger business end. I see an alternate station for about $50 on amzn that comes with multiple tips.
 
I know this opinion will be unpopular, but I really want to avoid APIs which essentially cheat the FS abstraction layer. Yes, it is an easy path to solving some types of performance issues, but this sort of design comes with long-term costs.
 
Have fun playing ...

I tried a WSON on the KurtE 4x Flash and one side worked but the other side exposed too little of the pad to get solder on 2 pads with the fat new tip on my iron :(
For the fun of it I will play around with the board for smaller chips and increase the length of the pads and order another set... As this size board is pretty cheap to make...

I do have two different soldering stations, one with small tip but still had difficulty...

I know this opinion will be unpopular, but I really want to avoid APIs which essentially cheat the FS abstraction layer. Yes, it is an easy path to solving some types of performance issues, but this sort of design comes with long-term costs.
I am not sure which ones cheat the abstraction?

I hear you, but taken to an extreme, all of the Teensy hardware should be defined within a Hardware Abstraction Layer, but as you know that comes with it's whole other sets of issues. And I am in no way suggesting that we go down that route!

As for short term versus long term, not sure where best to draw the line, that is, already with SD object you allow the sketch the ability to talk to the SDFat object...

And for example if I have a sketch that needs (or at least strongly wants) additional information about an FS, like lets say MTP stuff and I want the behavior of the code to be different maybe depending on which storage (FS) is acted on. As a developer I can see that if the library/core code provides the necessary data/hints, I will use them, if not I will either punt or roll my own.

Maybe sort of like the MTP storage class, where I can store additional information, like title, and maybe other things like capabilities... So then in the code when I need to decide things, I look through the table of storage looking for the matching FS pointer... Or I use some other token that I then map back tot he FS...

A valid opinion only if you'd provided a better solution. You're not providing *any* solution.
No solution means long term costs, short term. Now.
I.e. fast writes to SD are not possible.
Now.
Concurrend accesses (IRQ etc)are not handled. Now, and never will.
I am not sure Paul has said that nothing else will go into FS or File, so hard to say if some of these features will be implemented or not.

Again not sure about things, like fast writes can not be done to SD? Why not? We simply call through to the same write function. If talking about the preAllocate, again don't know if simply doing the seek operation can do it today for you or not?

Probably enough for one post. And again was just throwing out there something random. :eek:
 
Ok back to sflash5.

Can you remind me about this hardware? Is it one of the test boards I sent a couple months ago? Maybe one that looks like this?

sflash.jpg

I have 3 of those on my desk, but all are smaller chips. Looks like I might need to solder another board today....
 
Can you remind me about this hardware? Is it one of the test boards I sent a couple months ago? Maybe one that looks like this?


I have 3 of those on my desk, but all are smaller chips. Looks like I might need to solder another board today....

Close its the one with 2 Winbond W25Q512--one IM and one IQ, one Winbond N01 and one Winbond N02 chips:
IMG-0553.png
 
Oh, *that* board. Seems like I would have built one for myself, but can't find it now. Fortunately I do have spare parts.

w.jpg

Can you remind me of which chip is on each CS pin? Would be nice if I build another one the same way.
 
Back
Top