Teensyduino File System Integration, including MTP and MSC

@KurtE
@all - Sort of wondering what the normal usage patterns that are expected to work with MTP? For me, I see it as a way to get a few files back and forth between host and Teensy, not one that for example to the host works like a full filesystem that apps on the Host can manipulate...
Not sure about normal usage but if I have a T4.1 plus an SD Card I can see the following situation
1. QSPI
2. RAM
3. SPI on audio card
4. SDIO
5. SD Card on Audio board
6. Jump Drive or SDD on USB Host.

Probably less depending on what someone wants

EDIT: For me most of what I am looking at is maybe 2 at most
 
I modified Tim's SD2201 sketch to only write to the NAND2G 1 directory of the "ManyF" test files so that the 111 directory would be at the top level. This failed miserably and hung the sketch on 460.txt file:
Code:
 ========================================== Media Size=265289728	Used Size=262144
Make File:111/400.txt
	File size=400=400 end:zaaa.   1x   16

Make File:111/401.txt
	File size=401=401 end:zaaa.   2x   2E

Make File:111/402.txt
	File size=402=402 end:zaaa.   3x   46

Make File:111/403.txt
	File size=403=403 end:zaaa.   4x   5E

Make File:111/404.txt
	File size=404=404 end:zaaa.   5x   76

Make File:111/405.txt
	File size=405=405 end:zaaa.   6x   8E

Make File:111/406.txtFO:: set attribute creation failed
FO:: set attribute modified failed

	File size=406=406 end:zaaa.   7x   A6

Make File:111/407.txt
	File size=407=407 end:zaaa.   8x   BE

Make File:111/408.txt
	File size=408=408 end:zaaa.   9x   D6

Make File:111/409.txt
	File size=409=409 end:zaaa.  10x   EE

Make File:111/410.txt
	File size=410=410 end:zaaa.  11x  106

Make File:111/411.txt
	File size=411=411 end:zaaa.  12x  11E

Make File:111/412.txt
	File size=412=412 end:zaaa.  13x  136

Make File:111/413.txtFO:: set attribute modified failed

	File size=413=413 end:zaaa.  14x  14E

Make File:111/414.txt
	File size=414=414 end:zaaa.  15x  166

Make File:111/415.txtFO:: set attribute creation failed
FO:: set attribute modified failed

	File size=415=415 end:zaaa.  16x  17E

Make File:111/416.txt >> Fail Open << Make File:111/417.txt >> Fail Open << Make File:111/418.txt >> Fail Open << Make File:111/419.txtFO:: set attribute creation failed
FO:: set attribute modified failed

	File size=419=419 end:zaaa.  17x  197

Make File:111/420.txt >> Fail Open << Make File:111/421.txt >> Fail Open << Make File:111/422.txtFO:: set attribute modified failed

	File size=422=422 end:zaaa.  18x  1B0

Make File:111/423.txt >> Fail Open << Make File:111/424.txt >> Fail Open << Make File:111/425.txtFO:: set attribute creation failed

	File size=425=425 end:zaaa.  19x  1C9

Make File:111/426.txt >> Fail Open << Make File:111/427.txt >> Fail Open << Make File:111/428.txtFO:: set attribute creation failed
FO:: set attribute modified failed

	File size=428=428 end:zaaa.  20x  1E2

Make File:111/429.txtFO:: set attribute creation failed
FO:: set attribute modified failed

	File size=429=429 end:zaaa.  21x  1FB

Make File:111/430.txtFO:: set attribute modified failed

	File size=430=430 end:zaaa.  22x  214

Make File:111/431.txt >> Fail Open << Make File:111/432.txtFO:: set attribute creation failed

	File size=432=432 end:zaaa.  23x  22D

Make File:111/433.txtFO:: set attribute creation failed

	File size=433=433 end:zaaa.  24x  247

Make File:111/434.txt
	File size=434=434 end:zaaa.  25x  261

Make File:111/435.txtFO:: set attribute creation failed
FO:: set attribute modified failed

	File size=435=435 end:zaaa.  26x  27B

Make File:111/436.txt >> Fail Open << Make File:111/437.txt >> Fail Open << Make File:111/438.txt >> Fail Open << Make File:111/439.txt >> Fail Open << Make File:111/440.txt >> Fail Open << Make File:111/441.txt
	File size=441=441 end:zaaa.  27x  295

Make File:111/442.txt >> Fail Open << Make File:111/443.txtFO:: set attribute modified failed

	File size=443=443 end:zaaa.  28x  2AF

Make File:111/444.txtFO:: set attribute creation failed
FO:: set attribute modified failed

	File size=444=444 end:zaaa.  29x  2C9

Make File:111/445.txtFO:: set attribute creation failed

	File size=445=445 end:zaaa.  30x  2E3

Make File:111/446.txt >> Fail Open << Make File:111/447.txt >> Fail Open << Make File:111/448.txtFO:: set attribute creation failed
FO:: set attribute modified failed

	File size=448=448 end:zaaa.  31x  2FD

Make File:111/449.txtFO:: set attribute modified failed

	File size=449=449 end:zaaa.  32x  318

Make File:111/450.txtFO:: set attribute creation failed
FO:: set attribute modified failed

	File size=450=450 end:zaaa.  33x  333

Make File:111/451.txt >> Fail Open << Make File:111/452.txt >> Fail Open << Make File:111/453.txt >> Fail Open << Make File:111/454.txtFO:: set attribute creation failed
FO:: set attribute modified failed

	File size=454=454 end:zaaa.  34x  34E

Make File:111/455.txtFO:: set attribute creation failed

	File size=455=455 end:zaaa.  35x  369

Make File:111/456.txtFO:: set attribute modified failed

	File size=456=456 end:zaaa.  36x  384

Make File:111/457.txtFO:: set attribute modified failed

	File size=457=457 end:zaaa.  37x  39F

Make File:111/458.txt >> Fail Open << Make File:111/459.txtFO:: set attribute creation failed
FO:: set attribute modified failed

	File size=459=459 end:zaaa.  38x  3BA

Make File:111/460.txt

In MTP it looks like this:
Capture.PNG

I then switched to !G NAND Chip on the memory breakout board and it successfully wrote - without errors - all 1000 files:
Code:
....

Make File:111/1422.txt
	File size=1422=1422 end:zaaa.1023x DD68

Make File:111/1423.txt
	File size=1423=1423 end:zaaa.1024x DDBF

MakeData done.
took about an hour or so. But did notice that it would write files in batches of about 10 and then pause for about 20 or so seconds.

There is definitely something strange with that WIn2g chip for this kind of testing.

EDIT: I just finished downloading the files from Win1G chip to the PC and compared them to the files in the zip file. NO DIFFERENCES identified using Code Compare (1000 Files).
 
Last edited:
@KurtE
Not sure about normal usage but if I have a T4.1 plus an SD Card I can see the following situation
1. QSPI
2. RAM
3. SPI on audio card
4. SDIO
5. SD Card on Audio board
6. Jump Drive or SDD on USB Host.

Probably less depending on what someone wants

EDIT: For me most of what I am looking at is maybe 2 at most

For Normal/general use MTP seems a way to OFFLOAD data from a logging device, or to update configuration file(s) for the device in use.

Not seeing it as a device to replace a FLASH storage drive where it replaces for instance a USB to SD adapter for general transfer or usage.

Indeed, it might have more than one device where real time logging happens on one or the other Media at hand depending on speed or reliability and then is internally offloaded or segmented depending on the nature of the data to a second Media type. When connected to host those files would be offloaded or updated.
 
I modified Tim's SD2201 sketch to only write to the NAND2G 1 directory of the "ManyF" test files so that the 111 directory would be at the top level. This failed miserably and hung the sketch on 460.txt file:
...
took about an hour or so. But did notice that it would write files in batches of about 10 and then pause for about 20 or so seconds.

There is definitely something strange with that WIn2g chip for this kind of testing.

The SD2202 was updated to allow media swap with the #define DISK ...
> It could then even test that and have code to pick the begin based on the 'DISK'
> given that it could also make different calls to MakeD???() based on the 'DISK'
> Also the SD2202 version IIRC added this that may be quite useful: loop() { printDirectory(); }

So the problem is not in the MakeFiles.ino code ? or: MakeDataFiles( szStart[ 1 ], 1024, 400, 1 );

with: char szStart[][8] = { "", "ManyF", "ManyD10", "ManyD8", "MoreFD", "Ascii" };
Changing to ROOT should work with just: MakeDataFiles( szStart[ 0 ], 1024, 400, 1 );

This added text is coming from a 'low level' failure of this? : myFile = DISK.open(szPath, FILE_WRITE);
FO:: set attribute creation failed
FO:: set attribute modified failed

... gotta run to do some chores ...
 
On the question of anticipated usage, I would imagine nearly all uses will have no more than 4 filesystems "stores", and the majority will probably use just 1. Likewise, I would imagine most people will put a few dozen files in the root directory and probably not use subdirectories at all. For those using hierarchy, it's pretty unlikely the sorts of projects made for microcontrollers will nest more than 2 or 3 levels deep.

I do believe these very large tests have value in possibly catching rare bugs which we might miss with only smaller tests.

But when it comes to optimizing performance, we should probably focus on the likely usage scenarios. If the "large" tests show correct but slow behavior, that's probably fine. We have limited resources, especially RAM. People use Teensy for real-time projects where a slower file transfer that imposes less latency on real-time processes is probably better than one which competes faster by hogging the CPU.

Minimizing disruption to libraries like Audio, Servo, Wire, Serial1-8, OctoWS2811 during file transfers is one of the most important design goals, far more important than completing the transfer as fast as possible. The whole reason we're doing MTP rather than MSC (and the same reason Android switched from MSC to MTP) is so we can continue operating normally while the host reads and writes files.

I know this skirts all the more specific questions, but you did ask about expected uses and goals. Hopefully this helps?

Agree this is basic expected use and priority.
The system needs to be robust enough to not fail/fault when some odd situation is encountered.
Populating a 'WebServer' folder for Ethernet use might be worst case - but that would be odd/rare or maybe even while 'offline'. It needs to work reliably.

The SD2202 test of 3MB xfer that took almost 4 minutes on HDD ran in under 2 minutes against PC RamDrive. So the SPEED is there for raw transfer. That 3MB transfer has the ponderous DIR and filecount that took the Teensy about 35 seconds to parse from SD drive to tell the host what files existed, so the actual RamDrive transfer was closer to 1 minute for the 3 MB {thus 3 minutes against HDD} - and those 3MB of files consume 12 MB (with FS Slack space overhead) on the PC HDD.

This also points out that the MTP Host has issues of its own to deal with when it comes to getting the data stored to Media, so making the Teensy focus on speed would be misplaced if it compromises reliability or general function.

And the MakeFiles test was meant to test edge cases to make sure they weren't lurking undetected: File Count, Dir Depth, file sizes around the 500 and 512 byte boundaries and variations.
> if there are others they can be added
> the 1024 files differing by one byte was meant to catch each 400 to 1400 byte size, should also perhaps extend to 2048, 4096 region.
 
On the question of anticipated usage, I would imagine nearly all uses will have no more than 4 filesystems "stores", and the majority will probably use just 1. Likewise, I would imagine most people will put a few dozen files in the root directory and probably not use subdirectories at all. For those using hierarchy, it's pretty unlikely the sorts of projects made for microcontrollers will nest more than 2 or 3 levels deep.

I do believe these very large tests have value in possibly catching rare bugs which we might miss with only smaller tests.

But when it comes to optimizing performance, we should probably focus on the likely usage scenarios. If the "large" tests show correct but slow behavior, that's probably fine. We have limited resources, especially RAM. People use Teensy for real-time projects where a slower file transfer that imposes less latency on real-time processes is probably better than one which competes faster by hogging the CPU.

Minimizing disruption to libraries like Audio, Servo, Wire, Serial1-8, OctoWS2811 during file transfers is one of the most important design goals, far more important than completing the transfer as fast as possible. The whole reason we're doing MTP rather than MSC (and the same reason Android switched from MSC to MTP) is so we can continue operating normally while the host reads and writes files.

I know this skirts all the more specific questions, but you did ask about expected uses and goals. Hopefully this helps?

Thanks, that is answering some of the reals basics which is a good thing.

I totally agree that large tests are great for doing system wide stress tests. Unit tests and Component tests are also great... Oops sounds like I am not retired ;)

As I mentioned I think 1-4 storage units is probably the normal max... Our own stuff may have more.

I believe it was my Android(Kindle Fire) and/or IPad/IPhone when plugged in might have had like 2 storage named like: Internal, External,
I think some of them had then Top level Directory names, like: Pictures, Audio (or sounds), and maybe one or two others.
Under Pictures: it had a few others like: All(don't remember name), and different Album names and that was about it. So maybe 2 to 4 layers deep.

Yes need to handle more. Also noting that Pictures could have 1000 of them in it.

Usages as you mentioned: I typically would not be doing many of those things, playing Audio or servos at the same time, Hardware devices like Serial should not be impacted, much unless someone is turning on/off their interrupts.

MTP vs MSC (I always figured they went with MSC apple did not use it ;) just kidding)
I see good usage patterns for both. There are nice things being seen from the host as an FS. You don't have to run mtp specific code, instead you could double click on a file, open it up with your editor and save the file back out... As for overhead of one versus other, no idea...

As for having the ability for the MSC code to upload and download files with minimal interruption to other things, I am pretty sure we all agree with. The only disagreement might be on what approach to take to get there. Which I don't want to open that can of worms here :D

But ideas on possible usage patterns, and maybe places we should think about and have simple examples for:

a) simple logger - We have been there and done that

b) simple logger - BUT with a twist that MTP is not setup when the sketch starts - Example my never to be completed well monitor program, but suppose I have the Teensy running in my pump house for a month, collecting data, like when the three different things start and stop (Well 1, Well 2, Pressure Pump). I then decide I want to go analyze the data and so I plug in my laptop to the Teensy... Only at this point will there be USB connectivity... Which brings up a couple of side questions like:
1) detecting when the USB has started and then go though the MTP population of storage...
2) Security - maybe I only want someone who enters a password into the touch screen on it to be able to access MTP (like Apple asks do you trust...)
3) Probably detecting when it goes away... Ok to just disconnect, or should there in my logical case have button that says disconnect, which maybe can/should be used to signal to different storage units that they may want to do something like a sync operation.

c) As part of larger logical device or the like: example I want to produce a smart keyboard using a Teensy. So Would like to setup using USB Joystick and MTP - so maybe I can download to keyboard a new macro definition file, that special key X will output the following...

d) Audio or USB Audio - see downloading files to be played, maybe Play List files...

Guessing c) and d) may be beyond this round...
 
defragster said:
So the problem is not in the MakeFiles.ino code ? or: MakeDataFiles( szStart[ 1 ], 1024, 400, 1 );

Think you missed my post#948 where I identified an issue with LittleFS:
At Tim's suggestion I went ahead and tried his sketch to makefiles directly on the Win2G chip and to my surprise it failed to created second level directories:
Code:
root:
  |
  |- level 1
      |- level 2 (FAILS TO CREATE)
more details in that post. That would preclude me from using the sketch as is. In other words if I was running "ManyF" as shown in the sketch it would want to create a ManyF directory say on the Win1G storage device + 3 sub-directories: Win1G -> ManyF -> 111, Win1G -> ManyF -> 222, and Win1G -> ManyF -> 333. The sketch was 1 ], but I also told it to only create 1 directory. You tested on a HDD which you showed is fairly fast - on the Win1G Nand it took close to an 1 hour to just write 1 directory and it served its purpose for me to test whether it would transfer those files to the PC.

This added text is coming from a 'low level' failure of this?
Yep that is coming from file open function and seems to be an issue isolated to the Win2G nand chip as I mentioned in one of the posts.
 
Think you missed my post#948 where I identified an issue with LittleFS:
more details in that post. That would preclude me from using the sketch as is. In other words if I was running "ManyF" as shown in the sketch it would want to create a ManyF directory say on the Win1G storage device + 3 sub-directories: Win1G -> ManyF -> 111, Win1G -> ManyF -> 222, and Win1G -> ManyF -> 333. The sketch was 1 ], but I also told it to only create 1 directory. You tested on a HDD which you showed is fairly fast - on the Win1G Nand it took close to an 1 hour to just write 1 directory and it served its purpose for me to test whether it would transfer those files to the PC.

Yep that is coming from file open function and seems to be an issue isolated to the Win2G nand chip as I mentioned in one of the posts.

Indeed, reading bottom up and 15 minutes late as it was ... - just wanted to confirm there wasn't something uncaught in MakeFiles.ino.

Good you are set up to test that stuff as it will need it. LFS Integrity only ever went one level deep - which seemed reasonable for the space they provide.

Trying to keep up with Paul and USB MTP changes I was going for the big stuff that was failing/working before on SD - and my sanity being able to move the card back and forth to PC as needed.
 
Just in case anyone's wondering, the "Win2G" chip is from Winbond and the full part number is W25N02KVZEIR.

"Win1G" is (probably) Winbond W25N01GVZEIG.

Both of these are NAND flash, where some sectors may be defective. The NAND chips store error correcting codes with each sector. They also tend to have larger sectors and blocks than NOR chips, where all bytes within the chip are assumed to be reliable and don't need ECC or bad sector management. The NAND chips tend of have faster erase times (NOR erase is painfully slow) but NAND writes have extra latency for ECC and simply take longer because the minimum write length is so much longer. The NAND chips have a very different SPI / QSPI protocol too.

This sort of shorthand can be confusing because 1Gbit NOR flash chips are now selling and 2Gbit NOR have been announced. 4Gbit QSPI NAND are also apparently coming soon.
 
Just in case anyone's wondering, the "Win2G" chip is from Winbond and the full part number is W25N02KVZEIR.

"Win1G" is (probably) Winbond W25N01GVZEIG.

Both of these are NAND flash, where some sectors may be defective. The NAND chips store error correcting codes with each sector. They also tend to have larger sectors and blocks than NOR chips, where all bytes within the chip are assumed to be reliable and don't need ECC or bad sector management. The NAND chips tend of have faster erase times (NOR erase is painfully slow) but NAND writes have extra latency for ECC and simply take longer because the minimum write length is so much longer. The NAND chips have a very different SPI / QSPI protocol too.

This sort of shorthand can be confusing because 1Gbit NOR flash chips are now selling and 2Gbit NOR have been announced. 4Gbit QSPI NAND are also apparently coming soon.

Hi Paul - didn't realize about NOR 2Gbit Flash and 4Gbit QSPI NAND chips - we have always been working the Memory Board with has the W25N01GVZEIG and the W25N02KVZEIR so hence the shorthand.

Guess in the future will be clearer with the chip. The 1Gbit NOR flash - I should have remembered since I do have a W25Q01JV soldered up on a breakout but have not really used it yet for anything except to make sure it worked.

As for ECC - that is very true, and we do have the ability to read the ECC status as I did leave that function in the library. Winbond did not implement Bad Block Management on the W25N02 NAND chip. . I just ran a little test sketch to read the ECC for the W25N02KVZEIR flash and no errors were found - if I implemented it correctly. Think there is more on the LittleFS thread https://forum.pjrc.com/threads/58033-LittleFS-port-to-Teensy-SPIFlash?p=264385&viewfull=1#post264385.
 
The SD2202 was updated to allow media swap with the #define DISK ...
> It could then even test that and have code to pick the begin based on the 'DISK'
> given that it could also make different calls to MakeD???() based on the 'DISK'
> Also the SD2202 version IIRC added this that may be quite useful: loop() { printDirectory(); }

So the problem is not in the MakeFiles.ino code ? or: MakeDataFiles( szStart[ 1 ], 1024, 400, 1 );

with: char szStart[][8] = { "", "ManyF", "ManyD10", "ManyD8", "MoreFD", "Ascii" };
Changing to ROOT should work with just: MakeDataFiles( szStart[ 0 ], 1024, 400, 1 );

This added text is coming from a 'low level' failure of this? : myFile = DISK.open(szPath, FILE_WRITE);


... gotta run to do some chores ...

Mike - sorry this should have pointed to: void MakeDeepDirs( char* szRoot, int numDirs, int numFiles, uint32_t startSize, uint32_t growSize, int compoundGrow ) {

LIKE: MakeDeepDirs( szStart[ 0 ], 1, 6, 100, 400 );
> That makes one folder/Dir in the root with those Params 1 and 2 - and the other 3 up to desired file
Of course, that only works once with fixed folder name of "D0"

BTW: the SD2202 has the updated MakeFiles.ino packed in it. If I make worthy edits I'll publish SD2203 - looking to include the items mention in earlier post to differentiate same size files with growSize=0, and perhaps put a file count.txt in each created folder of SIZE==# files put in folder (excluding the count.txt?).
 
Just an FYI update, I pushed up some changes to my fork/branch (cache_storage_records) which still has the same 8 record cache...
Only differences is that I changed the max file name length to be 256-28, so with the other overhead records are 256 which map well to read/writes to storage.

I also added second define for max path length 260 and updates some of the code places to differentiate on usage...

I also as mentioned changed the Serial output for writes/reads of storage records to always output regardless of DEBUG level in storage...

I was thinking of merging but instead of thinking of trying a different approach. So far just playing...

For low memory machines default to more or less unchanged and rely totally on external storage.

But for higher memory setup, thinking about maybe a different approach: regardless can reduce overhead per record, for example maybe like:
Code:
	struct Record {
	  uint32_t dtModify;
	  uint32_t dtCreate;
	  uint16_t parent;
	  uint16_t child; // size stored here for files
	  uint16_t sibling;
	  uint8_t store_type; // 4-5 bits for store number, bit for directory bit for scanned 
	  uint8_t name_len; 
	  char name[MTP_MAX_FILENAME_LEN];
	};
So 16 bytes instead of 28... I am thinking of working with blocks of memory, example lets say 2K. With current uncached version with 256 bytes this 2k would hold 8 records.

Now was thinking of packing the 2K (could be 4K), and if we on average have file names of 16 byte names, with overhead of 16 so 32 bytes... we could pack in each 2K buffer maybe an average of 64 records.
Maybe a few less depending on how we allow access to individual records within each block... And how much space to leave in each one to make it easier to handle if files get renamed.

Without any additional information would need to walk the records, like know this block starts at record 50 and we want 55, we would walk from 50 to 55
Each block could have a header block, at start maybe keep the information like, which if we need a condition where the blocks are not consecutive, a link before/after... And could for example limit each block to not hold more than N records, and have 2*N bytes at start for where each record starts in our block...

But wondering if I am over thinking this...
 
Morning all
Did some tests last night with the W25N02 chip on the Memory Board. This assumes there are no bad blocks being hit. I did switch to @KurtE's branch with the index file changes last night.

1. Attempted to transfer the manyF/111 files again and hit the same issues as described before where a lot of files would fail to copy from the PC to the W25N02.
2. Did a quick format and took a different approach and avoid many small files
3. Used my mtp_test_integrity sketch to write 4 'B'ig files (25MB) each. Each complete write without an issue.
Code:
 Space Used = 100925440
Filesystem Size = 265289728
Directory
---------
0_bigfile.txt                         25122816
1_bigfile.txt                         25122816
2_bigfile.txt                         25122816
3_bigfile.txt                         25122816
on average each file too approximately 17.5 seconds.

4. Deleted those files and tied coping 91 files for a total of 202MB to the W25N02:
Code:
67 files copied successfully.
20 files did not copy but these all had long file names >= 50 characters.
4 just failed to copy

5. Note disk shows a full but....
Code:
 Space Used = 265289728
Filesystem Size = 265289728
however adding up the file sizes:
Code:
166433023 bytes
 
Morning all
Did some tests last night with the W25N02 chip on the Memory Board. This assumes there are no bad blocks being hit. I did switch to @KurtE's branch with the index file changes last night.

5. Note disk shows a full but....
Code:
 Space Used = 265289728
Filesystem Size = 265289728
however adding up the file sizes:
Code:
166433023 bytes

Morning all as well,

What I don't understand fully is the LittleFS configuration for these chips:
Specifically what is the logical SectorSize...

My quick look at data table is:
{{0xEF, 0xAA, 0x22}, 24, 2048, 131072, 265289728, 2000, 15000}, //Winbond W25N02G

config.block_size = info->erasesize;

Is the littlefs block size is then: 131072 or 128KB

What I don't understand enough about littleFS is what is the minimum number of blocks that a file might Use.
And I am assuming that each directory also takes a minimum of one block. But is the directory data also stored with it or another block? What about previous "generation?" blocks...

So for sure you would need to do round ups for each file to block sizes...
 
From my understanding LFS uses 2 blocks to maintain a linked list of the files and directories. Then for each file would need to calculated number of blocks rounded up and then convert back to bytes. I did this in an excel spreadsheet and:
Code:
 total file size = 166,433,023, 	 Total file Size + attribute space = 166,501,631 	 Expected total file size = 171,573,248.00 bytes
But not the a full disk?
 
As a follow on to previous post I did a low level format of the W25N02 chip and copied about 190 Mbytes to the chip - wanted to make sure I didn't hit 0 free space. This time
Code:
calculated space: 194,641,920 bytes
Space Used from Print Directory: 194,117,632 bytes
Delta: 524,288 bytes

So a lot closer to our estimate.

EDIT: Is it possible we are corrupting the storage when we try writing after disk is full?
 
USB Serial question with MTP:
> current is Seremu? Not standard USB Serial?
> Serial can work - but still slow to start IIRC?
> Wondering if DUAL_Serial could be used and that might allow a faster starting independent channel?

As a follow on to previous post I did a low level format of the W25N02 chip and copied about 190 Mbytes to the chip - wanted to make sure I didn't hit 0 free space. This time
Code:
calculated space: 194,641,920 bytes
Space Used from Print Directory: 194,117,632 bytes
Delta: 524,288 bytes

So a lot closer to our estimate.

EDIT: Is it possible we are corrupting the storage when we try writing after disk is full?

Quite possible! For LFS_Integrity a 'slack value' was put in to avoid getting 'write fail disk full'. Mostly tested on LFS_RAM's that were faster, smaller, and not suffering wear.

Can't say now for sure - but it seems space up to partial write failure was 'orphaned'/lost. Either way it voided the integrity test so it was avoided with test - from bigFile2MB():
Code:
		if ( toWrite > (65535 + (myfs.totalSize() - myfs.usedSize()) ) ) {
			Serial.print( "Disk too full! DO :: q or F");
			return;
		}

In the middle of MakeFiles edits:
> if compiled "MTP" it enables with MTP addFilesystem(SD {opps}) and loop() under #ifdef USB_MTPDISK
> When files in a folder are created with 'same size and not grown' they are given an index in the filename
> When a folder with files is created instead of just "D0" it is created "D0.#" with the number of files to be put there
-- Rather than yet another file for count of files, the folder name can be used to Verify the expected number of files
-- Allows mult dirs with diff # of files to coexist where each may be files of diff size and not corrupt the file count
>> Not yet done anything with start of 'Verify' DirWalk
 
Question: How is Windows informed of new files/folders to update the MTP view?
>> MakeFiles for instance is busy making files when it starts as MTP, added delay for USB to be online, but it takes some time.
>> If the Explorer view of MTP drive is opened during file creation the state of the Media at that point is shown, and not updated.
>> More upper level folders are under creation, but the view is not updated or changed- even with refresh

Update made to MakeFiles on github
> still WIP but it is back to prior function with fixes made from p#967 notice
 
Verify has moved along and is updated: Defragster/T4LockBeta/tree/main/MakeFiles

Moved along but not working fully - it skips dirs as written so far

> Does not yet verify file content
> All files in a DIR count toward DIR.6 where ".6" says 6 files are expected
> Verifies reported file size to match that in the file name #SIZE.txt or #SIZE_xxx.txt
> Dirs without a ".#' are ignored for verification!

-Will run NON MTP to create files on SD card.
-Running as MTP build it will add media content indicated in notSetup() {moved from setup()} on 20 sec delay in loop() for MTP to let SerMon connect.
> After running that and "MakeData done." shows you can hit 'Enter' in Sermon to do a Directory Verify
-> It no longer prints each file in those checked folders as another func() does the work and advance the openNextFile(), and dir 'printing' was removed in the Verify
--> it really isn't doing a PrintDirectory().

If run on unclean media some variations will result in 'detected errors' from unexpected files depending on what may have changed.

> Must use this version of Makefiles.ino to generate subdirectories
-> it has the folder name.COUNT of files used to verify no lost files

> It also allows making a dir of same sized files of chosen size
-> prior versions always expected a series of files with added content between files
-> these same size files get size_File#.txt, where File# changes if the content creation is changed.

Why is this helpful?: when Run as MTP.addFilesystem(DISK, "MakeFiles 2203");
> With known 'expected' file/folder content properly moved or edited MTP should never fail to leave Teensy Media Verifiably correct
-> This allows semi real time/'On Demand' verification testing without removing media or running another sketch
> Starting the sketch on EMPTY media will fill it with KNOWN content.
> COPY the data to PC and compare manually to SRC copy (like CodeCompare) to detect errors from Round Trip File copy
-> Or copy back to the MTP Media and hit Sermon 'Enter' and if ALL is well the content will verify
--> Either clean the media or put it in a subdirectory as desired
> Keep source on HOST PC and copy a folder over and do printDirectoryVerify()
> Make edits to MTP files on the HOST and confirm only the expected changes
-> delete a file from a counted directory: MISSING 1 FILE(S) 1 File Error(s)
-> rename a file like 1424 with wrong file size to 1425:
Code:
        DIR    333.1024 / 
1425.txt	XXXX	 File Size MISMATCH :1425
  DONE RESULTS BAD HERE  	XXXXXX	 1 File Error(s)
NOTE: Doing simple edits may result in OTHER MEDIA elements going bad!
-> I think I saw this once - but was busy editing other stuff so it may have been code reporting error, or restart file change.
> Files of arbitrary Size and number can be created in one or more folders

NOTE 2: After doing some edits one folder showed 404.txt? Going back gave wrong icon for all files

Note 3: Navigating down and back up resulted in PC Explorer confusion showing this path that does not exist:
Code:
This PC\Teensy\MakeFiles 2203\ManyF\222.1024\[B]222.1024\222.1024[/B]
Each time I double clicked 222.1024 it redrew as if in 'ManyF' after clicking into 333.1024 and then going 'back up'

Note 4: Right now MTP access to the Teensy SD Media is non-functional after browsing the media folders
> Data seems fine on the Teensy with DIR and fine on to restart MTP with Teensy repower.

TODO:
> Publish a new SD2203 {or SD2204 if I update the size of BLOCK#} ZIP for usable file content to copy 'Selected Folder(s)' from PC without having to run MakeFiles.
> Complete file content verification of expected content in 16 Byte Blocks: ...[zccc.3093x29ED4'\n'][zccc.3093x29ED5'\n']...
- each Block has same FILE#, and Each 0xBlock# is prior 0xBlock#++ to the end filler Bytes
- 5 hex char block# only good for 16MB of unique content, should have use 6-7 chars
-> okay now as long as no single run of MakeFiles passes this Mark or the 0xBlock#++ check would fail
- For REF: Keep and show Global File Count and Size in printDirectoryVerify() - it is only shown 'per directory'
- other feature creep ????
- ROOT file file verification isn't possible now - could add Folder on ROOT to key special case #_ROOT, or a File RootCnt.txt with SIZE of expected # files to trigger and verify file count
-> Add a func() to make files on Root with Key indicator
> Add MENU Commands
-> Only run/create notSetup() on command
-> Add Format Media
-> Run new printDirectoryVerify()
-> Run OLD printDirectory()
-> Run showMediaSpace();
-> Show Help
-> OTHER ????
--> TELL MTP host to RESYNC ???

Any questions or feedback or PR's I'm offline a few hours.
> Mike if you want to PR #ifdef in media other than: #define DISK SD
 
Last edited:
W25N02 NAND FLASH Having Issues on Teensy MicroMod

As mentioned in a couple of posts I am having failures where some files are failing to be copied from the PC to W25N02 NAND as well as when creating multiple files using Tim's file creation sketch. So decided to test the N02 NAND using the LittleFS SPI Test_Integrity sketch that is one of the examples.

Bottom line it worked without errors on the T4.1 but fails on the TMM.
I did test this a couple times with the TMM and it showed failures each time.

Basic approach was to first do a Low Level Format ('F') and then create 3 'B'ig files (would = 232,259,584 bytes used) which would fill most of the N02 NAND.

Teensy MicroMod Running LFS Test Integrity SPI Sketch
On attempting to create the third Big File (33,075,200 bytes) received the following error message that it could not write that file
Code:
tart Big write of 33075200 Bytes
	Big write KBytes per second  0.39 

Bytes Used: 199098368, Bytes Total:265289728

Big write ERR# -28 0xFFFFFFE4
doing a directory listing and then doing a 'b' to erase the files verify the files received some bad byte indicators
Code:
printDirectory SPI_NAND
--------------
FILE	0_bigfile.txt		132491264
FILE	1_bigfile.txt		66234368

 0 dirs with 2 files of Size 198725632 Bytes
 Total 2 files of Size 198725632 Bytes
Bytes Used: 199098368, Bytes Total:265289728


	 Loop Count: 8 (#fileCycle=0), Bytes read 147042820, written 0, #Files=1
	 ERROR COUNT =2
	dirV...	Filecount mismatch 1 != 2
0[  7.80 M](0.01513 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >
139035_us	[  7.80 M](0.01513 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >
Delete with read verify all #bigfile's
	Verify /0_bigfile.txt bytes 132491264 : ..................................................	GOOD! >>  bytes 132491264
	Big read&compare KBytes per second 1785.52 
	Verify /1_bigfile.txt bytes 66234368 : .......<Bad Byte!  1! = 0 [0x30] @9961239
0[ 10.10 M](2.32291 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >

	Read Count fail! :: read 9961239 != f.size 66234368
0[ 10.10 M](2.32291 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >

Teensy 4.1 Running LFS Test Integrity SPI Sketch
On the other hand, when repeating the above steps using the same N02 chip on the memory board all files created successfully and no errors appeared:
Code:
Big write /2_bigfile.txt took 24.86 Sec for 33073152 Bytes : file3.size()=33073152
	Big write KBytes per second 1330.53 

Bytes Used: 232259584, Bytes Total:265289728

[  3.85 M](0.43389 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 > d
printDirectory SPI_NAND
--------------
FILE	0_bigfile.txt		132491264
FILE	1_bigfile.txt		66234368
FILE	2_bigfile.txt		33073152

 0 dirs with 3 files of Size 231798784 Bytes
 Total 3 files of Size 231798784 Bytes
Bytes Used: 232259584, Bytes Total:265289728


	 Loop Count: 5 (#fileCycle=0), Bytes read 0, written 0, #Files=3
	dirV...82763_us	[  3.93 M](0.01623 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >

[  3.93 M](0.01623 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >
Start Big write of 16494592 Bytes.........................................................................................................................................................................................................
Big write /3_bigfile.txt took 12.67 Sec for 16492544 Bytes : file3.size()=16492544
	Big write KBytes per second 1301.53 

Bytes Used: 248774656, Bytes Total:265289728

[  4.26 M](0.23357 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 > d
printDirectory SPI_NAND
--------------
FILE	0_bigfile.txt		132491264
FILE	1_bigfile.txt		66234368
FILE	2_bigfile.txt		33073152
FILE	3_bigfile.txt		16492544

 0 dirs with 4 files of Size 248291328 Bytes
 Total 4 files of Size 248291328 Bytes
Bytes Used: 248774656, Bytes Total:265289728


	 Loop Count: 7 (#fileCycle=0), Bytes read 0, written 0, #Files=4
	dirV...147155_us	[  4.31 M](0.01987 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >

[  4.31 M](0.01987 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >
Delete with read verify all #bigfile's
	Verify /0_bigfile.txt bytes 132491264 : ..................................................	GOOD! >>  bytes 132491264
	Big read&compare KBytes per second 1785.10 
	Verify /1_bigfile.txt bytes 66234368 : ..................................................	GOOD! >>  bytes 66234368
	Big read&compare KBytes per second 1789.07 
	Verify /2_bigfile.txt bytes 33073152 : ..................................................	GOOD! >>  bytes 33073152
	Big read&compare KBytes per second 1802.46 
	Verify /3_bigfile.txt bytes 16492544 : ..................................................	GOOD! >>  bytes 16492544
	Big read&compare KBytes per second 1804.45 

[  6.67 M](2.37936 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >

To further test the N02 NAND with the T4.1 I copied one of the ManyF directories )1024 files ranging in size from 400bytes to 1423bytes in 1 byte increments) from the PC to the N02 NAND. However I am still seeing failures with copying these files to the N02 NAND.

So on the oft chance that there may be issues with copying a lot of small files to the N02 I tried copying another directory where I was seeing 0-byte files copied and some files just not copied. This time it copied successfully without errors and I was able to play the files in windows directly from the N02.
 
Update to previous post #970 on the Teensy MicroMod.

I removed the camera from the PJRC Breakout board and reran the SPI Integrity sketch using the N02 on Pin 4 of the memory breakout board. This time it showed Disk Full on start up when there should have been files from the previous test.
Code:
LittleFS Test : File Integrity
printDirectory SPI_NAND
--------------
	>>>	>>>>> No Dir
 Total 0 files of Size 0 Bytes
Bytes Used: 262144, Bytes Total:265289728


	WARNING: DISK FULL >>>>>  Bytes Used: 265289728, Bytes Total:265289728

So tried a low level format from the integrity sketch and while it showed that it formatted looked like it formatted the N02.
Code:
 Done Formatting Low Level in 56863777 us.

	 Updated filecount 0
F
printDirectory SPI_NAND
--------------
	>>>	>>>>> No Dir
 Total 0 files of Size 0 Bytes
Bytes Used: 262144, Bytes Total:265289728
but when I tried to write a 'B' file it failed:
Code:
>Disk too full! DO :: b or q or F

To verify that the memory board was working I reran the sketch using the N01 NAND on pin 3 and it worked without an issue - it read the directory on start and was able to write to the N01 NAND:
Code:
LittleFS Test : File Integrity
printDirectory SPI_NAND
--------------
DIR	extras / 
	FILE	Audio.zip		9365
	DIR	Sample Test Files / 
		DIR	2001 / 
			FILE	calculations.wav		426300
			FILE	completed.wav		276460
			FILE	dangerous_to_remain.wav		372892
			FILE	enough_info.wav		513388
			FILE	functional.wav		237356
			FILE	one_moment.wav		202236
			FILE	operational.wav		772140
.... there are more files but didn;t to take up space
and on write of a small file
Code:
Big write /0_2MBfile.txt took  1.71 Sec for 2048000 Bytes : file3.size()=2048000
	Big write KBytes per second 1195.34 

Bytes Used: 83099648, Bytes Total:131596288
 
That's very odd Mike!

So that is just T_4.1 .vs. the PJRC MMod breakout for SPI access - not using any SFun carrier boards wired to that 4X SPI flash MemBoard?

Wondering if there is some other 'iffy' connect on the MMod breakout from what seemed a bad PCB fabrication?

Could you hand wire an MRAM or something to that same CS==4 and test?

In the testing I did, the only "WARNING: DISK FULL >>>>> " on restart I saw seemed to have a reason at the time. They were 'long' ago now, and few in number, but never were something I saw as a problem.

Is taking the Camera off expected to have/cause some conflict? Or was that just incidental?

I can look to try that here - perhaps later today. Perhaps your M02 NAND@#4 is just exhausted from your testing? Is the "ECC" code or another edit in place?
 
defragster said:
That's very odd Mike!
Yes it is

So that is just T_4.1 .vs. the PJRC MMod breakout for SPI access - not using any SFun carrier boards wired to that 4X SPI flash MemBoard?
Tried Hardwiring the memory board to the ATP carrier to test the N01 and N02 NAND chips but neither was recognized, maybe the wires are too long. But the W25Q512 NOR Flash chips were recognized, and the sketches ran.

Code:
Is taking the Camera off expected to have/cause some conflict? Or was that just incidental?
Wasn't expecting the camera to cause any issues as it was on the 4bit port and doesn't use any of the SPI or CS pins for the memory board. The LCD was not attached either.

Unfortunately I do not have any other N02 NAND's to test - I do have a M02 that I may try on pin 4
 
I just checked Mouser and Digikey, both are out of stock on the W25N02KVZEIR NAND. So can not even get one so I can compare apples to apples. But in lieu of the N02 I used a M02 NAND 2Gbit Flash and reran the SPI integrity sketch - all tests passed.

Code:
printDirectory SPI_NAND
--------------
FILE	0_bigfile.txt		132491264
FILE	1_bigfile.txt		66234368
FILE	2_bigfile.txt		33073152

 0 dirs with 3 files of Size 231798784 Bytes
 Total 3 files of Size 231798784 Bytes
Bytes Used: 232259584, Bytes Total:265289728


	 Loop Count: 4 (#fileCycle=0), Bytes read 0, written 0, #Files=3
	dirV...85677_us	[  7.73 M](0.01681 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >

[  7.73 M](0.01681 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >
Delete with read verify all #bigfile's
	Verify /0_bigfile.txt bytes 132491264 : ..................................................	GOOD! >>  bytes 132491264
	Big read&compare KBytes per second 1757.49 
	Verify /1_bigfile.txt bytes 66234368 : ..................................................	GOOD! >>  bytes 66234368
	Big read&compare KBytes per second 1763.52 
	Verify /2_bigfile.txt bytes 33073152 : ..................................................	GOOD! >>  bytes 33073152
	Big read&compare KBytes per second 1778.09

EDTI: Tried to copy the ManyF/333 directory to the N02 chip but started in again with files not being written as described before. However, this time when I copied the 73Mbyte music directory to the N02 it worked no issues and was able to play wav and mp3 files directly off the chip without issue.

So probably going to have to reduce the clock speed down to 45Mhz - now the question is why the delta between the T4.1 and TMM.
 
Last edited:
Back
Top