LittleFS port to Teensy/SPIFlash

[mjs513]

@mjs513 - I figured my T41 with the soldering touchup was probably dry enough so tried running the test I mentioned above

Not sure of the debug output? But the timing looking interesting

Yeah the debug output is questionable but the timing is interesting. But not sure if it is surprising or not. SPIFFS assumes a flat file system (no directories) and think blocks of transfer are smaller so may make a difference. There are other limitations as well. I do have a old version of MTP for spiffs but not really all up to speed with the updates. File is there and the right size.

EDIT: think i have the clock boosted to 132 mhz as well so that may make up some of the difference but didn;t run the LittleFS version with the 132Mhz clock.

[defragster]

@Paul - did you see the note about considering an alternate set of params for LARGE LittleFS_RAM p#424? I didn't get them right in my edit - but larger blocks will minimize wasted 'metadata' overhead on larger media and big files. Write time was way down - assuming the date was written but just mis-ordered.

Using the original minimal bigfile() code here with with auto .quickformat()

W25Q128JVSIQ 16 MB Winbond date code 2011 - order from DigiKey 4/26/2020:

Big write took 134.09 Sec for 16564000 Bytes : file3.size()=16564000
	Big write KBytes per second 123.53

Changing that to HALFCUT 8 MB write - and same results from @KurtE prompt version:

Big write took 68.03 Sec for 8364000 Bytes : file3.size()=8364000
	Big write KBytes per second 122.94

16MB Flash date code summary:

 1905 - Big write KBytes per second 132.35
 1909 - Big write KBytes per second 134  @mjs513
 2011 - Big write KBytes per second 123  @defragster
 2028 - Big write KBytes per second 81.21

[PaulStoffregen]

did you see the note about considering an alternate set of params for LARGE LittleFS_RAM p#424?

Nope, missed it, and probably a lot of other details on this long thread. Saved a note to look at #424 later.

Robin & I are busy all day shipping the Black Friday orders, and it'll probably go into Saturday. I'm also planning to look at several audio lib issues and contributions after we've cleared the backlog, in prep for 1.54-beta6. If the only issues are performance, I probably won't do much more on LittleFS until after beta6 is packaged up.

If there are important but not critically urgent issues, please remind me when beta6 comes out (hopefully early next week).

[defragster]

Nope, missed it, and probably a lot of other details on this long thread. Saved a note to look at #424 later.

Robin & I are busy all day shipping the Black Friday orders, and it'll probably go into Saturday. I'm also planning to look at several audio lib issues and contributions after we've cleared the backlog, in prep for 1.54-beta6. If the only issues are performance, I probably won't do much more on LittleFS until after beta6 is packaged up.

If there are important but not critically urgent issues, please remind me when beta6 comes out (hopefully early next week).

Okay - the reason for the big file fail on the RAM is increased overhead of the small blocks on RAM - which may make sense when not using 8 or 16 MB of PSRAM. Moving to larger blocks should be faster with less FS overhead - and frees up some 100 KB's of space lost in small blocks.

I did a quick edit of the FS 'chipinfo' details - and failed AFAIK to get it right as file readback failed. If you get a minute that change would help RAM - and further testing.

> it was in looking for a way to test lookahead_size that I did the bigfile - increasing that did help on RAM - but the diff is obscured on FLASH with it's limited speed.

[PaulStoffregen]

I ordered a few Cypress FRAM chips from Digikey. Since they're limited to 40 MHz, these can only be supported on SPI. They're a lot more expensive than flash, but should give much higher write performance.

The Digikey 3 parts I ordered are 428-3323-ND, 428-4717-ND, 428-3413-ND.

[Frank B]

I must have some FRAMs, too. I can try to find them... since my move, its very hard to find things like these..lol
Didn't i add them to the audio-delay? Hmm.... so must have minimum one Audioshield with FRAM.. If I can find it...

[PaulStoffregen]

Oh, I forgot, I also had a small FRAM chip on this I2C test board from early this year (January seems like forever ago...)



This FRAM chip was on the I2C tests because it is one of the few chips which absolutely requires I2C repeated start.

[mjs513]

I ordered a few Cypress FRAM chips from Digikey. Since they're limited to 40 MHz, these can only be supported on SPI. They're a lot more expensive than flash, but should give much higher write performance.

The Digikey 3 parts I ordered are 428-3323-ND, 428-4717-ND, 428-3413-ND.

Good thing I ordered a bunch of those breakout boards that @KurtE designed. You really weren't kidding when you said they were expensive . Just ordered a set - probably won't be here until next late next week.

The only FRAM's I have laying around are the I2C ones from Adafruit.

Seems like yesterday we were playing with the I2C test board. Still use it once in a while

[defragster]

Good thing I ordered a bunch of those breakout boards that @KurtE designed. You really weren't kidding when you said they were expensive . Just ordered a set - probably won't be here until next late next week.

The only FRAM's I have laying around are the I2C ones from Adafruit.

Seems like yesterday we were playing with the I2C test board. Still use it once in a while

Indeed FRAM's not huge and more expensive for sure.

The LilttleFS No Re-Format on SPI gave 452 KB/sec (2 pages back - 16728000Bytes in 37 seconds - before KB/sec printed) - so Fram's (no QSPI support) should be in that neighborhood at same SPI speed? Better if they don't have waits doubling the return delay.

[Frank B]

My FRAMs are: CY15B104 , not sure about the more exact type, as I can't read the printed label.

@Paul, would be great if you could add your FRAMs to the audio effect_delay_ext, too.

[PaulStoffregen]

I'm not planning to work on effect_delay_ext.cpp. Too many other much more important things to do....

But if I was to work on it, the 2 things I would do are add support for the cheap PSRAM chip (over normal SPI, added to the bottom of the audio shield), and use of a buffer allocated in EXTMEM. I really don't see the value in supporting audio delay on a very expensive 4 Mbit FRAM chip when we have a cheap 64 Mbit PSRAM chip which works on the extremely fast QSPI and supports very high SPI clock speeds too.

[defragster]

Another idea for tracking Pre-Formatted Blocks:

Same 'method' as before - except alternate implementation:

Do what Paul does during lowLevelFormat() - read the full block for 0xff with : blockIsBlank().
And - in this case-  for the 4096 blocks alloc() and zero fill a 513 byte block for bit markers:
#ifdef DEBUGCF
			uint32_t *fake = (uint32_t *)checkformat;
			int ff=0;
#endif
		for (unsigned int block=0; block < config.block_count; block++) {
			if (blockIsBlank(&config, block, buffer)) {
				iiblk = block/8;
				jjbit = 1<<(block%8);
				checkformat[iiblk] = checkformat[iiblk] | jjbit;
			}
#ifdef DEBUGCF
			if ( (block/8)>3 && (0==(block/8)%4) && 0==block%8 ) Serial.printf( "\t0x%lx", fake[ff++]); // bugbug DEBUG
			if ( !((block/8)%40) && 0==block%8 ) Serial.printf( "\n%u", block ); // bugbug DEBUG
#endif
		}

Above done once when checkFormat(TRUE) called, and on every Format( BLOCK# ) - just resolves the BLOCK# to a bit from the array and skips formatting:

Since the lfs code always asks for a block to be formatted before using it - skip the format and the wait - and unset the bit - when a format is requested:
	if ( checkformat != nullptr) {
		uint32_t iiblk = block/8;
		uint8_t jjbit = 1<<(block%8);
		if ( checkformat[iiblk] & jjbit ) { // was set as formatted, unset as erase precedes usage
			checkformat[iiblk] &= ~jjbit;
			return 0;
		}
	}
        // Falls through to needed Format commands ...

This takes a 'bit' of RAM for every Formattable BLOCK on the media. Since it uses the scheme in place for checking for lowLevelFormat() to prevent re-Formatting - if that pattern {blockIsBlank()} works on any future media then this pattern should be able to evolve with it.
And it survives restart - as long as the written areas are not quickFormatted - the next used areas should be the same ones found on the next enable of LittleFS::checkFormat() to continue writing to Freshly Pre-Formatted space - with the exception perhaps of added files/directory updates if not done 'in place'.

Here is a test on LLformated media. 'Z'ippy on prints some view of the one bits above - edited output below with // inline comments
> 'Z' enables and shows bits ON for formatted regions after a restart where a format was done before - bitmap only created for myfs.checkFormat( TRUE ), otherwise NULL and free()'d to NULL on FALSE.
> Big write /0_bigfile.txt took 13.76 Sec for 8364000 Bytes : file3.size()=8364000 Big write KBytes per second 607.89 // unformatted area NOT formatted before use
> 'Z' toggle the checkFormat() OFF and ON to show resulting bitmap and then OFF
> Big write /1_bigfile.txt took 34.18 Sec for 4172000 Bytes : file3.size()=4172000 Big write KBytes per second 122.05 // unformatted area by default formatted before use
> 'Z' toggle the checkFormat() ON to show resulting bitmap

0	0xfffffffc	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
320	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
640	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
960	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
1280	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
1600	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
1920	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2240	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2560	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2880	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3200	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3520	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3840	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
 Done myfs.checkFormat() in 581461 us.  // 581 ms for READ of ALL the DISK as needed to assure all bytes 0xff <edit> : except those two root dir bits.
 CheckFormat On: 

printDirectory QSPI_DISK
--------------
 Total 0 files of Size 0 Bytes
Bytes Used: 8192, Bytes Total:16777216

Start Big write of 8364032 Bytes....................................................
Big write /0_bigfile.txt took 13.76 Sec for 8364000 Bytes : file3.size()=8364000
	Big write KBytes per second 607.89 

Bytes Used: 8388608, Bytes Total:16777216

 CheckFormat Off: 
[  1.38 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
0	0xfffffffc	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0x3fffffff	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1280	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1600	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1920	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2240	0xf0000000	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2560	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2880	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3200	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3520	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3840	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
 Done myfs.checkFormat() in 305777 us.  // 306 ms for READ of ALL the DISK as needed to assure all bytes 0xff - a bit faster with polluted space <edit> : two more bits gone on first row for the file name entry

 CheckFormat On: // showing above used space as 0's
 CheckFormat Off: // write another file without refusing requested formattins

Start Big write of 4173824 Bytes..........................
Big write /1_bigfile.txt took 34.18 Sec for 4172000 Bytes : file3.size()=4172000
	Big write KBytes per second 122.05 

Bytes Used: 12570624, Bytes Total:16777216

0	0xfffffffc	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0x3fffffff	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1280	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1600	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1920	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2240	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2560	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2880	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3200	0x0	0x0	0xfe000000	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3520	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3840	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
 Done myfs.checkFormat() in 168211 us.  // faster media scan for used areas

 CheckFormat On:  // on again to display debug print

I removed the other Pull Request and will orchestrate this one after another couple of test runs.

[Frank B]

My FRAMs are: CY15B104 , not sure about the more exact type, as I can't read the printed label.

It is always fascinating what you can do with a cell phone and image editing on the PC.

CY15B104Q1
B 607936
CYP 1707

[Frank B]

I'm not very interested in the audio-delay anymore, too.
The samplerate-adjustment in the other thread is a bit more interesting, but personally I don't need it, and at the moment I don't know wether I should work on it or not.

[defragster]

Cell phones are good at seeing the unseen.
I had to watch a Server boot - where it sits forever doing NOTHING - then you get a couple seconds to respond before it flashes away.
So I took a video and I could not make out my reflection in the matte LCD - but the small hires phone screen showed details not visible in the 'reflection'

[defragster]

@Paul - does that look like a better direction for faster FLASH writes when logging? The only reformats before the disk is full would be if file/directory creation alters in place metadata - AFAIK:

It was sitting there and I did these next steps: with checkFormat on from above:

printDirectory QSPI_DISK
--------------
FILE	0_bigfile.txt		8364000
FILE	1_bigfile.txt		4172000

Bytes Used: 12570624, Bytes Total:16777216

Big write /2_bigfile.txt took  3.41 Sec for 2080000 Bytes : file3.size()=2080000
	Big write KBytes per second 609.96 

Big write /3_bigfile.txt took  1.70 Sec for 1040000 Bytes : file3.size()=1040000
	Big write KBytes per second 610.22 

Big write /4_bigfile.txt took  0.85 Sec for 516000 Bytes : file3.size()=516000
	Big write KBytes per second 609.00 

printDirectory QSPI_DISK
--------------
FILE	0_bigfile.txt		8364000
FILE	1_bigfile.txt		4172000
FILE	2_bigfile.txt		2080000
FILE	3_bigfile.txt		1040000
FILE	4_bigfile.txt		516000

Bytes Used: 16220160, Bytes Total:16777216


 CheckFormat Off: 
[ 58.28 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
0	0x0	0x0	0xffc00000	0xffffffff	0xffffffff	0xffffffff	0x3fffffff	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1280	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1600	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1920	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2240	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2560	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2880	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3200	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3520	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3840	0x0	0x0	0x0	0x0	0x0	0x0	0x0
 Done myfs.checkFormat() in 48160 us.
 CheckFormat On: 

 CheckFormat Off: 
Big write /5_bigfile.txt took  2.09 Sec for 256000 Bytes : file3.size()=256000
	Big write KBytes per second 122.49 

[PaulStoffregen]

will orchestrate this one after another couple of test runs.

Before you put a lot more work in this, I hope you can spend a little time to think about whether you've building something that is generally useful to a large user base, or if you're optimizing the behavior for 1 fairly artificial benchmark. Indeed most people will start out with blank media. But after LittleFS has saved and deleted files, most of its sectors will be filled with leftover data. You only get this scenario with a virgin chip or after the extremely slow low level format which destroys all stored data. Your benchmark is testing this case, but perhaps a much better benchmark would explore the real performance seen when the media it littered with leftover data from previous use. That is the case which is far more likely to impact real applications. Knowing the long-term performance to expect would be far more valuable than knowing how fast it will be for only a short time before all the unused space is filled with leftover junk.

However, I do believe adding a check for already blank in the erase() function makes sense. It's a quick test even for all the bytes, and the most common case will find non-0xFF data in the first partial-sector read. I will merge that, if it checks all the bytes.

But please do keep in mind this sort of blank check is specific to NOR flash. When we later use NAND, leftover data in the ECC also matters.

To really improve performance, I believe we're missing 2 very difficult things. First, the program and erased functions need to become non-blocking, so they don't stall a user's program. But that alone won't help much, because LittleFS often needs to erase a sector when the user wants to actually write anything, due to its copy-on-write approach. The other missing piece is "trim" functionality inside lfs.c. My hunch is the only way we're ever going to have really nice performance for the common cases like data logging is with pre-erase of the sectors LittleFS has left filled with old data. Whether any of us can really work on lfs.c is a good question. I've looked through the code, and quite frankly it's a steep hill to climb. A trim capability probably needs to be added by ARM.

Like with flash SSD on PCs, use of a trim function doesn't show up in simple all-at-once speed benchmarks. The idea behind trim is to utilize the times the media isn't needed to pre-erase the leftover sectors. Benchmarks which measure latency with small accesses are probably more realistic for the sorts of ways people will really use LittleFS. The raw large file all-at-once performance probably only matters for MTP or other file transfer.

[PaulStoffregen]

FWIW, I'm planning to dust off the old SerialFlash library and try to put a File / FS wrapper on top of it, awkward as that may be. The main problem is SerialFlash needs to pre-allocate files to a fixed size which can not be altered after the file is created. It's probably going to need a non-standard extension to FS to create files. Probably won't work with MTP.

The main advantage of SerialFlash is you get the full raw speed of the media, with a subset of the usual File abstraction. But you lose the flexibility of generic filesystems like LittleFS.

Eventually I'm going to create web pages for these libs. I'm imaging SD/SdFat, LittleFS and SerialFlash will be a set with sections on each that give recommendations about which to use for various types of applications.

[defragster]

Before you put a lot more work in this, I hope you can spend a little time to think about whether you've building something that is generally useful to a large user base, or if you're optimizing the behavior for 1 fairly artificial benchmark. Indeed most people will start out with blank media. But after LittleFS has saved and deleted files, most of its sectors will be filled with leftover data. You only get this scenario with a virgin chip or after the extremely slow low level format which destroys all stored data. Your benchmark is testing this case, but perhaps a much better benchmark would explore the real performance seen when the media it littered with leftover data from previous use. That is the case which is far more likely to impact real applications. Knowing the long-term performance to expect would be far more valuable than knowing how fast it will be for only a short time before all the unused space is filled with leftover junk.

However, I do believe adding a check for already blank in the erase() function makes sense. It's a quick test even for all the bytes, and the most common case will find non-0xFF data in the first partial-sector read. I will merge that, if it checks all the bytes.

But please do keep in mind this sort of blank check is specific to NOR flash. When we later use NAND, leftover data in the ECC also matters.

To really improve performance, I believe we're missing 2 very difficult things. First, the program and erased functions need to become non-blocking, so they don't stall a user's program. But that alone won't help much, because LittleFS often needs to erase a sector when the user wants to actually write anything, due to its copy-on-write approach. The other missing piece is "trim" functionality inside lfs.c. My hunch is the only way we're ever going to have really nice performance for the common cases like data logging is with pre-erase of the sectors LittleFS has left filled with old data. Whether any of us can really work on lfs.c is a good question. I've looked through the code, and quite frankly it's a steep hill to climb. A trim capability probably needs to be added by ARM.

Like with flash SSD on PCs, use of a trim function doesn't show up in simple all-at-once speed benchmarks. The idea behind trim is to utilize the times the media isn't needed to pre-erase the leftover sectors. Benchmarks which measure latency with small accesses are probably more realistic for the sorts of ways people will really use LittleFS. The raw large file all-at-once performance probably only matters for MTP or other file transfer.

... the work is done now ... and, at the cost of 512 bytes in this case, when activated it does a bit test rather than a partial read. A full block read at time of write adds back 581461 us / 4096 == 141 us per block Full Read versus the above bit checking.

Indeed this addition is ONLY of value after a lowLevelFormat, it has no general use on a polluted disk that lfs isn't marching forward on the Pre-Formatted blocks. Anyone wanting to log at usable speeds - perhaps 600 KB/sec versus 125 KB/sec would want to use this. Not as fast as SD logging - but more consistent - without the power surges (noise?) and unexpected waits for Media housekeeping of the random 10-100 ms.

Noted above - as far as NAND or future chips - if the blockIsBlank() can/will evolve to prevent RE-formatting a ready block - then this can evolve with it - as noted in github reply - it may take another chipinfo setting in both cases to identify how many blocks are in a format unit or whatever the maath may be.

Doing test now on SPI versus the above QSPI ... while reading rest your post ...

Done Formatting Low Level in 348271858 us  // same 16MB flash as SPI on Audio Card

Full read to verify formatting means 1256 us to read a block to check for fully formatted {better than 85 ms per block on SPI} :: Done myfs.checkFormat() in 5148303 us

with bit check populated/enabled on SPI:
Start Big write of 8364032 Bytes....................................................
Big write /0_bigfile.txt took 18.44 Sec for 8364000 Bytes : file3.size()=8364000
	Big write KBytes per second 453.69 

Disable checkFormat and quickFormat to remove the half disk file to write the same file again with always format:

Odd - " CheckFormat Off: " and a 'q'uickFormat - should be treating the DISK as Formatted Empty and NOT using any new code:

	 Updated filecount 0
q
[ 11.83 M](0.00293 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
Start Big write of 8364032 Bytes............
Big write ERR# -5 0xFFFFFFFB 
Full Reformat - and Big File without the checkFormat enabled, will do another 'q'uickFormat then the same 2nd Big File - SAME FAILURE same '.' index on file write - the flash is bad or the quickFormat is bad:
>> WOULD say date code 1526 is OLD FLASH probably dead ??? - Looks to be a 25Q128FVS{Q or G?}

 Done Formatting Low Level in 218156296 us.
u 
	 Updated filecount 0
F

Big write /0_bigfile.txt took 192.49 Sec for 8364000 Bytes : file3.size()=8364000
	Big write KBytes per second 43.45 

Bytes Used: 8388608, Bytes Total:16777216

	 Updated filecount 0
q

[ 22.69 M](0.00290 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
Start Big write of 8364032 Bytes............
Big write ERR# -5 0xFFFFFFFB

Will have to replace that chip - recompiled now for T_4.0 with 16MB Flash ...

But above on SPI shows Big write KBytes per second 453.69 (quick) versus unaltered on same llFormatted blockas at Big write KBytes per second 43.45

The second SPI test is well underway on the T_4.0 with Audio card and 16MB 'similar' Flash - I cannot read the flash until I can pull the Audio board off the top. Will post full unedited log below when the second write completes.

[PaulStoffregen]

Indeed this addition is ONLY of value after a lowLevelFormat

Is the known erased blocks bitmap stored in volatile memory?

If someone uses this to build a data logging application, do they need to keep continuous power applied from the time of low level formatting throughout all data writing which requires the higher speed? What happens to their speed-sensitive application if the processor reboots due to power loss or any other reason?

[defragster]

Note sure if I missed anything in your post - except even if background lowlevel formatting is done - this code provides a 'bit' Formatted { Yes/No }. So if a background formatter were to know which which blocks were free - when LFS comes around it will still want to format them.

If this were extended to TWO bits per block Then it could provide the map for the background formatting: [ In Use, Formatted Free, Dirty Unused PLEASE FORMAT, Format in progress ].

So the SPI on T_4.0 completed - it is the SAME ancient date code - why I put these on Rev D audio cards ??? - Where I got them ??? Maybe fakes not digikey? And this is a BETTER picture - it is a "G"

Formatting Low Level:
	.........................................................................................................................

 Done Formatting Low Level in 344386069 us.
u 
	 Updated filecount 0
F
printDirectory SPI_DISK
--------------

 0 dirs with 0 files of Size 0 Bytes
 Total 0 files of Size 0 Bytes
Bytes Used: 8192, Bytes Total:16777216
[  7.00 M](5.73979 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
0	0xfffffffc	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
320	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
640	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
960	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
1280	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
1600	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
1920	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2240	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2560	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2880	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3200	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3520	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3840	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
 Done myfs.checkFormat() in 5149310 us.

 CheckFormat On: 
[  8.41 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
Start Big write of 8364032 Bytes....................................................
Big write /0_bigfile.txt took 18.33 Sec for 8364000 Bytes : file3.size()=8364000
	Big write KBytes per second 456.39 

Bytes Used: 8388608, Bytes Total:16777216

[  8.78 M](0.30678 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
 Done myfs.checkFormat() in 0 us.
 CheckFormat Off: 
[  8.85 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
0	0xfffffffc	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0x3fffffff	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1280	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1600	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1920	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2240	0xf0000000	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2560	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2880	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3200	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3520	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3840	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
 Done myfs.checkFormat() in 2733670 us.

 CheckFormat On: 
[  8.92 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
 Done myfs.checkFormat() in 0 us.
 CheckFormat Off: 
[  8.97 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >u 
	 Updated filecount 0
q
[  9.02 M](0.00293 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
Start Big write of 8364032 Bytes....................................................
Big write /0_bigfile.txt took 189.40 Sec for 8364000 Bytes : file3.size()=8364000
	Big write KBytes per second 44.16 

Bytes Used: 8388608, Bytes Total:16777216

[defragster]

Is the known erased blocks bitmap stored in volatile memory?

If someone uses this to build a data logging application, do they need to keep continuous power applied from the time of low level formatting throughout all data writing which requires the higher speed? What happens to their speed-sensitive application if the processor reboots due to power loss or any other reason?

Yes, this is stored in Volatile memory ::

		uint32_t iiblk = 1+(config.block_count /8);
		if ( checkformat == nullptr) checkformat = (uint8_t *)malloc( iiblk );

If the processor restarts through setup() the code would need to re-execute LittleFS::checkFormat() to check the current state of the media.

If that RAM is trashed - but the rest of the world is alive - and does not restart the CPU - then any references to that would be 'random' - just like all the rest of memory. If ON it would need call ON or OFF/ON to refresh the BitMap.

From what I can see LFS keeps track of where it last wrote and picks up writing there on restart. If the code wants to use quickFormat - it must enable it and the full media is again scanned as IF it were doing a LLformat with blockIsBlank()

Testing this now ...
>> Going back to T_4.1 with QSPI : ( BTW this proper message came up when the T_4.1 powered without the bad SPI Flash AUdio card plugged in :: Error starting SPI_DISK }

Did LLformat - did NOT engage quickFormat 
Did Big write '0'
Did RESET
Enable quickFormat
Did Big write '1'
Did RESET
Enable quickFormat
Did Big write '2'
Did RESET

...

Enable quickFormat
Did Big write '5'
Enable Off/On quickFormat just to show continued usage as detected
Did Big write '6'

LONG SPEW follows - startup Menu and then the 'Z'ippy quickFormat() debug table:

................................................................................................................

 Done Formatting Low Level in 100540974 us.
u 
	 Updated filecount 0
F
[  4.58 M](1.67568 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
 Done myfs.checkFormat() in 0 us.
 CheckFormat Off: 
[  5.13 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
0	0xfffffffc	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
320	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
640	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
960	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
1280	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
1600	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
1920	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2240	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2560	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2880	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3200	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3520	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3840	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
 Done myfs.checkFormat() in 581452 us.

 CheckFormat On: 
[  5.19 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
 Done myfs.checkFormat() in 0 us.
 CheckFormat Off: 
[  5.30 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
Start Big write of 8364032 Bytes....................................................
Big write /0_bigfile.txt took 66.72 Sec for 8364000 Bytes : file3.size()=8364000
	Big write KBytes per second 125.35 

Bytes Used: 8388608, Bytes Total:16777216

[  6.51 M](1.11218 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
0	0xfffffffc	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0x3fffffff	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1280	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1600	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1920	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2240	0xf0000000	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2560	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2880	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3200	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3520	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3840	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
 Done myfs.checkFormat() in 305779 us.

 CheckFormat On: 
[ 12.26 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
T:\tCode\littleFS\LFSintegrity\LFSintegrity.ino Nov 28 2020 03:28:10
LittleFS Test : File Integrity
printDirectory QSPI_DISK
--------------
FILE	0_bigfile.txt		8364000

 0 dirs with 1 files of Size 8364000 Bytes
 Total 1 files of Size 8364000 Bytes
Bytes Used: 8388608, Bytes Total:16777216
 0, 1-9 '#' passes continue loop before Pause
 'R' Restart Teensy
 'd' Directory of LittleFS
 'b' big file delete
 'B' BIG FILE RUN
 'c' Continuous Loop
 'h' Hundred loops
 'k' Thousand loops
 'F' LittleFS_ Low Level Format Disk 
 'q' LittleFS_ Quick Format Disk 
 'v' Verbose All Dir Prints - TOGGLE
 'p' Pause after all Dir prints - TOGGLE
 'l' Show count of loop()'s, Bytes Read,Written
 'm' Make ROOT dirs (needed after format !ROOTONLY)
 'u' Update Filecount
 'x' Directory filecount verify - TOGGLE
 'Z' ZIPPY write after 'F'ormat - TOGGLE
 '+' more add to delete cycles
 '-' fewer add to delete cycles
 '?' Help list
?[  0.02 M](0.02498 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
0	0xfffffffc	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0x3fffffff	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1280	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1600	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1920	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2240	0xf0000000	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2560	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2880	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3200	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3520	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3840	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
 Done myfs.checkFormat() in 305790 us.

 CheckFormat On: 
printDirectory QSPI_DISK
--------------
FILE	0_bigfile.txt		8364000

 0 dirs with 1 files of Size 8364000 Bytes
 Total 1 files of Size 8364000 Bytes
Bytes Used: 8388608, Bytes Total:16777216
[  0.15 M](0.00013 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
Start Big write of 4173824 Bytes..........................
Big write /1_bigfile.txt took  6.84 Sec for 4172000 Bytes : file3.size()=4172000
	Big write KBytes per second 610.38 

Bytes Used: 12570624, Bytes Total:16777216

[  0.38 M](0.11423 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
T:\tCode\littleFS\LFSintegrity\LFSintegrity.ino Nov 28 2020 03:28:10
LittleFS Test : File Integrity
printDirectory QSPI_DISK
--------------
FILE	0_bigfile.txt		8364000
FILE	1_bigfile.txt		4172000

 0 dirs with 2 files of Size 12536000 Bytes
 Total 2 files of Size 12536000 Bytes
Bytes Used: 12570624, Bytes Total:16777216
 0, 1-9 '#' passes continue loop before Pause
 'R' Restart Teensy
 'd' Directory of LittleFS
 'b' big file delete
 'B' BIG FILE RUN
 'c' Continuous Loop
 'h' Hundred loops
 'k' Thousand loops
 'F' LittleFS_ Low Level Format Disk 
 'q' LittleFS_ Quick Format Disk 
 'v' Verbose All Dir Prints - TOGGLE
 'p' Pause after all Dir prints - TOGGLE
 'l' Show count of loop()'s, Bytes Read,Written
 'm' Make ROOT dirs (needed after format !ROOTONLY)
 'u' Update Filecount
 'x' Directory filecount verify - TOGGLE
 'Z' ZIPPY write after 'F'ormat - TOGGLE
 '+' more add to delete cycles
 '-' fewer add to delete cycles
 '?' Help list
?[  0.01 M](0.01274 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
0	0xfffffffc	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0x3fffffff	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1280	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1600	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1920	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2240	0xf0000000	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0x7f	0x0
2560	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2880	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3200	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3520	0xfffffff0	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3840	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
 Done myfs.checkFormat() in 168224 us.

 CheckFormat On: 
printDirectory QSPI_DISK
--------------
FILE	0_bigfile.txt		8364000
FILE	1_bigfile.txt		4172000

 0 dirs with 2 files of Size 12536000 Bytes
 Total 2 files of Size 12536000 Bytes
Bytes Used: 12570624, Bytes Total:16777216
[  0.11 M](0.00020 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
Start Big write of 2082816 Bytes.............
Big write /2_bigfile.txt took  3.41 Sec for 2080000 Bytes : file3.size()=2080000
	Big write KBytes per second 609.35 

Bytes Used: 14655488, Bytes Total:16777216

[  0.29 M](0.05730 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
T:\tCode\littleFS\LFSintegrity\LFSintegrity.ino Nov 28 2020 03:28:10
LittleFS Test : File Integrity
printDirectory QSPI_DISK
--------------
FILE	0_bigfile.txt		8364000
FILE	1_bigfile.txt		4172000
FILE	2_bigfile.txt		2080000

 0 dirs with 3 files of Size 14616000 Bytes
 Total 3 files of Size 14616000 Bytes
Bytes Used: 14655488, Bytes Total:16777216
 0, 1-9 '#' passes continue loop before Pause
 'R' Restart Teensy
 'd' Directory of LittleFS
 'b' big file delete
 'B' BIG FILE RUN
 'c' Continuous Loop
 'h' Hundred loops
 'k' Thousand loops
 'F' LittleFS_ Low Level Format Disk 
 'q' LittleFS_ Quick Format Disk 
 'v' Verbose All Dir Prints - TOGGLE
 'p' Pause after all Dir prints - TOGGLE
 'l' Show count of loop()'s, Bytes Read,Written
 'm' Make ROOT dirs (needed after format !ROOTONLY)
 'u' Update Filecount
 'x' Directory filecount verify - TOGGLE
 'Z' ZIPPY write after 'F'ormat - TOGGLE
 '+' more add to delete cycles
 '-' fewer add to delete cycles
 '?' Help list
?[  0.05 M](0.04743 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
0	0xfffffffc	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0x3fffffff	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1280	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1600	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1920	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2240	0xf0000000	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0x7f	0x0
2560	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2880	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3200	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3520	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3840	0x0	0x0	0x0	0x0	0x0	0x0	0xfffffffe
 Done myfs.checkFormat() in 99642 us.

 CheckFormat On: 
printDirectory QSPI_DISK
--------------
FILE	0_bigfile.txt		8364000
FILE	1_bigfile.txt		4172000
FILE	2_bigfile.txt		2080000

 0 dirs with 3 files of Size 14616000 Bytes
 Total 3 files of Size 14616000 Bytes
Bytes Used: 14655488, Bytes Total:16777216
[  0.14 M](0.00024 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
Start Big write of 1040384 Bytes......
Big write /3_bigfile.txt took  1.73 Sec for 1040000 Bytes : file3.size()=1040000
	Big write KBytes per second 602.03 

Bytes Used: 15699968, Bytes Total:16777216

[  0.29 M](0.02925 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
T:\tCode\littleFS\LFSintegrity\LFSintegrity.ino Nov 28 2020 03:28:10
LittleFS Test : File Integrity
printDirectory QSPI_DISK
--------------
FILE	0_bigfile.txt		8364000
FILE	1_bigfile.txt		4172000
FILE	2_bigfile.txt		2080000
FILE	3_bigfile.txt		1040000

 0 dirs with 4 files of Size 15656000 Bytes
 Total 4 files of Size 15656000 Bytes
Bytes Used: 15699968, Bytes Total:16777216
 0, 1-9 '#' passes continue loop before Pause
 'R' Restart Teensy
 'd' Directory of LittleFS
 'b' big file delete
 'B' BIG FILE RUN
 'c' Continuous Loop
 'h' Hundred loops
 'k' Thousand loops
 'F' LittleFS_ Low Level Format Disk 
 'q' LittleFS_ Quick Format Disk 
 'v' Verbose All Dir Prints - TOGGLE
 'p' Pause after all Dir prints - TOGGLE
 'l' Show count of loop()'s, Bytes Read,Written
 'm' Make ROOT dirs (needed after format !ROOTONLY)
 'u' Update Filecount
 'x' Directory filecount verify - TOGGLE
 'Z' ZIPPY write after 'F'ormat - TOGGLE
 '+' more add to delete cycles
 '-' fewer add to delete cycles
 '?' Help list
?[  0.02 M](0.01803 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
printDirectory QSPI_DISK
--------------
FILE	0_bigfile.txt		8364000
FILE	1_bigfile.txt		4172000
FILE	2_bigfile.txt		2080000
FILE	3_bigfile.txt		1040000

 0 dirs with 4 files of Size 15656000 Bytes
 Total 4 files of Size 15656000 Bytes
Bytes Used: 15699968, Bytes Total:16777216
[  0.02 M](0.02083 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
0	0xfffffffc	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0x3fffffff	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1280	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1600	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1920	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2240	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2560	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2880	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3200	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3520	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3840	0x0	0x0	0x0	0x0	0x0	0x0	0xffe00000
 Done myfs.checkFormat() in 65283 us.

 CheckFormat On: 
[  0.19 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
Start Big write of 518144 Bytes...
Big write /4_bigfile.txt took  0.87 Sec for 516000 Bytes : file3.size()=516000
	Big write KBytes per second 590.66 

Bytes Used: 16220160, Bytes Total:16777216

[  0.29 M](0.01504 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
T:\tCode\littleFS\LFSintegrity\LFSintegrity.ino Nov 28 2020 03:28:10
LittleFS Test : File Integrity
printDirectory QSPI_DISK
--------------
FILE	0_bigfile.txt		8364000
FILE	1_bigfile.txt		4172000
FILE	2_bigfile.txt		2080000
FILE	3_bigfile.txt		1040000
FILE	4_bigfile.txt		516000

 0 dirs with 5 files of Size 16172000 Bytes
 Total 5 files of Size 16172000 Bytes
Bytes Used: 16220160, Bytes Total:16777216
 0, 1-9 '#' passes continue loop before Pause
 'R' Restart Teensy
 'd' Directory of LittleFS
 'b' big file delete
 'B' BIG FILE RUN
 'c' Continuous Loop
 'h' Hundred loops
 'k' Thousand loops
 'F' LittleFS_ Low Level Format Disk 
 'q' LittleFS_ Quick Format Disk 
 'v' Verbose All Dir Prints - TOGGLE
 'p' Pause after all Dir prints - TOGGLE
 'l' Show count of loop()'s, Bytes Read,Written
 'm' Make ROOT dirs (needed after format !ROOTONLY)
 'u' Update Filecount
 'x' Directory filecount verify - TOGGLE
 'Z' ZIPPY write after 'F'ormat - TOGGLE
 '+' more add to delete cycles
 '-' fewer add to delete cycles
 '?' Help list
?[  0.01 M](0.01237 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
Start Big write of 258048 Bytes.
Big write /5_bigfile.txt took  2.08 Sec for 256000 Bytes : file3.size()=256000
	Big write KBytes per second 122.80 

Bytes Used: 16478208, Bytes Total:16777216

printDirectory QSPI_DISK
--------------
FILE	0_bigfile.txt		8364000
FILE	1_bigfile.txt		4172000
FILE	2_bigfile.txt		2080000
FILE	3_bigfile.txt		1040000
FILE	4_bigfile.txt		516000
FILE	5_bigfile.txt		256000

 0 dirs with 6 files of Size 16428000 Bytes
 Total 6 files of Size 16428000 Bytes
Bytes Used: 16478208, Bytes Total:16777216
[  0.10 M](0.03556 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
0	0x0	0x0	0x0	0x0	0xffe00000	0xffffffff	0x3fffffff	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1280	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1600	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1920	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2240	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2560	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2880	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3200	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3520	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3840	0x0	0x0	0x0	0x0	0x0	0x0	0x0
 Done myfs.checkFormat() in 39683 us.

 CheckFormat On: 
[  0.21 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
Start Big write of 129024 Bytes
Big write /6_bigfile.txt took  0.21 Sec for 128000 Bytes : file3.size()=128000
	Big write KBytes per second 606.95 

Bytes Used: 16609280, Bytes Total:16777216

[  0.49 M](0.00402 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
 Done myfs.checkFormat() in 0 us.
 CheckFormat Off: 
[  0.56 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
0	0x0	0x0	0x0	0x0	0x0	0xffe00000	0x3fffffff	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1280	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1600	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1920	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2240	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2560	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2880	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3200	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3520	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3840	0x0	0x0	0x0	0x0	0x0	0x0	0x0
 Done myfs.checkFormat() in 35360 us.

 CheckFormat On: 
[  0.58 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 > d
printDirectory QSPI_DISK
--------------
FILE	0_bigfile.txt		8364000
FILE	1_bigfile.txt		4172000
FILE	2_bigfile.txt		2080000
FILE	3_bigfile.txt		1040000
FILE	4_bigfile.txt		516000
FILE	5_bigfile.txt		256000
FILE	6_bigfile.txt		128000

 0 dirs with 7 files of Size 16556000 Bytes
 Total 7 files of Size 16556000 Bytes
Bytes Used: 16609280, Bytes Total:16777216


	 Loop Count: 5 (#fileCycle=0), Bytes read 0, written 0, #Files=7
[  0.94 M](0.00035 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >

[  0.94 M](0.00035 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >

[defragster]

Well I managed to name a FORK after making a Clone and it seems right ...

...
github.com/Defragster/LittleFS
...

Heads up - I've been posting LittleFS::quickFormat() when I meant to say the added code is LittleFS::checkFormat()

My edits are there for review and trial / use.

Also added there :: Defragster/LittleFS/tree/main/examples/LFSintegrity

It now allows Multiple BigFiles as demo'd - and 'b' clears them all.

At all times as files come and go it should be tracking 'filecount' - and that is checked on each DIR and other displays of 'l' info. It may be possible to make an honest mistake - but if the file count ever fails to match ( DIR walk versus expected ) it will STOP and display that!

'Z'ippy toggled the LittleFS::checkFormat() - which can change underneath when a 'F'ormat lowLevel is done ... doesn't need to change for other LittleFS::quickFormat() - if active - leave it active and any changes will be tracked properly.

I create the Large files - and each is created with the '#' digit in the file name - I have not implemented any VERIFY like before on the other files.

Speaking of the other files: This will work JUST as well on the original LFSintegrity test of a series of files coming and going, until all freshly formatted media is consumed - the only FORMAT will be when a MetaData block has to be reflashed in place before continued usage.

The main reason I did this LARGE FILE test was to be able to evaluate the 'lookahead' change - and I was trying that on RAM when I ran into all the other issues ( 64 bit LFS() return values ) and wasted space on LARGE 8MB & 16 MB RAM_DISKS where the added slack space from small blocks had it failing on the same test that worked on FLASH.

The ELAPSED time "(0.02462 M elap)" on the 'l' Loops Left line may be more accurate now, but more useful for longer times when it works. But the way the loop/iterations cycle - they only repeat when followed in the same order.

For instance - Fresh lowLevelFormat - ROOTONLY: Restart and do a 'Z' and '3' then a '9':

 CheckFormat On: 
[ 11.99 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >3
:: /B_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 3072] ++ B 	Verify /B_file.txt bytes 3072 
:: /C_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 4096] ++ C 	Verify /C_file.txt bytes 4096 
:: /D_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 5120] ++ D 	Verify /D_file.txt bytes 5120 
:: /E_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 6144] ++ E 	Verify /E_file.txt bytes 6144 
:: /F_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 7168] ++ F 	Verify /F_file.txt bytes 7168 
:: /G_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 8192] ++ G 	Verify /G_file.txt bytes 8192 
[ 12.21 M](0.00135 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >9
:: /H_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 9216] ++ H 	Verify /H_file.txt bytes 9216 
:: /I_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 10240] ++ I 	Verify /I_file.txt bytes 10240 
:: /J_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 11264] ++ J 	Verify /J_file.txt bytes 11264 
:: /K_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 12288] ++ K 	Verify /K_file.txt bytes 12288 
:: /L_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 13312] ++ L 	Verify /L_file.txt bytes 13312 
:: /M_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 14336] ++ M 	Verify /M_file.txt bytes 14336 
:: /N_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 15360] ++ N 	Verify /N_file.txt bytes 15360 
:: /O_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 16384] ++ O 	Verify /O_file.txt bytes 16384 
:: /P_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 17408] ++ P 	Verify /P_file.txt bytes 17408 
:: /Q_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 18432] ++ Q 	Verify /Q_file.txt bytes 18432 
:: /R_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 19456] ++ R 	Verify /R_file.txt bytes 19456 
:: /S_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 20480] ++ S 	Verify /S_file.txt bytes 20480 
:: /T_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 21504] ++ T 	Verify /T_file.txt bytes 21504 
:: /U_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 22528] ++ U 	Verify /U_file.txt bytes 22528 
:: /V_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 23552] ++ V 	Verify /V_file.txt bytes 23552 
:: /W_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 24576] ++ W 	Verify /W_file.txt bytes 24576 
:: /X_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 25600] ++ X 	Verify /X_file.txt bytes 25600 
:: /Y_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 26624] ++ Y 	Verify /Y_file.txt bytes 26624 
[ 12.38 M](0.01521 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >
 Done myfs.checkFormat() in 0 us.

Now do a 'q', Restart (TyComm Ctrl+R), then a '3' and a '9' { NO 'Z' }:

printDirectory QSPI_DISK
--------------

 0 dirs with 0 files of Size 0 Bytes
 Total 0 files of Size 0 Bytes
Bytes Used: 8192, Bytes Total:16777216
[  0.07 M](0.07060 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >3
:: /B_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 3072] ++ B 	Verify /B_file.txt bytes 3072 
:: /C_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 4096] ++ C 	Verify /C_file.txt bytes 4096 
:: /D_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 5120] ++ D 	Verify /D_file.txt bytes 5120 
:: /E_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 6144] ++ E 	Verify /E_file.txt bytes 6144 
:: /F_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 7168] ++ F 	Verify /F_file.txt bytes 7168 
:: /G_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 8192] ++ G 	Verify /G_file.txt bytes 8192 
[  0.20 M](0.00624 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >9
:: /H_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 9216] ++ H 	Verify /H_file.txt bytes 9216 
:: /I_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 10240] ++ I 	Verify /I_file.txt bytes 10240 
:: /J_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 11264] ++ J 	Verify /J_file.txt bytes 11264 
:: /K_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 12288] ++ K 	Verify /K_file.txt bytes 12288 
:: /L_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 13312] ++ L 	Verify /L_file.txt bytes 13312 
:: /M_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 14336] ++ M 	Verify /M_file.txt bytes 14336 
:: /N_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 15360] ++ N 	Verify /N_file.txt bytes 15360 
:: /O_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 16384] ++ O 	Verify /O_file.txt bytes 16384 
:: /P_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 17408] ++ P 	Verify /P_file.txt bytes 17408 
:: /Q_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 18432] ++ Q 	Verify /Q_file.txt bytes 18432 
:: /R_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 19456] ++ R 	Verify /R_file.txt bytes 19456 
:: /S_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 20480] ++ S 	Verify /S_file.txt bytes 20480 
:: /T_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 21504] ++ T 	Verify /T_file.txt bytes 21504 
:: /U_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 22528] ++ U 	Verify /U_file.txt bytes 22528 
:: /V_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 23552] ++ V 	Verify /V_file.txt bytes 23552 
:: /W_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 24576] ++ W 	Verify /W_file.txt bytes 24576 
:: /X_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 25600] ++ X 	Verify /X_file.txt bytes 25600 
:: /Y_file.txt  QSPI_DISK +++ Add +++ [sz 0 add 26624] ++ Y 	Verify /Y_file.txt bytes 26624 
[  0.34 M](0.05544 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZ+-? loops left 0 >

Compare '3' First w/'Z' to Second elapsed : (0.00135 M elap) to (0.00624 M elap)

then Compare '9' First w/'Z' to Second elapsed : (0.01521 M elap) to (0.05544 M elap)

That has written only "written 356352, #Files=24" - so about 16,355,328 more of FREE space {opps used space is 421888} to go before it will format ( except when metadata rewrites are needed and they STAY in the same location { which wears on that spot - but prevents updating the MetaData block above it } )

BTW - I got bored after the '3' and '9' and walked away for 'h'undred - and that has used all available space. This is Root only and the files are not as small as sometimes.

Also - @Paul - 'Special Case' ? If I were using this to LOG something { assuming keeping track of 'something' } - I would only expect to log the disk toward FULL. It would have to stop when Full. Before it is FULL I would want to OFFLOAD the DATA for review/storage. Given the option then I would LLformat any used data block and clean the disk. The logging app would have fresh media running 5- 10 times faster than if it reformats every block as it goes. If the logging is slow - 100 KB / hour then I'd have 159 hours to wait before I had to offload - 6.65 days so when I pull the data I waste 100 seconds formatting the media and go away for 6 days. If I wanted faster logging - I'm limited by 45 KB/sec on SPI or 125 KB/sec on QSPI - unless it is preformatted - then I can get 425 KB/sec or 600 KB/sec.

And as noted p#471 - if the LFS can tell us when a block is in the 'garbage heap' another bit could allow background reformatting and tracking so it isn't formatted again later - without doing any additional read of the block to figure that out. Of course that 'bit sharing' could be a nightmare - so two bits in the master list and one bit each in the reformatter list where they trade notes perhaps.

printDirectory QSPI_DISK
--------------
FILE B_file.txt 3072
FILE C_file.txt 4096
FILE D_file.txt 5120
FILE E_file.txt 6144
FILE F_file.txt 7168
FILE G_file.txt 8192
FILE H_file.txt 9216
FILE I_file.txt 10240
FILE J_file.txt 11264
FILE K_file.txt 12288
FILE L_file.txt 13312
FILE M_file.txt 14336
FILE N_file.txt 15360
FILE O_file.txt 16384
FILE P_file.txt 17408
FILE Q_file.txt 18432
FILE R_file.txt 19456
FILE S_file.txt 20480
FILE T_file.txt 21504
FILE U_file.txt 22528
FILE V_file.txt 23552
FILE W_file.txt 24576
FILE X_file.txt 25600
FILE Y_file.txt 26624

0 dirs with 24 files of Size 356352 Bytes
Total 24 files of Size 356352 Bytes
Bytes Used: 421888, Bytes Total:16777216

[defragster]

> Posted a change in LFSintegrity to 'F'ormat to set the 'bCheckFormat = false;' since the code disables that on LLformat. Note: that also enable creation of Root Directories for the deeper integrity loops :: //#define ROOTONLY

@Paul this post #424 below was on your list.
Looking at the code I see these hardcoded values for 'block_size == 256' ???
> That would explain the jump to 4096 breaking the 'LFSintegrity'

	static int static_read(const struct lfs_config *c, lfs_block_t block,
	  lfs_off_t offset, void *buffer, lfs_size_t size) {
		//Serial.printf("    ram rd: block=%d, offset=%d, size=%d\n", block, offset, size);
		uint32_t index = block * 256 + offset;
		memcpy(buffer, (uint8_t *)(c->context) + index, size);
		return 0;
	}
	static int static_prog(const struct lfs_config *c, lfs_block_t block,
	  lfs_off_t offset, const void *buffer, lfs_size_t size) {
		//Serial.printf("    ram wr: block=%d, offset=%d, size=%d\n", block, offset, size);
		uint32_t index = block * 256 + offset;
		memcpy((uint8_t *)(c->context) + index, buffer, size);
		return 0;
	}
	static int static_erase(const struct lfs_config *c, lfs_block_t block) {
		uint32_t index = block * 256;
		memset((uint8_t *)(c->context) + index, 0xFF, 256);
		return 0;
	}

Back to the real issue in #422 (tiny alloc blocks in BIG PSRAM) doing this for LittleFS_RAM() { } :: bool begin(()
> it runs maybe 3-5X faster - but I get LFSintegrity file verify failure so something is mismatched

		if ( size > 1024*1024 ) {
			config.read_size = 64;
			config.prog_size = 64;
			config.block_size = 4096;
			config.block_count = size / 4096;
			config.block_cycles = 900;
			config.cache_size = 256;
			//config.lookahead_size = 64;  // was 64
			config.lookahead_size = 512; // (config.block_count / 8 + 7) & 0xFFFFFFF8;
		}
		else {
			config.read_size = 64;
			config.prog_size = 64;
			config.block_size = 256;
			config.block_count = size / 256;
			config.block_cycles = 50;
			config.cache_size = 64;
			config.lookahead_size = 64;  // was 64
			//config.lookahead_size = (config.block_count / 8 + 7) & 0xFFFFFFF8;
		}

Makes 16 MB PSRAM do this:

Big write took  2.48 Sec for 16728000 Bytes : file3.size()=16728000
	Big write KBytes per second 6758.09 
Bytes Used: 16777216, Bytes Total:16777216

TWO THINGS:
> The Speed 2.48 versus 13 seconds for a ~16 MB file
> AND that is a full size bigfile now fitting 16728000 bytes versus 16572000
-> using :: uint32_t toWrite = (myfs.totalSize()) - myfs.usedSize() - 40960; // allow for slack space :: WORKS on FLASH?


Notes on:: config.block_cycles
> For RAM this should be MAX as it won't wear out
> For Flash - since it may always store DIR data in the same spot given the same program behavior - it should tend to be smaller perhaps or it may always hit the same spot and then stay there re-formatting for a large number of cycles.

I added explicit timing for LLformat and bigfile() - but my quick failed elapsed timer on command line never got addressed - if I get that working the goal was timing between user input on a number of iterations and I could test the Files in directory that may be more like standard logging.

[KurtE]

Just an FYI - The newer versions of the github for windows app is getting a little nicer for doing things like forks and the like. For example if you have github project you simply cloned down to your machine without forking and makes changes to it. It may prompt you or have button that says something like: it sees you are making changes would you like to fork it... It will then do the stuff to do it, like fork a copy of the project to your github project. Rename the origin remote to be upstream remote and create a new origin to your newly forked... Actually after it creates the fork up on github it will give you a few options, will you want to make contributions to the original project or make your copy not related to the original... I discussed the wanting to make contributions.

Will be interesting to see if there are simple fixes/extensions to the library the speed it up. As I mentioned I was especially seeing it with MTP where I could easily copy a 300kb file to SD cards and the like either the built in one or one using normal SPI. But the same copy would fail to the flash of propshield and many times to the QSPI flash. Again probably secondary issue where if the copy took over maybe half second the progress dialog would show up and the drive going away sound would happen and the Teensy drive does not work unless you reset.

Now to play some more. But first sync up to everything.