LittleFS port to Teensy/SPIFlash

I also could not reproduce the problem using 1.54-beta10 and running on a Teensy 4.1 with 1 PSRAM and 1 Flash chip soldered to the bottom side.

DOING: 'B' - starts BIG write and does not complete

When I run it here, it prints this:

Code:
Big write /1_bigfile.txt took  0.50 Sec for 2074624 Bytes : file3.size()=2074624
	Big write KBytes per second 4157.79 

Bytes Used: 6277120, Bytes Total:8388608

printDirectory RAM_DISK
--------------
FILE	0_bigfile.txt		4169728
FILE	1_bigfile.txt		2074624
FILE	B_file.txt		1536
FILE	C_file.txt		2048

 0 dirs with 4 files of Size 6247936 Bytes
 Total 4 files of Size 6247936 Bytes
Bytes Used: 6277120, Bytes Total:8388608
[  0.11 M](0.00896 M elap) Awaiting input 0123456789RdwcghkFqvplmusSBbyYxfan+-? loops left 0 >


or
DOING '1' - a single loop write to one small file and it hangs.

I get this result here:

Code:
:: /B_file.txt  RAM_DISK +++ Add [sz 1536 add 1536] @KB/sec 327.02 {81.21}  ++ B   Verify /B_file.txt 3072B  @KB/sec 2357.64 
:: /C_file.txt  RAM_DISK +++ Add [sz 2048 add 2048] @KB/sec 487.04 {110.97}  ++ C   Verify /C_file.txt 4096B  @KB/sec 2386.95 
[  1.12 M](0.00067 M elap) Awaiting input 0123456789RdwcghkFqvplmusSBbyYxfan+-? loops left 0 >

Ran it several times, could not get it to hang.

I ran the exact from from msg #873. Didn't edit the defines or many changes in any way.
 
I tried running it as well on T4.1 with 2 memory chips...

Note: I have hacked the code:

To add in some memory dump and maybe check stack and heap usages...
Which I included. Note: older style memory output from the uncanney eyes large version...
Code:
C:\Users\kurte\Documents\Arduino\Teensy Tests\LFSIntegrity_debug\LFSIntegrity_debug.ino Jun 19 2021 07:15:58
LittleFS Test : File Integrity
IOMUXC_GPR_GPR17:aaaaaabf IOMUXC_GPR_GPR16:200007 IOMUXC_GPR_GPR14:aa0000
Initial Stack pointer: 20068000
ITCM allocated: 98304  DTCM allocated: 425984
ITCM init range: 0 - 11518 Count: 70936
DTCM init range: 20000000 - 20002580 Count: 9600
DTCM cleared range: 20002580 - 200052c0 Count: 11584
Now fill rest of DTCM with known pattern(200052c4 - 20067f8c
Ram2: variables:12384
[COLOR="#FF0000"]EXTRAM: variables: 8388608, size in MB: 16[/COLOR]

printDirectory RAM_DISK
--------------

 0 dirs with 0 files of Size 0 Bytes
 Total 0 files of Size 0 Bytes
Bytes Used: 4096, Bytes Total:8388608
 0, 1-9 '#' passes continue loop before Pause
 '$' Print memory info T4.x only
 'a' Auto formatUnused() during iterations - TOGGLE
 'R' Restart Teensy
 'd' Directory of LittleFS
 'w' WIPE Directory of LittleFS
 'b' big file delete
 'B' BIG FILE MAKE
 'S' FILE 2MB MAKE
 's' FILE 2MB delete
 'c' Continuous Loop
 'g' run speedBench()
 'h' Hundred loops
 'k' Thousand loops
 'F' LittleFS_ Low Level Format Disk 
 'f' LittleFS::formatUnused( ALL ) : DATA PRESERVED 
 'q' LittleFS_ Quick Format Disk 
 'v' Verbose All Dir Prints - TOGGLE
 'p' Pause after all Dir prints - TOGGLE
 'l' Show count of loop()'s, Bytes Read,Written
 'm' Make ROOT dirs (needed after q/F format !ROOTONLY)
 'u' Update Filecount
 'x' Directory filecount verify - TOGGLE
 'n' No verify on Write- TOGGLE
 '+' more add to delete cycles
 '-' fewer add to delete cycles
 'y' reclaim 1 block :: myfs.formatUnused( 1 )
 'Y' reclaim 15 blocks :: myfs.formatUnused( 15 )
 '?' Help list
?[  0.02 M](0.02042 M elap) Awaiting input 0123456789RdwcghkFqvplmusSBbyYxfan$+-? loops left 0 >  Estimate Stack and Heap usage

Estimated max stack usage: 1308
Estimated Heap usage: 4000
$
The Extram stuff will only show up on T4.1..
And if size used > you have it will show another line like: ***** External memory used larger than you have *****

Does not build on T4 as is.. Exceeds memory...
Suggest maybe ifdefs for which Teensy and for example only do the EXTMEM for T4.1...
 

Attachments

  • LFSIntegrity_debug-210619a.zip
    9.1 KB · Views: 44
Okay - I need to clean something.

Something not right in that it worked okay at first - until restart - then it started going wholly bad

On the RAMDISK - does the restart noted here from the Beta 10 thread work? :: start RAM :: 'B'igfile OR '1' loop :: 'R'estart >> does it restart to function and repeat?
 
Clean Install and all is well. Did 15 sec Restore of both T_4.0 and 4.1

Win 10 : IDE 1.8.15 unzip : TD 1.54 B 10 install

Using IDE and TeensyLoader and IDE SerMon.

Various : 'B' and '1'/'#' and 'R' to make files and then software restart and no problems on either 4.0 or 4.1 with DMAMEM LittleFS_RAM

TeensyLoader No-AUTO: Button, Reboot :: Did show error on 'B'igfile - but FS was safe and good after that file was deleted.
Code:
Delete with read verify all #bigfile's
	Verify /0_bigfile.txt bytes 202752 : <Bad Byte!  0! = [0x0] @1

	Read Count fail! :: read 1 != f.size 202752

	Big read&compare KBytes per second  0.00

Something must have been corrupt. From the initial odd behavior see on Beta 10 install that was ODD then it got worse? Not idea what that may have been - but GONE now.

ref:
Code:
Memory Usage on Teensy 4.1:
  FLASH: code:76216, data:9720, headers:8268   free for files:8032260
   RAM1: variables:21184, code:70488, padding:27816   free for local variables:404800
   RAM2: variables:502400  free for malloc/new:21888
Using library LittleFS at version 1.0.0 in folder: T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS 
Using library SPI at version 1.0 in folder: T:\arduino-1.8.15\hardware\teensy\avr\libraries\SPI 
Using library SdFat at version 2.0.5-beta.1 in folder: T:\arduino-1.8.15\hardware\teensy\avr\libraries\SdFat
 
Can anyone remind me why we ended up with 2 header files? Unless there is a strong need for a separate header, I'd really like to see a single LittleFS.h before we release 1.54.

Separate .cpp files is not an issue.

@Defragster - LFSintegrity.ino needs comments added at the top which explain its purpose and give a clear description to first-time users of how to run it and what to expect. When anyone opens the example, the first ~20 lines of the program are their first impression. A good first impression, which helps them understand the program's purpose is essential. Especially important is for people to quickly understand whether the example is *not* the thing they need for whatever goals they may have.

Perhaps the name should be changed, at least dropping "LFS". Maybe IntegrityTestSuite.ino? The name should be chosen so that first-time users who see it in the File > Examples menu understand it's for testing the library, rather than an example they're meant to use for code to copy into their own application.

Ideally, I'd really like to see more human readable output. For example, imagine someone with no experience using the library chooses LFSintegrity as their first attempt to do something. What is a new user supposed to think of this sort of result?

Code:
:: /B_file.txt  RAM_DISK +++ Add [sz 1536 add 1536] @KB/sec 327.02 {81.21}  ++ B   Verify /B_file.txt 3072B  @KB/sec 2357.64 
:: /C_file.txt  RAM_DISK +++ Add [sz 2048 add 2048] @KB/sec 487.04 {110.97}  ++ C   Verify /C_file.txt 4096B  @KB/sec 2386.95 
[  1.12 M](0.00067 M elap) Awaiting input 0123456789RdwcghkFqvplmusSBbyYxfan+-? loops left 0 >

LFSintegrity.ino has a lot of very good work in it, but for a non-beta release, it needs at least some more work so it doesn't create a bad first-use experience for new users.
 
Trying to debug this and recalling that SIZE's are returned as 64 bit.
Learned this before - and seeing some print anomalies trying to debug ... doing cleanup ...

These common methods:
Code:
	virtual uint64_t size() {
	uint64_t usedSize() {
	uint64_t totalSize() {

But see this for file.size() returning uint64_t :: internally using typedef int32_t lfs_soff_t;

Code:
	virtual [B]uint64_t[/B] size() {
		if (!file) return 0;
		[B][U]lfs_soff_t size[/U][/B] = lfs_file_size(lfs, file);
		if (size < 0) size = 0;
		return [B]size[/B];
	}

That uses ( internally consistent checking above NEG on SIGNED int32_t ) : lfs_soff_t lfs_file_size(lfs_t *lfs, lfs_file_t *file) {

But can only handle 31 bit numbers and returning 64 bits?

Only looking because LFSIntegrity has %llu to print .size - but forgot above lesson and adding early exit on bad file size used a uint32_t and seeing size of this woke me up ... : 2305891319005839368

Also saw this where disk is FULL with 20KB of files using 489KB of disk space doing RAM Restarts - :
Code:
printDirectory RAM_DISK
--------------
FILE	B_file.txt		3072
FILE	C_file.txt		4096
FILE	D_file.txt		2560
FILE	E_file.txt		3072
FILE	F_file.txt		3584
FILE	G_file.txt		4096

 0 dirs with 6 files of Size 20480 Bytes
 Total 6 files of [B]Size 20480 Bytes[/B]
[B]Bytes Used: 489984[/B], Bytes Total:489984
 
Can anyone remind me why we ended up with 2 header files? Unless there is a strong need for a separate header, I'd really like to see a single LittleFS.h before we release 1.54.

Separate .cpp files is not an issue.

@Defragster - LFSintegrity.ino needs comments added at the top which explain its purpose and give a clear description to first-time users of how to run it and what to expect. When anyone opens the example, the first ~20 lines of the program are their first impression. A good first impression, which helps them understand the program's purpose is essential. Especially important is for people to quickly understand whether the example is *not* the thing they need for whatever goals they may have.

Perhaps the name should be changed, at least dropping "LFS". Maybe IntegrityTestSuite.ino? The name should be chosen so that first-time users who see it in the File > Examples menu understand it's for testing the library, rather than an example they're meant to use for code to copy into their own application.

Ideally, I'd really like to see more human readable output. For example, imagine someone with no experience using the library chooses LFSintegrity as their first attempt to do something. What is a new user supposed to think of this sort of result?

Code:
:: /B_file.txt  RAM_DISK +++ Add [sz 1536 add 1536] @KB/sec 327.02 {81.21}  ++ B   Verify /B_file.txt 3072B  @KB/sec 2357.64 
:: /C_file.txt  RAM_DISK +++ Add [sz 2048 add 2048] @KB/sec 487.04 {110.97}  ++ C   Verify /C_file.txt 4096B  @KB/sec 2386.95 
[  1.12 M](0.00067 M elap) Awaiting input 0123456789RdwcghkFqvplmusSBbyYxfan+-? loops left 0 >

LFSintegrity.ino has a lot of very good work in it, but for a non-beta release, it needs at least some more work so it doesn't create a bad first-use experience for new users.

RE: 2nd Header :: IIRC @mjs513 was pushing the envelope for NAND and wasn't sure it would get done for TD 1.54 - so kept was separate.
> at this point it managed to mature and work, so suspect it could be merged into single header for release.

RE: LFSIntegrity : Good and valid points, the code and layout confuses me too.
I thought LFSIntegrity was a lot to type the last 4,000 times, so "IntegrityTestSuite.ino" is much better :) ... but at least that could be said as ITS.ino ... Integrity also okay
> as noted putting in some more checks - will try to leave it cleaner and add comments

Glad I made it as it gives me confidence in the function from a DEV level - but it has grown to be a BEAST and it is cryptic to be sure with reduced output to focus on proper function ( knowing what it should say ) and popping out errors.
Goal was to exemplify interfaces in source code and see them used - as run it is a 'busy work' pointless program that proves function internally and spits out the cryptic sausage making process in case something goes wrong.
... with lots of feature creep and no real plan/expectation it would end up where it is.
 
Can anyone remind me why we ended up with 2 header files? Unless there is a strong need for a separate header, I'd really like to see a single LittleFS.h before we release 1.54.

Separate .cpp files is not an issue.

......
LFSintegrity.ino has a lot of very good work in it, but for a non-beta release, it needs at least some more work so it doesn't create a bad first-use experience for new users.
Think I was keeping it separate for devel and not corrupt the LittleFS header. To be honest probably should have merged the too. Would keep the .cpp's separate though as you said.

Ok - on my to do list. whats your target for 1.55? Was debugging an issue with my RA8875's

EDIT: Testing a merge right now.
 
Last edited:
defragster said:
Something must have been corrupt. From the initial odd behavior see on Beta 10 install that was ODD then it got worse? Not idea what that may have been - but GONE now.
Glad you got it working :). Was getting me worried.
 
@Paul/@all RE: LFSIntegrity
As noted it was a Swiss Army knife morphing as Media were added and test avenues crept in - and still are ...

For more useful User Examples - something that made no sense during DEV time when core code was common - perhaps split it into an Integrity Folder with cleaner custom clones to the media at hand::
Code:
Examples / Integrity
  -> TestRam
  -> TestPSRAM
  -> TestSPI [ NOR | NAND ??]
  -> TestQSPI [ NOR | NAND ??]
  -> TestSPIMRAM

That would allow removing/minimizing the Complex MultiPlexing start of the file ... i.e. #ifdef Hell. Isolating the hardware setup from the common running code in each case.

Nothing worse than trying to test Interface X - and forgetting to set one 'define' and seeing it FAIL ... and not knowing if Hardware or Software.
 
Glad you got it working :). Was getting me worried.

Worried ... Me too. But was end of day ... and it was easier to sleep throwing the issue out to @all 'test' ... assuming PJRC changes to .LD files or startup.c that had just shown need for change.
 
@degragster - @PaulStoffregen - @KurtE

Just pushed changes to a my fork of the LittleFS reposititory: https://github.com/mjs513/LittleFS . Tested SPI-NAND with the memory board as well SPI Flash on the same board. All worked. If you want to test feel free and let me know if there are any problems.

EDIT: Ok just submitted the PR.
 
RESTART testing was never part of my process here until seeing when RAM disk could warm restart.

With that finding - all the dCache devices (QSPI FLASH) need the static_sync add of dcache_flush - if they were to restart Warm or Cold they should be SAFE and given RAM findings - they will be corrupted if dCache is not flushed.
Code:
	static int static_sync(const struct lfs_config *c) {
		if ( c->context >= (void *)0x20200000 ) {
			//Serial.printf("    arm_dcache_flush_delete: ptr=0x%x, size=%d\n", c->context, c->block_count * c->block_size);
			arm_dcache_flush_delete(c->context, c->block_count * c->block_size );
		}
		return 0;
	}

Currently they are void funcs:
Code:
	static int static_sync(const struct lfs_config *c) {
		return 0;
	}


Again I think the _delete may be overkill but was at hand when I did copy/paste ???

I've got work to do for Integrity release ... Does the Multi Example seem like the best end for cleaner examples?
 
defragster said:
I've got work to do for Integrity release ... Does the Multi Example seem like the best end for cleaner examples?
Personally -yes i do - makes a good example for all the storage and a good reference.
 
Personally -yes i do - makes a good example for all the storage and a good reference.

Cool ... Occurred to me as way to make it more understandable and usable for test and ref. ... one FS Media at a time.

There are other examples where a CORE bus/concept is presented with sub elements where each does just one thing - that for LittleFS seemed to apply. Especially to hope to get it small enough for a user to read.

Otherwise that top part only works with 'inside' knowledge. The underlying code will still be the same odd cyclic looping - but at least the #ifdef and settings would be ready to run.

Will see if I can get focus on it after resolving the new issue I'm seeing - clean the common CORE loop behavior - then add comments and prettify the Media type intro portions above and in setup. First step might be backwards - exploding LFSIntegrity into IntegrityTestSuite.ino with each Media having ALL the top {setup() and above } replicated in sperate #ifdef XXX, so I can switch between them to test one last time ... then parse them out with no #ifdef
>> Using Paul's name idea maybe:: LittleFS / Examples / IntegritySuite / [ RAM, PSRAM, SPI, QSPI, ... ]

MRAM using SPI are not so different from SPI FLASH - except being much smaller like RAM .vs. PSRAM the size increments need tweaking to avoid over filling the disk causing OUT OF SPACE errors. I've been hitting that doing the RAM test as I didn't tweak them down.

Not sure how to fix the cryptic output ... it started doing a full DIR after each file ... that was a ton of spew on faster RAM, but a FULL dir is SLOW on FLASH media so the Verbose got set OFF.
> the initial test idea was just to do random like create, append, delete forcing all the underlying Media area to cycle through and DIR entries to some and go and get updated. Doing that for 16MB or 64MB took lots of files in lots of directories and a mile of Spew. Once it was stable that jibberish just says it is working with enough info to see where it went wrong if a failure does show up. So the fix would be another 4th level Less verbose hiding that ...
 
Added "D" to Integrity, it walks the dir reading every file up to the returned file.size(). Reports Error if it doesn't get that many and also stops reads with error when it exceeds the reported .size()

Added .flush() before close and that doesn't help for the problem noted before and shown below:

latest repro on DMAMEM RAM disk
> 'F'ormat fresh
> '1' - one iteration
> 'R'estart Teensy
> 'Enter' to exit setup
> '1' - one iteration
> 'R'estart Teensy
> 'Enter' to exit setup
> '1' - one iteration
> 'R'estart Teensy
> 'Enter' to exit setup
> "D" shows 1 file size error

Output showing the error in the end and just before the the reported SPASM of myfs.usedSize() giving >> Bytes Used: 489984
This also repros using "B"igfile - but these small files are more trivial and shouldn't be as stressful to move and manipulate.
After a restart the '1' iteration always starts at the same point - so it finds the prior files and extends them, which is not how "B"igfiles works.
This only happens because of the restart - I assume the "lfs CORE' code is caching something internally - unless there is something unique about the code for :: myfs.usedSize()
>> This is important because this easy to do and restore repro on RAM disk would most likely be the same or WORSE on larger FLASH disks - they need the added _sync() flush of dcache - and they'll need ALL FS data updated onto the media.
- > as noted I added file.flush() after write before .close and that didn't help. Is there another unfound way to purge all lfs core elements that is missing?
Code:
// RESTART #2

printDirectory RAM_DISK
--------------
FILE	B_file.txt		3072
FILE	C_file.txt		4096

 0 dirs with 2 files of Size 7168 Bytes
 Total 2 files of Size 7168 Bytes
[B][COLOR="#FF0000"]Bytes Used: 8192[/COLOR][/B], Bytes Total:489984
[  0.03 M](0.02992 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >
Walk all Files verify Read Size:	
..0 Errors found

	 Loop Count: 1 (#fileCycle=0), Bytes read 0, written 0, #Files=2
	dirV...91_us	[  0.08 M](0.00004 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >

[  0.08 M](0.00004 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >1
:: /B_file.txt     [U]arm_dcache_flush_delete[/U]: ptr=0x20200000, size=489984
 RAM_DISK +++ Add [sz 3072 add 3072] @KB/sec  0.00 {1349.74}  ++ B   Verify /B_file.txt 4608B  @KB/sec 2691.59 
:: /C_file.txt     [U]arm_dcache_flush_delete[/U]: ptr=0x20200000, size=489984
 RAM_DISK +++ Add [sz 4096 add 4096] @KB/sec  0.00 {1579.03}  ++ C   Verify /C_file.txt 6144B  @KB/sec 2707.80 
[  0.12 M](0.00011 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 > [B][COLOR="#FF0000"]RESTART Teensy ...[/COLOR][/B]
T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS\examples\LFSintegrity\LFSintegrity.ino Jun 20 2021 00:55:24
LittleFS Test : File Integrity
printDirectory RAM_DISK
--------------
FILE	B_file.txt		4608
FILE	C_file.txt		6144

 0 dirs with 2 files of Size 10752 Bytes
 Total 2 files of Size 10752 Bytes
[B][COLOR="#FF0000"]Bytes Used: 489984[/COLOR][/B], Bytes Total:489984

// RESTART #3

printDirectory RAM_DISK
--------------
FILE	B_file.txt		4608
FILE	C_file.txt		6144

 0 dirs with 2 files of Size 10752 Bytes
 Total 2 files of Size 10752 Bytes
[B][COLOR="#FF0000"]Bytes Used: 489984[/COLOR][/B], Bytes Total:489984
[  0.10 M](0.10200 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >
Walk all Files verify Read Size:	
..
[B][COLOR="#FF0000"] File Size Error:: C_file.txt found 6145 Bytes for Size 6144 
1 Errors found

	 Loop Count: 1 (#fileCycle=0), Bytes read 0, written 0, #Files=2
	 ERROR COUNT =1[/COLOR][/B]
 
Given post #890 a test was added when DIR is shown - and a DIR is shown on start in setup().

It is consistent as noted above after the 3rd 'R'estart process from a fresh format the following appears.

Too late to look harder now - but I glanced a bit in lfs core and didn't see anything to follow. I saw that a READ during WRITE does a flush internally - so I swapped a read() for flush() and no diff. It also seemed like there was a flush done on each close() - so I removed the flush() as that didn't change anything.

But in :: T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS\src\LittleFS.h I did see this:
Code:
	uint64_t usedSize() {
		if (!mounted) return 0;
		int blocks = lfs_fs_size(&lfs);
		[B][COLOR="#FF0000"]if (blocks < 0 || (lfs_size_t)blocks > config.block_count) return totalSize();[/COLOR][/B]
		return blocks * config.block_size;
	}

Code:
[B][U]Updated code is here[/U][/B] ::   [B][ATTACH]25086._xfImport[/ATTACH][/B]

T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS\examples\LFSintegrity\LFSintegrity.ino

Code:
T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS\examples\LFSintegrity\LFSintegrity.ino Jun 20 2021 01:46:07
LittleFS Test : File Integrity
printDirectory RAM_DISK
--------------
FILE	B_file.txt		4608
FILE	C_file.txt		6144

 0 dirs with 2 files of Size 10752 Bytes
 Total 2 files of Size 10752 Bytes
Bytes Used: 489984, Bytes Total:489984


	[B][COLOR="#FF0000"]WARNING: DISK FULL >>>>>  Bytes Used: 489984, Bytes Total:489984[/COLOR][/B]

This edit to the above code prints the compared values in usedSize() { where uint32_t is == lfs_size_t }:
Code:
	uint64_t usedSize() {
		if (!mounted) return 0;
		[B][COLOR="#FF0000"]int blocks[/COLOR][/B] = lfs_fs_size(&lfs);
		if (blocks < 0 || (lfs_size_t)blocks > config.block_count) {
			[B]Serial.printf("    usedSize(): (lfs_size_t)blocks=%lu, config.block_count=%lu\n", (lfs_size_t)blocks, config.block_count);[/B]
			return totalSize();
		}
		return blocks * config.block_size;
	}


Code:
LittleFS Test : File Integrity
printDirectory RAM_DISK
--------------
FILE	B_file.txt		4608
FILE	C_file.txt		6144

 0 dirs with 2 files of Size 10752 Bytes
 Total 2 files of Size 10752 Bytes
    [B]usedSize(): (lfs_size_t)blocks=4294967212, config.block_count=1914[/B]
Bytes Used: 489984, Bytes Total:489984
    [B]usedSize(): (lfs_size_t)blocks=4294967212, config.block_count=1914[/B]
    [B]usedSize(): (lfs_size_t)blocks=4294967212, config.block_count=1914[/B]


	WARNING: DISK FULL >>>>>  Bytes Used: 489984, Bytes Total:489984

And it seems this 'lfs core' code is returning an ERROR: T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS\src\littlefs\lfs.c
Code:
lfs_ssize_t lfs_fs_size(lfs_t *lfs) {
    LFS_TRACE("lfs_fs_size(%p)", (void*)lfs);
    lfs_size_t size = 0;
    int err = lfs_fs_traverseraw(lfs, lfs_fs_size_count, &size, false);
    if (err) {
        LFS_TRACE("lfs_fs_size -> %d", err);
        return err;
    }

    LFS_TRACE("lfs_fs_size -> %d", err);
    return size;
}

AFAIK the "lfs_fs_traverseraw()" is the internal version of the block used file chain WALK I used to find the USED/UNUSED disk blocks for formatUnused() - so that is what is failing ???
>>> It is probably failing for the same reason the READ of the corrupted file was doing a FOREVER "while ( file3.available() ) {" loop reading the file in bigVerify() : It is walking a file chain without an END marker saved on disk.
-> Seems that unsaved FS element is critical and needs to be flushed out of lfs.

Also - no DIFF but the arm_dcache_flush_delete() was changed to arm_dcache_flush().
Are these the expected numbers for this DMAMEM disk? Does thelfe core code store anything AFTER the DISK size area? {c->block_count * c->block_size}:: arm_dcache_flush: ptr=0x20200000, size=489984

Zzzzz...
 
Last edited:
With that finding - all the dCache devices (QSPI FLASH) need the static_sync add of dcache_flush - if they were to restart Warm or Cold they should be SAFE and given RAM findings - they will be corrupted if dCache is not flushed.

I don't understand why this would be needed. Those other devices don't leave data stored in RAM.
 
To explain a bit further, here is the FlexSPI block diagram from page 1623 in the reference manual.

screenshot.png

The important concept is 2 paths exist to gain access to the QSPI memory chips. Ordinary memory access (cache misses) access via the 64 bit AHB path. Software can also "manually" access the memory using the IPS 32 bit path.

As far as I know, all the QPI NOR & NAND flash code is exclusively using the IPS path.

LittleFS_Program is the only device which uses both. Reads are done using memcpy(), which of course uses the 64 bit AHB path. Writes are done on the 32 bit IPS path. The write code already does use arm_dcache_delete(), so after data is written future reads will not fetch stale data from the CPU cache. There should never need to be a case of flushing the cache, because writing can never happen over the AHB path.

But it's entirely possible I've missed some important case. If we really do need to add "sync" callbacks on the non-RAM storage devices, please explain why. I just don't see it, but I'm willing to listen to any explanation or try running any test case.
 
Good Morning all: Sorry been distracted. Distractions still ongoing :D

@defragster - One thing with LFSIntegrety sketch that I always wished was some of the sections like TEST_RAM
Actually had some configurations depending on which board you are building for.

Example only try to use EXTMEM if you are running on Teensy41.
#ifdef ARDUINO_TEENSY41


Also with Teensy41. Maybe put explicit option to enable EXTMEM in test...
Might also want some of the debug code I put in, like I did in the debug version I posted, that sees your sketch is configured to use 8mb of PSRAM, but you T4.1 does not have any...
 
I don't understand why this would be needed. Those other devices don't leave data stored in RAM.

Makes sense, Thanks.

Opps, Hadn't thought the process through: QSPI Flash writes can not be queued in cache and handled later (as DMAMEM and PSRAM can do), because in the case of the QSPI Flash the 'helper code' does immediate write to Flash.

Though - if the LFS Core is not flushing everything - the problem the RAM drive is seeing could show up on FLASH as well with 'unplanned restart'. Not sure if that was directly tested as the RAM case is doing.
 
Good Morning all: Sorry been distracted. Distractions still ongoing :D

@defragster - One thing with LFSIntegrety sketch that I always wished was some of the sections like TEST_RAM
Actually had some configurations depending on which board you are building for.

Example only try to use EXTMEM if you are running on Teensy41.
#ifdef ARDUINO_TEENSY41


Also with Teensy41. Maybe put explicit option to enable EXTMEM in test...
Might also want some of the debug code I put in, like I did in the debug version I posted, that sees your sketch is configured to use 8mb of PSRAM, but you T4.1 does not have any...

Notes above showed PSRAM separate from RAM1 or RAM2 Example usage, and indeed for general consumption the RAM1 / RAM2 variant would need a clear way to differentiate.
Because, while 'we' know PSRAM uses the same code, it is a case that shouldn't be confusing to use, or users might get false failures after soldering.

Not sure I'm following in the RAM section currently there are TWO (cryptic) lines - one line can be changed (as commented) or a third line added:
Code:
#ifdef TEST_RAM
LittleFS_RAM myfs;
[B]// RUNTIME :: extern "C" uint8_t external_psram_size;
//EXTMEM char buf[MYPSRAM * 1024 * 1024];	// USE DMAMEM for more memory than ITCM allows - or remove
DMAMEM char buf[490000];	[B][COLOR="#FF0000"]// USE DMAMEM for more memory than ITCM allows - or remove[/COLOR][/B]
[/B]char szDiskMem[] = "RAM_DISK";
#elif defined(TEST_SPI)

When testing RAM1 changing of adding an alteration of the DMAMEM line like - with change for available RAM:
Code:
//DMAMEM char buf[490000];	// [B][COLOR="#FF0000"]USE DMAMEM for more memory than ITCM allows - or remove[/COLOR][/B]
char buf[390000];	// USE DMAMEM for more memory than ITCM allows - or remove
 
RE post #891 and corruption on DMAMEM RAM DRIVE.

Just move to PSRAM and the same behavior does NOT repeat?

Did similar allocations and 'R'estart cycles and all is well?

Just enabled 12 directories and doing larger runs All seems proper with PSRAM usage, not yet triggered the behavior seen with DMAMEM?

This bodes well for FLASH - but DMAMEM a puzzle why the Warm Restart is getting corrupted?

Post #891 CODE: In adding dirs I found Copy/Paste Edit error in this function about line 603 - it wasn't recursing on itself - but the donor code :(
Code:
int DirectoryVerify(File dir) {
	int errCnt = 0;
	while (true) {
		File entry =  dir.openNextFile();
		if (! entry) {
			break;
		}
		if (entry.isDirectory()) {
			Serial.print("\tD");
			errCnt += [B][COLOR="#FF0000"]DirectoryVerify[/COLOR](entry);[/B]
 
Notes above showed PSRAM separate from RAM1 or RAM2 Example usage, and indeed for general consumption the RAM1 / RAM2 variant would need a clear way to differentiate.
Because, while 'we' know PSRAM uses the same code, it is a case that shouldn't be confusing to use, or users might get false failures after soldering.

Not sure I'm following in the RAM section currently there are TWO (cryptic) lines - one line can be changed (as commented) or a third line added:
Code:
#ifdef TEST_RAM
LittleFS_RAM myfs;
[B]// RUNTIME :: extern "C" uint8_t external_psram_size;
//EXTMEM char buf[MYPSRAM * 1024 * 1024];	// USE DMAMEM for more memory than ITCM allows - or remove
DMAMEM char buf[490000];	[B][COLOR="#FF0000"]// USE DMAMEM for more memory than ITCM allows - or remove[/COLOR][/B]
[/B]char szDiskMem[] = "RAM_DISK";
#elif defined(TEST_SPI)

When testing RAM1 changing of adding an alteration of the DMAMEM line like - with change for available RAM:
Code:
//DMAMEM char buf[490000];	// [B][COLOR="#FF0000"]USE DMAMEM for more memory than ITCM allows - or remove[/COLOR][/B]
char buf[390000];	// USE DMAMEM for more memory than ITCM allows - or remove

Thanks Defragster... Was just sort of wondering for the main RAM test and maybe others if it might make sense to have, it some how preconfigured for different boards...
That is if you are going back and forth testing on T4.1 with, T4.1 withouth, Micromod, T4, T3.6, T3.5, T3.2 would it make sense to have default setup for each board so quick compile for that board will give you sort of the default testing...

Probably not a bit deal, just a thought
 
PSRAM seems safe for use with Software Restart - between operations all is properly stored.

Did 20 or 30 or more Restarts with "same" operations as RAM2 showed corruption with and it worked perfectly each time.

Where "same" was then with more folders and multiple "B" or "S" files to make sure media space was filled and reused. And each restart was a'5' iteration set not '1' to create and invoke more file changes across the dirs.

In case anyone wants to see here is the current sketch :
Code:
[B][ATTACH]25089._xfImport[/ATTACH][/B]
> going to do some quick repeats of process on a 1G QSPI NAND and get to another non-Teensy task I've been doing background wise but needs to be done ...


Not sure how DMAMEM is 'cracking' - if the _FLUSH is right - but it is showing that anomaly regularly.

Back to PSRAM - FUNNY I bumped from 8MB to two CHIP 16MB between uploads and the Disk Survived and Extended! And ran repeating fillings from there ???
> Not sure the mount code could or should look for that? Ideally it should not work ... going to go from 16 to 8 and see what shows.
>> Downsizing 16 to 8 MB did not Mount old drive but presented a fresh formatted drive
- > back from 8 to 16 MB and it picked up running again!

Updated the "D" file walk length verify - put out fir DIR or FILE letter:
Code:
Walk all Files verify Read Size:	
00	D1 FGNOPQRSTUVW
	D1 HIPQRSTUVWXY
	D1 
1	D1 ABCDEFGHIJKLMNOPQRSTUVWXYZ
	D2 ABCDEFGHIJKLMNOPQRSTUVWXYZ
	D3 ABCDEFGHIJKLMNOPQRSTUVWXYZ
	D4 BCDEFGHIJK
	D5 DEFGHIJKLM
	D6 FGHIJKLMNO
	D7 ABCDEFGHIJKLMNOPQ
	D8 BCJKLMNOPQRS
	D9 DELMNOPQRSTU
ABCDEFGHIJKLMNOPQRSTUVWXYZ	0 Errors found
 
Back
Top