LittleFS port to Teensy/SPIFlash

Printable View

Show 40 post(s) from this thread on one page

06-19-2021, 02:25 PM
KurtE

1 Attachment(s)

I tried running it as well on T4.1 with 2 memory chips...

Note: I have hacked the code:

To add in some memory dump and maybe check stack and heap usages...
Which I included. Note: older style memory output from the uncanney eyes large version...

Code:

C:\Users\kurte\Documents\Arduino\Teensy Tests\LFSIntegrity_debug\LFSIntegrity_debug.ino Jun 19 2021 07:15:58 LittleFS Test : File Integrity IOMUXC_GPR_GPR17:aaaaaabf IOMUXC_GPR_GPR16:200007 IOMUXC_GPR_GPR14:aa0000 Initial Stack pointer: 20068000 ITCM allocated: 98304 DTCM allocated: 425984 ITCM init range: 0 - 11518 Count: 70936 DTCM init range: 20000000 - 20002580 Count: 9600 DTCM cleared range: 20002580 - 200052c0 Count: 11584 Now fill rest of DTCM with known pattern(200052c4 - 20067f8c Ram2: variables:12384 EXTRAM: variables: 8388608, size in MB: 16 printDirectory RAM_DISK -------------- 0 dirs with 0 files of Size 0 Bytes Total 0 files of Size 0 Bytes Bytes Used: 4096, Bytes Total:8388608 0, 1-9 '#' passes continue loop before Pause '$' Print memory info T4.x only 'a' Auto formatUnused() during iterations - TOGGLE 'R' Restart Teensy 'd' Directory of LittleFS 'w' WIPE Directory of LittleFS 'b' big file delete 'B' BIG FILE MAKE 'S' FILE 2MB MAKE 's' FILE 2MB delete 'c' Continuous Loop 'g' run speedBench() 'h' Hundred loops 'k' Thousand loops 'F' LittleFS_ Low Level Format Disk 'f' LittleFS::formatUnused( ALL ) : DATA PRESERVED 'q' LittleFS_ Quick Format Disk 'v' Verbose All Dir Prints - TOGGLE 'p' Pause after all Dir prints - TOGGLE 'l' Show count of loop()'s, Bytes Read,Written 'm' Make ROOT dirs (needed after q/F format !ROOTONLY) 'u' Update Filecount 'x' Directory filecount verify - TOGGLE 'n' No verify on Write- TOGGLE '+' more add to delete cycles '-' fewer add to delete cycles 'y' reclaim 1 block :: myfs.formatUnused( 1 ) 'Y' reclaim 15 blocks :: myfs.formatUnused( 15 ) '?' Help list ?[ 0.02 M](0.02042 M elap) Awaiting input 0123456789RdwcghkFqvplmusSBbyYxfan$+-? loops left 0 > Estimate Stack and Heap usage Estimated max stack usage: 1308 Estimated Heap usage: 4000 $

The Extram stuff will only show up on T4.1..
And if size used > you have it will show another line like: ***** External memory used larger than you have *****

Does not build on T4 as is.. Exceeds memory...
Suggest maybe ifdefs for which Teensy and for example only do the EXTMEM for T4.1...
06-19-2021, 03:05 PM
defragster

Okay - I need to clean something.

Something not right in that it worked okay at first - until restart - then it started going wholly bad

On the RAMDISK - does the restart noted here from the Beta 10 thread work? :: start RAM :: 'B'igfile OR '1' loop :: 'R'estart >> does it restart to function and repeat?
06-19-2021, 04:39 PM
defragster

Clean Install and all is well. Did 15 sec Restore of both T_4.0 and 4.1

Win 10 : IDE 1.8.15 unzip : TD 1.54 B 10 install

Using IDE and TeensyLoader and IDE SerMon.

Various : 'B' and '1'/'#' and 'R' to make files and then software restart and no problems on either 4.0 or 4.1 with DMAMEM LittleFS_RAM

TeensyLoader No-AUTO: Button, Reboot :: Did show error on 'B'igfile - but FS was safe and good after that file was deleted.

Code:

Delete with read verify all #bigfile's Verify /0_bigfile.txt bytes 202752 : <Bad Byte! 0! = [0x0] @1 Read Count fail! :: read 1 != f.size 202752 Big read&compare KBytes per second 0.00

Something must have been corrupt. From the initial odd behavior see on Beta 10 install that was ODD then it got worse? Not idea what that may have been - but GONE now.

ref:

Code:

Memory Usage on Teensy 4.1: FLASH: code:76216, data:9720, headers:8268 free for files:8032260 RAM1: variables:21184, code:70488, padding:27816 free for local variables:404800 RAM2: variables:502400 free for malloc/new:21888 Using library LittleFS at version 1.0.0 in folder: T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS Using library SPI at version 1.0 in folder: T:\arduino-1.8.15\hardware\teensy\avr\libraries\SPI Using library SdFat at version 2.0.5-beta.1 in folder: T:\arduino-1.8.15\hardware\teensy\avr\libraries\SdFat
06-19-2021, 06:29 PM
PaulStoffregen

Can anyone remind me why we ended up with 2 header files? Unless there is a strong need for a separate header, I'd really like to see a single LittleFS.h before we release 1.54.

Separate .cpp files is not an issue.

@Defragster - LFSintegrity.ino needs comments added at the top which explain its purpose and give a clear description to first-time users of how to run it and what to expect. When anyone opens the example, the first ~20 lines of the program are their first impression. A good first impression, which helps them understand the program's purpose is essential. Especially important is for people to quickly understand whether the example is *not* the thing they need for whatever goals they may have.

Perhaps the name should be changed, at least dropping "LFS". Maybe IntegrityTestSuite.ino? The name should be chosen so that first-time users who see it in the File > Examples menu understand it's for testing the library, rather than an example they're meant to use for code to copy into their own application.

Ideally, I'd really like to see more human readable output. For example, imagine someone with no experience using the library chooses LFSintegrity as their first attempt to do something. What is a new user supposed to think of this sort of result?

Code:

:: /B_file.txt RAM_DISK +++ Add [sz 1536 add 1536] @KB/sec 327.02 {81.21} ++ B Verify /B_file.txt 3072B @KB/sec 2357.64 :: /C_file.txt RAM_DISK +++ Add [sz 2048 add 2048] @KB/sec 487.04 {110.97} ++ C Verify /C_file.txt 4096B @KB/sec 2386.95 [ 1.12 M](0.00067 M elap) Awaiting input 0123456789RdwcghkFqvplmusSBbyYxfan+-? loops left 0 >

LFSintegrity.ino has a lot of very good work in it, but for a non-beta release, it needs at least some more work so it doesn't create a bad first-use experience for new users.
06-19-2021, 06:49 PM
defragster

Trying to debug this and recalling that SIZE's are returned as 64 bit.
Learned this before - and seeing some print anomalies trying to debug ... doing cleanup ...

These common methods:

Code:

virtual uint64_t size() { uint64_t usedSize() { uint64_t totalSize() {

But see this for file.size() returning uint64_t :: internally using typedef int32_t lfs_soff_t;

Code:

virtual uint64_t size() { if (!file) return 0; lfs_soff_t size = lfs_file_size(lfs, file); if (size < 0) size = 0; return size; }

That uses ( internally consistent checking above NEG on SIGNED int32_t ) : lfs_soff_t lfs_file_size(lfs_t *lfs, lfs_file_t *file) {

But can only handle 31 bit numbers and returning 64 bits?

Only looking because LFSIntegrity has %llu to print .size - but forgot above lesson and adding early exit on bad file size used a uint32_t and seeing size of this woke me up ... : 2305891319005839368

Also saw this where disk is FULL with 20KB of files using 489KB of disk space doing RAM Restarts - :

Code:

printDirectory RAM_DISK -------------- FILE B_file.txt 3072 FILE C_file.txt 4096 FILE D_file.txt 2560 FILE E_file.txt 3072 FILE F_file.txt 3584 FILE G_file.txt 4096 0 dirs with 6 files of Size 20480 Bytes Total 6 files of Size 20480 Bytes Bytes Used: 489984, Bytes Total:489984
06-19-2021, 07:05 PM
defragster

Quote:

Originally Posted by PaulStoffregen

Can anyone remind me why we ended up with 2 header files? Unless there is a strong need for a separate header, I'd really like to see a single LittleFS.h before we release 1.54.

Separate .cpp files is not an issue.

@Defragster - LFSintegrity.ino needs comments added at the top which explain its purpose and give a clear description to first-time users of how to run it and what to expect. When anyone opens the example, the first ~20 lines of the program are their first impression. A good first impression, which helps them understand the program's purpose is essential. Especially important is for people to quickly understand whether the example is *not* the thing they need for whatever goals they may have.

Perhaps the name should be changed, at least dropping "LFS". Maybe IntegrityTestSuite.ino? The name should be chosen so that first-time users who see it in the File > Examples menu understand it's for testing the library, rather than an example they're meant to use for code to copy into their own application.

Ideally, I'd really like to see more human readable output. For example, imagine someone with no experience using the library chooses LFSintegrity as their first attempt to do something. What is a new user supposed to think of this sort of result?

Code:

:: /B_file.txt RAM_DISK +++ Add [sz 1536 add 1536] @KB/sec 327.02 {81.21} ++ B Verify /B_file.txt 3072B @KB/sec 2357.64 :: /C_file.txt RAM_DISK +++ Add [sz 2048 add 2048] @KB/sec 487.04 {110.97} ++ C Verify /C_file.txt 4096B @KB/sec 2386.95 [ 1.12 M](0.00067 M elap) Awaiting input 0123456789RdwcghkFqvplmusSBbyYxfan+-? loops left 0 >

LFSintegrity.ino has a lot of very good work in it, but for a non-beta release, it needs at least some more work so it doesn't create a bad first-use experience for new users.

RE: 2nd Header :: IIRC @mjs513 was pushing the envelope for NAND and wasn't sure it would get done for TD 1.54 - so kept was separate.
> at this point it managed to mature and work, so suspect it could be merged into single header for release.

RE: LFSIntegrity : Good and valid points, the code and layout confuses me too.
I thought LFSIntegrity was a lot to type the last 4,000 times, so "IntegrityTestSuite.ino" is much better :) ... but at least that could be said as ITS.ino ... Integrity also okay
> as noted putting in some more checks - will try to leave it cleaner and add comments

Glad I made it as it gives me confidence in the function from a DEV level - but it has grown to be a BEAST and it is cryptic to be sure with reduced output to focus on proper function ( knowing what it should say ) and popping out errors.
Goal was to exemplify interfaces in source code and see them used - as run it is a 'busy work' pointless program that proves function internally and spits out the cryptic sausage making process in case something goes wrong.
... with lots of feature creep and no real plan/expectation it would end up where it is.
06-19-2021, 07:26 PM
mjs513

Quote:

Originally Posted by PaulStoffregen

Can anyone remind me why we ended up with 2 header files? Unless there is a strong need for a separate header, I'd really like to see a single LittleFS.h before we release 1.54.

Separate .cpp files is not an issue.

......
LFSintegrity.ino has a lot of very good work in it, but for a non-beta release, it needs at least some more work so it doesn't create a bad first-use experience for new users.

Think I was keeping it separate for devel and not corrupt the LittleFS header. To be honest probably should have merged the too. Would keep the .cpp's separate though as you said.

Ok - on my to do list. whats your target for 1.55? Was debugging an issue with my RA8875's

EDIT: Testing a merge right now.
06-19-2021, 07:28 PM
mjs513

Quote:

Originally Posted by defragster

Something must have been corrupt. From the initial odd behavior see on Beta 10 install that was ODD then it got worse? Not idea what that may have been - but GONE now.

Glad you got it working :). Was getting me worried.
06-19-2021, 07:57 PM
defragster

@Paul/@all RE: LFSIntegrity
As noted it was a Swiss Army knife morphing as Media were added and test avenues crept in - and still are ...

For more useful User Examples - something that made no sense during DEV time when core code was common - perhaps split it into an Integrity Folder with cleaner custom clones to the media at hand::

Code:

Examples / Integrity -> TestRam -> TestPSRAM -> TestSPI [ NOR | NAND ??] -> TestQSPI [ NOR | NAND ??] -> TestSPIMRAM

That would allow removing/minimizing the Complex MultiPlexing start of the file ... i.e. #ifdef Hell. Isolating the hardware setup from the common running code in each case.

Nothing worse than trying to test Interface X - and forgetting to set one 'define' and seeing it FAIL ... and not knowing if Hardware or Software.
06-19-2021, 08:02 PM
defragster

Quote:

Originally Posted by mjs513

Glad you got it working :). Was getting me worried.

Worried ... Me too. But was end of day ... and it was easier to sleep throwing the issue out to @all 'test' ... assuming PJRC changes to .LD files or startup.c that had just shown need for change.
06-19-2021, 08:26 PM
mjs513

@degragster - @PaulStoffregen - @KurtE

Just pushed changes to a my fork of the LittleFS reposititory: https://github.com/mjs513/LittleFS . Tested SPI-NAND with the memory board as well SPI Flash on the same board. All worked. If you want to test feel free and let me know if there are any problems.

EDIT: Ok just submitted the PR.
06-19-2021, 08:57 PM
defragster

RESTART testing was never part of my process here until seeing when RAM disk could warm restart.

With that finding - all the dCache devices (QSPI FLASH) need the static_sync add of dcache_flush - if they were to restart Warm or Cold they should be SAFE and given RAM findings - they will be corrupted if dCache is not flushed.

Code:

static int static_sync(const struct lfs_config *c) { if ( c->context >= (void *)0x20200000 ) { //Serial.printf(" arm_dcache_flush_delete: ptr=0x%x, size=%d\n", c->context, c->block_count * c->block_size); arm_dcache_flush_delete(c->context, c->block_count * c->block_size ); } return 0; }

Currently they are void funcs:

Code:

static int static_sync(const struct lfs_config *c) { return 0; }

Again I think the _delete may be overkill but was at hand when I did copy/paste ???

I've got work to do for Integrity release ... Does the Multi Example seem like the best end for cleaner examples?
06-19-2021, 10:09 PM
mjs513

Quote:

Originally Posted by defragster

I've got work to do for Integrity release ... Does the Multi Example seem like the best end for cleaner examples?

Personally -yes i do - makes a good example for all the storage and a good reference.
06-19-2021, 10:52 PM
defragster

Quote:

Originally Posted by mjs513

Personally -yes i do - makes a good example for all the storage and a good reference.

Cool ... Occurred to me as way to make it more understandable and usable for test and ref. ... one FS Media at a time.

There are other examples where a CORE bus/concept is presented with sub elements where each does just one thing - that for LittleFS seemed to apply. Especially to hope to get it small enough for a user to read.

Otherwise that top part only works with 'inside' knowledge. The underlying code will still be the same odd cyclic looping - but at least the #ifdef and settings would be ready to run.

Will see if I can get focus on it after resolving the new issue I'm seeing - clean the common CORE loop behavior - then add comments and prettify the Media type intro portions above and in setup. First step might be backwards - exploding LFSIntegrity into IntegrityTestSuite.ino with each Media having ALL the top {setup() and above } replicated in sperate #ifdef XXX, so I can switch between them to test one last time ... then parse them out with no #ifdef
>> Using Paul's name idea maybe:: LittleFS / Examples / IntegritySuite / [ RAM, PSRAM, SPI, QSPI, ... ]

MRAM using SPI are not so different from SPI FLASH - except being much smaller like RAM .vs. PSRAM the size increments need tweaking to avoid over filling the disk causing OUT OF SPACE errors. I've been hitting that doing the RAM test as I didn't tweak them down.

Not sure how to fix the cryptic output ... it started doing a full DIR after each file ... that was a ton of spew on faster RAM, but a FULL dir is SLOW on FLASH media so the Verbose got set OFF.
> the initial test idea was just to do random like create, append, delete forcing all the underlying Media area to cycle through and DIR entries to some and go and get updated. Doing that for 16MB or 64MB took lots of files in lots of directories and a mile of Spew. Once it was stable that jibberish just says it is working with enough info to see where it went wrong if a failure does show up. So the fix would be another 4th level Less verbose hiding that ...
06-20-2021, 08:16 AM
defragster

Added "D" to Integrity, it walks the dir reading every file up to the returned file.size(). Reports Error if it doesn't get that many and also stops reads with error when it exceeds the reported .size()

Added .flush() before close and that doesn't help for the problem noted before and shown below:

latest repro on DMAMEM RAM disk
> 'F'ormat fresh
> '1' - one iteration
> 'R'estart Teensy
> 'Enter' to exit setup
> '1' - one iteration
> 'R'estart Teensy
> 'Enter' to exit setup
> '1' - one iteration
> 'R'estart Teensy
> 'Enter' to exit setup
> "D" shows 1 file size error

Output showing the error in the end and just before the the reported SPASM of myfs.usedSize() giving >> Bytes Used: 489984
This also repros using "B"igfile - but these small files are more trivial and shouldn't be as stressful to move and manipulate.
After a restart the '1' iteration always starts at the same point - so it finds the prior files and extends them, which is not how "B"igfiles works.
This only happens because of the restart - I assume the "lfs CORE' code is caching something internally - unless there is something unique about the code for :: myfs.usedSize()
>> This is important because this easy to do and restore repro on RAM disk would most likely be the same or WORSE on larger FLASH disks - they need the added _sync() flush of dcache - and they'll need ALL FS data updated onto the media.
- > as noted I added file.flush() after write before .close and that didn't help. Is there another unfound way to purge all lfs core elements that is missing?

Code:

// RESTART #2 printDirectory RAM_DISK -------------- FILE B_file.txt 3072 FILE C_file.txt 4096 0 dirs with 2 files of Size 7168 Bytes Total 2 files of Size 7168 Bytes Bytes Used: 8192, Bytes Total:489984 [ 0.03 M](0.02992 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 > Walk all Files verify Read Size: ..0 Errors found Loop Count: 1 (#fileCycle=0), Bytes read 0, written 0, #Files=2 dirV...91_us [ 0.08 M](0.00004 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 > [ 0.08 M](0.00004 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >1 :: /B_file.txt arm_dcache_flush_delete: ptr=0x20200000, size=489984 RAM_DISK +++ Add [sz 3072 add 3072] @KB/sec 0.00 {1349.74} ++ B Verify /B_file.txt 4608B @KB/sec 2691.59 :: /C_file.txt arm_dcache_flush_delete: ptr=0x20200000, size=489984 RAM_DISK +++ Add [sz 4096 add 4096] @KB/sec 0.00 {1579.03} ++ C Verify /C_file.txt 6144B @KB/sec 2707.80 [ 0.12 M](0.00011 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 > RESTART Teensy ... T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS\examples\LFSintegrity\LFSintegrity.ino Jun 20 2021 00:55:24 LittleFS Test : File Integrity printDirectory RAM_DISK -------------- FILE B_file.txt 4608 FILE C_file.txt 6144 0 dirs with 2 files of Size 10752 Bytes Total 2 files of Size 10752 Bytes Bytes Used: 489984, Bytes Total:489984 // RESTART #3 printDirectory RAM_DISK -------------- FILE B_file.txt 4608 FILE C_file.txt 6144 0 dirs with 2 files of Size 10752 Bytes Total 2 files of Size 10752 Bytes Bytes Used: 489984, Bytes Total:489984 [ 0.10 M](0.10200 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 > Walk all Files verify Read Size: .. File Size Error:: C_file.txt found 6145 Bytes for Size 6144 1 Errors found Loop Count: 1 (#fileCycle=0), Bytes read 0, written 0, #Files=2 ERROR COUNT =1
06-20-2021, 09:14 AM
defragster

1 Attachment(s)

Given post #890 a test was added when DIR is shown - and a DIR is shown on start in setup().

It is consistent as noted above after the 3rd 'R'estart process from a fresh format the following appears.

Too late to look harder now - but I glanced a bit in lfs core and didn't see anything to follow. I saw that a READ during WRITE does a flush internally - so I swapped a read() for flush() and no diff. It also seemed like there was a flush done on each close() - so I removed the flush() as that didn't change anything.

But in :: T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS\src\LittleFS.h I did see this:

Code:

uint64_t usedSize() { if (!mounted) return 0; int blocks = lfs_fs_size(&lfs); if (blocks < 0 || (lfs_size_t)blocks > config.block_count) return totalSize(); return blocks * config.block_size; }

Code:

Updated code is here :: Attachment 25086 T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS\examples\LFSintegrity\LFSintegrity.ino

Code:

T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS\examples\LFSintegrity\LFSintegrity.ino Jun 20 2021 01:46:07 LittleFS Test : File Integrity printDirectory RAM_DISK -------------- FILE B_file.txt 4608 FILE C_file.txt 6144 0 dirs with 2 files of Size 10752 Bytes Total 2 files of Size 10752 Bytes Bytes Used: 489984, Bytes Total:489984 WARNING: DISK FULL >>>>> Bytes Used: 489984, Bytes Total:489984

This edit to the above code prints the compared values in usedSize() { where uint32_t is == lfs_size_t }:

Code:

uint64_t usedSize() { if (!mounted) return 0; int blocks = lfs_fs_size(&lfs); if (blocks < 0 || (lfs_size_t)blocks > config.block_count) { Serial.printf(" usedSize(): (lfs_size_t)blocks=%lu, config.block_count=%lu\n", (lfs_size_t)blocks, config.block_count); return totalSize(); } return blocks * config.block_size; }

Code:

LittleFS Test : File Integrity printDirectory RAM_DISK -------------- FILE B_file.txt 4608 FILE C_file.txt 6144 0 dirs with 2 files of Size 10752 Bytes Total 2 files of Size 10752 Bytes usedSize(): (lfs_size_t)blocks=4294967212, config.block_count=1914 Bytes Used: 489984, Bytes Total:489984 usedSize(): (lfs_size_t)blocks=4294967212, config.block_count=1914 usedSize(): (lfs_size_t)blocks=4294967212, config.block_count=1914 WARNING: DISK FULL >>>>> Bytes Used: 489984, Bytes Total:489984

And it seems this 'lfs core' code is returning an ERROR: T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS\src\littlefs\lfs.c

Code:

lfs_ssize_t lfs_fs_size(lfs_t *lfs) { LFS_TRACE("lfs_fs_size(%p)", (void*)lfs); lfs_size_t size = 0; int err = lfs_fs_traverseraw(lfs, lfs_fs_size_count, &size, false); if (err) { LFS_TRACE("lfs_fs_size -> %d", err); return err; } LFS_TRACE("lfs_fs_size -> %d", err); return size; }

AFAIK the "lfs_fs_traverseraw()" is the internal version of the block used file chain WALK I used to find the USED/UNUSED disk blocks for formatUnused() - so that is what is failing ???
>>> It is probably failing for the same reason the READ of the corrupted file was doing a FOREVER "while ( file3.available() ) {" loop reading the file in bigVerify() : It is walking a file chain without an END marker saved on disk.
-> Seems that unsaved FS element is critical and needs to be flushed out of lfs.

Also - no DIFF but the arm_dcache_flush_delete() was changed to arm_dcache_flush().
Are these the expected numbers for this DMAMEM disk? Does thelfe core code store anything AFTER the DISK size area? {c->block_count * c->block_size}:: arm_dcache_flush: ptr=0x20200000, size=489984

Zzzzz...
06-20-2021, 10:21 AM
PaulStoffregen

Quote:

Originally Posted by defragster

With that finding - all the dCache devices (QSPI FLASH) need the static_sync add of dcache_flush - if they were to restart Warm or Cold they should be SAFE and given RAM findings - they will be corrupted if dCache is not flushed.

I don't understand why this would be needed. Those other devices don't leave data stored in RAM.
06-20-2021, 10:47 AM
PaulStoffregen

1 Attachment(s)

To explain a bit further, here is the FlexSPI block diagram from page 1623 in the reference manual.

Attachment 25087

The important concept is 2 paths exist to gain access to the QSPI memory chips. Ordinary memory access (cache misses) access via the 64 bit AHB path. Software can also "manually" access the memory using the IPS 32 bit path.

As far as I know, all the QPI NOR & NAND flash code is exclusively using the IPS path.

LittleFS_Program is the only device which uses both. Reads are done using memcpy(), which of course uses the 64 bit AHB path. Writes are done on the 32 bit IPS path. The write code already does use arm_dcache_delete(), so after data is written future reads will not fetch stale data from the CPU cache. There should never need to be a case of flushing the cache, because writing can never happen over the AHB path.

But it's entirely possible I've missed some important case. If we really do need to add "sync" callbacks on the non-RAM storage devices, please explain why. I just don't see it, but I'm willing to listen to any explanation or try running any test case.
06-20-2021, 01:03 PM
KurtE

Good Morning all: Sorry been distracted. Distractions still ongoing :D

@defragster - One thing with LFSIntegrety sketch that I always wished was some of the sections like TEST_RAM
Actually had some configurations depending on which board you are building for.

Example only try to use EXTMEM if you are running on Teensy41.
#ifdef ARDUINO_TEENSY41

Also with Teensy41. Maybe put explicit option to enable EXTMEM in test...
Might also want some of the debug code I put in, like I did in the debug version I posted, that sees your sketch is configured to use 8mb of PSRAM, but you T4.1 does not have any...
06-20-2021, 05:08 PM
defragster

Quote:

Originally Posted by PaulStoffregen

I don't understand why this would be needed. Those other devices don't leave data stored in RAM.

Makes sense, Thanks.

Opps, Hadn't thought the process through: QSPI Flash writes can not be queued in cache and handled later (as DMAMEM and PSRAM can do), because in the case of the QSPI Flash the 'helper code' does immediate write to Flash.

Though - if the LFS Core is not flushing everything - the problem the RAM drive is seeing could show up on FLASH as well with 'unplanned restart'. Not sure if that was directly tested as the RAM case is doing.
06-20-2021, 05:24 PM
defragster

Quote:

Originally Posted by KurtE

Good Morning all: Sorry been distracted. Distractions still ongoing :D

@defragster - One thing with LFSIntegrety sketch that I always wished was some of the sections like TEST_RAM
Actually had some configurations depending on which board you are building for.

Example only try to use EXTMEM if you are running on Teensy41.
#ifdef ARDUINO_TEENSY41

Also with Teensy41. Maybe put explicit option to enable EXTMEM in test...
Might also want some of the debug code I put in, like I did in the debug version I posted, that sees your sketch is configured to use 8mb of PSRAM, but you T4.1 does not have any...

Notes above showed PSRAM separate from RAM1 or RAM2 Example usage, and indeed for general consumption the RAM1 / RAM2 variant would need a clear way to differentiate.
Because, while 'we' know PSRAM uses the same code, it is a case that shouldn't be confusing to use, or users might get false failures after soldering.

Not sure I'm following in the RAM section currently there are TWO (cryptic) lines - one line can be changed (as commented) or a third line added:

Code:

#ifdef TEST_RAM LittleFS_RAM myfs; // RUNTIME :: extern "C" uint8_t external_psram_size; //EXTMEM char buf[MYPSRAM * 1024 * 1024]; // USE DMAMEM for more memory than ITCM allows - or remove DMAMEM char buf[490000]; // USE DMAMEM for more memory than ITCM allows - or remove char szDiskMem[] = "RAM_DISK"; #elif defined(TEST_SPI)

When testing RAM1 changing of adding an alteration of the DMAMEM line like - with change for available RAM:

Code:

//DMAMEM char buf[490000]; // USE DMAMEM for more memory than ITCM allows - or remove char buf[390000]; // USE DMAMEM for more memory than ITCM allows - or remove
06-20-2021, 06:09 PM
defragster

RE post #891 and corruption on DMAMEM RAM DRIVE.

Just move to PSRAM and the same behavior does NOT repeat?

Did similar allocations and 'R'estart cycles and all is well?

Just enabled 12 directories and doing larger runs All seems proper with PSRAM usage, not yet triggered the behavior seen with DMAMEM?

This bodes well for FLASH - but DMAMEM a puzzle why the Warm Restart is getting corrupted?

Post #891 CODE: In adding dirs I found Copy/Paste Edit error in this function about line 603 - it wasn't recursing on itself - but the donor code :(

Code:

int DirectoryVerify(File dir) { int errCnt = 0; while (true) { File entry = dir.openNextFile(); if (! entry) { break; } if (entry.isDirectory()) { Serial.print("\tD"); errCnt += DirectoryVerify(entry);
06-20-2021, 07:12 PM
KurtE

Quote:

Originally Posted by defragster

Notes above showed PSRAM separate from RAM1 or RAM2 Example usage, and indeed for general consumption the RAM1 / RAM2 variant would need a clear way to differentiate.
Because, while 'we' know PSRAM uses the same code, it is a case that shouldn't be confusing to use, or users might get false failures after soldering.

Not sure I'm following in the RAM section currently there are TWO (cryptic) lines - one line can be changed (as commented) or a third line added:

Code:

#ifdef TEST_RAM LittleFS_RAM myfs; // RUNTIME :: extern "C" uint8_t external_psram_size; //EXTMEM char buf[MYPSRAM * 1024 * 1024]; // USE DMAMEM for more memory than ITCM allows - or remove DMAMEM char buf[490000]; // USE DMAMEM for more memory than ITCM allows - or remove char szDiskMem[] = "RAM_DISK"; #elif defined(TEST_SPI)

When testing RAM1 changing of adding an alteration of the DMAMEM line like - with change for available RAM:

Code:

//DMAMEM char buf[490000]; // USE DMAMEM for more memory than ITCM allows - or remove char buf[390000]; // USE DMAMEM for more memory than ITCM allows - or remove

Thanks Defragster... Was just sort of wondering for the main RAM test and maybe others if it might make sense to have, it some how preconfigured for different boards...
That is if you are going back and forth testing on T4.1 with, T4.1 withouth, Micromod, T4, T3.6, T3.5, T3.2 would it make sense to have default setup for each board so quick compile for that board will give you sort of the default testing...

Probably not a bit deal, just a thought
06-20-2021, 07:55 PM
defragster

1 Attachment(s)

PSRAM seems safe for use with Software Restart - between operations all is properly stored.

Did 20 or 30 or more Restarts with "same" operations as RAM2 showed corruption with and it worked perfectly each time.

Where "same" was then with more folders and multiple "B" or "S" files to make sure media space was filled and reused. And each restart was a'5' iteration set not '1' to create and invoke more file changes across the dirs.

In case anyone wants to see here is the current sketch :

Code:

Attachment 25089

> going to do some quick repeats of process on a 1G QSPI NAND and get to another non-Teensy task I've been doing background wise but needs to be done ...

Not sure how DMAMEM is 'cracking' - if the _FLUSH is right - but it is showing that anomaly regularly.

Back to PSRAM - FUNNY I bumped from 8MB to two CHIP 16MB between uploads and the Disk Survived and Extended! And ran repeating fillings from there ???
> Not sure the mount code could or should look for that? Ideally it should not work ... going to go from 16 to 8 and see what shows.
>> Downsizing 16 to 8 MB did not Mount old drive but presented a fresh formatted drive
- > back from 8 to 16 MB and it picked up running again!

Updated the "D" file walk length verify - put out fir DIR or FILE letter:

Code:

Walk all Files verify Read Size: 00 D1 FGNOPQRSTUVW D1 HIPQRSTUVWXY D1 1 D1 ABCDEFGHIJKLMNOPQRSTUVWXYZ D2 ABCDEFGHIJKLMNOPQRSTUVWXYZ D3 ABCDEFGHIJKLMNOPQRSTUVWXYZ D4 BCDEFGHIJK D5 DEFGHIJKLM D6 FGHIJKLMNO D7 ABCDEFGHIJKLMNOPQ D8 BCJKLMNOPQRS D9 DELMNOPQRSTU ABCDEFGHIJKLMNOPQRSTUVWXYZ 0 Errors found
06-20-2021, 08:57 PM
defragster

Quick update QSPI NAND passes restart Integrity tests. Good like PSRAM, and no signs of corruption like on the DMAMEM Ramdisk.

Slower than (PS)RAM and larger media size a double whammy for test time. But did a few cycles and no issues.

Code:

12 dirs with 32 files of Size 88495360 Bytes Total 220 files of Size 91009536 Bytes Bytes Used: 119930880, Bytes Total:131596288 [ 0.31 M](0.31412 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 > [ 0.33 M](0.32963 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 > Walk all Files verify Read Size: 00 D1 LMNOPQRSTUVW D1 NOPQRSTUVWXY D1 11 D1 ABCDEFGHIJKLMNOPQRSTUVWXYZ 22 D2 ABCDEFGHIJKLMNOPQRSTUVWXYZ D3 ABCDEFGHIJKLMNOPQRSTUVWXYZ D4 ABCDEFGHIJKLMNOPQRSTUVWXYZ D5 BCDEFGHIJKLM D6 DEFGHIJKLMNO D7 FGHIJKLMNOPQ D8 HIJKLMNOPQRS D9 JKLMNOPQRSTU ABCDEFGHIJKLMNOPQRSTUVWXYZ 0 Errors found

Show 40 post(s) from this thread on one page