Forum Rule: Always post complete source code & details to reproduce any issue!
Page 36 of 38 FirstFirst ... 26 34 35 36 37 38 LastLast
Results 876 to 900 of 940

Thread: LittleFS port to Teensy/SPIFlash

  1. #876
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,219
    I tried running it as well on T4.1 with 2 memory chips...

    Note: I have hacked the code:

    To add in some memory dump and maybe check stack and heap usages...
    Which I included. Note: older style memory output from the uncanney eyes large version...
    Code:
    C:\Users\kurte\Documents\Arduino\Teensy Tests\LFSIntegrity_debug\LFSIntegrity_debug.ino Jun 19 2021 07:15:58
    LittleFS Test : File Integrity
    IOMUXC_GPR_GPR17:aaaaaabf IOMUXC_GPR_GPR16:200007 IOMUXC_GPR_GPR14:aa0000
    Initial Stack pointer: 20068000
    ITCM allocated: 98304  DTCM allocated: 425984
    ITCM init range: 0 - 11518 Count: 70936
    DTCM init range: 20000000 - 20002580 Count: 9600
    DTCM cleared range: 20002580 - 200052c0 Count: 11584
    Now fill rest of DTCM with known pattern(200052c4 - 20067f8c
    Ram2: variables:12384
    EXTRAM: variables: 8388608, size in MB: 16
    
    printDirectory RAM_DISK
    --------------
    
     0 dirs with 0 files of Size 0 Bytes
     Total 0 files of Size 0 Bytes
    Bytes Used: 4096, Bytes Total:8388608
     0, 1-9 '#' passes continue loop before Pause
     '$' Print memory info T4.x only
     'a' Auto formatUnused() during iterations - TOGGLE
     'R' Restart Teensy
     'd' Directory of LittleFS
     'w' WIPE Directory of LittleFS
     'b' big file delete
     'B' BIG FILE MAKE
     'S' FILE 2MB MAKE
     's' FILE 2MB delete
     'c' Continuous Loop
     'g' run speedBench()
     'h' Hundred loops
     'k' Thousand loops
     'F' LittleFS_ Low Level Format Disk 
     'f' LittleFS::formatUnused( ALL ) : DATA PRESERVED 
     'q' LittleFS_ Quick Format Disk 
     'v' Verbose All Dir Prints - TOGGLE
     'p' Pause after all Dir prints - TOGGLE
     'l' Show count of loop()'s, Bytes Read,Written
     'm' Make ROOT dirs (needed after q/F format !ROOTONLY)
     'u' Update Filecount
     'x' Directory filecount verify - TOGGLE
     'n' No verify on Write- TOGGLE
     '+' more add to delete cycles
     '-' fewer add to delete cycles
     'y' reclaim 1 block :: myfs.formatUnused( 1 )
     'Y' reclaim 15 blocks :: myfs.formatUnused( 15 )
     '?' Help list
    ?[  0.02 M](0.02042 M elap) Awaiting input 0123456789RdwcghkFqvplmusSBbyYxfan$+-? loops left 0 >  Estimate Stack and Heap usage
    
    Estimated max stack usage: 1308
    Estimated Heap usage: 4000
    $
    The Extram stuff will only show up on T4.1..
    And if size used > you have it will show another line like: ***** External memory used larger than you have *****

    Does not build on T4 as is.. Exceeds memory...
    Suggest maybe ifdefs for which Teensy and for example only do the EXTMEM for T4.1...
    Attached Files Attached Files

  2. #877
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,492
    Okay - I need to clean something.

    Something not right in that it worked okay at first - until restart - then it started going wholly bad

    On the RAMDISK - does the restart noted here from the Beta 10 thread work? :: start RAM :: 'B'igfile OR '1' loop :: 'R'estart >> does it restart to function and repeat?

  3. #878
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,492
    Clean Install and all is well. Did 15 sec Restore of both T_4.0 and 4.1

    Win 10 : IDE 1.8.15 unzip : TD 1.54 B 10 install

    Using IDE and TeensyLoader and IDE SerMon.

    Various : 'B' and '1'/'#' and 'R' to make files and then software restart and no problems on either 4.0 or 4.1 with DMAMEM LittleFS_RAM

    TeensyLoader No-AUTO: Button, Reboot :: Did show error on 'B'igfile - but FS was safe and good after that file was deleted.
    Code:
    Delete with read verify all #bigfile's
    	Verify /0_bigfile.txt bytes 202752 : <Bad Byte!  0! = [0x0] @1
    
    	Read Count fail! :: read 1 != f.size 202752
    
    	Big read&compare KBytes per second  0.00
    Something must have been corrupt. From the initial odd behavior see on Beta 10 install that was ODD then it got worse? Not idea what that may have been - but GONE now.

    ref:
    Code:
    Memory Usage on Teensy 4.1:
      FLASH: code:76216, data:9720, headers:8268   free for files:8032260
       RAM1: variables:21184, code:70488, padding:27816   free for local variables:404800
       RAM2: variables:502400  free for malloc/new:21888
    Using library LittleFS at version 1.0.0 in folder: T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS 
    Using library SPI at version 1.0 in folder: T:\arduino-1.8.15\hardware\teensy\avr\libraries\SPI 
    Using library SdFat at version 2.0.5-beta.1 in folder: T:\arduino-1.8.15\hardware\teensy\avr\libraries\SdFat

  4. #879
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    24,497
    Can anyone remind me why we ended up with 2 header files? Unless there is a strong need for a separate header, I'd really like to see a single LittleFS.h before we release 1.54.

    Separate .cpp files is not an issue.

    @Defragster - LFSintegrity.ino needs comments added at the top which explain its purpose and give a clear description to first-time users of how to run it and what to expect. When anyone opens the example, the first ~20 lines of the program are their first impression. A good first impression, which helps them understand the program's purpose is essential. Especially important is for people to quickly understand whether the example is *not* the thing they need for whatever goals they may have.

    Perhaps the name should be changed, at least dropping "LFS". Maybe IntegrityTestSuite.ino? The name should be chosen so that first-time users who see it in the File > Examples menu understand it's for testing the library, rather than an example they're meant to use for code to copy into their own application.

    Ideally, I'd really like to see more human readable output. For example, imagine someone with no experience using the library chooses LFSintegrity as their first attempt to do something. What is a new user supposed to think of this sort of result?

    Code:
    :: /B_file.txt  RAM_DISK +++ Add [sz 1536 add 1536] @KB/sec 327.02 {81.21}  ++ B   Verify /B_file.txt 3072B  @KB/sec 2357.64 
    :: /C_file.txt  RAM_DISK +++ Add [sz 2048 add 2048] @KB/sec 487.04 {110.97}  ++ C   Verify /C_file.txt 4096B  @KB/sec 2386.95 
    [  1.12 M](0.00067 M elap) Awaiting input 0123456789RdwcghkFqvplmusSBbyYxfan+-? loops left 0 >
    LFSintegrity.ino has a lot of very good work in it, but for a non-beta release, it needs at least some more work so it doesn't create a bad first-use experience for new users.

  5. #880
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,492
    Trying to debug this and recalling that SIZE's are returned as 64 bit.
    Learned this before - and seeing some print anomalies trying to debug ... doing cleanup ...

    These common methods:
    Code:
    	virtual uint64_t size() {
    	uint64_t usedSize() {
    	uint64_t totalSize() {
    But see this for file.size() returning uint64_t :: internally using typedef int32_t lfs_soff_t;

    Code:
    	virtual uint64_t size() {
    		if (!file) return 0;
    		lfs_soff_t size = lfs_file_size(lfs, file);
    		if (size < 0) size = 0;
    		return size;
    	}
    That uses ( internally consistent checking above NEG on SIGNED int32_t ) : lfs_soff_t lfs_file_size(lfs_t *lfs, lfs_file_t *file) {

    But can only handle 31 bit numbers and returning 64 bits?

    Only looking because LFSIntegrity has %llu to print .size - but forgot above lesson and adding early exit on bad file size used a uint32_t and seeing size of this woke me up ... : 2305891319005839368

    Also saw this where disk is FULL with 20KB of files using 489KB of disk space doing RAM Restarts - :
    Code:
    printDirectory RAM_DISK
    --------------
    FILE	B_file.txt		3072
    FILE	C_file.txt		4096
    FILE	D_file.txt		2560
    FILE	E_file.txt		3072
    FILE	F_file.txt		3584
    FILE	G_file.txt		4096
    
     0 dirs with 6 files of Size 20480 Bytes
     Total 6 files of Size 20480 Bytes
    Bytes Used: 489984, Bytes Total:489984

  6. #881
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,492
    Quote Originally Posted by PaulStoffregen View Post
    Can anyone remind me why we ended up with 2 header files? Unless there is a strong need for a separate header, I'd really like to see a single LittleFS.h before we release 1.54.

    Separate .cpp files is not an issue.

    @Defragster - LFSintegrity.ino needs comments added at the top which explain its purpose and give a clear description to first-time users of how to run it and what to expect. When anyone opens the example, the first ~20 lines of the program are their first impression. A good first impression, which helps them understand the program's purpose is essential. Especially important is for people to quickly understand whether the example is *not* the thing they need for whatever goals they may have.

    Perhaps the name should be changed, at least dropping "LFS". Maybe IntegrityTestSuite.ino? The name should be chosen so that first-time users who see it in the File > Examples menu understand it's for testing the library, rather than an example they're meant to use for code to copy into their own application.

    Ideally, I'd really like to see more human readable output. For example, imagine someone with no experience using the library chooses LFSintegrity as their first attempt to do something. What is a new user supposed to think of this sort of result?

    Code:
    :: /B_file.txt  RAM_DISK +++ Add [sz 1536 add 1536] @KB/sec 327.02 {81.21}  ++ B   Verify /B_file.txt 3072B  @KB/sec 2357.64 
    :: /C_file.txt  RAM_DISK +++ Add [sz 2048 add 2048] @KB/sec 487.04 {110.97}  ++ C   Verify /C_file.txt 4096B  @KB/sec 2386.95 
    [  1.12 M](0.00067 M elap) Awaiting input 0123456789RdwcghkFqvplmusSBbyYxfan+-? loops left 0 >
    LFSintegrity.ino has a lot of very good work in it, but for a non-beta release, it needs at least some more work so it doesn't create a bad first-use experience for new users.
    RE: 2nd Header :: IIRC @mjs513 was pushing the envelope for NAND and wasn't sure it would get done for TD 1.54 - so kept was separate.
    > at this point it managed to mature and work, so suspect it could be merged into single header for release.

    RE: LFSIntegrity : Good and valid points, the code and layout confuses me too.
    I thought LFSIntegrity was a lot to type the last 4,000 times, so "IntegrityTestSuite.ino" is much better ... but at least that could be said as ITS.ino ... Integrity also okay
    > as noted putting in some more checks - will try to leave it cleaner and add comments

    Glad I made it as it gives me confidence in the function from a DEV level - but it has grown to be a BEAST and it is cryptic to be sure with reduced output to focus on proper function ( knowing what it should say ) and popping out errors.
    Goal was to exemplify interfaces in source code and see them used - as run it is a 'busy work' pointless program that proves function internally and spits out the cryptic sausage making process in case something goes wrong.
    ... with lots of feature creep and no real plan/expectation it would end up where it is.

  7. #882
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    7,225
    Quote Originally Posted by PaulStoffregen View Post
    Can anyone remind me why we ended up with 2 header files? Unless there is a strong need for a separate header, I'd really like to see a single LittleFS.h before we release 1.54.

    Separate .cpp files is not an issue.

    ......
    LFSintegrity.ino has a lot of very good work in it, but for a non-beta release, it needs at least some more work so it doesn't create a bad first-use experience for new users.
    Think I was keeping it separate for devel and not corrupt the LittleFS header. To be honest probably should have merged the too. Would keep the .cpp's separate though as you said.

    Ok - on my to do list. whats your target for 1.55? Was debugging an issue with my RA8875's

    EDIT: Testing a merge right now.
    Last edited by mjs513; 06-19-2021 at 07:48 PM.

  8. #883
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    7,225
    Quote Originally Posted by defragster
    Something must have been corrupt. From the initial odd behavior see on Beta 10 install that was ODD then it got worse? Not idea what that may have been - but GONE now.
    Glad you got it working . Was getting me worried.

  9. #884
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,492
    @Paul/@all RE: LFSIntegrity
    As noted it was a Swiss Army knife morphing as Media were added and test avenues crept in - and still are ...

    For more useful User Examples - something that made no sense during DEV time when core code was common - perhaps split it into an Integrity Folder with cleaner custom clones to the media at hand::
    Code:
    Examples / Integrity
      -> TestRam
      -> TestPSRAM
      -> TestSPI [ NOR | NAND ??]
      -> TestQSPI [ NOR | NAND ??]
      -> TestSPIMRAM
    That would allow removing/minimizing the Complex MultiPlexing start of the file ... i.e. #ifdef Hell. Isolating the hardware setup from the common running code in each case.

    Nothing worse than trying to test Interface X - and forgetting to set one 'define' and seeing it FAIL ... and not knowing if Hardware or Software.

  10. #885
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,492
    Quote Originally Posted by mjs513 View Post
    Glad you got it working . Was getting me worried.
    Worried ... Me too. But was end of day ... and it was easier to sleep throwing the issue out to @all 'test' ... assuming PJRC changes to .LD files or startup.c that had just shown need for change.

  11. #886
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    7,225
    @degragster - @PaulStoffregen - @KurtE

    Just pushed changes to a my fork of the LittleFS reposititory: https://github.com/mjs513/LittleFS . Tested SPI-NAND with the memory board as well SPI Flash on the same board. All worked. If you want to test feel free and let me know if there are any problems.

    EDIT: Ok just submitted the PR.

  12. #887
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,492
    RESTART testing was never part of my process here until seeing when RAM disk could warm restart.

    With that finding - all the dCache devices (QSPI FLASH) need the static_sync add of dcache_flush - if they were to restart Warm or Cold they should be SAFE and given RAM findings - they will be corrupted if dCache is not flushed.
    Code:
    	static int static_sync(const struct lfs_config *c) {
    		if ( c->context >= (void *)0x20200000 ) {
    			//Serial.printf("    arm_dcache_flush_delete: ptr=0x%x, size=%d\n", c->context, c->block_count * c->block_size);
    			arm_dcache_flush_delete(c->context, c->block_count * c->block_size );
    		}
    		return 0;
    	}
    Currently they are void funcs:
    Code:
    	static int static_sync(const struct lfs_config *c) {
    		return 0;
    	}

    Again I think the _delete may be overkill but was at hand when I did copy/paste ???

    I've got work to do for Integrity release ... Does the Multi Example seem like the best end for cleaner examples?

  13. #888
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    7,225
    Quote Originally Posted by defragster
    I've got work to do for Integrity release ... Does the Multi Example seem like the best end for cleaner examples?
    Personally -yes i do - makes a good example for all the storage and a good reference.

  14. #889
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,492
    Quote Originally Posted by mjs513 View Post
    Personally -yes i do - makes a good example for all the storage and a good reference.
    Cool ... Occurred to me as way to make it more understandable and usable for test and ref. ... one FS Media at a time.

    There are other examples where a CORE bus/concept is presented with sub elements where each does just one thing - that for LittleFS seemed to apply. Especially to hope to get it small enough for a user to read.

    Otherwise that top part only works with 'inside' knowledge. The underlying code will still be the same odd cyclic looping - but at least the #ifdef and settings would be ready to run.

    Will see if I can get focus on it after resolving the new issue I'm seeing - clean the common CORE loop behavior - then add comments and prettify the Media type intro portions above and in setup. First step might be backwards - exploding LFSIntegrity into IntegrityTestSuite.ino with each Media having ALL the top {setup() and above } replicated in sperate #ifdef XXX, so I can switch between them to test one last time ... then parse them out with no #ifdef
    >> Using Paul's name idea maybe:: LittleFS / Examples / IntegritySuite / [ RAM, PSRAM, SPI, QSPI, ... ]

    MRAM using SPI are not so different from SPI FLASH - except being much smaller like RAM .vs. PSRAM the size increments need tweaking to avoid over filling the disk causing OUT OF SPACE errors. I've been hitting that doing the RAM test as I didn't tweak them down.

    Not sure how to fix the cryptic output ... it started doing a full DIR after each file ... that was a ton of spew on faster RAM, but a FULL dir is SLOW on FLASH media so the Verbose got set OFF.
    > the initial test idea was just to do random like create, append, delete forcing all the underlying Media area to cycle through and DIR entries to some and go and get updated. Doing that for 16MB or 64MB took lots of files in lots of directories and a mile of Spew. Once it was stable that jibberish just says it is working with enough info to see where it went wrong if a failure does show up. So the fix would be another 4th level Less verbose hiding that ...

  15. #890
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,492
    Added "D" to Integrity, it walks the dir reading every file up to the returned file.size(). Reports Error if it doesn't get that many and also stops reads with error when it exceeds the reported .size()

    Added .flush() before close and that doesn't help for the problem noted before and shown below:

    latest repro on DMAMEM RAM disk
    > 'F'ormat fresh
    > '1' - one iteration
    > 'R'estart Teensy
    > 'Enter' to exit setup
    > '1' - one iteration
    > 'R'estart Teensy
    > 'Enter' to exit setup
    > '1' - one iteration
    > 'R'estart Teensy
    > 'Enter' to exit setup
    > "D" shows 1 file size error

    Output showing the error in the end and just before the the reported SPASM of myfs.usedSize() giving >> Bytes Used: 489984
    This also repros using "B"igfile - but these small files are more trivial and shouldn't be as stressful to move and manipulate.
    After a restart the '1' iteration always starts at the same point - so it finds the prior files and extends them, which is not how "B"igfiles works.
    This only happens because of the restart - I assume the "lfs CORE' code is caching something internally - unless there is something unique about the code for :: myfs.usedSize()
    >> This is important because this easy to do and restore repro on RAM disk would most likely be the same or WORSE on larger FLASH disks - they need the added _sync() flush of dcache - and they'll need ALL FS data updated onto the media.
    - > as noted I added file.flush() after write before .close and that didn't help. Is there another unfound way to purge all lfs core elements that is missing?
    Code:
    // RESTART #2
    
    printDirectory RAM_DISK
    --------------
    FILE	B_file.txt		3072
    FILE	C_file.txt		4096
    
     0 dirs with 2 files of Size 7168 Bytes
     Total 2 files of Size 7168 Bytes
    Bytes Used: 8192, Bytes Total:489984
    [  0.03 M](0.02992 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >
    Walk all Files verify Read Size:	
    ..0 Errors found
    
    	 Loop Count: 1 (#fileCycle=0), Bytes read 0, written 0, #Files=2
    	dirV...91_us	[  0.08 M](0.00004 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >
    
    [  0.08 M](0.00004 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >1
    :: /B_file.txt     arm_dcache_flush_delete: ptr=0x20200000, size=489984
     RAM_DISK +++ Add [sz 3072 add 3072] @KB/sec  0.00 {1349.74}  ++ B   Verify /B_file.txt 4608B  @KB/sec 2691.59 
    :: /C_file.txt     arm_dcache_flush_delete: ptr=0x20200000, size=489984
     RAM_DISK +++ Add [sz 4096 add 4096] @KB/sec  0.00 {1579.03}  ++ C   Verify /C_file.txt 6144B  @KB/sec 2707.80 
    [  0.12 M](0.00011 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 > RESTART Teensy ...
    T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS\examples\LFSintegrity\LFSintegrity.ino Jun 20 2021 00:55:24
    LittleFS Test : File Integrity
    printDirectory RAM_DISK
    --------------
    FILE	B_file.txt		4608
    FILE	C_file.txt		6144
    
     0 dirs with 2 files of Size 10752 Bytes
     Total 2 files of Size 10752 Bytes
    Bytes Used: 489984, Bytes Total:489984
    
    // RESTART #3
    
    printDirectory RAM_DISK
    --------------
    FILE	B_file.txt		4608
    FILE	C_file.txt		6144
    
     0 dirs with 2 files of Size 10752 Bytes
     Total 2 files of Size 10752 Bytes
    Bytes Used: 489984, Bytes Total:489984
    [  0.10 M](0.10200 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >
    Walk all Files verify Read Size:	
    ..
     File Size Error:: C_file.txt found 6145 Bytes for Size 6144 
    1 Errors found
    
    	 Loop Count: 1 (#fileCycle=0), Bytes read 0, written 0, #Files=2
    	 ERROR COUNT =1

  16. #891
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,492
    Given post #890 a test was added when DIR is shown - and a DIR is shown on start in setup().

    It is consistent as noted above after the 3rd 'R'estart process from a fresh format the following appears.

    Too late to look harder now - but I glanced a bit in lfs core and didn't see anything to follow. I saw that a READ during WRITE does a flush internally - so I swapped a read() for flush() and no diff. It also seemed like there was a flush done on each close() - so I removed the flush() as that didn't change anything.

    But in :: T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS\src\LittleFS.h I did see this:
    Code:
    	uint64_t usedSize() {
    		if (!mounted) return 0;
    		int blocks = lfs_fs_size(&lfs);
    		if (blocks < 0 || (lfs_size_t)blocks > config.block_count) return totalSize();
    		return blocks * config.block_size;
    	}
    Code:
    Updated code is here ::   LFSintegrity.ino
    
    T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS\examples\LFSintegrity\LFSintegrity.ino
    Code:
    T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS\examples\LFSintegrity\LFSintegrity.ino Jun 20 2021 01:46:07
    LittleFS Test : File Integrity
    printDirectory RAM_DISK
    --------------
    FILE	B_file.txt		4608
    FILE	C_file.txt		6144
    
     0 dirs with 2 files of Size 10752 Bytes
     Total 2 files of Size 10752 Bytes
    Bytes Used: 489984, Bytes Total:489984
    
    
    	WARNING: DISK FULL >>>>>  Bytes Used: 489984, Bytes Total:489984
    This edit to the above code prints the compared values in usedSize() { where uint32_t is == lfs_size_t }:
    Code:
    	uint64_t usedSize() {
    		if (!mounted) return 0;
    		int blocks = lfs_fs_size(&lfs);
    		if (blocks < 0 || (lfs_size_t)blocks > config.block_count) {
    			Serial.printf("    usedSize(): (lfs_size_t)blocks=%lu, config.block_count=%lu\n", (lfs_size_t)blocks, config.block_count);
    			return totalSize();
    		}
    		return blocks * config.block_size;
    	}

    Code:
    LittleFS Test : File Integrity
    printDirectory RAM_DISK
    --------------
    FILE	B_file.txt		4608
    FILE	C_file.txt		6144
    
     0 dirs with 2 files of Size 10752 Bytes
     Total 2 files of Size 10752 Bytes
        usedSize(): (lfs_size_t)blocks=4294967212, config.block_count=1914
    Bytes Used: 489984, Bytes Total:489984
        usedSize(): (lfs_size_t)blocks=4294967212, config.block_count=1914
        usedSize(): (lfs_size_t)blocks=4294967212, config.block_count=1914
    
    
    	WARNING: DISK FULL >>>>>  Bytes Used: 489984, Bytes Total:489984
    And it seems this 'lfs core' code is returning an ERROR: T:\arduino-1.8.15\hardware\teensy\avr\libraries\LittleFS\src\littlefs\lfs.c
    Code:
    lfs_ssize_t lfs_fs_size(lfs_t *lfs) {
        LFS_TRACE("lfs_fs_size(%p)", (void*)lfs);
        lfs_size_t size = 0;
        int err = lfs_fs_traverseraw(lfs, lfs_fs_size_count, &size, false);
        if (err) {
            LFS_TRACE("lfs_fs_size -> %d", err);
            return err;
        }
    
        LFS_TRACE("lfs_fs_size -> %d", err);
        return size;
    }
    AFAIK the "lfs_fs_traverseraw()" is the internal version of the block used file chain WALK I used to find the USED/UNUSED disk blocks for formatUnused() - so that is what is failing ???
    >>> It is probably failing for the same reason the READ of the corrupted file was doing a FOREVER "while ( file3.available() ) {" loop reading the file in bigVerify() : It is walking a file chain without an END marker saved on disk.
    -> Seems that unsaved FS element is critical and needs to be flushed out of lfs.

    Also - no DIFF but the arm_dcache_flush_delete() was changed to arm_dcache_flush().
    Are these the expected numbers for this DMAMEM disk? Does thelfe core code store anything AFTER the DISK size area? {c->block_count * c->block_size}:: arm_dcache_flush: ptr=0x20200000, size=489984

    Zzzzz...
    Last edited by defragster; 06-20-2021 at 09:27 AM.

  17. #892
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    24,497
    Quote Originally Posted by defragster View Post
    With that finding - all the dCache devices (QSPI FLASH) need the static_sync add of dcache_flush - if they were to restart Warm or Cold they should be SAFE and given RAM findings - they will be corrupted if dCache is not flushed.
    I don't understand why this would be needed. Those other devices don't leave data stored in RAM.

  18. #893
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    24,497
    To explain a bit further, here is the FlexSPI block diagram from page 1623 in the reference manual.

    Click image for larger version. 

Name:	screenshot.png 
Views:	12 
Size:	72.3 KB 
ID:	25087

    The important concept is 2 paths exist to gain access to the QSPI memory chips. Ordinary memory access (cache misses) access via the 64 bit AHB path. Software can also "manually" access the memory using the IPS 32 bit path.

    As far as I know, all the QPI NOR & NAND flash code is exclusively using the IPS path.

    LittleFS_Program is the only device which uses both. Reads are done using memcpy(), which of course uses the 64 bit AHB path. Writes are done on the 32 bit IPS path. The write code already does use arm_dcache_delete(), so after data is written future reads will not fetch stale data from the CPU cache. There should never need to be a case of flushing the cache, because writing can never happen over the AHB path.

    But it's entirely possible I've missed some important case. If we really do need to add "sync" callbacks on the non-RAM storage devices, please explain why. I just don't see it, but I'm willing to listen to any explanation or try running any test case.

  19. #894
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,219
    Good Morning all: Sorry been distracted. Distractions still ongoing

    @defragster - One thing with LFSIntegrety sketch that I always wished was some of the sections like TEST_RAM
    Actually had some configurations depending on which board you are building for.

    Example only try to use EXTMEM if you are running on Teensy41.
    #ifdef ARDUINO_TEENSY41


    Also with Teensy41. Maybe put explicit option to enable EXTMEM in test...
    Might also want some of the debug code I put in, like I did in the debug version I posted, that sees your sketch is configured to use 8mb of PSRAM, but you T4.1 does not have any...

  20. #895
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,492
    Quote Originally Posted by PaulStoffregen View Post
    I don't understand why this would be needed. Those other devices don't leave data stored in RAM.
    Makes sense, Thanks.

    Opps, Hadn't thought the process through: QSPI Flash writes can not be queued in cache and handled later (as DMAMEM and PSRAM can do), because in the case of the QSPI Flash the 'helper code' does immediate write to Flash.

    Though - if the LFS Core is not flushing everything - the problem the RAM drive is seeing could show up on FLASH as well with 'unplanned restart'. Not sure if that was directly tested as the RAM case is doing.

  21. #896
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,492
    Quote Originally Posted by KurtE View Post
    Good Morning all: Sorry been distracted. Distractions still ongoing

    @defragster - One thing with LFSIntegrety sketch that I always wished was some of the sections like TEST_RAM
    Actually had some configurations depending on which board you are building for.

    Example only try to use EXTMEM if you are running on Teensy41.
    #ifdef ARDUINO_TEENSY41


    Also with Teensy41. Maybe put explicit option to enable EXTMEM in test...
    Might also want some of the debug code I put in, like I did in the debug version I posted, that sees your sketch is configured to use 8mb of PSRAM, but you T4.1 does not have any...
    Notes above showed PSRAM separate from RAM1 or RAM2 Example usage, and indeed for general consumption the RAM1 / RAM2 variant would need a clear way to differentiate.
    Because, while 'we' know PSRAM uses the same code, it is a case that shouldn't be confusing to use, or users might get false failures after soldering.

    Not sure I'm following in the RAM section currently there are TWO (cryptic) lines - one line can be changed (as commented) or a third line added:
    Code:
    #ifdef TEST_RAM
    LittleFS_RAM myfs;
    // RUNTIME :: extern "C" uint8_t external_psram_size;
    //EXTMEM char buf[MYPSRAM * 1024 * 1024];	// USE DMAMEM for more memory than ITCM allows - or remove
    DMAMEM char buf[490000];	// USE DMAMEM for more memory than ITCM allows - or remove
    char szDiskMem[] = "RAM_DISK";
    #elif defined(TEST_SPI)
    When testing RAM1 changing of adding an alteration of the DMAMEM line like - with change for available RAM:
    Code:
    //DMAMEM char buf[490000];	// USE DMAMEM for more memory than ITCM allows - or remove
    char buf[390000];	// USE DMAMEM for more memory than ITCM allows - or remove

  22. #897
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,492
    RE post #891 and corruption on DMAMEM RAM DRIVE.

    Just move to PSRAM and the same behavior does NOT repeat?

    Did similar allocations and 'R'estart cycles and all is well?

    Just enabled 12 directories and doing larger runs All seems proper with PSRAM usage, not yet triggered the behavior seen with DMAMEM?

    This bodes well for FLASH - but DMAMEM a puzzle why the Warm Restart is getting corrupted?

    Post #891 CODE: In adding dirs I found Copy/Paste Edit error in this function about line 603 - it wasn't recursing on itself - but the donor code
    Code:
    int DirectoryVerify(File dir) {
    	int errCnt = 0;
    	while (true) {
    		File entry =  dir.openNextFile();
    		if (! entry) {
    			break;
    		}
    		if (entry.isDirectory()) {
    			Serial.print("\tD");
    			errCnt += DirectoryVerify(entry);

  23. #898
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,219
    Quote Originally Posted by defragster View Post
    Notes above showed PSRAM separate from RAM1 or RAM2 Example usage, and indeed for general consumption the RAM1 / RAM2 variant would need a clear way to differentiate.
    Because, while 'we' know PSRAM uses the same code, it is a case that shouldn't be confusing to use, or users might get false failures after soldering.

    Not sure I'm following in the RAM section currently there are TWO (cryptic) lines - one line can be changed (as commented) or a third line added:
    Code:
    #ifdef TEST_RAM
    LittleFS_RAM myfs;
    // RUNTIME :: extern "C" uint8_t external_psram_size;
    //EXTMEM char buf[MYPSRAM * 1024 * 1024];	// USE DMAMEM for more memory than ITCM allows - or remove
    DMAMEM char buf[490000];	// USE DMAMEM for more memory than ITCM allows - or remove
    char szDiskMem[] = "RAM_DISK";
    #elif defined(TEST_SPI)
    When testing RAM1 changing of adding an alteration of the DMAMEM line like - with change for available RAM:
    Code:
    //DMAMEM char buf[490000];	// USE DMAMEM for more memory than ITCM allows - or remove
    char buf[390000];	// USE DMAMEM for more memory than ITCM allows - or remove
    Thanks Defragster... Was just sort of wondering for the main RAM test and maybe others if it might make sense to have, it some how preconfigured for different boards...
    That is if you are going back and forth testing on T4.1 with, T4.1 withouth, Micromod, T4, T3.6, T3.5, T3.2 would it make sense to have default setup for each board so quick compile for that board will give you sort of the default testing...

    Probably not a bit deal, just a thought

  24. #899
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,492
    PSRAM seems safe for use with Software Restart - between operations all is properly stored.

    Did 20 or 30 or more Restarts with "same" operations as RAM2 showed corruption with and it worked perfectly each time.

    Where "same" was then with more folders and multiple "B" or "S" files to make sure media space was filled and reused. And each restart was a'5' iteration set not '1' to create and invoke more file changes across the dirs.

    In case anyone wants to see here is the current sketch : > going to do some quick repeats of process on a 1G QSPI NAND and get to another non-Teensy task I've been doing background wise but needs to be done ...


    Not sure how DMAMEM is 'cracking' - if the _FLUSH is right - but it is showing that anomaly regularly.

    Back to PSRAM - FUNNY I bumped from 8MB to two CHIP 16MB between uploads and the Disk Survived and Extended! And ran repeating fillings from there ???
    > Not sure the mount code could or should look for that? Ideally it should not work ... going to go from 16 to 8 and see what shows.
    >> Downsizing 16 to 8 MB did not Mount old drive but presented a fresh formatted drive
    - > back from 8 to 16 MB and it picked up running again!

    Updated the "D" file walk length verify - put out fir DIR or FILE letter:
    Code:
    Walk all Files verify Read Size:	
    00	D1 FGNOPQRSTUVW
    	D1 HIPQRSTUVWXY
    	D1 
    1	D1 ABCDEFGHIJKLMNOPQRSTUVWXYZ
    	D2 ABCDEFGHIJKLMNOPQRSTUVWXYZ
    	D3 ABCDEFGHIJKLMNOPQRSTUVWXYZ
    	D4 BCDEFGHIJK
    	D5 DEFGHIJKLM
    	D6 FGHIJKLMNO
    	D7 ABCDEFGHIJKLMNOPQ
    	D8 BCJKLMNOPQRS
    	D9 DELMNOPQRSTU
    ABCDEFGHIJKLMNOPQRSTUVWXYZ	0 Errors found

  25. #900
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,492
    Quick update QSPI NAND passes restart Integrity tests. Good like PSRAM, and no signs of corruption like on the DMAMEM Ramdisk.

    Slower than (PS)RAM and larger media size a double whammy for test time. But did a few cycles and no issues.

    Code:
     12 dirs with 32 files of Size 88495360 Bytes
     Total 220 files of Size 91009536 Bytes
    Bytes Used: 119930880, Bytes Total:131596288
    [  0.31 M](0.31412 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >
    [  0.33 M](0.32963 M elap) Awaiting input 0123456789RdDwcghkFqvplmusSBbyYxfan+-? loops left 0 >
    Walk all Files verify Read Size:	
    00	D1 LMNOPQRSTUVW
    	D1 NOPQRSTUVWXY
    	D1 
    11	D1 ABCDEFGHIJKLMNOPQRSTUVWXYZ
    22	D2 ABCDEFGHIJKLMNOPQRSTUVWXYZ
    	D3 ABCDEFGHIJKLMNOPQRSTUVWXYZ
    	D4 ABCDEFGHIJKLMNOPQRSTUVWXYZ
    	D5 BCDEFGHIJKLM
    	D6 DEFGHIJKLMNO
    	D7 FGHIJKLMNOPQ
    	D8 HIJKLMNOPQRS
    	D9 JKLMNOPQRSTU
    ABCDEFGHIJKLMNOPQRSTUVWXYZ	0 Errors found

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •