Forum Rule: Always post complete source code & details to reproduce any issue!
Page 16 of 33 FirstFirst ... 6 14 15 16 17 18 26 ... LastLast
Results 376 to 400 of 811

Thread: LittleFS port to Teensy/SPIFlash

  1. #376
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    7,530
    copied the latest code again (one hour old, from github).
    code from "official" wmxz example.
    Click image for larger version. 

Name:	2020-11-20 19_19_42-Suche.png 
Views:	11 
Size:	33.4 KB 
ID:	22552
    RAM1 seems to work - everything else not.


    Edit:Correcton, RAM1 does not work, too.
    This seems to stay forever:
    Click image for larger version. 

Name:	2020-11-20 19_25_54-Start.png 
Views:	9 
Size:	24.2 KB 
ID:	22553


    Doing other things now..

  2. #377
    Senior Member wwatson's Avatar
    Join Date
    Aug 2017
    Posts
    338
    Quote Originally Posted by PaulStoffregen View Post
    Maybe it's time to package up 1.54-beta5? I had hoped to play with MSC and bring it into USBHost_t36 and with FS/File interface, but maybe we need this now so everyone can easily sync up with the massive number of changes over the last few weeks.
    Please check out these these links:
    https://github.com/wwatson4506/SdFat-beta

    https://github.com/wwatson4506/SD

    and:
    https://github.com/wwatson4506/MSC/t...C-non-blocking.

    I have Arduino 1.8.13, TD 1.54B5 working with FS, SD, SdFat and MSC-non-blocking but not in a non-blocking async mode. Not sure if this is what you are after. It has taken me some time to understand SdFat-beta and adapt MSC to it. I think I am close.

    There are two versions that I have been working on. One version modifies SdFat.h, SD.h and other SdFat files. Sort of in-lining with SdFat.h and other SdFat files plus adding the actual SdFat block device driver directory USBmsController to SdFat. I have another version that parallels SdFat.h and SD.h. This is not on GitHub. Both appear to work with FS.h.

    I started from scratch on one of my laptops and installed the latest TDB5 with a fresh copy of Arduino 1.8.13 and tested all of the modified SD example sketches along with the existing sketches including SDIO and external SPI cards. They all seem to work. I have a modified version of bench.ino and SdInfo.ino that are also working with MSC. The example sketches are obvious ending the filename with 'USB'.
    Some observations are:
    1. The speed of transfer is not much better than FatFs. Limited by MSC blocking. I tested all of the USB drive types that I have including USB SD card readers.
    2. Setting SdFat PRE_ALLOCATE = true does not work with bench.ino properly. When closing the file it does not terminate the file by the amount of bytes written but by the size of the pre allocation. This adds garbage to the end of a file. When opening bench.dat with a text editor it seems to fail. Using a hex editor shows the difference.
    3. In the bench.ino sketch if BUF_SIZE < 2 * 512 (sector size = 512) then SdFat does not use multi sector writes. This can be seen in FatFile.cpp line 1395.

    There is still a lot of unfinished work to do with the MSC version of SdFat. This just proof of concept again.
    I think maybe I should release MSC-non-blocking to the wild

  3. #378
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,133
    @Paul :: Back to QSPI LittleFS:

    Just realized had not tested operation after Low Level format of QSPI drive with SPEW on looking to see if the underlying code detects formatted space before forcing format.

    Note: I did confirm second of two back to back lowLevelFormat()'s runs without ANY 'waited # us' so blockIsBlank() is valid within the LittleFS.cpp
    <edit> NOTE: tested the blockIsBlank() test to take : "BiB wait 143 us"
    > would be nice if any USED LittleFS Flash block was known to always write a value into a given block location? Then just that Area could be tested for non-blank?

    With 'waited' SPEW back on this shows during the low level format as expected:
    Code:
    Formatting Low Level:
    	.  waited 26885 us
      waited 27402 us
      waited 26538 us
    ...
      waited 23747 us
      waited 26121 us
      waited 23558 us
    .  waited 25682 us
      waited 23587 us
      waited 25991 us
      waited 24630 us
    ...
    Then making directories first thing on the formatted disk:
    Code:
    ...
     Done Formatting Low Level.
    ...
    	 Making Root Dirs
      waited 24632 us
      waited 84 us
      waited 123 us
      waited 24787 us
      waited 97 us
      waited 123 us
      waited 24531 us
      waited 98 us
      waited 123 us
      waited 24658 us
      waited 99 us
      waited 123 us
      waited 27343 us
    ...
    That then continues during initial file creation using LFSintegrity '1':
    Code:
    [  5.09 M](  0.00 elap) Awaiting input 0123456789RdchkFqvplmux? loops left 0 >1
    :: /B_file.txt   waited 105 us
      waited 29825 us
      waited 371 us
      waited 371 us
      waited 371 us
      waited 372 us
      waited 373 us
      waited 372 us
      waited 372 us
      waited 371 us
      waited 373 us
      waited 371 us
      waited 372 us
      waited 372 us
      waited 94 us
     QSPI_DISK +++ Add +++ [sz 0 add 3072] ++ B 	Verify /B_file.txt bytes 3072 
    :: /C_file.txt   waited 26235 us
      waited 366 us
      waited 371 us
      waited 307 us
      waited 23703 us
      waited 372 us
    ...
    It seems the Core LittleFS code always does a block format on any block before using it without checking like the lowLevelFormat()? LittleFS_QSPIFlash::erase() ???

  4. #379
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,383
    Quote Originally Posted by defragster View Post
    > would be nice if any USED LittleFS Flash block was known to always write a value into a given block location? Then just that Area could be tested for non-blank?
    LittleFS uses a superblock to hold metadata about the entire filesystem. I believe just erasing the superblock is enough, though I will admit I have not carefully studied the details of how LittleFS manages everything. But I have seen in the code that the superblock can be expanded or relocated. How exactly that is done, and whether erasing only the original superblock is enough, I do not know.

    Some of the things I do know, or believe I might sort-of know, are from watching what LittleFS actually does with the media using Serial.printf() to show (or "spew") info about each media access. I left all those in the code, just commented out.

    Code:
            static int static_read(const struct lfs_config *c, lfs_block_t block,
              lfs_off_t offset, void *buffer, lfs_size_t size) {
                    //Serial.printf("   qspi rd: block=%d, offset=%d, size=%d\n", block, offset, size);
                    return ((LittleFS_QSPIFlash *)(c->context))->read(block, offset, buffer, size);
            }
            static int static_prog(const struct lfs_config *c, lfs_block_t block,
              lfs_off_t offset, const void *buffer, lfs_size_t size) {
                    //Serial.printf("   qspi wr: block=%d, offset=%d, size=%d\n", block, offset, size);
                    return ((LittleFS_QSPIFlash *)(c->context))->prog(block, offset, buffer, size);
            }
            static int static_erase(const struct lfs_config *c, lfs_block_t block) {
                    //Serial.printf("   qspi er: block=%d\n", block);
                    return ((LittleFS_QSPIFlash *)(c->context))->erase(block);
            }
    My intention was to just get LittleFS working well on Teensy hardware, rather then studying how it actually accesses the media. But I did notice formatting fresh media always seems to write to the same 2 blocks. Maybe just erasing those 2 would be enough? How to test this I to not know, especially the case where the superblock has been expanded by lots of activity. A really cautious approach might involve editing lfs.c to turn on a LED or do something else noticeable inside the superblock-expanding code to confirm it changed the superblock allocation, and then try erasing only those 2 sectors, and watch whether a mount operation actually tries to access any other blocks.

    While I think this would be interesting info to know, I'm not personally planning to do anything about it (other than write this one message).


    It seems the Core LittleFS code always does a block format on any block before using it without checking like the lowLevelFormat()? LittleFS_QSPIFlash::erase() ???
    Are you proposing to change the public API? We are currently in the transition from the very fluid time only days ago, before any beta installers had LittleFS, but not yet a non-beta 1.54 release. I'm less inclined to change the public API than only days ago, but if a change really is worthwhile, now is the time to make that case. After 1.54 is released, only the most dire problems would justify a public API change that would break people's programs.

  5. #380
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,133
    Quote Originally Posted by PaulStoffregen View Post
    LittleFS uses a superblock to hold metadata about the entire filesystem. I believe just erasing the superblock is enough, though I will admit I have not carefully studied the details of how LittleFS manages everything. But I have seen in the code that the superblock can be expanded or relocated. How exactly that is done, and whether erasing only the original superblock is enough, I do not know.

    ...
    My intention was to just get LittleFS working well on Teensy hardware
    ...
    While I think this would be interesting info to know, I'm not personally planning to do anything about it (other than write this one message).

    Are you proposing to change the public API?
    ...
    ... And you did Very Well!

    Not proposing anything - just noting that the complete LLformat did not actually set the QSPI Flash up for one full pass across the media of no Format access as hoped/expected. Unlike LLFormat that reads a block ( in 142-143 us ) to check if it needs formatting - the LittleFS core just safely formats everything as it wants to use it.

    Reading some of the SPEC and it looks like all the blocks are tagged that would take them to non 0xff... state in the first 32 bits - except 0xff... is noted as okay.

    Added quick test to read 64 bytes and skip the format if 0xff and it did prevent unneeded pre-formatting { this was changing the public API } and it ran fine for MkDir and some small number of iterations of LFSintegrity (using larger files 10-50K)

    But - it didn't last very long as the code seemed to return some blocks #200-700 and hit them again it time less than the whole 16MB should have been 'wear leveled' across the 4096 blocks.
    Result: no Value in LLFormat before use as designed to get maximal first use write speeds.


    Also odd how they come up with totalSize() from block used counts - wondering if there is really that much overhead that this is true:
    Code:
     Total 92 files of Size 514048 Bytes
    Bytes Used: 909312, Bytes Total:16777216
    But that doesn't impair general use as far as tested.

  6. #381
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,383
    Quote Originally Posted by defragster View Post
    But - it didn't last very long as the code seemed to return some blocks #200-700 and hit them again it time less than the whole 16MB should have been 'wear leveled' across the 4096 blocks.
    Maybe try experimenting with the config.block_cycles setting?

    https://github.com/PaulStoffregen/Li...tleFS.cpp#L460

    Here's the only documentation (that I know about).

    Code:
        // Number of erase cycles before littlefs evicts metadata logs and moves 
        // the metadata to another block. Suggested values are in the
        // range 100-1000, with large values having better performance at the cost
        // of less consistent wear distribution.
        //
        // Set to -1 to disable block-level wear-leveling.
    I just picked a number roughly in the middle of the range this comment recommends. So far, I've done absolutely no testing whether it really is wear leveling or how this affects performance.

  7. #382
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,133
    Quote Originally Posted by PaulStoffregen View Post
    Maybe try experimenting with the config.block_cycles setting?

    https://github.com/PaulStoffregen/Li...tleFS.cpp#L460

    ...
    I just picked a number roughly in the middle of the range this comment recommends. So far, I've done absolutely no testing whether it really is wear leveling or how this affects performance.
    Will give that a look - thanks for the insight. Was considering doing a heat map array of blocks formatted to see the pattern - having that will give a way to alter and see the diff. But bottom line seems it purposefully reuses some blocks in place to keep from rewriting the whole of the larger block of core Meta Data.

    I need to fix my quick and dirty elapsed time between commands monitor as it seems to get reset prematurely as written.

    The LFSintegrity was designed more for cyclic/random abuse - so maybe that is why it triggers the 'early' re-format/reuse as it is forcing extra meta data changes {files create/extend, delete} - as hoped. For a FAST_LOG mode just creating and extending a single file perhaps on a LLFormat might be possible with an internal mode change.

  8. #383
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,133
    First glance using the 'unsafe=1' hack to skip formatting if the first 64 bytes were 0xff - after a LLformat seems to works safely and avoid having the format before use when not needed saving 100 seconds?

    @Paul::
    > Bottom line - Notes below don't show any issues with data wear leveling - unless those DIR entry blocks get updated in place too long before being moved.
    -> the FULL FLASH does get covered to the end once in 6 iterations of LFSintegrity as it runs, iteration #7 comes around and properly and uses previously unused blocks at media start.
    -> The only formats during that time are when DIR blocks need to be updated with file changes as done.

    Creating the directories noted these _NF_# blocks were not formatted, it did format block 0 which was not formatted during the LLformat:
    Code:
    	_NF_222__	_NF_224__	_NF_226__	_NF_228__	_NF_230__	_NF_232__	_NF_234__	_NF_236__	_NF_238__	_NF_240__	_NF_242__	_NF_244__	_NF_246__	_NF_248__	_NF_250__	_NF_252__
      FMT blk 0 	  waited 27069 us		_NF_254__	_NF_256__	_NF_258__	_NF_260__	_NF_262__	_NF_264__	_NF_266__	_NF_268__	_NF_270__	_NF_272__	_NF_274__	_NF_276__
    Those would be the blocks the DIRS are stored in and those blocks see multiple reformats - { stored in pairs :: metadata pair is stored in two blocks, with one block
    providing a backup during erase cycles} - after 6 iterations:
    Code:
    	 Block Heat Map
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
    1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
    	 Format No/Yes    0 - 63
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
    	 Format No/Yes    64 - 127
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
    	 Format No/Yes    128 - 191
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 
    	 Format No/Yes    192 - 255
    1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 
    3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
    	 Format No/Yes    256 - 319
    ...
    The index numbers are after the blocks with top line the blocks NOT reformatted on request ONCE - then the lower line shows # when format was needed.

    After those reformatted blocks in the 6 iterations the output shows only skipped formats on top row - and no block format commands issued to the end of the media:
    Code:
    ...
    1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
    	 Format No/Yes    3968 - 4035
    1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
    	 Format No/Yes    4032 - 4095
    As shown above there are untouched blocks around and before the blocks allocated for the directories. On the seventh iteration those blocks are then given _NF_ indicators and then the data is moved into rewritten blocks where formatting is needed:
    Code:
    	 Block Heat Map
    0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 
    6 6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
    	 Format No/Yes    0 - 63
    1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
    	 Format No/Yes    64 - 127
    1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
    	 Format No/Yes    128 - 191
    1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 3 3 5 4 3 3 3 3 5 4 3 3 3 3 5 4 3 3 3 3 5 4 3 3 3 3 5 4 3 3 3 3 5 4 
    	 Format No/Yes    192 - 255
    1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 
    3 3 3 3 5 4 3 3 3 3 5 4 3 3 3 3 5 4 3 3 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 
    ...
    Another '9' iterations and it looks like it has paraded around the FLASH again and hit the DIR blocks some more:
    Code:
    ...
    1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 
    1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 1 0 8 8 10 9 8 8 8 8 10 9 8 8 8 8 10 9 8 8 8 8 10 9 8 8 8 8 10 9 8 8 8 8 10 9 
    	 Format No/Yes    192 - 255
    1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 
    8 8 8 8 10 9 8 8 8 8 10 9 8 8 8 8 10 9 8 8 0 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 
    	 Format No/Yes    256 - 319
    1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 
    2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 
    	 Format No/Yes    320 - 383
    ...

  9. #384
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,133
    Added to LFSintegrity is a 'B'ig file write { uint32_t xx, toWrite = (16 * 1024 * 1024) - myfs.usedSize() - 40960; }

    Using a fresh lowLevelFormatted QSPI drive (NO Dir root folders)
    'B'big file write takes 27 seconds, then 'd'ir:
    Code:
    Big write took 27497895 us for 16728000 Bytes : file3.size()=16728000[/B]
    
    printDirectory QSPI_DISK
    --------------
    FILE	bigfile.txt		16728000
    
     0 dirs with 1 files of Size 16728000 Bytes
     Total 1 files of Size 16728000 Bytes
    Bytes Used: 16769024, Bytes Total:16777216
    HEATMAP shows NO FORMATTED blocks - including and DIR blocks, all were tested blank and format skipped like:
    Code:
    ...
    1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
    	 Format No/Yes    192 - 255
    ...
    Then 'b'ig file delete on DIRTY disk:
    'B'ig file write takes 135 seconds { ALL blocks took a FORMAT } and 'd'ir:
    Code:
    Big write took 135054336 us for 16728000 Bytes : file3.size()=16728000[/B]
    
    printDirectory QSPI_DISK
    --------------
    FILE	bigfile.txt		16728000
    
     0 dirs with 1 files of Size 16728000 Bytes
     Total 1 files of Size 16728000 Bytes
    Bytes Used: 16769024, Bytes Total:16777216
    NOTE: This allows File storage of 15.9 MB of data in 27 seconds instead of 135 seconds

    That is 605 KB/sec instead of 121 KB/sec with ~4090 regular waits over 20 ms. Not perfect - but would allow best the FLASH can offer and ~5 times better without awkward waits for 'quick' logging.
    > Going to format and test one more time - the same should apply to !^MB SPI FLASH - will test.

    Propose: Make the unconditional Format conditionally conditional by adding a method to set/unset a flag for user use ideally after USER does a lowLevelFormat.

    Here is the MOD made to LittleFS func - currently with hardcoded "unsafe=1" would instead have a method to enable, that has to be OFF during LLformat, then set ON by user:
    Code:
    extern uint16_t blockHeat[2][9000]; // this is DEBUG
    int unsafe = 1;
    int LittleFS_QSPIFlash::erase(lfs_block_t block)
    {
    	int format = 0;
    
    	if ( 1 == unsafe ) { // this var would be set ZERO in LLformat() - user method to set On/Off - Off on start
    		char readBuf[16];
    		memset(readBuf, 0, 16);
    		config.read(&config, block, 0, readBuf, 16);
    		for (unsigned int i = 0; i < 16; i++) {
    			if (readBuf[i] != 0xFF) {
    				format = 1;
    				break;
    			}
    		}
    		if ( !format ) {
    			//Serial.printf("\t_NF_%u__", block);
    			blockHeat[0][block]++;
    			return 0; // no apparent need to FORMAT
    		}
    		//Serial.printf("\n  FMT blk %u \t", (unsigned int)block);
    		blockHeat[1][block]++;
    	}
    
    	flexspi2_ip_command(10, 0x00800000);
    	const uint32_t addr = block * config.block_size;
    	flexspi2_ip_command(12, 0x00800000 + addr);
    	// TODO: detect errors, return LFS_ERR_IO
    	return wait(erasetime);
    }
    // DEBUG blockHeat is how the heat map of formatting was tracked
    Note: initial test was 64 Bytes - here reduced to 16 Bytes - before retesting - no faster

    And current write code for testing results presented - not sure how much usable space per block, so wrote 4,000B:
    Code:
    void bigFile( int doThis ) {
    	char myFile[] = "/bigfile.txt";
    
    	if ( 0 == doThis ) {	// delete File
    		myfs.remove(myFile);
    		filecount--;
    	}
    	else {	// FILL DISK
    		lfs_ssize_t resW = 1;
    		uint32_t xx, toWrite = (16 * 1024 * 1024) - myfs.usedSize() - 40960;
    		xx = toWrite;
    		char someData[4000];
    		memset( someData, 'z', 4000 );
    		Serial.printf( "\nStart Big write of %u Bytes", xx);
    		uint32_t timeMe = micros();
    		file3 = myfs.open(myFile, FILE_WRITE);
    		while ( toWrite > 4000 && resW > 0 ) {
    			resW = file3.write( someData , 4000 );
    			toWrite -= 4000;
    		}
    		xx -= toWrite;
    		file3.close();
    		filecount++
    		timeMe = micros() - timeMe;
    		file3 = myfs.open(myFile, FILE_WRITE);
    		if ( resW < 0 ) {
    			Serial.printf( "\nBig write error %u 0x%X Bytes", resW, resW );
    		}
    		Serial.printf( "\nBig write took %u us for %u Bytes : file3.size()=%u", timeMe, xx, file3.size() );
    		file3.close();
    	}
    }

  10. #385
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,133
    Opps : of course LittleFS_SPIFlash::erase(lfs_block_t block) is it own function and would take alternate code to implement the byte read test for 'unsafe'.

    For fun here is a unedited current LFSintegrity repro session showing:: 'F'ormat, 'B'ig file, 'd'ir, 'b'ig delete, 'B'ig file, 'd'ir : where the second 'B' will be on dirty media needing format before use:
    Code:
    Formatting Low Level:
    	.........................................................................................................................
    
     Done Formatting Low Level.
    u 
    	 Updated filecount 0
    F
    [  2.17 M](  1.87 elap) Awaiting input 0123456789RdchkFqvplmuxsBb? loops left 0 >
    Start Big write of 16728064 Bytes
    Big write took 27493108 us for 16728000 Bytes : file3.size()=16728000
    [  4.61 M](  2.44 elap) Awaiting input 0123456789RdchkFqvplmuxsBb? loops left 0 > d
    printDirectory QSPI_DISK
    --------------
    FILE	bigfile.txt		16728000
    
     0 dirs with 1 files of Size 16728000 Bytes
     Total 1 files of Size 16728000 Bytes
    Bytes Used: 16769024, Bytes Total:16777216
    
    
    	 Loop Count: 5 (#fileCycle=0), Bytes read 0, written 0, #Files=1
    [  4.73 M](  0.13 elap) Awaiting input 0123456789RdchkFqvplmuxsBb? loops left 0 >
    
    [  4.73 M](  0.00 elap) Awaiting input 0123456789RdchkFqvplmuxsBb? loops left 0 >b
    [  4.79 M](  0.06 elap) Awaiting input 0123456789RdchkFqvplmuxsBb? loops left 0 >
    Start Big write of 16728064 Bytes
    Big write took 135182612 us for 16728000 Bytes : file3.size()=16728000
    [  7.19 M](  2.40 elap) Awaiting input 0123456789RdchkFqvplmuxsBb? loops left 0 > d
    printDirectory QSPI_DISK
    --------------
    FILE	bigfile.txt		16728000
    
     0 dirs with 1 files of Size 16728000 Bytes
     Total 1 files of Size 16728000 Bytes
    Bytes Used: 16769024, Bytes Total:16777216
    
    
    	 Loop Count: 8 (#fileCycle=0), Bytes read 0, written 0, #Files=1

  11. #386
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,133
    Update on Q/SPI prevention of re-Format after lowLevelFormat - coding put into both SPI and QSPI and tested as follows:

    Code:
    QSPI:
    108.1 secs :: lowLevelFormat
     27.5  secs :: Big File write > 16728000 Bytes << CLEAN MEDIA - no pre-format
    135.6 secs :: Big File write > 16728000 Bytes << DIRTY MEDIA
    
    SPI:
    348.4 secs :: lowLevelFormat
     36.9  secs :: Big File write > 16728000 Bytes << CLEAN MEDIA - no pre-format
    384.6 secs :: Big File write > 16728000 Bytes << DIRTY MEDIA
    Code into the class forked and pull request :: PaulStoffregen/LittleFS/pull/6

    QSPI: - Commas added for readbility
    Code:
    Formatting Low Level:
    	.........................................................................................................................
    
     Done Formatting Low Level in 108,143,703 us.
    u 
    	 Updated filecount 0
    F
    [  2.74 M](  1.86 elap) Awaiting input 0123456789RdchkFqvplmuxBb? loops left 0 >
    Start Big write of 16728064 Bytes....................................................
    Big write took 27,499,494 us for 16728000 Bytes : file3.size()=16728000
    [  4.04 M](  1.30 elap) Awaiting input 0123456789RdchkFqvplmuxBb? loops left 0 > d
    printDirectory QSPI_DISK
    --------------
    FILE	bigfile.txt		16728000
    
     0 dirs with 1 files of Size 16728000 Bytes
     Total 1 files of Size 16728000 Bytes
    Bytes Used: 16769024, Bytes Total:16777216
    
    
    	 Loop Count: 4 (#fileCycle=0), Bytes read 0, written 0, #Files=1
    [  4.49 M](  0.45 elap) Awaiting input 0123456789RdchkFqvplmuxBb? loops left 0 >b
    
    [  4.49 M](  0.00 elap) Awaiting input 0123456789RdchkFqvplmuxBb? loops left 0 >
    Start Big write of 16728064 Bytes....................................................
    Big write took 135,800,088 us for 16728000 Bytes : file3.size()=16728000
    [  6.87 M](  2.38 elap) Awaiting input 0123456789RdchkFqvplmuxBb? loops left 0 > d
    printDirectory QSPI_DISK
    --------------
    FILE	bigfile.txt		16728000
    
     0 dirs with 1 files of Size 16728000 Bytes
     Total 1 files of Size 16728000 Bytes
    Bytes Used: 16769024, Bytes Total:16777216
    SPI : same 16M FLASH on PJRC Audio card - Commas added for readbility
    Code:
    Formatting Low Level:
    	.........................................................................................................................
    
     Done Formatting Low Level in 348,352,624 us.
    u 
    	 Updated filecount 0
    F
    [ 14.14 M](  6.42 elap) Awaiting input 0123456789RdchkFqvplmuxBb? loops left 0 >
    Start Big write of 16728064 Bytes....................................................
    Big write took 36,958,825 us for 16728000 Bytes : file3.size()=16728000
    [ 15.15 M](  1.01 elap) Awaiting input 0123456789RdchkFqvplmuxBb? loops left 0 > d
    printDirectory SPI_DISK
    --------------
    FILE	bigfile.txt		16728000
    
     0 dirs with 1 files of Size 16728000 Bytes
     Total 1 files of Size 16728000 Bytes
    Bytes Used: 16769024, Bytes Total:16777216
    
    
    	 Loop Count: 6 (#fileCycle=0), Bytes read 0, written 0, #Files=1
    [ 15.89 M](  0.74 elap) Awaiting input 0123456789RdchkFqvplmuxBb? loops left 0 >b
    
    [ 15.89 M](  0.00 elap) Awaiting input 0123456789RdchkFqvplmuxBb? loops left 0 >
    Start Big write of 16728064 Bytes....................................................
    Big write took 384,570,810 us for 16728000 Bytes : file3.size()=16728000
    [ 22.42 M](  6.53 elap) Awaiting input 0123456789RdchkFqvplmuxBb? loops left 0 > d
    printDirectory SPI_DISK
    --------------
    FILE	bigfile.txt		16728000
    
     0 dirs with 1 files of Size 16728000 Bytes
     Total 1 files of Size 16728000 Bytes
    Bytes Used: 16769024, Bytes Total:16777216

    Here is the Big File Write code - User does myfs.lowLevelFormat('.'); then just adds: myfs.checkFormat( true ); then call 'false' after.
    Code:
    void bigFile( int doThis ) {
    	char myFile[] = "/bigfile.txt";
    
    	if ( 0 == doThis ) {	// delete File
    		myfs.remove(myFile);
    		filecount--;
    	}
    	else {	// FILL DISK
    		myfs.checkFormat( true );
    		lfs_ssize_t resW = 1;
    		uint32_t xx, toWrite = (16 * 1024 * 1024) - myfs.usedSize() - 40960;
    		xx = toWrite;
    		char someData[4000];
    		memset( someData, 'z', 4000 );
    		Serial.printf( "\nStart Big write of %u Bytes", xx);
    		uint32_t timeMe = micros();
    		file3 = myfs.open(myFile, FILE_WRITE);
    		int hh = 0;
    		while ( toWrite > 4000 && resW > 0 ) {
    			resW = file3.write( someData , 4000 );
    			hh++;
    			if ( !(hh % 80) ) Serial.print('.');
    			toWrite -= 4000;
    		}
    		xx -= toWrite;
    		file3.close();
    		filecount++;
    		timeMe = micros() - timeMe;
    		file3 = myfs.open(myFile, FILE_WRITE);
    		if ( resW < 0 ) {
    			Serial.printf( "\nBig write error %u 0x%X Bytes", resW, resW );
    		}
    		Serial.printf( "\nBig write took %u us for %u Bytes : file3.size()=%u", timeMe, xx, file3.size() );
    		file3.close();
    		myfs.checkFormat( false );
    	}
    }

  12. #387
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,133
    If anyone looks the PR DIFF is TINY ... writing the code much easier than testing ... have Three Teensy testing again ...

    Changed LFSintegrity to have a TOGGLE 'Z'ippy to request no reformat of blocks that already start with 16 Bytes of 0xff. Removed FORCE ON in BigFile to know it isn't using it.

    T_4.0 and a 4.1 with Audio Flash for SPI, and one T_4.1 now set for QSPI.

    Going to do a 'F' on all three, then a 'Z'ippy and 'B'ig file, then 'b'yebye delete Big File, then 'Z'ippt toggle Off and 'B'ig file, then 'b'yebye delete Big File
    then 'Z'ippy toggle ON , 'm'ake dirs and a long run of prior Iterations to 28 subdirs and root while 'Z'ippy on to make sure it never gets a false positive.

    Will add LFSintegrity to my fork of LittleFS when I see them started with working code and post link >>:

    QSPI looks like this now with units change to second and added not on KB/sec:
    Code:
     Done Formatting Low Level in 107938776 us.
    u 
    	 Updated filecount 0
    F
    [  2.19 M](  1.94 elap) Awaiting input 0123456789RdchkFqvplmuxBbZ? loops left 0 > CheckFormat on: 
    [ 12.73 M]( 10.54 elap) Awaiting input 0123456789RdchkFqvplmuxBbZ? loops left 0 >
    Start Big write of 16728064 Bytes....................................................
    Big write took 27.54 Sec for 16728000 Bytes : file3.size()=2054847098
    	Big write KBytes per second 607.41 
    
    [ 13.24 M](  0.50 elap) Awaiting input 0123456789RdchkFqvplmuxBbZ? loops left 0 > d
    printDirectory QSPI_DISK
    --------------
    FILE	bigfile.txt		16728000
    
     0 dirs with 1 files of Size 16728000 Bytes
     Total 1 files of Size 16728000 Bytes
    Bytes Used: 16769024, Bytes Total:16777216
    
    
    	 Loop Count: 4 (#fileCycle=0), Bytes read 0, written 0, #Files=1
    [ 13.71 M](  0.48 elap) Awaiting input 0123456789RdchkFqvplmuxBbZ? loops left 0 >
    
    [ 13.71 M](  0.00 elap) Awaiting input 0123456789RdchkFqvplmuxBbZ? loops left 0 >b
    [ 14.03 M](  0.32 elap) Awaiting input 0123456789RdchkFqvplmuxBbZ? loops left 0 > CheckFormat off: 
    [ 14.38 M](  0.35 elap) Awaiting input 0123456789RdchkFqvplmuxBbZ? loops left 0 >
    Start Big write of 16728064 Bytes....................................................
    Big write took 135.73 Sec for 16728000 Bytes : file3.size()=2054847098
    	Big write KBytes per second 123.25 
    
    [ 16.67 M](  2.29 elap) Awaiting input 0123456789RdchkFqvplmuxBbZ? loops left 0 >b CheckFormat on:
    Summary for T_4.0 SPI - first ZIPPY - second without::
    Code:
    Big write took 36.80 Sec for 16728000 Bytes : file3.size()=4383
    	Big write KBytes per second 454.58 
    
    
    Big write took 381.32 Sec for 16728000 Bytes : file3.size()=4383
    	Big write KBytes per second 43.87
    Summary for T_4.1 SPI - first ZIPPY - second without::
    Code:
    Big write took 36.99 Sec for 16728000 Bytes : file3.size()=0
    	Big write KBytes per second 452.23 
    
    Big write took 385.39 Sec for 16728000 Bytes : file3.size()=0
    	Big write KBytes per second 43.41
    Last edited by defragster; 11-24-2020 at 07:06 AM.

  13. #388
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,383
    Quote Originally Posted by defragster View Post
    Update on Q/SPI prevention of re-Format after lowLevelFormat - coding put into both SPI and QSPI and tested as follows:
    ....
    Code into the class forked and pull request :: PaulStoffregen/LittleFS/pull/6
    This code worries me.

    The main problem is it assumes an entire flash block is blank and does not require erase if only the first 16 bytes are 0xFF. The huge problem is it never examines the other 4080 bytes (or more, when we support chips with larger sector size). If any of those bytes are not blank, this will cause data corruption!

    Yes, I understand it only does this if checkFormat(true) was previously called. But just because a dangerous behavior is not the default does not make it any safer for users who do choose to use this (proposed) public API.

    I probably will not merge this pull request.

  14. #389
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,133
    'block' seems to be the single unit of tracking and allocation?

    Indeed this is only 'safe' if every "used" 'block' starts with some header tagging/info that assures somewhere in the first 16 bytes ( originally test was 64 B that did not affect the speed on QSPI ).
    > in reading the SPEC.md that seemed to be the case
    > it would be up to the user smart enough to call myfs.checkFormat( true ) to have waited through a myfs.lowLevelFormat(), which cannot be enforced at runtime because a "restart" after formatting works as well
    - > LLformat need only be done once - in the limited testing done - the detection works as long as LittleFS was behind making the block dirty, it just isn't fast anymore.
    > if areas on the media were not LLformatted, and held data other than LittleFS data the test is not valid.

    Is a 'block' always the min format unit - on any size flash where that 'block' is the allocation unit used by LittleFS? Regardless of size of the media?
    from spec: "In addition to the logical block size (which usually matches the erase block size)"
    > If a larger media 'format unit' can hold many blocks then this won't work on those media if not used sequentially from the start
    > unless each erase 'block' in that 'format unit' is tested to see all logical blocks in the unit as pristine
    Last edited by defragster; 11-24-2020 at 11:48 AM.

  15. #390
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,383
    LittleFS uses 3 sizes, which are documented this way in lfs.h:

    Code:
        // Minimum size of a block read. All read operations will be a
        // multiple of this value.
        lfs_size_t read_size;
    
        // Minimum size of a block program. All program operations will be a
        // multiple of this value.
        lfs_size_t prog_size;
    
        // Size of an erasable block. This does not impact ram consumption and
        // may be larger than the physical erase size. However, non-inlined files
        // take up at minimum one block. Must be a multiple of the read
        // and program sizes.
        lfs_size_t block_size;
    Sizes supported by flash chips vary. The begin() function reads the JEDEC ID and uses a table of known chips. So far, I have been setting read_size to the same as prog_size.

    Winbond's NOR flash chips use 3 different erase sizes: 4K, 32K, 64K. You can't erase less than 4K at a time. Winbond's chips have a "programming page" size of 256 bytes. With NOR flash, partial page programming is generally supported, so we could implement prog_size of 128 or 64 bytes. Whether that would be beneficial, I do not know.

    Not all NOR flash chips are created equal. Many of the Spansion parts have much larger minimum erase size, and some have larger program page size.

    Use of ECC in NAND flash generally means partial page programming isn't allowed. But I'm planning to create new LittleFS classes for NAND chips, rather than trying to overload this relatively simple code with the complexity of NAND's bad block management.

  16. #391
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    6,095
    Quote Originally Posted by PaulStoffregen View Post
    ......
    Use of ECC in NAND flash generally means partial page programming isn't allowed. But I'm planning to create new LittleFS classes for NAND chips, rather than trying to overload this relatively simple code with the complexity of NAND's bad block management.
    Think that is a great idea. For the last couple of days been playing with the W25N01G code to convert the QSPI lib to a SPI proof of concept lib along with looking at incorporating it into the current LittleFS lib. Trying to combine it with what is there now would really add a whole lot more complexity. I can be done but...... Just my 2 cents to agree with you

  17. #392
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,133
    Quote Originally Posted by PaulStoffregen View Post
    LittleFS uses 3 sizes, which are documented this way in lfs.h:

    Code:
        // Minimum size of a block read. All read operations will be a
        // multiple of this value.
        lfs_size_t read_size;
    
        // Minimum size of a block program. All program operations will be a
        // multiple of this value.
        lfs_size_t prog_size;
    
        // Size of an erasable block. This does not impact ram consumption and
        // may be larger than the physical erase size. However, non-inlined files
        // take up at minimum one block. Must be a multiple of the read
        // and program sizes.
        lfs_size_t block_size;
    Sizes supported by flash chips vary. The begin() function reads the JEDEC ID and uses a table of known chips. So far, I have been setting read_size to the same as prog_size.

    Winbond's NOR flash chips use 3 different erase sizes: 4K, 32K, 64K. You can't erase less than 4K at a time. Winbond's chips have a "programming page" size of 256 bytes. With NOR flash, partial page programming is generally supported, so we could implement prog_size of 128 or 64 bytes. Whether that would be beneficial, I do not know.

    Not all NOR flash chips are created equal. Many of the Spansion parts have much larger minimum erase size, and some have larger program page size.

    Use of ECC in NAND flash generally means partial page programming isn't allowed. But I'm planning to create new LittleFS classes for NAND chips, rather than trying to overload this relatively simple code with the complexity of NAND's bad block management.
    Thanks - I've been chasing circles looking for the root of those defines - even just grepping the LittleFS folder I was not getting to the actual number part. Then seeing odd 128's - now I assume means 128 32 bit words filling a 4096 block.

    Unless we can safely iterate over the start of all the logical block allocations in an erase unit to be sure then it won't work

  18. #393
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,383
    Quote Originally Posted by defragster View Post
    Then seeing odd 128's - now I assume means 128 32 bit words filling a 4096 block.
    I don't know which "odd 128" you mean. I'm happy to explain the rationale behind anything I've written, but if you want a useful answer you do need to be more specific with the question. In so many cases, the rationale of choices I've made is merely a conservative default just to get the things working, with little or no idea if that choice is optimal.

    128 32 bit words is 512 bytes, not 4096, so maybe not a great assumption. Probably best to ask more specifically.

  19. #394
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,133
    Quote Originally Posted by PaulStoffregen View Post
    I don't know which "odd 128" you mean. I'm happy to explain the rationale behind anything I've written, but if you want a useful answer you do need to be more specific with the question. In so many cases, the rationale of choices I've made is merely a conservative default just to get the things working, with little or no idea if that choice is optimal.

    128 32 bit words is 512 bytes, not 4096, so maybe not a great assumption. Probably best to ask more specifically.
    That wasn't a question - and not properly phrased either : 4096/128 == 32 byte blocks. ( that was from hard coded LittleFS_Program )

    Seeing the chipinfo explains it with regard to FLASH.


    Question - if the 64MB Flash to be treated the same way as the 16MB - or is it organized differently?

    Yes the NOR in it's own world will have lots of extra work.

    I didn't look to try this on _PROGRAM flash - it seems to have a different method - as might NOR.

    Would be nice if the myfs.checkFormat( true ); found a way to work - ugly to format in advance - but slow to log and format at the same time - and quicker but not fast when writing the first time.


    Also I noted the wait(uint32_t microseconds) calls yield() : What happens if yield is gone a long time and gives a false positive timeout because some _Event had some work to do - or worse yet tried to access the Q/SPI FLASH, assuming the 'weak yield()' is even being called?

  20. #395
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,383
    I found this github issue about LittleFS performance. Sounds similar to the 10X slower speed you're seeing in these tests.

    https://github.com/littlefs-project/littlefs/issues/75

    The first reply talks about increasing the lookahead buffer to improve large file performance.

    Here's the documentation from lfs.h:

    Code:
        // Size of the lookahead buffer in bytes. A larger lookahead buffer
        // increases the number of blocks found during an allocation pass. The
        // lookahead buffer is stored as a compact bitmap, so each byte of RAM
        // can track 8 blocks. Must be a multiple of 8.
        lfs_size_t lookahead_size;
    Currently the code is setting this to the same as progsize, which probably doesn't make much sense.

    Maybe try this?

    Code:
        config.lookahead_size = config.block_count / 8;
    This should give it enough memory to track the status of every block in the entire chip.

    They do say it must be a multiple of 8. I believe that should happen automatically with every chip we currently support. But a really cautious approach might round up to the nearest 8, maybe like this.

    Code:
        config.lookahead_size = (config.block_count / 8 + 7) & 0xFFFFFFF8;
    If this makes a significant speed improvement for large file writes, send a pull request for only this change and I'll merge it.

    There might be some other gems hidden in that issue. I'll admit, I didn't read the whole thing. And realistically, I probably won't have time to do much on LittleFS over the next couple weeks, other than add support for more NOR flash chips which are similar to the ones we already support.

  21. #396
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,383
    Quote Originally Posted by defragster View Post
    Question - if the 64MB Flash to be treated the same way as the 16MB - or is it organized differently?
    I read the Winbond W25Q512JV datasheet a few days ago. My first impression was the structure looked nearly identical to W25Q128JV, but the address is 32 bits rather than 24 bits. It has legacy 24 bit commands and a special register to write the high bits, and new 32 bit commands. I'm planning to use those new 32 bit commands. I have some of those chips on order from Digikey, hopefully arriving soon...

    It's been years since I've read the Spansion, Macronix and Micron datasheets. But my recollection is their larger parts differ in several ways.

  22. #397
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    6,095
    Quote Originally Posted by PaulStoffregen View Post
    I read the Winbond W25Q512JV datasheet a few days ago. My first impression was the structure looked nearly identical to W25Q128JV, but the address is 32 bits rather than 24 bits. It has legacy 24 bit commands and a special register to write the high bits, and new 32 bit commands. I'm planning to use those new 32 bit commands. I have some of those chips on order from Digikey, hopefully arriving soon...

    It's been years since I've read the Spansion, Macronix and Micron datasheets. But my recollection is their larger parts differ in several ways.
    Thought this sounded familiar so went back and it sound familiar and went back. It seems it may be similar to the W25Q256JV. See this post where I went and took a it: https://forum.pjrc.com/threads/62217...l=1#post248442. Just for reference

  23. #398
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    8,183
    Sounds like this should be fun!

    It would be nice to find a solution to speed up the littleFS as at least on one of my T4.1s with the Flash/PSRAM MTP file copies work faster and more reliably to external SD Card reader than it does to the FLASH memory.

    I need to solder up the new one to see if it is similar.

    Side Note: with the FS.h file and the like, especially with MTP and the like.

    Is there a standard way for other parts of your program like MTP to know when a file is created and/or changes size... i.e. something like the File Change notify you can get in windows and the like.

    That is suppose we have an FS or two (or more) showing up on a PC and I have open directory explorer windows open to show the contents. If the user does some form of manipulation on the PC, then MTP would know about it and either rely on the PC notify it of changes or deduce it and try to update all effected files...

    But suppose instead you have a folder window on the PC and the Teensy sketch does something like create a new log file and/or append to an existing one... How would MTP (or other sub-systems) have a clue if anything changed in the FS...

    Probably as a test, could maybe add an MTP function/method, that says something like mtpd.fsChanged(???) where you maybe pass in which FS object, and some indication of what may have changed...

  24. #399
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,133
    Will look into the "config.lookahead_size =" - but the real issue is 24ms stall on every added/redone block - of course this can only go away with pre-formatting for 'purposeful faster logging' - similar to pre-alloc of a disk ( SD/Hdd/SSD ) file on open. WIll tweak if needed the BigFile to run on PSRAM and see if it makes any difference to start ... but other things first ...

    In the LFSintegrity the DIR Walk integrity count - it does pause some long unknown time - seems a couple seconds with dozens of folders with dozens of files. Will add a timer for that as I do it on every DIR listing and the 'l' command too and the delay is annoying without a 'wait ...' note of what is happening. Having a File change for general purposes would be better than doing that.

    LittleFS does work reliably - and other than being less than fast on FLASH - it is a great first step.

    Looks like I'll have some 64MB Flash - and more spare T_4.1's - end of next week from ProtoSupplies.

    In my 'annecdotal' testing that format pre-test works. Is there a 'Teensy' bit somewhere in the LittleFS structures that could indicate when '.checkFormat()' is supported - that could be set and maintained on supported chips when a LLformat has been completed?

  25. #400
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,383
    Quote Originally Posted by KurtE View Post
    Is there a standard way for other parts of your program like MTP to know when a file is created and/or changes size... i.e. something like the File Change notify you can get in windows and the like.
    Today, no, but this probably does need to be added soon.

    Sadly, I haven't been able to keep up with the forum and MTP development over the last few days. Again PJRC is overwhelmed by several large orders, so Robin & I have been putting in long hours since last Friday just to keep up. My hope is to get more time for software over the upcoming (USA) holiday 4-day weekend.

    Maybe better to discuss this on the MTP thread. Just wanted to say that do anticipate needing to add a public API for services like MTP to get notification of changes.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •