LittleFS port to Teensy/SPIFlash

Just an FYI - The newer versions of the github for windows app is getting a little nicer for doing things like forks and the like. For example if you have github project you simply cloned down to your machine without forking and makes changes to it. It may prompt you or have button that says something like: it sees you are making changes would you like to fork it... It will then do the stuff to do it, like fork a copy of the project to your github project. Rename the origin remote to be upstream remote and create a new origin to your newly forked... Actually after it creates the fork up on github it will give you a few options, will you want to make contributions to the original project or make your copy not related to the original... I discussed the wanting to make contributions.

Will be interesting to see if there are simple fixes/extensions to the library the speed it up. As I mentioned I was especially seeing it with MTP where I could easily copy a 300kb file to SD cards and the like either the built in one or one using normal SPI. But the same copy would fail to the flash of propshield and many times to the QSPI flash. Again probably secondary issue where if the copy took over maybe half second the progress dialog would show up and the drive going away sound would happen and the Teensy drive does not work unless you reset.

Now to play some more. But first sync up to everything.
 
As always - it is a Trade OFF ... Cluster Size ... Block Size ....

With edits taking out the hardcoded 256 (block_size) shown in p#474:
Code:
private:
	static int static_read(const struct lfs_config *c, lfs_block_t block,
	  lfs_off_t offset, void *buffer, lfs_size_t size) {
		//Serial.printf("    ram rd: block=%d, offset=%d, size=%d\n", block, offset, size);
		uint32_t index = block * c->block_size + offset;
		memcpy(buffer, (uint8_t *)(c->context) + index, size);
		return 0;
	}
	static int static_prog(const struct lfs_config *c, lfs_block_t block,
	  lfs_off_t offset, const void *buffer, lfs_size_t size) {
		//Serial.printf("    ram wr: block=%d, offset=%d, size=%d\n", block, offset, size);
		uint32_t index = block * c->block_size + offset;
		memcpy((uint8_t *)(c->context) + index, buffer, size);
		return 0;
	}
	static int static_erase(const struct lfs_config *c, lfs_block_t block) {
		uint32_t index = block * c->block_size;
		memset((uint8_t *)(c->context) + index, 0xFF, c->block_size);
		return 0;
	}
	static int static_sync(const struct lfs_config *c) {
		return 0;
	}

Then the bigfile() fill the disk works - like it does on FLASH!

However - I tested first on an 8MB PSRAM 4.1 - and the first LFSintegrity iteration '5' with 28 Dirs + Root FAILS because the number of directories creates 28 blobs of Folders with up to 26 files in each - and the larger block size then eats more disk space for another reason

ArrGHH - so where is the middle ground ... as always ... depends on how the disk is used and filled.
> block_size=2048 also fails with 16MB RAM_DISK on bigfile() full disk that works on FLASH , but it works with 8 MB RAM_DISK, and the 8MB RAM_DISK LFSintegrity '5' works

For test purposes I created a parallel to LittleFS_RAM so I could just edit #define in sketch to test:
Code:
class LittleFS_RAM2 : public LittleFS
{
public:
	LittleFS_RAM2() { }
...

@Paul QUESTION: in addition to something like changing the RAM's "chipinfo" setup params like : if ( size > 1024*1024 ) {
I see this - there are already TWO .begin()'s:
Code:
	bool begin(uint32_t size) {
#if defined(__IMXRT1062__)
		return begin(extmem_malloc(size), size);
#else
		return begin(malloc(size), size);
#endif
	}
	bool begin(void *ptr, uint32_t size) {

Perhaps we could add a third .begin() to override block_size when not provided? Then if 'Jane' likes few big files all on ROOT and it fails on RAM she could change block_size to 4096, and if 'Jim' likes a folder per month and a file per day { and has battery backup } he could change block_size toward 256.
> There is no single perfect answer depending on use case. Adding the begin( ... , block_size ) would be great for anothe rpage or two of documentation filler as well explaining the difference.


NOTE ALSO BTW: The LFSintegrity was updated to preserve Minute ELAPSED display - it was zeroed on '0' that is used for ERROR halt - or user Abort the iterations.
 
Just an FYI - The newer versions of the github for windows app is getting a little nicer for doing things like forks and the like. For example if you have github project you simply cloned down to your machine without forking and makes changes to it. It may prompt you or have button that says something like: it sees you are making changes would you like to fork it... It will then do the stuff to do it, like fork a copy of the project to your github project. Rename the origin remote to be upstream remote and create a new origin to your newly forked... Actually after it creates the fork up on github it will give you a few options, will you want to make contributions to the original project or make your copy not related to the original... I discussed the wanting to make contributions.

Will be interesting to see if there are simple fixes/extensions to the library the speed it up. As I mentioned I was especially seeing it with MTP where I could easily copy a 300kb file to SD cards and the like either the built in one or one using normal SPI. But the same copy would fail to the flash of propshield and many times to the QSPI flash. Again probably secondary issue where if the copy took over maybe half second the progress dialog would show up and the drive going away sound would happen and the Teensy drive does not work unless you reset.

Now to play some more. But first sync up to everything.

Indeed I was presented those options and it was helpful, somewhat. I had a clone and did the PR with 'share my work' - then deleting the PR destroyed my local copy as it was a subset of Paul's - I had to agree to DELETE. No Biggie.

So this time I did a CLONE to PAUL, but then could not post the 'defragster' WIP version { except in a piecemeal PR } because it was tied to Paul's - so I did a FORK - which needed a unique local DIR name - so now I have doc\github:[ LittleFS and LittleFS_qf ] folders. Which is fine and better than having my machine get totally hosed like it used to.

In playing just now again with LittleFS_RAM I noticed the myfs.checkFormat() works just like it does for FLASH - The MAP is created and Updated - the only thing missing is the piece where a format is refused if the bit is set that would avoid the:
Code:
	static int static_erase(const struct lfs_config *c, lfs_block_t block) {
		uint32_t index = block * c->block_size;
		[B]memset((uint8_t *)(c->context) + index, 0xFF, c->block_size);[/B]
		return 0;
	}

So nice 'CLASSY' work Paul! Impressive mapping the LittleFS across the various Media to work!

OH - if you grab LFSintegrity - it has to come with the Altered LittleFS from defragster - although also works just commenting out in the 'Z' case the line :: bCheckFormat = myfs.checkFormat( bCheckFormat );

... I should sleep more after 2+ hours sleep - but lunch is too close now ... so nap after.
 
Added file read verify of each bigfile on delete to LFSintegrity. Has to have the number of characters indicated by file.size() and each char is the same as the index '#' of the file.

@KurtE - if you are excited to play with RAM2 that has the alternate block size - here set at 2048 hand type twice in the first if() clause. It has the hardcoded 256's removed. LFSintegrity has #if for RAM2 - but you can rename to LittleFS_RAM just as well:
Code:
class [B]LittleFS_RAM2[/B] : public LittleFS
{
public:
	[B]LittleFS_RAM2[/B]() { }
	bool begin(uint32_t size) {
#if defined(__IMXRT1062__)
		return begin(extmem_malloc(size), size);
#else
		return begin(malloc(size), size);
#endif
	}
	bool begin(void *ptr, uint32_t size) {
		//Serial.println("configure "); delay(5);
		configured = false;
		if (!ptr) return false;
		memset(ptr, 0xFF, size); // always start with blank slate
		size = size & 0xFFFFFF00;
		memset(&lfs, 0, sizeof(lfs));
		memset(&config, 0, sizeof(config));
		config.context = ptr;
		config.read = &static_read;
		config.prog = &static_prog;
		config.erase = &static_erase;
		config.sync = &static_sync;
		if ( size > 1024*1024 ) {
			config.read_size = 256;
			config.prog_size = 256;
[B]			config.block_size = 2048;
			config.block_count = size / 2048;[/B]
			config.block_cycles = 900;
			config.cache_size = 256;
			//config.lookahead_size = 64;  // was 64
			config.lookahead_size = 512; // (config.block_count / 8 + 7) & 0xFFFFFFF8;
		}
		else {
			config.read_size = 64;
			config.prog_size = 64;
			config.block_size = 256;
			config.block_count = size / 256;
			config.block_cycles = 50;
			config.cache_size = 64;
			config.lookahead_size = 64;
		}
		config.name_max = LFS_NAME_MAX;
		config.file_max = 0;
		config.attr_max = 0;
		configured = true;
		if (lfs_format(&lfs, &config) < 0) return false;
		//Serial.println("formatted");
		if (lfs_mount(&lfs, &config) < 0) return false;
		//Serial.println("mounted atfer format");
		mounted = true;
		checkformat = nullptr;
		return true;
	}
private:
	static int static_read(const struct lfs_config *c, lfs_block_t block,
	  lfs_off_t offset, void *buffer, lfs_size_t size) {
		//Serial.printf("    ram rd: block=%d, offset=%d, size=%d\n", block, offset, size);
		uint32_t index = block * c->block_size + offset;
		memcpy(buffer, (uint8_t *)(c->context) + index, size);
		return 0;
	}
	static int static_prog(const struct lfs_config *c, lfs_block_t block,
	  lfs_off_t offset, const void *buffer, lfs_size_t size) {
		//Serial.printf("    ram wr: block=%d, offset=%d, size=%d\n", block, offset, size);
		uint32_t index = block * c->block_size + offset;
		memcpy((uint8_t *)(c->context) + index, buffer, size);
		return 0;
	}
	static int static_erase(const struct lfs_config *c, lfs_block_t block) {
		uint32_t index = block * c->block_size;
		memset((uint8_t *)(c->context) + index, 0xFF, c->block_size);
		return 0;
	}
	static int static_sync(const struct lfs_config *c) {
		return 0;
	}
};
 
FWIW, I'm planning to dust off the old SerialFlash library and try to put a File / FS wrapper on top of it, awkward as that may be. The main problem is SerialFlash needs to pre-allocate files to a fixed size which can not be altered after the file is created. It's probably going to need a non-standard extension to FS to create files. Probably won't work with MTP. :(

The main advantage of SerialFlash is you get the full raw speed of the media, with a subset of the usual File abstraction. But you lose the flexibility of generic filesystems like LittleFS.

Eventually I'm going to create web pages for these libs. I'm imaging SD/SdFat, LittleFS and SerialFlash will be a set with sections on each that give recommendations about which to use for various types of applications.

Paul - before I got distracted looking for how LFS 'deletes' blocks - I noted a lot of refs to 'look_ahead' - that is the WINDOW it seems - how far ahead it will look to find a contiguous space for a file? That could explain the 10+ second search for RAM_DISK to fail when out of space with bigfile() as it piece meal walking the media trying to find the space without looking ahead enough at once to see there was never going to be enough space.

On freshly formatted media a file will be created contiguously - or after if the look_ahead can see a spot for it. Although I'm not sure how much of each block is 'data' given some bytes of 'meta tags' applied within each block so that will create some breaks if not 'hacked'.

Once written something that never changes is never moved AFAIK. Things that point to it may move - but it never moves unchanged data - it just updates anything that moves with pointers to where it resides - when changes to the meta data ( file/folder updates ) causes the block to fill.

Maybe that is correct to your understanding as well? If so the issue would be stepping over the 'meta markings' used for disk integrity ID'ing each block that would be embedded in the 'SerialFlash' data stream, unless you wanted to disincorporate those bytes as well - which may work - as long as nobody walks the blocks expecting to find them - which in normal use it probably wouldn't except in the course of a normal 'block to block' read perhaps?
 
Another update to LFSintegrity.
> ack true errors
> clean up side effects of errors
> added MAXFILL to compare differnece of ( myfs.totalSize() - myfs.usedSize() ) to not add to files when space running tight and tend more to delete files that can grow while 26 deep in 28 directories. This allows longer unattended runs to fill disk without overfilling

Added 'trial' updates to RAM2 copy to cascade .begin once more to allow USER to take the default BLOCK size Up or Down to fit use case.
Passes in blockSize=1024 unless over ridden on RAM_DISK over 1MB. Smaller drives use original 256 Bytes.
Few but Large file logging better with 4096, more directory clutter folders+files can be wasteful of prime space.
Code:
class LittleFS_RAM2 : public LittleFS
{
public:
	LittleFS_RAM2() { }
	[B]bool begin(uint32_t size)[/B] {
#if defined(__IMXRT1062__)
		return begin(extmem_malloc(size), size);
#else
		return begin(malloc(size), size);
#endif
	}
	[B]bool begin(void *ptr, uint32_t size)[/B] {
		return begin(extmem_malloc(size), size, 1024 );	
	}
	[B]bool begin(void *ptr, uint32_t size, uint32_t blockSize )[/B] {
		//Serial.println("configure "); delay(5);
		configured = false;
		if (!ptr) return false;
		memset(ptr, 0xFF, size); // always start with blank slate
		size = size & 0xFFFFFF00;
		memset(&lfs, 0, sizeof(lfs));
		memset(&config, 0, sizeof(config));
		config.context = ptr;
		config.read = &static_read;
		config.prog = &static_prog;
		config.erase = &static_erase;
		config.sync = &static_sync;
		if ( size > 1024*1024 ) {
			config.read_size = 256;
			config.prog_size = 256;
			config.block_size = blockSize;
			config.block_count = size / blockSize;
			config.block_cycles = 900;
			config.cache_size = 256;
			config.lookahead_size = (config.block_count / 8 + 7) & 0xFFFFFFF8;
		}
		else {
			config.read_size = 64;
			config.prog_size = 64;
			config.block_size = 256;
			config.block_count = size / 256;
			config.block_cycles = 50;
			config.cache_size = 64;
			config.lookahead_size = 64;
		}
		config.name_max = LFS_NAME_MAX;
		config.file_max = 0;
		config.attr_max = 0;
		configured = true;
		if (lfs_format(&lfs, &config) < 0) return false;
		//Serial.println("formatted");
		if (lfs_mount(&lfs, &config) < 0) return false;
		//Serial.println("mounted atfer format");
		mounted = true;
		checkformat = nullptr;
		[B]Serial.printf("RAM2 BlockSize %lu\n", config.block_size );[/B]
		return true;
	}
private:
	static int static_read(const struct lfs_config *c, lfs_block_t block,
	  lfs_off_t offset, void *buffer, lfs_size_t size) {
		//Serial.printf("    ram rd: block=%d, offset=%d, size=%d\n", block, offset, size);
		uint32_t index = block * c->block_size + offset;
		memcpy(buffer, (uint8_t *)(c->context) + index, size);
		return 0;
	}
	static int static_prog(const struct lfs_config *c, lfs_block_t block,
	  lfs_off_t offset, const void *buffer, lfs_size_t size) {
		//Serial.printf("    ram wr: block=%d, offset=%d, size=%d\n", block, offset, size);
		uint32_t index = block * c->block_size + offset;
		memcpy((uint8_t *)(c->context) + index, buffer, size);
		return 0;
	}
	static int static_erase(const struct lfs_config *c, lfs_block_t block) {
		uint32_t index = block * c->block_size;
		memset((uint8_t *)(c->context) + index, 0xFF, c->block_size);
		return 0;
	}
	static int static_sync(const struct lfs_config *c) {
		return 0;
	}
};
 
Regarding notes about LittleFS formatting and tracking I now have a bitmap of blocks IN USE by LittleFS.

Nothing to publish yet - but after scanning the core little fs code I found a helpful function : int err = lfs_fs_traverse(&lfs, cb_usedBlocks, checkused);

That passes a callback that lfs calls with each block in use.

With prior checkFormat() bitmap tracking of the Already Formatted blocks [ not in use and not needing format before use ]

Now having a list of blocks in use Paul's note about background formatting dirty unused blocks is possible to execute - with added safety checks that lfs doesn't access the system while a block format is in progress - but that isn't a new issue.

So a quick look at the maps for fun's sake - TOP map is prior where 1 bit is a Formatted block and LOWER map it the new list where 1 bit shows the lfs considers the block in use.
Code:
0	0XFFFFFFFC	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
320	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
640	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
960	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0X1FFFFF	0X0	0X0	0X0
1280	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0
1600	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0
1920	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0
2240	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0
2560	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0
2880	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0
3200	0X0	0XFFFC0000	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3520	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3840	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff

 lfs Used Blocks Map: #used is 2079 : lfs traverse return 0 (neg on err)

0	0x3	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0xffe00000	0xffffffff	0xffffffff	0xffffffff
1280	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
1600	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
1920	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2240	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2560	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2880	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3200	0xffffffff	0x3ffff	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3520	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3840	0x0	0x0	0x0	0x0	0x0	0x0	0x0
 lfs Used Blocks Map: #used is 2079

 Done myfs.checkFormat() in 311037 us.

So far the new LittleFS::checkUsed() is just called on the call so far on the prior checkFormat().

Here is the code in case it seems worth looking at for use:
Code:
static int cb_usedBlocks( void *inData, lfs_block_t block )
{
	static lfs_block_t maxBlock;
	static uint32_t totBlock;
	if ( nullptr == inData ) {
		uint32_t totRet = totBlock;
		maxBlock = block;
		totBlock = 0;
		return totRet; // exit after init, end, or bad call
	}
	totBlock++;
	if ( block > maxBlock ) return block; // this is beyond media blocks
	uint32_t iiblk = block/8;
	uint8_t jjbit = 1<<(block%8);
	uint8_t *myData = inData;
	myData[iiblk] = myData[iiblk] | jjbit;
	return 0;
}

FLASHMEM
bool LittleFS::checkUsed(bool readCheck) {
	if ( !configured || !readCheck ) {
		if ( checkused != nullptr) {
			free( checkused );
		}
		checkused = nullptr;
	}
	else if ( readCheck ) {
		cb_usedBlocks( nullptr, config.block_count ); // init and pass block_count
		uint32_t iiblk = 1+(config.block_count /8);
		if ( checkused == nullptr) checkused = (uint8_t *)malloc( iiblk );
		if ( checkused == nullptr) return (checkused != nullptr);
		memset(checkused, 0, iiblk);

		int err = lfs_fs_traverse(&lfs, cb_usedBlocks, checkused);
		uint32_t totBlock = cb_usedBlocks( nullptr, config.block_count ); // get total blocks
#ifdef DEBUGCF
		Serial.printf( "\n\n lfs Used Blocks Map: #used is %lu : lfs traverse return %i (neg on err)\n", totBlock, err );
		uint32_t *fake = (uint32_t *)checkused;
		int ff=0;
		for (unsigned int block=0; block < config.block_count; block++) {
			if ( (block/8)>3 && (0==(block/8)%4) && 0==block%8 ) Serial.printf( "\t0x%lx", fake[ff++]); // bugbug DEBUG
			if ( !((block/8)%40) && 0==block%8 ) Serial.printf( "\n%u", block ); // bugbug DEBUG
		}
		Serial.printf( "\n lfs Used Blocks Map: #used is %lu\n", totBlock );
#endif
		if ( err < 0 )
		{
			if ( checkused != nullptr)
				free( checkused );
			checkused = nullptr;
		}
	}
	return checkused != nullptr;
}

Note:
> XOR of those two blocks should never show a 1
> Invert of the second USED and XOR'd with 2nd list of pre-Formatted would show blocks ( ~in_use & not_formatted )
> A WWW search shows this lfs_fs_traverse() was in common use to find the media free space, before they introduced lfs_fs_size for that.
> Code search shows the lookahead is a group of bytes they fill with a similar map to find free space for allocations
> @mjs513 noted somewhere that SPIFFS is being deprecatated - I saw that today as the SPIFFS coders have gone idle, whereas the LiltleFS is active
> as before for SPIFFS - there are python tools for ESP (?) - to preload a disk image onto a device from within the IDE

their notes:
Code:
// Finds the current size of the filesystem
//
// Note: Result is best effort. If files share COW structures, the returned
// size may be larger than the filesystem actually is.
//
// Returns the number of allocated blocks, or a negative error code on failure.
lfs_ssize_t lfs_fs_size(lfs_t *lfs);

// Traverse through all blocks in use by the filesystem
//
// The provided callback will be called with each block address that is
// currently in use by the filesystem. This can be used to determine which
// blocks are in use or how much of the storage is available.
//
// Returns a negative error code on failure.
int lfs_fs_traverse(lfs_t *lfs, int (*cb)(void*, lfs_block_t), void *data);

    // Size of the lookahead buffer in bytes. A larger lookahead buffer
    // increases the number of blocks found during an allocation pass. The
    // lookahead buffer is stored as a compact bitmap, so each byte of RAM
    // can track 8 blocks. Must be a multiple of 8.
    lfs_size_t lookahead_size;
 
Last edited:
Hi,
Could this file system be made to operate with multiple flash chips on SPI, each with individual CS pins? That would be very useful for some audio applications.
 
Could this file system be made to operate with multiple flash chips on SPI, each with individual CS pins?

Yes. It already is designed to operate this way!

Just create multiple LittleFS_SPIFlash instances, one for each chip. For example:

Code:
LittleFS_SPIFlash chip1;
LittleFS_SPIFlash chip2;
LittleFS_SPIFlash extrachip;

Then in setup(), call each instance's begin() function with the CS pin number, and optionally the SPI port if you have connected them to SPI1 or SPI2 rather than the main SPI.

Code:
void setup() {
  chip1.begin(10);
  chip2.begin(6);
  extrachip.begin(31, SPI1);

Currently LittleFS_QSPIFlash supports only a single instance, since there is only 1 place on the bottom of Teensy 4.1 to add a QSPI flash chip. While theoretically (with very difficult soldering) a flash chip could also be added to the smaller pads, we currently only support PSRAM on those pads.
 
@Paul - can you give a quick note on p#481 development - right direction worth some cleanup? And update to defragster github for further review/test?
> on 16MB QSPI the LittleFS : Traverse takes 320us to ~15ms to get the used disk space depending on how full the disk is.

@AlanK - And indeed as far as Multiple LittleFS drives - it has been tested on the MTP interface where 4,5,6? mounted drives are presented over USB to Host computer for I/O. It isn't part of the single disk testing like LFSintegrity because they are independent - but do work together as seen with MTP.
 
Not something for TD 1.54 ? - but want to do some garbage collection on discarded LittleFS flash blocks in 'spare time' now with that time not needed during a write?

Just call with the number of blocks to free up:
Code:
uint32_t LittleFS::formatUnused(uint32_t blockCnt) {
	checkUsed( true );
	checkFormat( true );
	if ( nullptr == checkformat || nullptr == checkused ) return 0;
	uint32_t ii=0, jj=0, tt;
	uint8_t bb;
	uint8_t *lfsUsed = checkused; // 1 bits are lfs Used
	uint8_t *preFmt = checkformat; // 1 bits are pre Formatted block
	while ( ii<config.block_count && jj<blockCnt ) {
		if ( 0==(ii%8) ) bb=1;
		tt = ii/8;
		if ( !(lfsUsed[tt] & bb ) && !(preFmt[tt] & bb) ) { // block not already formatted or used
			(*config.erase)(&config, ii);
			preFmt[tt] |= bb; // mark as pre-formatted
			jj++;
		}
		ii++;
		bb = bb<<1;
	}
	return jj;
}

To make code look cool I pulled the debug prints but they put out lines like this in LFSintegrity - where a set of 15 Blocks is formatted in 541.807 ms - but that might be 389.491 ms (26 ms /block) with full cache integrity and trust.
Code:
 UnUsed Block 253 formatted. # 1 of 15	  lfsU=0x0 preF=0x1f (after0x3f)
 UnUsed Block 254 formatted. # 2 of 15	  lfsU=0x0 preF=0x3f (after0x7f)
 UnUsed Block 255 formatted. # 3 of 15	  lfsU=0x0 preF=0x7f (after0xff)
 UnUsed Block 256 formatted. # 4 of 15	  lfsU=0x0 preF=0x0 (after0x1)
 UnUsed Block 257 formatted. # 5 of 15	  lfsU=0x0 preF=0x1 (after0x3)
 UnUsed Block 258 formatted. # 6 of 15	  lfsU=0x0 preF=0x3 (after0x7)
 UnUsed Block 259 formatted. # 7 of 15	  lfsU=0x0 preF=0x7 (after0xf)
 UnUsed Block 260 formatted. # 8 of 15	  lfsU=0x0 preF=0xf (after0x1f)
 UnUsed Block 261 formatted. # 9 of 15	  lfsU=0x0 preF=0x1f (after0x3f)
 UnUsed Block 262 formatted. # 10 of 15	  lfsU=0x0 preF=0x3f (after0x7f)
 UnUsed Block 263 formatted. # 11 of 15	  lfsU=0x0 preF=0x7f (after0xff)
 UnUsed Block 264 formatted. # 12 of 15	  lfsU=0x0 preF=0x0 (after0x1)
 UnUsed Block 265 formatted. # 13 of 15	  lfsU=0x0 preF=0x1 (after0x3)
 UnUsed Block 266 formatted. # 14 of 15	  lfsU=0x0 preF=0x3 (after0x7)
 UnUsed Block 267 formatted. # 15 of 15	  lfsU=0x0 preF=0x7 (after0xf)
 formatUnused Block formatted :: # 15 of 15

	 Done Formatting Low Level in 541807 us.

[  7.51 M](0.00903 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZt+-yY? loops left 0 >
 No lfschange could have skipped ???

 checkUsed:: lfs Used Blocks Map: #used is 117 : lfs traverse return 0 (neg on err)

0	0x3	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0xffffc000	0xffffffff	0xffffffff	0xffffffff	0x1	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1280	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1600	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1920	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2240	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2560	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2880	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3200	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3520	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3840	0x0	0x0	0x0	0x0	0x0	0x0	0x0
 checkUsed:: lfs Used Blocks Map: #used is 117

 lfs Used&Format overlap errs?: {eol}

 Done myfs.checkUsed() in 2400 us.

[ 14.12 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZt+-yY? loops left 0 >

 checkFormat:: Pre-Formatted Blocks Map

0	0XFFFFFFFC	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0XFFF	0X0
320	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0
640	0X0	0X0	0X0	0X0	0X0	0XFFFFFFFE	0xffffffff	0xffffffff	0xffffffff	0xffffffff
960	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
1280	0xffffffff	0X7FF	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0
1600	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0
1920	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0
2240	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0
2560	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0
2880	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0
3200	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0
3520	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0	0X0
3840	0X0	0X0	0X0	0X0	0X0	0X0	0X0
 Done myfs.checkFormat() in 136037 us.
 
Last edited:
@Paul - update to using maintained BitMap to skip formatting blocks before use when they are known to be formatted.
>> Today's improvement is using the LFS reported used block bitmap to skip calling blockIsBlank() to check as they are known dirty and known to be left alone.

Now working is :: myfs.formatUnused( 33000 ); // Runtime LLformat of ONLY unused and dirty blocks.
> it can be any smaller number where myfs.formatUnused( 4 ) would format 4 blocks for example in about 104 ms - iff there are 4 dirty blocks.

To test I wrote 33,000 for 'f'ormat because the sketch doesn't know how many blocks are on the disk, it will stop at that number - or when no more UNUSED blocks are found unformatted
> 16 MB Flash with 4096 byte blocks has 4096
> 16 MB RAM with 512 Byte blocks has 32767 of them ( @KurtE - this goes to my suggestion of selective sizing based on RAM size provided - Any feedback? - this is wasteful/wrong )

So bottom line using LFSintegrity on 16MB with bigfile() creates an 8MB file then run again creates a file of 4MB.
> Deleting the files then running :: myfs.formatUnused( 33000 ); takes 80 seconds to format those Disk Blocks, but leaves the file system intact and with other allocated files VALID
> Running a 'F' full LLformat Totally wipes the disk of those 2 bigfile()'s (and the FS) and takes 80 seconds - leaving a formatted media and a new FS.

So with the bitmaps to reference ( what blocks are unformatted ) and ( what blocks are not IN USE ) the code here can selectively format as need the unused blocks at runtime.
> This uses the existing blockIsBlank() code to verify all blocks not reported in use by the LFS Traverse() code.
> When moving to alternate 'BLOCK' allocations on larger disks NOR|NAND I assume this function or similar will be used to identify and prevent reFormatting? That would need to be accounted for in this code then as well.

And when the unused blocks are kept formatted - and the bitmap is used to see it as formatted the PJRC code can return to LFS for immediate writing without testing or formatting for faster writes.
> This doesn't matter when disk is used for static storage
> But write throughput is 5-10 times faster on any active logging use when the media blocks are kept formatted for use, during 'downtime'

For 4096 blocks each bitmap is malloc() of 512 bytes. Only allocated when activated.

The blockIsBlank() calls to fill the one bitmap are the same as when called before current format code. And the bit is unset for that block assuming LFS is putting it to use, as it does.

Calling the lfs traverse code to create the IN USE bitmap can take 2-10(?) ms to fill that bitmap depending on how many disk blocks are IN USE.

Having the IN USE bitmap is faster with fewer calls to blockIsBlank() when IN USE blocks do not need to be read or considered for formatting.

Knowing a block is already formatted by bitcheck, and having it pre-formatted speeds the return to lfs writing.

Having this info could allow background no wait formatting - but I'm not sure how that would work if I/O to the same media could occur.

Just like it isn't yet known what happens if yield() during FLASH wait() code allows user EVENT code to attempt a LittleFS media access?

This code does allow any logging sketch to pre-format to speed disk writes significantly when that speed is a usability factor.

Except for verification and cleanup this is a completed effort to do these things.
 
Last edited:
Evening all
Just soldered up a W25Q512JV chip to a breakout board and using SPI as opposed to QSPI. Running @defragster's BigFileTest sketch:
Code:
Started SPI_DISK
Bytes Used: 8192, Bytes Total:67108864
Start Big write of 33528000 Bytes

Big write took 466.31 Sec for 33528000 Bytes : file3.size()=33528000
	Big write KBytes per second 71.90 
------------------------------------------------------------
	myfs.quickFormat() Started
		myfs.quickFormat() Completed
Bytes Used: 8192, Bytes Total:67108864

Big write took 461.13 Sec for 33528000 Bytes : file3.size()=33528000
	Big write KBytes per second 72.71

PS> looks like 30Mhz is max clock before LittleFS errors out on the begin.

EDIT: no more T4.1s available to test QSPI.
 
Not seeing any tests of the 64MB WSON QSPI or SPI since Paul added that chip to the list?

QSPI 64MB Flash :: ( WSON 8 ) on a fresh T_4.1 it worked. Only 432 KB/sec even with checking preFormat to skip that - that chip was formatted out of the box. With normal inline Format write speed 84.6 KB/sec.
>> First bigfile() write on factory formatted media – using the no PreFormat code:
Big write /0_bigfile.txt took 77.58 Sec for 33527808 Bytes : file3.size()=33527808
Big write KBytes per second 432.17

>> Doing a quick format and then creating the same file with default format on write:
Big write /0_bigfile.txt took 396.37 Sec for 33527808 Bytes : file3.size()=33527808
Big write KBytes per second 84.59

Reformat of the half used media takes 320 seconds.

SPI 64MB FLASH :: Put another 64MB on that Audio board that prior post showed FLASH FAIL after removing the old 16MB.
> Wow just checking the Formatted media for Format with the Blank Check takes 20.6 seconds - that is effectively a FULL 'low level direct' 64MB READ of the chip at 3.10 MB/sec.
-- > Going to have to edit format bitmap fill for fast format to just read 8 blocks at a time
>> First bigfile() write on factory formatted media – using the no PreFormat code:
Big write /0_bigfile.txt took 94.53 Sec for 33527808 Bytes : file3.size()=33527808
Big write KBytes per second 354.68
>> Reading that file back to verify on delete :: Big read&compare KBytes per second 1223.60

>> Doing the delete to empty and then creating the same file with default format on write: {good I print dots on writing - this takes a long time - soo many dots!}
Big write /0_bigfile.txt took 448.65 Sec for 33527808 Bytes : file3.size()=33527808
Big write KBytes per second 74.73

As currently configured? The 64MB is slower, and the SPI isn't as much slower as QSPI as the smaller chip. But writing without (uneeded) format is still ~5 times faster.

Started an SPI 'f'ast format on live disk takes ... will update after it finishes ... something over 10 minutes based on QSPI time above.
Only half the media was unformatted ( bigfile was not deleted ) - 5.62 minutes { cool the elap time is RIGHT! }
Code:
 formatUnused Block formatted :: # 8156 of 16384

	 formatUnused :: Done Formatting Low Level in 337237812 us.

[ 27.04 M](5.62063 M elap) Awaiting input 0123456789RdchkFqvplmuxBbZztyYf+-? loops left 0 > d
printDirectory SPI_DISK
--------------
FILE	0_bigfile.txt		33527808
FILE	B_file.txt		3072
FILE	C_file.txt		4096
FILE	D_file.txt		5120
FILE	E_file.txt		6144
FILE	F_file.txt		7168
FILE	G_file.txt		8192
FILE	H_file.txt		9216
FILE	I_file.txt		10240
FILE	J_file.txt		11264
FILE	K_file.txt		12288

 0 dirs with 11 files of Size 33604608 Bytes
 Total 11 files of Size 33604608 Bytes
Bytes Used: 33701888, Bytes Total:67108864

And after the 'f'ast bitmap selective disk online format the bigfile verified and deleted okay and the LFSintegrity iterations run fine where each add or delete do a read verify of the file contents.

I need to cleanup the edits and put on github with the latest LFS integrity.

NOTE: Soldering the WSON 8 onto the 8 pin pads went clean and easy - but chip has a big center underside pad. Following a suggestion by @KenHahn (protosupplies.com) I covered that pad on these two 64MB chips with Kapton tape before placing and soldering. It is thin enough that it didn't interfere with soldering
 
Last edited:
Evening all
Just soldered up a W25Q512JV chip to a breakout board and using SPI as opposed to QSPI. Running @defragster's BigFileTest sketch:
...
PS> looks like 30Mhz is max clock before LittleFS errors out on the begin.

EDIT: no more T4.1s available to test QSPI.

Bummer it doesn't have more headroom for QSPI speed :( Something odd the SPI so close to QSPI ???

Also sad not to have spare T_4.1's. I added 2 more on that order from @KenHahn - and still have some in wrappers. Time to order more @mjs513 ... if only they had been BFri special.

Apparently I did (2) with 1GB NAND - found them and marked them for future use.
 
Updated LFSintegrity on github

It should compile as uploaded with line #3 :: #define RELEASE // Use this to exclude defragster new functions not in Beta5

It is all it was before but multiple 'B'igfile half of remaining space to a file [0-8], and 'b' deletes them all. They are read verified before deletion.
> If the directory integrity is maintained all files add/del should be accounted for in static counter to match what DIR shows ( when a DIR is done or the 'l' command )
> When doing a long run - if new file 'iteration' would not leave "#define MAXFILL 66000" space the file add is skipped and it counts a warning but should continue until files get deleted.

I'll review my LittleFS bitmap src changes and post tomorrow on that repostitory. Until then RAM2 won't work and RELEASE must be defined ( tried to catch the 5 new added commands that would break )

The added breaking CLASS commands are still there just 'gutless' to compile and the '?' HELP denoted then with a "__" : double underbar - they still print - just no action.
 
W25Q512JV LittleFS/MTP Transfer File Test

Will have to brush off and maybe test with @defragster's LFSIntegrity sketch. But just loaded up my MTP test sketch and just testes with various file sizes.

Seems to suffer from the issue that we discussed before. Any file transfer over about 370k basically crashes MTP - note think I may have missed the latest update.

Some Basic Testing
Using one of test LittleFS test sketches from way back when that creates several files and directories and then run's Pauls PrintDirectory test worked with out an issue. Note I did update SD, LittleFS and USBHost_t36 before the testing (note probably should have updated the core as well):
Code:
D:\Users\Merli\Documents\Arduino\LittleFS tests\littlefs_teensy_test3\littlefs_teensy_test3.ino Dec  3 2020 06:40:01
LittleFS Timer Test
printDirectory
--------------
LIDAR.csv		75599
PRINTOUTPUT1.txt		628
PRINTOUTPUT2.txt		630
Sensor-Calibration-Procedure-v11.pdf		0
file1.txt		32
file10.txt		16
file2.txt		32
file20.txt		16
file3.txt		32
file30.txt		16
ili9341_simpletest.py		1598
mtpindex.dat		0
structuredData / 
	logger.txt		480
test1 / 
	file1.txt		16
test2 / 
	file2.txt		16
test3 / 
	file3.txt		16

Using println and printf to printoutput file
opened PRINTOUTPUT1.txt
opened PRINTOUTPUT2.txt
printDirectory
--------------
LIDAR.csv		75599
PRINTOUTPUT1.txt		1256
PRINTOUTPUT2.txt		1260
Sensor-Calibration-Procedure-v11.pdf		0
file1.txt		32
file10.txt		16
file2.txt		32
file20.txt		16
file3.txt		32
file30.txt		16
ili9341_simpletest.py		1598
mtpindex.dat		0
structuredData / 
	logger.txt		480
test1 / 
	file1.txt		16
test2 / 
	file2.txt		16
test3 / 
	file3.txt		16

-------------------------------
File1 contents:
-------------------------------
File1 size: 1256
abcdefghijklmnopqrstuvwxyz
Rec: 0, Float: 26.400000, Int: 98
abcdefghijklmnopqrstuvwxyz
Rec: 1, Float: 27.400000, Int: 99
abcdefghijklmnopqrstuvwxyz
Rec: 2, Float: 28.400000, Int: 100
abcdefghijklmnopqrstuvwxyz
Rec: 3, Float: 29.400000, Int: 101
abcde
ghijklmnopqrstuvwxyz
Rec: 4, Float: 30.400000, Int: 102
abcdefghijklmnopqrstuvwxyz
Rec: 5, Float: 31.400000, Int: 103
abcdefghijklmnopqrstuvwxyz
Rec: 6, Float: 32.400000, Int: 104
abcdefghijklmnopqrstuvwxyz
Rec: 7, Float: 33.400000, Int: 105
abcdefghi
klmnopqrstuvwxyz
Rec: 8, Float: 34.400000, Int: 106
abcdefghijklmnopqrstuvwxyz
Rec: 9, Float: 35.400000, Int: 107
abcdefghijklmnopqrstuvwxyz
Rec: 0, Float: 26.400000, Int: 98
abcdefghijklmnopqrstuvwxyz
Rec: 1, Float: 27.400000, Int: 99
abcdefghijklmno
qrstuvwxyz
Rec: 2, Float: 28.400000, Int: 100
abcdefghijklmnopqrstuvwxyz
Rec: 3, Float: 29.400000, Int: 101
abcdefghijklmnopqrstuvwxyz
Rec: 4, Float: 30.400000, Int: 102
abcdefghijklmnopqrstuvwxyz
Rec: 5, Float: 31.400000, Int: 103
abcdefghijklmnopqrs
uvwxyz
Rec: 6, Float: 32.400000, Int: 104
abcdefghijklmnopqrstuvwxyz
Rec: 7, Float: 33.400000, Int: 105
abcdefghijklmnopqrstuvwxyz
Rec: 8, Float: 34.400000, Int: 106
abcdefghijklmnopqrstuvwxyz
Rec: 9, Float: 35.400000, Int: 107

-------------------------------
File3 byte conversion test:
-------------------------------

Data_0: true
Data_1: 1.3574999571
Data_2: 314159
Data_3: 142
19
The Quick Brown Fox
Init Done - array loaded
...... ...... ......
create folder
  failed


2nd Directory contents:
printDirectory
--------------
LIDAR.csv		75599
PRINTOUTPUT1.txt		1256
PRINTOUTPUT2.txt		1260
Sensor-Calibration-Procedure-v11.pdf		0
file1.txt		32
file10.txt		16
file2.txt		32
file20.txt		16
file3.txt		32
file30.txt		16
ili9341_simpletest.py		1598
mtpindex.dat		0
structuredData / 
	logger.txt		960
test1 / 
	file1.txt		16
test2 / 
	file2.txt		16
test3 / 
	file3.txt		16


true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox

printDirectory
--------------
LIDAR.csv		75599
PRINTOUTPUT1.txt		1256
PRINTOUTPUT2.txt		1260
Sensor-Calibration-Procedure-v11.pdf		0
file1.txt		64
file10.txt		32
file2.txt		64
file20.txt		32
file3.txt		64
file30.txt		32
ili9341_simpletest.py		1598
mtpindex.dat		0
structuredData / 
	logger.txt		960
test1 / 
	file1.txt		16
test2 / 
	file2.txt		16
test3 / 
	file3.txt		16

printDirectory
--------------
LIDAR.csv		75599
PRINTOUTPUT1.txt		1256
PRINTOUTPUT2.txt		1260
Sensor-Calibration-Procedure-v11.pdf		0
file1.txt		64
file10.txt		32
file2.txt		64
file20.txt		32
file3.txt		64
file30.txt		32
ili9341_simpletest.py		1598
mtpindex.dat		0
structuredData / 
	logger.txt		960
test1 / 
	file1.txt		32
test2 / 
	file2.txt		32
test3 / 
	file3.txt		32

printDirectory
--------------
LIDAR.csv		75599
PRINTOUTPUT1.txt		1256
PRINTOUTPUT2.txt		1260
Sensor-Calibration-Procedure-v11.pdf		0
file1.txt		64
file10.txt		32
file2.txt		64
file20.txt		32
file3.txt		64
file30.txt		32
ili9341_simpletest.py		1598
mtpindex.dat		0
structuredData / 
	logger.txt		960
test1 / 
	file1.txt		32
test2 / 
	file2.txt		32
test3 / 
	file3.txt		32

Disk Usuage:
Bytes Used: 135168, Bytes Total:67108864
run 1, errors 0
printDirectory
--------------
LIDAR.csv		75599
PRINTOUTPUT1.txt		1256
PRINTOUTPUT2.txt		1260
Sensor-Calibration-Procedure-v11.pdf		0
file1.txt		64
file10.txt		32
file2.txt		64
file20.txt		32
file3.txt		64
file30.txt		32
ili9341_simpletest.py		1598
mtpindex.dat		0
structuredData / 
	logger.txt		960
test1 / 
	file1.txt		32
test2 / 
	file2.txt		32
test3 / 
	file3.txt		32
Going back to MTP to see if all the files are there, the answer is yes. Teensy does show the Winbond chip as a64MB drive.
Here is a screen capture for reference:
Capture.PNG
 
Latest CORE Updates + New Version of MTP LittleFS Testing

Ok folks just a heads up. Downloaded the latest and greatest changes to the Core and MTP.

While LittleFS still works new version of MTP fails - Teensy does not show up as a Portable device.
1. Reverting changes to MTP does not help Teensy does not show up.
2. Reverting changes to CORE and using updated version of MTP works to show Teensy as a portable device.

Something in the latest changes to the Core seems to break MTP.
 
Thanks @mjs513 (and @defragster) I thought it was just me, when I last tried MTP.

My gut tells me, that some of the more recent changes to core included, I think migrating in some of the stuff that was in USB2 library. Was unclear with the status and I did not notice any new messages on the MTP threads, so was waiting to see if there were some messages. Things like, with the new stuff do we still need a hacked up usb_desc.h file to allow us to also still use Serial. Is there a new device ID for MTP and Serial? ...

I will try to take a look. Just been having too much fun playing with camera and shooting self in foot with CS pins ;)

Yesterdays mailbox run did include a package from Digikey with W25Q512JVEIM chips... I have not done anything with it yet. Not sure if to wait for my breakout boards from OSHPark (which is out to fabrication), and then try it with T4.1 board with bottom pads broke out... Or use up my last T4.1 or...

Probably at some point need to order a few more.
 
@KurtE

Agree that it was probably some change to USB stuff in the core for MTP but didn't look at it - had to get some more sleep. Think I may know what the problem may be. Wondering if you have do an include usb_mtp.h even though its in the core? Not sure didn't dig far enough yet.

Yeah - probably have to order a few more T4.1's as well.
 
Obviously distractions here are just filling this thread - not looked at MTP - or updated cores.

@KurtE and mjs513 - did you get email noting physical mail may be coming with some parts? Maybe wait a bit given T_4.1 shortages there. My Black Friday pack to arrive today from PJRC - and it has some added parts I'll see in ~2 hours.

The bitmap work in LittleFS looks like a path to some optimized usage on FLASH writes. It has overhead - nothing is free the FLASH is slow to format - it just does upfront PREP work so that 'run time' writes to new blocks are not ALL held up by a request to format the block.

Even when the Format is not needed to read the block to see that :: Format with the Blank Check takes 20.6 seconds - that is effectively a FULL 'low level direct' 64MB READ of the chip at 3.10 MB/sec.

Getting that 20 seconds out of the way to fill the bitmap saves 20/16384 seconds per block - and if the block is Preformatted saves another 25 ms per block. On 64MB Flash there are 16384 blocks of 4K - so the bitmap grows to 2K.

As noted from p#489 the low level block read to check formatting is 3.1 MB/sec. When a 32MB 'b'igfile file is deleted the LFS read ( byte read with compare ) only gets 1.095 MB/sec. So this larger Flash seems less efficient for LFS - perhaps it need more Cache and lookahead as well.

On current disk image the Traverse of used LFS blocks takes 27.4 ms to fill the bitmap used. So even the effort to speed things up is taking more time and more memory :(

PRE format - having to REFILL the bitmaps - takes 10.3 seconds for ONE BLOCK and given the same bitmap overhead to PRE format 17 blocks takes only 10.98 seconds.

With a bit more WORK to integrate into the FORMAT to track LFS changes versus others the bitmaps could be trusted and not discarded and rebuilt on each call.
 
Just figured that I would give a little status on Flash Chips. Received my Cyprus 8Mbit Fram and put it on a breakout board as usual. In prep for trying to incorporate it into LittleFS I threw together a little sketch to test it as standalone:
Code:
#include <SPI.h>

#define SPICONFIG SPISettings(30000000, MSBFIRST, SPI_MODE0)
uint8_t pin = 6;

  uint8_t xData[2048], buffer[2048];

void setup() {
  pinMode(pin, OUTPUT);
  
  Serial.begin(115000);
  delay(3000);
  SPI.begin();
  
  GetJedecID();

  readStatusRegister(0x05, false);


  uint32_t address = 0;

  writeEnable(true);
  
  bool status = readStatusRegister(0x05, false);
  Serial.println("Write Enabled");

  
  memset(buffer, 0xff, 2048);
  for (uint16_t i = 0; i < 32; i++) buffer[i] =  2;
  writeData(address, buffer, 32);

    writeEnable(false);
  delay(100);

  readData(address, buffer, 32);

 Serial.println();
  for (uint16_t j = 0; j < 12; j++) {
    for (uint16_t i = 0; i < 16; i++) {
      Serial.printf("0x%02x, ", buffer[j * 16 + i]);
    } Serial.println();
  }

}

void loop() {
  // put your main code here, to run repeatedly:

}

// JEDEC ID is available even if chip is busy, so no need to stall
void GetJedecID(void) {
 uint8_t data[9];
 
  SPI.beginTransaction(SPICONFIG);
    digitalWrite(pin, LOW);
    SPI.transfer(0x9f);  //0x9f - JEDEC register
    for(uint8_t i = 0; i<8; i++)
      data[i] = SPI.transfer(0);
    digitalWrite(pin, HIGH); // Chip deselect
    SPI.endTransaction();
    for(uint8_t i = 0; i<3; i++) Serial.println(data[i], HEX);
    
 }

bool writeEnable(bool enable) {
   uint8_t status;
  if(enable == true) {
    SPI.beginTransaction(SPICONFIG);
    digitalWrite(pin, LOW);
    SPI.transfer(0x06);
    digitalWrite(pin, HIGH); // Chip deselect
    SPI.endTransaction();
  } else {
    SPI.beginTransaction(SPICONFIG);
    digitalWrite(pin, LOW);
    SPI.transfer(0x04);
    digitalWrite(pin, HIGH); // Chip deselect
    SPI.endTransaction();
  }
  
  status = readStatusRegister(0x05, true);
  Serial.print("Status WEL: ");  Serial.println(status, BIN);
  Serial.print("Wel Bit Enabled:  "); Serial.println(status & (1<<1));
  if(status & (1<<1) != 2) return false;
  return status & (0x02);
 }

 uint8_t readStatusRegister(uint16_t reg, bool dump)
{
  uint8_t val;
    SPI.beginTransaction(SPICONFIG);
    digitalWrite(pin, LOW);
    SPI.transfer(0x05);  //0x05 - read status register
    SPI.transfer(reg);
    val = SPI.transfer(0x00);
    digitalWrite(pin, HIGH); // Chip deselect
    SPI.endTransaction();

  if(dump) {
    Serial.printf("Status of reg 0x%x: \n", reg);
    Serial.printf("(HEX: ) 0x%02X, (Binary: )", val);
    Serial.println(val, BIN);
    Serial.println();
  }
  
  return val;
}


void writeData(uint32_t address, uint8_t *data, uint32_t length)
{

    SPI.beginTransaction(SPICONFIG);
    digitalWrite(pin, LOW);
    SPI.transfer(0x02);  //0x05 - read status register
   SPI.transfer( (address >> 16) & 0xFF);
   SPI.transfer( (address >> 8) & 0xFF);
   SPI.transfer( (address >> 0) & 0xFF);
    SPI.transfer(data, length);
    digitalWrite(pin, HIGH); // Chip deselect
    SPI.endTransaction();

    writeEnable(false);

}


bool readData(uint32_t address, uint8_t *data, uint32_t length)
{
    SPI.beginTransaction(SPICONFIG);
    digitalWrite(pin, LOW);
    SPI.transfer(0x03);  //0x05 - read status register
   SPI.transfer( (address >> 16) & 0xFF);
   SPI.transfer( (address >> 8) & 0xFF);
   SPI.transfer( (address >> 0) & 0xFF);
   
  for(uint32_t i = 0; i < length; i++) {
    data[i] = SPI.transfer(0);
  }  
    digitalWrite(pin, HIGH); // Chip deselect
    SPI.endTransaction();

    
}
Not pretty but it works if I turn power off and on again:
Code:
3
2E
C2
Status of reg 0x5: 
(HEX: ) 0x42, (Binary: )1000010

Status WEL: 1000010
Wel Bit Enabled:  2
Write Enabled
Status of reg 0x5: 
(HEX: ) 0x40, (Binary: )1000000

Status WEL: 1000000
Wel Bit Enabled:  0
Status of reg 0x5: 
(HEX: ) 0x40, (Binary: )1000000

Status WEL: 1000000
Wel Bit Enabled:  0

0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 
0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 0x02, 
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 
0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff,
First 3 bytes that are printed is the JEDEC for this chip.

Now guess to see if I can get it to work in LittleFS :)
 
@mjs513 - maybe not news? Just clone RAM as a start in LittleFS.h - AFAIK?

Renamed - and the outdented config. lines are the ones I mucked with for RAM2 to get fewer and larger BLOCKS
The begin( size ) needs resolved based on chip at hand ( like the FLASH do ) ?

Add in your code for READ and PROG ( write ) ?
And the erase will need to put those bits to the media - not just RAM

The other FLASH ones are blown up in LittleFS.cpp - but this may be a simpler and valid start? ... if you didn't already see it ...

Code:
class LittleFS_MRAM : public LittleFS
{
public:
	LittleFS_MRAM() { }
	bool begin(uint32_t size) {
#if defined(__IMXRT1062__)
		return begin(extmem_malloc(size), size);
#else
		return begin(malloc(size), size);
#endif
	}
	bool begin(void *ptr, uint32_t size) {
		//Serial.println("configure "); delay(5);
		configured = false;
		if (!ptr) return false;
		memset(ptr, 0xFF, size); // always start with blank slate
		size = size & 0xFFFFFF00;
		memset(&lfs, 0, sizeof(lfs));
		memset(&config, 0, sizeof(config));
		config.context = ptr;
		config.read = &static_read;
		config.prog = &static_prog;
		config.erase = &static_erase;
		config.sync = &static_sync;
			config.read_size = 256;
			config.prog_size = 256;
			config.block_size = 4096;
			config.block_count = size / 4096;
			config.block_cycles = 900;
			config.cache_size = 256;
			config.lookahead_size = 512;
		config.name_max = LFS_NAME_MAX;
		config.file_max = 0;
		config.attr_max = 0;
		configured = true;
		if (lfs_format(&lfs, &config) < 0) return false;
		//Serial.println("formatted");
		if (lfs_mount(&lfs, &config) < 0) return false;
		//Serial.println("mounted atfer format");
		mounted = true;
		checkformat = nullptr;
		checkused = nullptr;
		lfschange = true;
		return true;
	}
private:
	static int static_read(const struct lfs_config *c, lfs_block_t block,
	  lfs_off_t offset, void *buffer, lfs_size_t size) {
		//Serial.printf("    ram rd: block=%d, offset=%d, size=%d\n", block, offset, size);
		uint32_t index = block * c->block_size + offset;
		memcpy(buffer, (uint8_t *)(c->context) + index, size);
		return 0;
	}
	static int static_prog(const struct lfs_config *c, lfs_block_t block,
	  lfs_off_t offset, const void *buffer, lfs_size_t size) {
		//Serial.printf("    ram wr: block=%d, offset=%d, size=%d\n", block, offset, size);
		uint32_t index = block * c->block_size + offset;
		memcpy((uint8_t *)(c->context) + index, buffer, size);
		return 0;
	}
	static int static_erase(const struct lfs_config *c, lfs_block_t block) {
		uint32_t index = block * c->block_size;
		memset((uint8_t *)(c->context) + index, 0xFF, c->block_size);
		return 0;
	}
	static int static_sync(const struct lfs_config *c) {
		return 0;
	}
};
 
As a follow-up to using the Cypress FRAM chip. After some more testing its just not working properly. Appears that SPI implementation that I posted works once in a while. Not sure what I am doing or what pullups or caps I need to add.

If anyone has any ideas let me know.

Thanks
 
Back
Top