LittleFS port to Teensy/SPIFlash

@KurtE
Just got back from some errands. Can't see anything wrong from the pictures to be honest or it just could be my eyes. I just added and tested basically a complete wipe of the QSPI disk based on Frank's stuff to my version of littleFS. With that it completely wipes the QSPI flash chip. After begin all I call is myfs.initialize(); then it does it thing and format's it for the FS like it does now. I am attaching if you want to give it a try. If Paul likes it I can submit a PR.
 

Attachments

  • src.zip
    8.2 KB · Views: 47
Also had to use FILE_WRITE when opening the file. SDFat used file.open("bench.dat", O_RDWR | O_CREAT | O_TRUNC).

reason is SD defines mode in open as uint8_t, while SDFAT expects 32 bit.
This prohibits the use of O_ type modes.
 
reason is SD defines mode in open as uint8_t, while SDFAT expects 32 bit.
This prohibits the use of O_ type modes.

Yep - think we touched on this issue earlier in the thread as well. Was just documenting the changes I made from the SDFat benchmark sketch
 
@KurtE
Think we are cross-posting now with these edits?

Check the benchmark sketch is configure for QSPI not Program. The sketch as posted is configured for Program.

The results you are getting is what I am seeing when testing LittleFS_Program so that may be the issue

I ran the updated format, and then edited the program to use QSPI both at top and begin...

And now: :D
Code:
LittleFS Test
QSPI flash begin
Flash ID: EF 40 18
Flash size is 16.00 Mbyte
attempting to mount existing media
success
started
FILE_SIZE_MB = 5
BUF_SIZE = 512 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
78.77,51849,834,6499
78.58,55570,836,6515

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
21737.74,129,14,23
21643.64,130,14,23

Done
Now to see about if I can see it on the PC... And copy files to it.
 
Really fun getting here in AM some days with 2-3 folks East Coast and Early risers posting after I logged off :)

Fixing no #ifdef class .format() ...

@KurtE - the 'winbond' font looks funny - maybe normal - suppose it was from digikey and not a fraud? But ones here from digikey match the font if not the cut depth:
20201117_100813.jpg

@mjs513 - here is a Spew Free Dir Verify from the above LFSintegrity - it returns a filecount to compare to static count:
Code:
unsigned int printDirectoryFilecount(File dir) {
	unsigned int filecnt = 0;
	while (true) {
		File entry =  dir.openNextFile();
		if (! entry) {
			// no more files
			break;
		}
		if (entry.isDirectory()) {
			filecnt+=printDirectoryFilecount(entry);
		} else {
			filecnt++;
		}
		entry.close();
	}
	return filecnt;
}
 
@defragster - Now you know how I feel when I wake up some days and you all have been working something new as well :)
thanks for the updated function - will make it easier to for testing.

@KurtE - cool that its now working after the format. Think maybe I have to issue the PR to Paul?
 
Thanks Paul - will be nice when this is all in the next beta. :) Now back to playing.

Indeed I just pulled three ZIPs to integrate yet again ... but easier that than pushing another beta for each change when it just begets another change :)

@defragster - Now you know how I feel when I wake up some days and you all have been working something new as well :)
thanks for the updated function - will make it easier to for testing.

@KurtE - cool that its now working after the format. Think maybe I have to issue the PR to Paul?

I started to add code block for the printDirectoryFilecount() - but decided to drop the whole sketch that had it in.

Having a Flash Level Format ( QSPI and SPI ) in FLASHMEM could be nice. Though if the right post seen above did a pre-format - it is still slow to write at 58KB/sec? Does the low level code always format space it wants to use?

Now to integrate updated sources ... and see about LFSintegrity edits ...
 
Indeed I just pulled three ZIPs to integrate yet again ... but easier that than pushing another beta for each change when it just begets another change :)



I started to add code block for the printDirectoryFilecount() - but decided to drop the whole sketch that had it in.

Having a Flash Level Format ( QSPI and SPI ) in FLASHMEM could be nice. Though if the right post seen above did a pre-format - it is still slow to write at 58KB/sec? Does the low level code always format space it wants to use?

Now to integrate updated sources ... and see about LFSintegrity edits ...

Oops forgot to add FLASHMEM to the function. Never mind. Its ok. Haven't done anything with SPI flash. Guess that's next.

Now I have to go update my copies of everything as well
 
Sorry I spoke too soon. I still called format instead of your initialize...

And still had issues. So converted sketch over at first to call initialize instead of begin... It hung in the call to: flexspi2_ip_command(15, 0x00800000);

So converted sketch to now call begin followed by initialize...
Which was the problem all along, in that I just used a raw chip...

so simple sketch now:
Code:
#include <LittleFS.h>
#include <SPI.h>

LittleFS_QSPIFlash myfs;

void setup() {
  //  pinMode(13, OUTPUT);
  //  digitalWrite(13, HIGH);
  SPI.begin();
  while (!Serial) ; // wait
  Serial.println("LittleFS Test"); delay(5);

  if (!myfs.begin()) {
    Serial.println("Begin Failed");
  }
  Serial.println("*** Enter anything to continue to initialize ***");
  while (!Serial.available()) ;
  Serial.println("Configured QSPI");
  myfs.initialize();
  Serial.println("End");
}

void loop() {}

Output:
Code:
LittleFS Test
QSPI flash begin
Flash ID: EF 40 18
Flash size is 16.00 Mbyte
attempting to mount existing media
success
*** Enter anything to continue to initialize ***
Configured QSPI
Erasing... (may take some time)
............................................................................
Chip erased in 38 seconds.
attempting to mount existing media
couldn't mount media, attemping to format
attempting to mount freshly formatted media
success
End
Note: I do get compile warnings with the initialize as you have return false;
from void function.

Now to try the real sketch:
mtp-test_combined... :( it is still not happy?

Will have to try some other tests...
 
will be nice when this is all in the next beta.

Yes, there have been so many changes to multiple repositories! It's pretty amazing you're keeping up with so many commits.

I am hoping to publish 1.54-beta5 this week.

But first we probably need to revisit the thorny open modes topic. I've thought much about this over the last several days since we all exchanged so many messages about it. I know this is going to be unpopular, but I've more or less decided not provide a mechanism for filesystem specific flags from the abstract FS.h layer.
 
Kurte said:
Note: I do get compile warnings with the initialize as you have return false;
from void function.

Now to try the real sketch:
mtp-test_combined... it is still not happy?

Will have to try some other tests...
Well just fixed that issue in the PR and on my PC. So hopefully its right now. Yeah, initialize has to be called after begin. All the chip initialization after begin since it has all the LUTS setup there.

EDIT: forgot - mtp_test_combined still not working. That;s really strange since its working for me. Have no clue why its not working on your end. Just out of curiosity do you still have the issue say if you use SPIFlash?

Think I am going to have to go and solder up a new T4.1 with PSRAM and Flash as a double check. Speaking of that did you check with the Beta T4.1?
 
My question for everyone about open modes is what functionality do we *need* that isn't possible, or isn't efficient enough, using the abstract functions?

For example, with the just-now-added truncate() function, you can achieve the same as SdFat's O_TRUNC and LittleFS's LFS_O_TRUNC by just opening the file and then calling file.truncate(0).

Likewise, you can call fs.exists(pathname) before opening to get the result as using LittleFS's LFS_O_EXCL flag.

I am willing to add more abstract open modes. But they need to be abstract to be done at FS.h and used by all programs, so we don't end up with just more of the same situation we have now, where programs have hard-coded dependency on a particular library. We've seen where this ultimately ends up, where people can't use 2 different libraries / examples together, where including one header automatically pulls in a library you didn't want and can't build together with the one you did wish to use. The abstraction layer is meant to solve all those problems long-term, but it can only do so it we resist the urge to take the easy short-term solutions the sacrifice it's abstractness. (is "abstractness" even a real word?)

To add another open mode, there needs to be reason why something is needed which can't be done with the abstraction layer functions & existing modes, or is too inefficient when done with them.
 
Yes, there have been so many changes to multiple repositories! It's pretty amazing you're keeping up with so many commits.

I am hoping to publish 1.54-beta5 this week.

But first we probably need to revisit the thorny open modes topic. I've thought much about this over the last several days since we all exchanged so many messages about it. I know this is going to be unpopular, but I've more or less decided not provide a mechanism for filesystem specific flags from the abstract FS.h layer.

Sounds good to have beta - not sure what all needs to sync up at different times.

As for File open mode, sort of interesting question as it likely break existing programs who use SD that use the additional flags...

I guess the real question may not really be filesystem specific flags versus a standardized set of options, such as a subset of the fcntl.h options...

Again since I personally don't use many files with Teensy stuff, I can adapt to most anything, but would be nice to be a reasonable subset and not limited to:
READ - which is read only and starts at start of file.
WRITE - which is read and write and is put at end of file...

But for the files I use with Teensy - usually test runs like this - not a big deal. Although if we actually have a working MTP (hopefully with Serial) where I can extract files when I want to or copy in new stuff... That could change.

@mjs513 - The LittleFS_FileSize is working fine with that board/chip... Will now try back to mtp to see if that makes difference.
 
To be honest, out of the one's listed by Kurt earlier in the thread, I've only used the ones I highlighted in red in most of my sketches:
Code:
// open flags
[COLOR="#FF0000"]    LFS_O_RDONLY = 1,         // Open a file as read only        <===> FILE_READ
    LFS_O_WRONLY = 2,         // Open a file as write only
    LFS_O_RDWR   = 3,         // Open a file as read and write                                 <===> FILE_WRITE
    LFS_O_CREAT  = 0x0100,    // Create a file if it does not exist[/COLOR]
    LFS_O_EXCL   = 0x0200,    // Fail if a file already exists                                    <===>  fs.exists(pathname) 
    LFS_O_TRUNC  = 0x0400,    // Truncate the existing file to zero size                   <===> truncate() function
[COLOR="#FF0000"]    LFS_O_APPEND = 0x0800,    // Move to end of file on every write
[/COLOR] Listing just for info,
 
Last edited:
As for File open mode, sort of interesting question as it likely break existing programs who use SD that use the additional flags...

Officially SD has only ever published 2 open modes.

Are there significant cases of programs open files in undocumented ways?


but would be nice to be a reasonable subset and not limited to:
READ - which is read only and starts at start of file.
WRITE - which is read and write and is put at end of file...

I agree in principle. I'm pretty sure we need at least 1 more mode, to write starting at the beginning rather than the end. Even though you could just open with writing at the end and then use seek(0), for very large files seeking to the end is expensive.

Now is the time to think of what other modes are truly needed (so they get into 1.54-beta5)
 
More on my failure CHIP... It does work for smaller files.

That is I was able to copy source files and the like to it, also a smaller image, create a folder... copy files files there, but not the two larger images.
screenshot.jpg

Wonder if there is some form of timeout?

Sort of an interesting thing, is I can drag and drop files from another folder into these folders and do a copy....
It also looks like I maybe can do a drag and drop out of this folder and drop to another folder.

But if I try to drag a file out of this directory to sublimetext it shows it as a no drop, where from other directories I can drop the file to open it... Wonder what the difference is?
 
More on my failure CHIP... It does work for smaller files.

That is I was able to copy source files and the like to it, also a smaller image, create a folder... copy files files there, but not the two larger images.
View attachment 22504

Wonder if there is some form of timeout?

Sort of an interesting thing, is I can drag and drop files from another folder into these folders and do a copy....
It also looks like I maybe can do a drag and drop out of this folder and drop to another folder.

But if I try to drag a file out of this directory to sublimetext it shows it as a no drop, where from other directories I can drop the file to open it... Wonder what the difference is?

Problem CHIP? What does LFSintegrity do running on that?
This writes smaller files a byte at a time - changing either will do bigger with SUBADD having a multiplier if set to 1,000
Code:
#define SUBADD 1	// bytes added each pass (*times file number)
#define BIGADD 10	// bytes added each pass - bigger will quickly consume more space

SubLime is PICKY about where you drag to drop - perhaps this isn't news - it has to go on the file list bar not above - or the text window I've found even bringing in a source file from disk
 
As for File open mode, sort of interesting question as it likely break existing programs who use SD that use the additional flags...
My preference would be to always support a transparent mode, where implementation is restricting option, So I vote for an 'oflag_t mode = O_READ' open

Edit: As an example: "File file = SD.sdfs.open(fileName, O_CREAT | O_WRITE | O_TRUNC);" is not correct
and has to be replaced by "File file = File(new SDFile(msd->sdfs.open(fileName, O_CREAT | O_WRITE | O_TRUNC)));"
 
Last edited:
Edit: As an example: "File file = SD.sdfs.open(fileName, O_CREAT | O_WRITE | O_TRUNC);" is not correct
and has to be replaced by "File file = File(new SDFile(msd->sdfs.open(fileName, O_CREAT | O_WRITE | O_TRUNC)));"

This is exactly the wrong way to use the File API. Code written in this manner won't be portable to LittleFS or other future filesystems!
 
This is exactly the wrong way to use the File API. Code written in this manner won't be portable to LittleFS or other future filesystems!

Sure, thats why I gave this example, which could easily be avoided having an additional future oriented open method. One always can maintain the backward compatible methods.
BTW, FILE_WRITE is not at all transparent.
 
Problem CHIP? What does LFSintegrity do running on that?
This writes smaller files a byte at a time - changing either will do bigger with SUBADD having a multiplier if set to 1,000
Code:
#define SUBADD 1	// bytes added each pass (*times file number)
#define BIGADD 10	// bytes added each pass - bigger will quickly consume more space

SubLime is PICKY about where you drag to drop - perhaps this isn't news - it has to go on the file list bar not above - or the text window I've found even bringing in a source file from disk
I have been running the LFSIntegrety sketch. Not sure of best options... But
#define SUBADD 5 // bytes added each pass (*times file number)
#define BIGADD 50 // bytes added each pass - bigger will quickly consume more space

Ran I think fine. I tried hitting h... got tired so then entered numbers...

Code:
[ 25.22 M] Awaiting input 0123456789rdchkfvplmu? loops left 0 > d
printDirectory QSPI_DISK
--------------
DIR	0_dir / 

 0 dirs with 0 files of Size 0 Bytes
DIR	1_dir / 

 0 dirs with 0 files of Size 0 Bytes
DIR	2_dir / 

 0 dirs with 0 files of Size 0 Bytes
DIR	3_dir / 
	FILE	B_file.txt		55

 0 dirs with 1 files of Size 55 Bytes
DIR	4_dir / 
	FILE	C_file.txt		60
	FILE	D_file.txt		65

 0 dirs with 2 files of Size 125 Bytes
DIR	5_dir / 
	FILE	E_file.txt		70
	FILE	F_file.txt		75

 0 dirs with 2 files of Size 145 Bytes
DIR	6_dir / 
	FILE	G_file.txt		80
	FILE	H_file.txt		85

 0 dirs with 2 files of Size 165 Bytes
DIR	7_dir / 
	FILE	I_file.txt		90
	FILE	J_file.txt		95

 0 dirs with 2 files of Size 185 Bytes
DIR	8_dir / 
	FILE	K_file.txt		100
	FILE	L_file.txt		105

 0 dirs with 2 files of Size 205 Bytes
DIR	9_dir / 
	FILE	M_file.txt		110
	FILE	N_file.txt		115

 0 dirs with 2 files of Size 225 Bytes
DIR	Folder / 
	FILE	IMG_1260.jpg		121305
	FILE	LittleFS.cpp		19766
	FILE	LittleFS.h		13378
	FILE	T4.1-Cardlike.jpg		0

 0 dirs with 4 files of Size 154449 Bytes
FILE	IMG_1255.jpg		96181
FILE	O_file.txt		120
FILE	P_file.txt		125
FILE	T4.1-Cardlike.jpg		0
FILE	TEST.txt		66
FILE	mtpindex.dat		0
FILE	test.txt		66

 11 dirs with 7 files of Size 96558 Bytes
 Total 24 files of Size 252112 Bytes
Bytes Used: 364544, Bytes Total:16777216


	 Loop Count: 39 (#fileCycle=2199), Bytes read 427296, written 159025, #Files=24
[ 31.38 M] Awaiting input 0123456789rdchkfvplmu? loops left 0 >
Back to playing.
 
@KurtE

LPSIntegrity looks like it working but still now sure why you can not copy a 214Kb file. As a test i tried coping a 4.7Mb pdf file to flash and it failed but a 1.5mb pdf file worked (yes it did open when I clicked on it). Was also able to load a 2.7mb stl file and then double click and it opened.

Just thought of something. @defragster was having a issue early on and he re-downloaded @WMXZ'z USB2 respository: https://github.com/WMXZ-EU/USB2. Maybe yours needs to be updated?
 
Back
Top