LittleFS port to Teensy/SPIFlash

@KurtE - as noted that looks like a usable CHIP - at least for that sketch usage.

It does many files with byte writes and with many files it started small with RAMDISK, I did test failure on disk full there - I have not gone so large as to test FLASH DISK full yet.

... just now put latest 3 from github on my system and did some code edits and adds about to compile - ... lots of phone, post, github distractions ... and lunch soon ...
 
I want to be perfectly clear. This sort of code will not be supported in Teensyduino 1.54.

FS & File are an abstraction layer, not a "transparency" layer.

File file = File(new SDFile(msd->sdfs.open(fileName, O_CREAT | O_WRITE | O_TRUNC)));

Ok, I see where I went wrong. The SDFile constructor really should be private with friend class SDClass. This usage is not allowed, and the use of "public" in the code today is a bug.

You will need to write this instead.

Code:
  File file = msd->open(fileName, FILE_WRITE);
  file.truncate(0);

I will add FILE_WRITE_BEGIN soon. If you want something other than FILE_READ, FILE_WRITE or FILE_WRITE_BEGIN, now is the time to talk about that abstract open modes are necessary. If you really need another abstract open mode, I will add it.

But you can not have "transparent" modes through the File class. Do not write your code that way. It will not work with Teensyduino 1.54. I'm try to say "no" as politely but clearly as possible. This coding style just will not be supported.
 
Now is the time to think of what other modes are truly needed (so they get into 1.54-beta5)

All modes that are common and used in programs. If we don't add them now, it will happen never :) So, even if it hurts, let's better do it now - then we can relax after that.
I can imagine to write somewhere in the middle of a file. This is useful for saving structs - kind of a database. (COBOL used this a lot..)
So, random access for read and write, and append, truncate.


I must admit - i did not follow this thread - so please excuse if it is possible already.
 
Last edited:
One more detail is the definition of the "msd" pointer. If you're designing code like MTP which is meant to work with filesystems, you should probably define the pointer this way:

Code:
  FS *msd = &SD;

If you're write this sort of definition

Code:
  SDClass *msd = &SD;

then you're unnecessarily limiting yourself to only SD cards. The purpose of the FS class is to allow this sort of code to automatically work with LittleFS and SD, and other future filesystems we don't even imagine today.
 
So, random access for read and write, and append, truncate.

All of this is already in the File API, though admittedly the truncate function was added only hours ago.

Why should the API duplicate functionality like truncate() in the open() function?
 
@KurtE

LPSIntegrity looks like it working but still now sure why you can not copy a 214Kb file. As a test i tried coping a 4.7Mb pdf file to flash and it failed but a 1.5mb pdf file worked (yes it did open when I clicked on it). Was also able to load a 2.7mb stl file and then double click and it opened.

Just thought of something. @defragster was having a issue early on and he re-downloaded @WMXZ'z USB2 respository: https://github.com/WMXZ-EU/USB2. Maybe yours needs to be updated?

I believe my USB2 is up to date with his latest change back in May...

I have tried a few different files.
It worked on a screenshot file that was 181kb but failed on another one that was 299kb. It is almost like it is doing a partial copy and the code hung or did not ACK or something and then the progress box, sits
there for N seconds, and then I here a windows chime of device went away...

@KurtE - as noted that looks like a usable CHIP - at least for that sketch usage.

It does many files with byte writes and with many files it started small with RAMDISK, I did test failure on disk full there - I have not gone so large as to test FLASH DISK full yet.

... just now put latest 3 from github on my system and did some code edits and adds about to compile - ... lots of phone, post, github distractions ... and lunch soon ...

I hear you, I just also updated all three libraries (which probably overwrite @mjs513 changes. It looks like the new version has a new method of lowLevelFormat... Have not looked yet.

Wondering are all of you running at the default FlexSPI2 speed or have you upped the speed?

@all - which test sketch are you running? Combined or combined1? Does Combin1 work with QSPI and RAM and/or SPI?
 
I believe my USB2 is up to date with his latest change back in May...

I have tried a few different files.
It worked on a screenshot file that was 181kb but failed on another one that was 299kb. It is almost like it is doing a partial copy and the code hung or did not ACK or something and then the progress box, sits
there for N seconds, and then I here a windows chime of device went away...



I hear you, I just also updated all three libraries (which probably overwrite @mjs513 changes. It looks like the new version has a new method of lowLevelFormat... Have not looked yet.

Wondering are all of you running at the default FlexSPI2 speed or have you upped the speed?

@all - which test sketch are you running? Combined or combined1? Does Combin1 work with QSPI and RAM and/or SPI?

I am running Combined. Combined1 is just a copy of Combined to see if I could get QSPI/Ram working together or QSPI/SPI etc. It fails and gives me an error that I can not figure out how to fix.

Also just using the default FlexSPI2 speed - didn't want to start playing with that yet.

That result is still puzzling to me about copying files. In the storage class or MTP classes I am working with I don't remember seeing anything that would limit the transfers or buffer sizes that would cause a problem. Wonder if its something in USB1_MTP class which I guess is linked to USB_DEV/DESC etc. Buffer size increase or decrease or other issue, especially since we are using it other storage types and not SD cards.

EDIT: Strange thought - maybe lowering the T4.1 clock?
 
@all - I probably should have done this earlier as well, but I get the same behavior when I plug in the propshield into this board, change to SPI and only pin 6...

Come up. I can copy the same file to the propshield memory and it errors on the larger file.
 
@all - I probably should have done this earlier as well, but I get the same behavior when I plug in the propshield into this board, change to SPI and only pin 6...

Come up. I can copy the same file to the propshield memory and it errors on the larger file.

Can you try a different jpg file about the same size. Been testing pdf's as I mentioned, 1 4.9mb pdf worked no issue another pdf of 1.9mb failed.


EDIT: OK going to use that PDF file that I have a test case. I just pasted it to a sdcard and it seems to work so have a little more work to do. Going to check I incorporated the changes correctly. Maybe retest with the original master to see if it works. Unfortunatlely now diner time here so may not be completed uintil tomorrow morning
 
Can you try a different jpg file about the same size. Been testing pdf's as I mentioned, 1 4.9mb pdf worked no issue another pdf of 1.9mb failed.


EDIT: OK going to use that PDF file that I have a test case. I just pasted it to a sdcard and it seems to work so have a little more work to do. Going to check I incorporated the changes correctly. Maybe retest with the original master to see if it works. Unfortunatlely now diner time here so may not be completed uintil tomorrow morning

Quick update: I tried with different files and it looks like it hits up against some maximum time timeout or the like or maybe Teensy gets into some deadlock or???

For the heck of I tried by using RAM (This one has an 8mb PSRAM installed). So edited to enable RAM, Changed the EXTRAM to allocate 4mb size buffer.

I then tried to copy the larger files to this extram disk and it appeared to not have any problems doing so. However the file sizes look wrong!

screenshot2.jpg

Hopefully you can see that for example the file I copied T4.1-cardlike.jpg on my main machine is 384kb, the resulting file on the RAM disk says it is 36.8kb... Maybe we just have some limitation on max file size on RAM disk or???

Have a good Diner!
 
Modified/Updated LFSintegrity below.

Uses myfs.quickFormat(); and myfs.lowLevelFormat();
Two commands now Upper case 'R' and 'F'
Added 'x' TOGGLE the dirwalk to compare files counts (starts off) {ON even slow on RAMDISK}
> doing 'l' or 'd'ir does verify file count and will indicate if compare fails
Added 'm'ake rootDirs for use after a format - or file open fails
Code can now make NUMDIRS instead of fixed at 10 - numbered 1-NUMDIRS
Better early exit of an interation on error is '0' entered
Code:
 'R' Restart Teensy
 'd' Directory of LittleFS
 'c' Continuous Loop
 'h' Hundred loops
 'k' Thousand loops
 'F' LittleFS_ Low Level Format Disk 
 'q' LittleFS_ Quick Format Disk 
 'v' Verbose All Dir Prints - TOGGLE
 'p' Pause after all Dir prints - TOGGLE
 'l' Show count of loop()'s, Bytes Read,Written
 'm' Make ROOT dirs (needed after format !ROOTONLY)
 'u' Update Filecount
 'x' Directory filecount verify - TOGGLE
 '?' Help list

Still nothing exotic on file: Open new or extend and write more, then remove in some fashion:
Code:
#include <LittleFS.h>

// bDirVerify :: UI TOGGLE printDirectoryFilecount() each iteration - always test on DIR display
// NUMDIRS :: Rewrite Dirs 0 is root and 1-NUMDIRS to fit into sprintf used to make Dir name.

//#define ROOTONLY // NORMAL is NOT DEFINED!
#define NUMDIRS 28  // When not ROOTONLY must be 1 or more

#define TEST_RAM
//#define TEST_SPI
//#define TEST_QSPI
//#define TEST_PROG

// Set for SPI usage
const int FlashChipSelect = 6; // digital pin for flash chip CS pin

#ifdef TEST_RAM
LittleFS_RAM myfs;
DMAMEM char buf[490000];	// USE DMAMEM for more memory than ITCM allows - or remove
char szDiskMem[] = "RAM_DISK";
#elif defined(TEST_SPI)
//const int FlashChipSelect = 21; // Arduino 101 built-in SPI Flash
#define FORMATSPI
//#define FORMATSPI2
LittleFS_SPIFlash myfs;
char szDiskMem[] = "SPI_DISK";
#elif defined(TEST_PROG)
LittleFS_Program myfs;
char szDiskMem[] = "PRO_DISK";
#else // TEST_QSPI
LittleFS_QSPIFlash myfs;
char szDiskMem[] = "QSPI_DISK";
#endif

File file3;

#define SUBADD 10	// bytes added each pass (*times file number)
#define BIGADD 100	// bytes added each pass - bigger will quickly consume more space
#define MAXNUM 26	// ALPHA A-Z is 26, less for fewer files
#define DELDELAY 0 	// delay before DEL files : delayMicroseconds
#define ADDDELAY 0 	// delay on ADD FILE : delayMicroseconds

const uint32_t lowOffset = 'a' - 'A';
const uint32_t lowShift = 13;
uint32_t errsLFS = 0;
uint32_t lCnt = 0;
uint32_t LoopCnt = 0;
uint32_t rdCnt = 0;
uint32_t wrCnt = 0;
unsigned int filecount = 0;

void setup() {
	while (!Serial) ; // wait
	Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);
	Serial.println("LittleFS Test : File Integrity"); delay(5);

#ifdef TEST_RAM
	if (!myfs.begin(buf, sizeof(buf))) {
#elif defined(TEST_SPI)
#ifdef FORMATSPI
	if (!myfs.begin( FlashChipSelect )) {
#elif defined(FORMATSPI2)
	pinMode(FlashChipSelect, OUTPUT);
	digitalWriteFast(FlashChipSelect, LOW);
	SPI2.setMOSI(50);
	SPI2.setMISO(54);
	SPI2.setSCK(49);
	SPI2.begin();
	if (!myfs.begin(51, SPI2)) {
#endif
#elif defined(TEST_PROG)
	if (!myfs.begin(1024 * 1024 * 4)) {
#else
	if (!myfs.begin()) {
#endif
		Serial.printf("Error starting %s\n", szDiskMem);
		checkInput( 1 );
	}
	// parseCmd( 'F' ); // ENABLE this if disk won't allow startup
	printDirectory();
	parseCmd( '?' );
#ifndef ROOTONLY // make subdirs if !ROOTONLY
	makeRootDirs();
#endif
	checkInput( 1 );
	filecount = printDirectoryFilecount( myfs.open("/") );  // Set base value of filecount for disk
	printDirectory();
}

void makeRootDirs() {
	char szDir[16];
	for ( uint32_t ii = 1; ii <= NUMDIRS; ii++ ) {
		sprintf( szDir, "/%lu_dir", ii );
		myfs.mkdir( szDir );
	}
}

int loopLimit = 0; // -1 continuous, otherwise # to count down to 0
bool pauseDir = false;  // Start Pause on each off
bool showDir =  false;  // false Start Dir on each off
bool bDirVerify =  false;  // false Start Dir on each off
void loop() {
	char szDir[16];
	LoopCnt++;
	uint32_t chStep;
	if ( loopLimit != 0 ) {
#ifdef ROOTONLY // ii=1-NUMDIRS are subdirs. #0 is Root
		for ( uint32_t ii = 0; ii < 1 && ( loopLimit != 0 ); ii++ )
#else
		for ( uint32_t ii = 0; ii < NUMDIRS && ( loopLimit != 0 ); ii++ )
#endif
		{
			if ( ii == 0 )
				sprintf( szDir, "/" );
			else
				sprintf( szDir, "/%lu_dir", ii );
			chStep = fileCycle(szDir);
			while ( chStep != fileCycle(szDir) && ( loopLimit != 0 ) ) checkInput( 0 ); // user input can 0 loopLimit
		}
		checkInput( 0 );
		if ( loopLimit > 0 ) // -1 means continuous
			loopLimit--;
	}
	else
		checkInput( 1 );
}

char szInputs[] = "0123456789RdchkFqvplmux?";
void checkInput( int step ) { // prompt for input without user input with step != 0
	static uint32_t lastTime;
	uint32_t nowTime = micros();

	char retVal = 0, temp;
	char *pTemp;
	if ( step != 0 ) {
		nowTime -= lastTime;
		Serial.printf( "[%6.2f M](%6.2f elap) Awaiting input %s loops left %d >", millis() / 60000.0, nowTime / 60000000.0, szInputs, loopLimit );
		lastTime = micros();
	}
	else {
		if ( !Serial.available() ) return;
		nowTime -= lastTime;
		Serial.printf( "[%6.2f M](%6.2f elap) Awaiting input %s loops left %d >", millis() / 60000.0, nowTime / 60000000.0, szInputs, loopLimit );
		//Serial.printf( "[%6.2f] Awaiting input %s loops left %d >", millis() / 60000.0, szInputs, loopLimit );
		lastTime = micros();
		while ( Serial.available() ) {
			temp = Serial.read( );
			if ( (pTemp = strchr(szInputs, temp)) ) {
				retVal = pTemp[0];
				parseCmd( retVal );
			}
		}
	}
	while ( !Serial.available() );
	while ( Serial.available() ) {
		temp = Serial.read();
		if ( (pTemp = strchr(szInputs, temp)) ) {
			retVal = pTemp[0];
			parseCmd( retVal );
		}
	}
	Serial.print( '\n' );
	if ( '?' == retVal ) checkInput( 1 ); // recurse on '?' to allow command show and response
	return;
}
void parseCmd( char chIn ) { // pass chIn == '?' for help
	switch (chIn ) {
	case '?':
		Serial.printf( "%s\n", " 0, 1-9 '#' passes continue loop before Pause\n\
 'R' Restart Teensy\n\
 'd' Directory of LittleFS\n\
 'c' Continuous Loop\n\
 'h' Hundred loops\n\
 'k' Thousand loops\n\
 'F' LittleFS_ Low Level Format Disk \n\
 'q' LittleFS_ Quick Format Disk \n\
 'v' Verbose All Dir Prints - TOGGLE\n\
 'p' Pause after all Dir prints - TOGGLE\n\
 'l' Show count of loop()'s, Bytes Read,Written\n\
 'm' Make ROOT dirs (needed after format !ROOTONLY)\n\
 'u' Update Filecount\n\
 'x' Directory filecount verify - TOGGLE\n\
 '?' Help list" );
		break;
	case 'R':
		Serial.print(" RESTART Teensy ...");
		delay(100);
		SCB_AIRCR = 0x05FA0004;
		break;
	case '0':
	case '1':
	case '2':
	case '3':
	case '4':
	case '5':
	case '6':
	case '7':
	case '8':
	case '9':
		loopLimit = chIn - '0';
		break;
	case 'c':
		loopLimit = -1;
		break;
	case 'd':
		Serial.print( " d\n" );
		printDirectory();
		Serial.print( '\n' );
		parseCmd( 'l' );
		checkInput( 1 );
		chIn = 0;
		break;
	case 'h':
		loopLimit = 100;
		break;
	case 'k':
		loopLimit = 1000;
		break;
	case 'F': // Low Level format
		Serial.print( "\nFormatting Low Level:\n\t" );
		myfs.lowLevelFormat('.');
		Serial.print( "\n Done Formatting Low Level.\n" );
		errsLFS = 0; // No Errors on new Format
		parseCmd( 'u' );
		break;
	case 'q': // quick format
		myfs.quickFormat();
		errsLFS = 0; // No Errors on new Format
		parseCmd( 'u' );
		break;
	case 'v': // verbose dir
		showDir = !showDir;
		showDir ? Serial.print(" Verbose on: ") : Serial.print(" Verbose off: ");
		chIn = 0;
		break;
	case 'p': // pause on dirs
		pauseDir = !pauseDir;
		pauseDir ? Serial.print(" Pause on: ") : Serial.print(" Pause off: ");
		chIn = 0;
		break;
	case 'x': // dir filecount Verify
		bDirVerify = !bDirVerify;
		bDirVerify ? Serial.print(" FileCnt on: ") : Serial.print(" FileCnt off: ");
		dirVerify();
		chIn = 0;
		break;

	case 'l': // Show Loop Count
		Serial.printf("\n\t Loop Count: %u (#fileCycle=%u), Bytes read %u, written %u, #Files=%u\n", LoopCnt, lCnt, rdCnt, wrCnt, filecount );
		if ( 0 != errsLFS )
			Serial.printf("\t ERROR COUNT =%u\n", errsLFS );
		dirVerify();
		chIn = 0;
		break;
	case 'm':
		Serial.printf("m \n\t Making Root Dirs\n" );
		makeRootDirs();
		parseCmd( 'd' );
		chIn = 0;
		break;
	case 'u': // Show Loop Count
		filecount = printDirectoryFilecount( myfs.open("/") );
		Serial.printf("u \n\t Updated filecount %u\n", filecount );
		chIn = 0;
		break;

	default:
		Serial.println( chIn ); // never see without unhandled char in szInputs[]
		break;
	}
	if ( 0 != chIn ) Serial.print( chIn );
}

uint32_t fTot, totSize;
void printDirectory() {
	fTot = 0, totSize = 0;
	Serial.printf("printDirectory %s\n--------------\n", szDiskMem);
	printDirectory(myfs.open("/"), 0);
	Serial.printf(" %Total %u files of Size %u Bytes\n", fTot, totSize);
	Serial.printf("Bytes Used: %llu, Bytes Total:%llu\n", myfs.usedSize(), myfs.totalSize());
}

unsigned int printDirectoryFilecount(File dir) {
	unsigned int filecnt = 0;
	while (true) {
		File entry =  dir.openNextFile();
		if (! entry) {
			// no more files
			break;
		}
		if (entry.isDirectory()) {
			filecnt += printDirectoryFilecount(entry);
		} else {
			filecnt++;
		}
		entry.close();
	}
	return filecnt;
}

void printDirectory(File dir, int numTabs) {
	//dir.whoami();
	uint32_t fSize = 0, dCnt = 0, fCnt = 0;
	if ( 0 == dir ) {
		Serial.printf( "\t>>>\t>>>>> No Dir\n" );
		return;
	}
	while (true) {
		File entry =  dir.openNextFile();
		if (! entry) {
			// no more files
			Serial.printf("\n %u dirs with %u files of Size %u Bytes\n", dCnt, fCnt, fSize);
			fTot += fCnt;
			totSize += fSize;
			break;
		}
		for (uint8_t i = 0; i < numTabs; i++) {
			Serial.print('\t');
		}

		if (entry.isDirectory()) {
			Serial.print("DIR\t");
			dCnt++;
		} else {
			Serial.print("FILE\t");
			fCnt++;
			fSize += entry.size();
		}
		Serial.print(entry.name());
		if (entry.isDirectory()) {
			Serial.println(" / ");
			printDirectory(entry, numTabs + 1);
		} else {
			// files have sizes, directories do not
			Serial.print("\t\t");
			Serial.println(entry.size(), DEC);
		}
		entry.close();
		//Serial.flush();
	}
}

uint32_t cCnt = 0;
uint32_t fileCycle(const char *dir) {
	static char szFile[] = "_file.txt";
	char szPath[150];
	int ii;
	lCnt++;
	byte nNum = lCnt % MAXNUM;
	char chNow = 'A' + lCnt % MAXNUM;
	lfs_ssize_t resW = 1;

	if ( dir[1] == 0 )	// catch root
		sprintf( szPath, "/%c%s", chNow, szFile );
	else
		sprintf( szPath, "%s/%c%s", dir, chNow, szFile );
	if ( cCnt >= 3 && myfs.exists(szPath) ) { // DELETE ALL KNOWN FILES
		if ( nNum == 1 ) {
			Serial.print( "\n == == ==   DELETE PASS START  == == == = \n");
			if ( showDir ) {
				printDirectory();
				Serial.print( " == == ==   DELETE PASS START  == == == = \n");
			}
			delayMicroseconds(DELDELAY);
		}
	}
	Serial.printf( ":: %s ", szPath );
	if ( cCnt >= 3 && myfs.exists(szPath) ) { // DELETE ALL KNOWN FILES
		readVerify( szPath, chNow );
		myfs.remove(szPath);
		filecount--;
		Serial.printf(" %s ----DEL----", szDiskMem);
		Serial.printf(" -- %c", chNow);
		if ( showDir ) {
			Serial.print("\n");
			printDirectory(myfs.open(dir), 1);
		}
		if ( pauseDir ) checkInput( 1 );
		Serial.println();
	}
	else {
		if ( nNum == 0 ) {
			nNum = 10;
			cCnt++;
			if ( cCnt >= 5 ) cCnt = 0;
		}
		file3 = myfs.open(szPath, FILE_WRITE);
		if ( 0 == file3 ) {
			Serial.printf( "\tXXX\tXXX\tXXX\tXXX\tFail File open {mkdir?}\n" );
			delayMicroseconds(300000);
			checkInput( 1 );	// PAUSE on CmdLine
		}
		else {
			delayMicroseconds(ADDDELAY);
			char mm = chNow + lowOffset;
			uint32_t jj = file3.size() + 1;
			for ( ii = 0; ii < (nNum * SUBADD + BIGADD ) && resW > 0; ii++ ) {
				if ( 0 == ((ii + jj) / lowShift) % 2 )
					resW = file3.write( &mm , 1 );
				else
					resW = file3.write( &chNow , 1 );
				wrCnt++;
				// if ( lCnt%100 == 50 ) mm='x'; // GENERATE ERROR to detect on DELETE read verify
			}
			file3.close();
			Serial.printf(" %s +++ Add +++ [sz %u add %u]", szDiskMem, jj - 1, ii);
			if (resW < 0) {
				Serial.printf( "\n\twrite fail %i\n", resW );
				parseCmd( '0' );
				errsLFS++;
				checkInput( 1 );	// PAUSE on CmdLine
			}
			else if ( jj == 1 ) filecount++; // File Added
			Serial.printf(" ++ %c ", chNow);
			readVerify( szPath, chNow );
			if ( showDir ) {
				Serial.print("\n");
				printDirectory(myfs.open(dir), 1);
			}
		}
		if ( pauseDir ) checkInput( 1 );
		Serial.print("\n");
		delayMicroseconds(ADDDELAY);
	}
	checkInput( 0 ); // user stop request?
	if ( bDirVerify ) dirVerify();
	return cCnt;
}

void dirVerify() {
	if ( filecount != printDirectoryFilecount( myfs.open("/") ) ) {
		Serial.printf( "\tFilecount mismatch %u != %u\n", filecount, printDirectoryFilecount( myfs.open("/") ) );
		parseCmd( '0' );
		errsLFS++;
		checkInput( 1 );	// PAUSE on CmdLine
	}
}

void readVerify( char szPath[], char chNow ) {
	file3 = myfs.open(szPath);
	if ( 0 == file3 ) {
		Serial.printf( "\tV\t Fail File open %s\n", szPath );
		parseCmd( '0' );
		errsLFS++;
		checkInput( 1 );
	}
	char mm;
	char chNow2 = chNow + lowOffset;
	uint32_t ii = 0;
	while ( file3.available() ) {
		file3.read( &mm , 1 );
		rdCnt++;
		//Serial.print( mm ); // show chars as read
		ii++;
		if ( 0 == (ii / lowShift) % 2 ) {
			if ( chNow2 != mm ) {
				Serial.printf( "<Bad Byte!  %c! = %c [0x%X] @%u\n", chNow2, mm, mm, ii );
				parseCmd( '0' );
				errsLFS++;
				checkInput( 1 );
				break;
			}
		}
		else {
			if ( chNow != mm ) {
				Serial.printf( "<Bad Byte!  %c! = %c [0x%X] @%u\n", chNow, mm, mm, ii );
				parseCmd( '0' );
				errsLFS++;
				checkInput( 1 );
				break;
			}
		}
	}
	Serial.printf( "\tVerify %s bytes %u ", szPath, ii );
	if (ii != file3.size()) {
		Serial.printf( "\n\tRead Count fail! :: read %u != f.size %u", ii, file3.size() );
		parseCmd( '0' );
		errsLFS++;
		checkInput( 1 );	// PAUSE on CmdLine
	}
	file3.close();
}
 
Last edited:
Quick update: I tried with different files and it looks like it hits up against some maximum time timeout or the like or maybe Teensy gets into some deadlock or???

For the heck of I tried by using RAM (This one has an 8mb PSRAM installed). So edited to enable RAM, Changed the EXTRAM to allocate 4mb size buffer.

I then tried to copy the larger files to this extram disk and it appeared to not have any problems doing so. However the file sizes look wrong!

View attachment 22507

Hopefully you can see that for example the file I copied T4.1-cardlike.jpg on my main machine is 384kb, the resulting file on the RAM disk says it is 36.8kb... Maybe we just have some limitation on max file size on RAM disk or???

Have a good Diner!

First will do but was playing with the T3.5 and Littlefs. Sort of work. Still problem with that on pdf I have that does work on a SD card.

Anyway, the jpg that I have from downloading from your post is only 214kb. That one works using RAM, SPI and on QSPI. When I put it on RAM it does show correct file size of 214 KB.

Really have to dig into the code tomorrow to see what changed and see what it says - I am using a buffer of 40Kb. Getting same behavior even using the master branch. Maybe have to look over the spec.
 
Posted code in #336 without actually testing a lowLevelFormat.

Just did that on SPI disk and it completed. Added print notice before and after for user ...

I saw a progress 'char' could be passed ... I tried that and it shows WAY TOO many ...
 
I've added FILE_WRITE_BEGIN mode for FS open().

https://github.com/PaulStoffregen/cores/commit/5443bfd17949fed7cfd6e4b1f788fdfb3be48383

https://github.com/PaulStoffregen/LittleFS/commit/92e2520367df71df59e28fe2df57edbb0b0cd5ff

https://github.com/PaulStoffregen/SD/commit/030244e90f2cfae5d2b5ba064b17c19916be7e37

This provides a more efficient way to open a file for writing (and read) where you intend to use random access with seek() or just overwrite the existing data. While you can just use seek(0) to go back to the beginning after open(filename, FILE_WRITE), this avoids the unnecessary overhead of the finding the end of the file.
 
Did an LFSintegrity run to better fill 16M - then 'F'ormat for '.' timing. The dots were under 3 secs each with full llFormat in 96 seconds.

Using:
Code:
#define NUMDIRS 28  // When not ROOTONLY must be 1 or more
...
#define SUBADD 1000	// bytes added each pass (*times file number)
#define BIGADD 1000	// bytes added each pass - bigger will quickly consume more space
#define MAXNUM 26	// ALPHA A-Z is 26, less for fewer files

This is 16MB QSPI:
Code:
 Total 705 files of Size 13965000 Bytes
Bytes Used: 15450112, Bytes Total:16777216
	 Loop Count: 20 (#fileCycle=5925), Bytes read 149734100, written 58843000, #Files=705

So that is enough slack space it seems as it was running at this point for a bit

Doing Repro on SPI now ... took longer to fill - about 2X
Code:
 Total 726 files of Size 15141000 Bytes
Bytes Used: 16654336, Bytes Total:16777216

	 Loop Count: 20 (#fileCycle=5176), Bytes read 116625000, written 51623000, #Files=726

SPI format is longer too 5:48 or 348 seconds ... 8.7 seconds each for 40 dot chars. On the same chip ...

Back to filling the disk - after SPI Format restarted and it filled to FULL:
Code:
:: /27_dir/X_file.txt  SPI_DISK +++ Add +++ [sz 0 add 24000] ++ X 	Verify /27_dir/X_file.txt bytes 24000 
:: /27_dir/Y_file.txt  SPI_DISK +++ Add +++ [sz 0 add 257]	write fail -28[ 53.98 M] Awaiting input 0123456789RdchkFqvplmux? loops left 4 >0
 ++ Y 	Verify /27_dir/Y_file.txt bytes 0 
[ 54.81 M] Awaiting input 0123456789RdchkFqvplmux? loops left 0 >
[ 54.86 M] Awaiting input 0123456789RdchkFqvplmux? loops left 0 >
	 Loop Count: 32 (#fileCycle=7148), Bytes read 162737000, written 73709257, #Files=457
	Filecount mismatch 457 != 458

...
 Total 458 files of Size 15623000 Bytes
Bytes Used: 16777216, Bytes Total:16777216

	 Loop Count: 32 (#fileCycle=7148), Bytes read 162737000, written 73709257, #Files=457
	Filecount mismatch 457 != 458

Error -28 is :: LFS_ERR_NOSPC = -28, // No space left on device

So it acted right up to then on 16M SPI - except I counted the file as added - when it wasn't

Format of that took over 9 minutes ... ICK ????
 
I want to be perfectly clear. This sort of code will not be supported in Teensyduino 1.54.
As you realized later, this coding style WAS supported, at least for a while.
Anyhow, my exercise was only to make the case that I suggested a coding style like (this is an example!)
Code:
File file = SD.open(filename,(O_EXCL | O_WRONLY | O_CREAT));
if(file) {...}
and not
Code:
File file;
if(!SD.exists(filename))  file = SD.open(filename,FILE_WRITE); else { /* handle case */ }
file.seek(0);
(Yes, I know FILE_WRITE_BEGIN was added, so that code could be simplified)

or the one I showed earlier, and you mistakenly assumed it is my suggestion.

I do not care if the the suggested additional line is more general
Code:
File open(char *filename, uint8_t mode); // Arduino mode
....
File open(char *filename, uint32_t mode); // 32 bit mode for more flexibility
//or
File open(char *filename, oflag_t mode);  // SDFat int mode
Anyhow, I will close this argument and PRs I made
 
Drive wasn't filled - format took 6.14 minutes (New ( 6.14 elap)sed time on command) this time:
Code:
Bytes Used: 15691776, Bytes Total:16777216
[  0.26 M](  0.17 elap) Awaiting input 0123456789RdchkFqvplmux? loops left 0 >
Formatting Low Level: .........................................................................................................................

 Done Formatting Low Level.
u 
	 Updated filecount 0
F
[  6.40 M](  6.14 elap)

That is with the 120 chars printed as in current PR updated so dots aren't so long (6-9 secs) between - but on TyComm with non-prop fonts it can push under scroll bar to finish or narrow window.
These are 3 secs between - and larger disks later may take even longer than 6-9 minutes - but better for now.

So that's the lowLevelFormat 'progress' story ... the good news is a nearly FULL Q/SPI drive works up until all space is allocated - then will return an expected error.
Wear leveling is limited to newly freed blocks - but with haphazard cycling by LPSintegrity test - they are probably well distributed.
 
@KurtE

Looks like MTP for storage areas isn't working so well after the last couple of updates. Still trying to figure out whats wrong. Going to take awhile I think. Need another diversion.
 
@mjs513 - Let me know if there is another iteration of tests I should run on it...

I might play a little with other diversion.
 
@mjs513 - Let me know if there is another iteration of tests I should run on it...

I might play a little with other diversion.

I will go on to other distractions. I just tested the updated MTPResponder with Builtin_SDCard and a breakout card reader and MTP for SD is failing reading anything but the Builtin_sdcard. So may be another issue. Going to post on the other thread now. Begining to get tired of MTP, think I need a diversion just to clear my head.
 
Newest version of LFSintegrity.ino in post #336

Updated with LittleFS constrained progress char print on LLformat.

Now with elapsed time track on prompt line from time last command sent until prompt appears : like issuing 'F'ormat or other.

<edit>: Updated again above - counts errors and does auto set of '0' to not continue right away on error.
 
Last edited:
One more detail is the definition of the "msd" pointer. If you're designing code like MTP which is meant to work with filesystems, you should probably define the pointer this way:

Code:
  FS *msd = &SD;

If you're write this sort of definition

Code:
  SDClass *msd = &SD;

then you're unnecessarily limiting yourself to only SD cards. The purpose of the FS class is to allow this sort of code to automatically work with LittleFS and SD, and other future filesystems we don't even imagine today.

Evening Paul - been keeping this post in the back of my mind since you posted it and guess I am ready at this point to ask how does this really work. The reason I am asking is that @WMXZ came up with an interesting way to possibly incorporate everything in a single MTP library but now running into conflicts between littileFS and SDClass which you kind of allude to with the second format.

Right now in the storage class I have a slight mod based on what I was working on to his existing coded so you could have multiple RAM storage devices along with his multiple SD Cards. But getting conflicts in SDClass:
Code:
In file included from D:\Users\Merli\Documents\Arduino\libraries\MTP_t4-master\src/Storage.h:32:0,
                 from D:\Users\Merli\Documents\Arduino\libraries\MTP_t4-master\src/MTP.h:36,
                 from D:\Users\Merli\Documents\Arduino\libraries\MTP_t4-master\examples\mtp-test\mtp-test.ino:6:
F:\arduino-1.8.13-beta4\hardware\teensy\avr\libraries\SD\src/SD.h:159:49: error: 'FILE_READ' was not declared in this scope
  File open(const char *filepath, uint8_t mode = FILE_READ) {
                                                 ^
F:\arduino-1.8.13-beta4\hardware\teensy\avr\libraries\SD\src/SD.h: In member function 'virtual File SDClass::open(const char*, uint8_t)':
F:\arduino-1.8.13-beta4\hardware\teensy\avr\libraries\SD\src/SD.h:161:15: error: 'FILE_WRITE' was not declared in this scope
   if (mode == FILE_WRITE) flags = O_RDWR | O_CREAT | O_AT_END;
               ^
In file included from D:\Users\Merli\Documents\Arduino\libraries\MTP_t4-master\src/MTP.h:36:0,
                 from D:\Users\Merli\Documents\Arduino\libraries\MTP_t4-master\examples\mtp-test\mtp-test.ino:6:
D:\Users\Merli\Documents\Arduino\libraries\MTP_t4-master\src/Storage.h: At global scope:
D:\Users\Merli\Documents\Arduino\libraries\MTP_t4-master\src/Storage.h:195:52: error: 'FILE_READ' was not declared in this scope
   void OpenFileByIndex(uint32_t i, uint32_t mode = FILE_READ) ;
believe it coming from this portion of the code;
Code:
#include "SD.h"
#ifndef FILE_WRITE_BEGIN
  #define FILE_WRITE_BEGIN 2
#endif

// following is a device specific base class for storage classs
[B][COLOR="#FF0000"]extern SDClass sdx[];[/COLOR][/B]

#ifdef USE_RAM
#include "LittleFS.h"
[COLOR="#FF0000"][B]extern LittleFS_RAM ramfs[];[/B][/COLOR]

class mSD_Base
{ 
  public:
  File sd_open(uint32_t store, const char *filename, uint32_t mode) 
  { if(!cs || (cs[store]<256)) return sdx[store].open(filename,mode); else return ramfs[store].open(filename,mode);
  }
  bool sd_mkdir(uint32_t store, char *filename) 
  { if(!cs || (cs[store]<256)) return sdx[store].mkdir(filename); else return ramfs[store].mkdir(filename);
  }

  bool sd_rename(uint32_t store, char *oldfilename, char *newfilename) 
  { if(!cs || (cs[store]<256)) return sdx[store].rename(oldfilename,newfilename); else return ramfs[store].rename(oldfilename,newfilename);
    
  }
  bool sd_remove(uint32_t store, const char *filename) 
  { if(!cs || (cs[store]<256)) return sdx[store].remove(filename); else return ramfs[store].remove(filename);
  }
  bool sd_rmdir(uint32_t store, char *filename) 
  { if(!cs || (cs[store]<256)) return sdx[store].rmdir(filename); else return ramfs[store].rmdir(filename);
  }
    
  uint32_t sd_totalClusterCount(uint32_t store) 
  { if(!cs || (cs[store]<256)) return sdx[store].sdfs.clusterCount(); else return ramfs[store].totalSize()/512;
  }
  uint32_t sd_freeClusterCount(uint32_t store)  
  { if(!cs || (cs[store]<256)) return sdx[store].sdfs.freeClusterCount(); else return (ramfs[store].totalSize()-ramfs.usedSize())/512;
  }
  uint32_t sd_sectorsPerCluster(uint32_t store) 
  [COLOR="#FF0000"]{ if(!cs || (cs[store]<256)) return sdx[store].sdfs.sectorsPerCluster(); else return 1;[/COLOR]
  }

  bool setCs(const int *csx) { cs = csx; return true; }

  private:
  const int * cs = 0;
};
but I am lost on how to incorporate your recommendation. If you get a minute can you advise. If anyone else wants to jump to the discussion feel free. Stuck on getting everything to play nice together. Of course there are other issues but those I think I have a solution for.
 
One more detail is the definition of the "msd" pointer. If you're designing code like MTP which is meant to work with filesystems, you should probably define the pointer this way:

Code:
  FS *msd = &SD;

If you're write this sort of definition

Code:
  SDClass *msd = &SD;

then you're unnecessarily limiting yourself to only SD cards. The purpose of the FS class is to allow this sort of code to automatically work with LittleFS and SD, and other future filesystems we don't even imagine today.

I thought so (use of interfaces), but by extending lost this concept.
Will revisit my MTP implementation in this light.
 
Back
Top