Forum Rule: Always post complete source code & details to reproduce any issue!
Page 14 of 33 FirstFirst ... 4 12 13 14 15 16 24 ... LastLast
Results 326 to 350 of 811

Thread: LittleFS port to Teensy/SPIFlash

  1. #326
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,485
    I want to be perfectly clear. This sort of code will not be supported in Teensyduino 1.54.

    FS & File are an abstraction layer, not a "transparency" layer.

    Quote Originally Posted by WMXZ View Post
    File file = File(new SDFile(msd->sdfs.open(fileName, O_CREAT | O_WRITE | O_TRUNC)));
    Ok, I see where I went wrong. The SDFile constructor really should be private with friend class SDClass. This usage is not allowed, and the use of "public" in the code today is a bug.

    You will need to write this instead.

    Code:
      File file = msd->open(fileName, FILE_WRITE);
      file.truncate(0);
    I will add FILE_WRITE_BEGIN soon. If you want something other than FILE_READ, FILE_WRITE or FILE_WRITE_BEGIN, now is the time to talk about that abstract open modes are necessary. If you really need another abstract open mode, I will add it.

    But you can not have "transparent" modes through the File class. Do not write your code that way. It will not work with Teensyduino 1.54. I'm try to say "no" as politely but clearly as possible. This coding style just will not be supported.

  2. #327
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    7,658
    Quote Originally Posted by PaulStoffregen View Post
    Now is the time to think of what other modes are truly needed (so they get into 1.54-beta5)
    All modes that are common and used in programs. If we don't add them now, it will happen never So, even if it hurts, let's better do it now - then we can relax after that.
    I can imagine to write somewhere in the middle of a file. This is useful for saving structs - kind of a database. (COBOL used this a lot..)
    So, random access for read and write, and append, truncate.


    I must admit - i did not follow this thread - so please excuse if it is possible already.
    Last edited by Frank B; 11-17-2020 at 09:57 PM.

  3. #328
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,485
    One more detail is the definition of the "msd" pointer. If you're designing code like MTP which is meant to work with filesystems, you should probably define the pointer this way:

    Code:
      FS *msd = &SD;
    If you're write this sort of definition

    Code:
      SDClass *msd = &SD;
    then you're unnecessarily limiting yourself to only SD cards. The purpose of the FS class is to allow this sort of code to automatically work with LittleFS and SD, and other future filesystems we don't even imagine today.

  4. #329
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,485
    Quote Originally Posted by Frank B View Post
    So, random access for read and write, and append, truncate.
    All of this is already in the File API, though admittedly the truncate function was added only hours ago.

    Why should the API duplicate functionality like truncate() in the open() function?

  5. #330
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    8,217
    Quote Originally Posted by mjs513 View Post
    @KurtE

    LPSIntegrity looks like it working but still now sure why you can not copy a 214Kb file. As a test i tried coping a 4.7Mb pdf file to flash and it failed but a 1.5mb pdf file worked (yes it did open when I clicked on it). Was also able to load a 2.7mb stl file and then double click and it opened.

    Just thought of something. @defragster was having a issue early on and he re-downloaded @WMXZ'z USB2 respository: https://github.com/WMXZ-EU/USB2. Maybe yours needs to be updated?
    I believe my USB2 is up to date with his latest change back in May...

    I have tried a few different files.
    It worked on a screenshot file that was 181kb but failed on another one that was 299kb. It is almost like it is doing a partial copy and the code hung or did not ACK or something and then the progress box, sits
    there for N seconds, and then I here a windows chime of device went away...

    Quote Originally Posted by defragster View Post
    @KurtE - as noted that looks like a usable CHIP - at least for that sketch usage.

    It does many files with byte writes and with many files it started small with RAMDISK, I did test failure on disk full there - I have not gone so large as to test FLASH DISK full yet.

    ... just now put latest 3 from github on my system and did some code edits and adds about to compile - ... lots of phone, post, github distractions ... and lunch soon ...
    I hear you, I just also updated all three libraries (which probably overwrite @mjs513 changes. It looks like the new version has a new method of lowLevelFormat... Have not looked yet.

    Wondering are all of you running at the default FlexSPI2 speed or have you upped the speed?

    @all - which test sketch are you running? Combined or combined1? Does Combin1 work with QSPI and RAM and/or SPI?

  6. #331
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    7,658
    Quote Originally Posted by PaulStoffregen View Post
    All of this is already in the File API, though admittedly the truncate function was added only hours ago.

    Why should the API duplicate functionality like truncate() in the open() function?
    That's not needed.

  7. #332
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    6,125
    Quote Originally Posted by KurtE View Post
    I believe my USB2 is up to date with his latest change back in May...

    I have tried a few different files.
    It worked on a screenshot file that was 181kb but failed on another one that was 299kb. It is almost like it is doing a partial copy and the code hung or did not ACK or something and then the progress box, sits
    there for N seconds, and then I here a windows chime of device went away...



    I hear you, I just also updated all three libraries (which probably overwrite @mjs513 changes. It looks like the new version has a new method of lowLevelFormat... Have not looked yet.

    Wondering are all of you running at the default FlexSPI2 speed or have you upped the speed?

    @all - which test sketch are you running? Combined or combined1? Does Combin1 work with QSPI and RAM and/or SPI?
    I am running Combined. Combined1 is just a copy of Combined to see if I could get QSPI/Ram working together or QSPI/SPI etc. It fails and gives me an error that I can not figure out how to fix.

    Also just using the default FlexSPI2 speed - didn't want to start playing with that yet.

    That result is still puzzling to me about copying files. In the storage class or MTP classes I am working with I don't remember seeing anything that would limit the transfers or buffer sizes that would cause a problem. Wonder if its something in USB1_MTP class which I guess is linked to USB_DEV/DESC etc. Buffer size increase or decrease or other issue, especially since we are using it other storage types and not SD cards.

    EDIT: Strange thought - maybe lowering the T4.1 clock?

  8. #333
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    8,217
    @all - I probably should have done this earlier as well, but I get the same behavior when I plug in the propshield into this board, change to SPI and only pin 6...

    Come up. I can copy the same file to the propshield memory and it errors on the larger file.

  9. #334
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    6,125
    Quote Originally Posted by KurtE View Post
    @all - I probably should have done this earlier as well, but I get the same behavior when I plug in the propshield into this board, change to SPI and only pin 6...

    Come up. I can copy the same file to the propshield memory and it errors on the larger file.
    Can you try a different jpg file about the same size. Been testing pdf's as I mentioned, 1 4.9mb pdf worked no issue another pdf of 1.9mb failed.


    EDIT: OK going to use that PDF file that I have a test case. I just pasted it to a sdcard and it seems to work so have a little more work to do. Going to check I incorporated the changes correctly. Maybe retest with the original master to see if it works. Unfortunatlely now diner time here so may not be completed uintil tomorrow morning

  10. #335
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    8,217
    Quote Originally Posted by mjs513 View Post
    Can you try a different jpg file about the same size. Been testing pdf's as I mentioned, 1 4.9mb pdf worked no issue another pdf of 1.9mb failed.


    EDIT: OK going to use that PDF file that I have a test case. I just pasted it to a sdcard and it seems to work so have a little more work to do. Going to check I incorporated the changes correctly. Maybe retest with the original master to see if it works. Unfortunatlely now diner time here so may not be completed uintil tomorrow morning
    Quick update: I tried with different files and it looks like it hits up against some maximum time timeout or the like or maybe Teensy gets into some deadlock or???

    For the heck of I tried by using RAM (This one has an 8mb PSRAM installed). So edited to enable RAM, Changed the EXTRAM to allocate 4mb size buffer.

    I then tried to copy the larger files to this extram disk and it appeared to not have any problems doing so. However the file sizes look wrong!

    Click image for larger version. 

Name:	screenshot2.jpg 
Views:	18 
Size:	96.9 KB 
ID:	22507

    Hopefully you can see that for example the file I copied T4.1-cardlike.jpg on my main machine is 384kb, the resulting file on the RAM disk says it is 36.8kb... Maybe we just have some limitation on max file size on RAM disk or???

    Have a good Diner!

  11. #336
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,203
    Modified/Updated LFSintegrity below.

    Uses myfs.quickFormat(); and myfs.lowLevelFormat();
    Two commands now Upper case 'R' and 'F'
    Added 'x' TOGGLE the dirwalk to compare files counts (starts off) {ON even slow on RAMDISK}
    > doing 'l' or 'd'ir does verify file count and will indicate if compare fails
    Added 'm'ake rootDirs for use after a format - or file open fails
    Code can now make NUMDIRS instead of fixed at 10 - numbered 1-NUMDIRS
    Better early exit of an interation on error is '0' entered
    Code:
     'R' Restart Teensy
     'd' Directory of LittleFS
     'c' Continuous Loop
     'h' Hundred loops
     'k' Thousand loops
     'F' LittleFS_ Low Level Format Disk 
     'q' LittleFS_ Quick Format Disk 
     'v' Verbose All Dir Prints - TOGGLE
     'p' Pause after all Dir prints - TOGGLE
     'l' Show count of loop()'s, Bytes Read,Written
     'm' Make ROOT dirs (needed after format !ROOTONLY)
     'u' Update Filecount
     'x' Directory filecount verify - TOGGLE
     '?' Help list
    Still nothing exotic on file: Open new or extend and write more, then remove in some fashion:
    Code:
    #include <LittleFS.h>
    
    // bDirVerify :: UI TOGGLE printDirectoryFilecount() each iteration - always test on DIR display
    // NUMDIRS :: Rewrite Dirs 0 is root and 1-NUMDIRS to fit into sprintf used to make Dir name.
    
    //#define ROOTONLY // NORMAL is NOT DEFINED!
    #define NUMDIRS 28  // When not ROOTONLY must be 1 or more
    
    #define TEST_RAM
    //#define TEST_SPI
    //#define TEST_QSPI
    //#define TEST_PROG
    
    // Set for SPI usage
    const int FlashChipSelect = 6; // digital pin for flash chip CS pin
    
    #ifdef TEST_RAM
    LittleFS_RAM myfs;
    DMAMEM char buf[490000];	// USE DMAMEM for more memory than ITCM allows - or remove
    char szDiskMem[] = "RAM_DISK";
    #elif defined(TEST_SPI)
    //const int FlashChipSelect = 21; // Arduino 101 built-in SPI Flash
    #define FORMATSPI
    //#define FORMATSPI2
    LittleFS_SPIFlash myfs;
    char szDiskMem[] = "SPI_DISK";
    #elif defined(TEST_PROG)
    LittleFS_Program myfs;
    char szDiskMem[] = "PRO_DISK";
    #else // TEST_QSPI
    LittleFS_QSPIFlash myfs;
    char szDiskMem[] = "QSPI_DISK";
    #endif
    
    File file3;
    
    #define SUBADD 10	// bytes added each pass (*times file number)
    #define BIGADD 100	// bytes added each pass - bigger will quickly consume more space
    #define MAXNUM 26	// ALPHA A-Z is 26, less for fewer files
    #define DELDELAY 0 	// delay before DEL files : delayMicroseconds
    #define ADDDELAY 0 	// delay on ADD FILE : delayMicroseconds
    
    const uint32_t lowOffset = 'a' - 'A';
    const uint32_t lowShift = 13;
    uint32_t errsLFS = 0;
    uint32_t lCnt = 0;
    uint32_t LoopCnt = 0;
    uint32_t rdCnt = 0;
    uint32_t wrCnt = 0;
    unsigned int filecount = 0;
    
    void setup() {
    	while (!Serial) ; // wait
    	Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);
    	Serial.println("LittleFS Test : File Integrity"); delay(5);
    
    #ifdef TEST_RAM
    	if (!myfs.begin(buf, sizeof(buf))) {
    #elif defined(TEST_SPI)
    #ifdef FORMATSPI
    	if (!myfs.begin( FlashChipSelect )) {
    #elif defined(FORMATSPI2)
    	pinMode(FlashChipSelect, OUTPUT);
    	digitalWriteFast(FlashChipSelect, LOW);
    	SPI2.setMOSI(50);
    	SPI2.setMISO(54);
    	SPI2.setSCK(49);
    	SPI2.begin();
    	if (!myfs.begin(51, SPI2)) {
    #endif
    #elif defined(TEST_PROG)
    	if (!myfs.begin(1024 * 1024 * 4)) {
    #else
    	if (!myfs.begin()) {
    #endif
    		Serial.printf("Error starting %s\n", szDiskMem);
    		checkInput( 1 );
    	}
    	// parseCmd( 'F' ); // ENABLE this if disk won't allow startup
    	printDirectory();
    	parseCmd( '?' );
    #ifndef ROOTONLY // make subdirs if !ROOTONLY
    	makeRootDirs();
    #endif
    	checkInput( 1 );
    	filecount = printDirectoryFilecount( myfs.open("/") );  // Set base value of filecount for disk
    	printDirectory();
    }
    
    void makeRootDirs() {
    	char szDir[16];
    	for ( uint32_t ii = 1; ii <= NUMDIRS; ii++ ) {
    		sprintf( szDir, "/%lu_dir", ii );
    		myfs.mkdir( szDir );
    	}
    }
    
    int loopLimit = 0; // -1 continuous, otherwise # to count down to 0
    bool pauseDir = false;  // Start Pause on each off
    bool showDir =  false;  // false Start Dir on each off
    bool bDirVerify =  false;  // false Start Dir on each off
    void loop() {
    	char szDir[16];
    	LoopCnt++;
    	uint32_t chStep;
    	if ( loopLimit != 0 ) {
    #ifdef ROOTONLY // ii=1-NUMDIRS are subdirs. #0 is Root
    		for ( uint32_t ii = 0; ii < 1 && ( loopLimit != 0 ); ii++ )
    #else
    		for ( uint32_t ii = 0; ii < NUMDIRS && ( loopLimit != 0 ); ii++ )
    #endif
    		{
    			if ( ii == 0 )
    				sprintf( szDir, "/" );
    			else
    				sprintf( szDir, "/%lu_dir", ii );
    			chStep = fileCycle(szDir);
    			while ( chStep != fileCycle(szDir) && ( loopLimit != 0 ) ) checkInput( 0 ); // user input can 0 loopLimit
    		}
    		checkInput( 0 );
    		if ( loopLimit > 0 ) // -1 means continuous
    			loopLimit--;
    	}
    	else
    		checkInput( 1 );
    }
    
    char szInputs[] = "0123456789RdchkFqvplmux?";
    void checkInput( int step ) { // prompt for input without user input with step != 0
    	static uint32_t lastTime;
    	uint32_t nowTime = micros();
    
    	char retVal = 0, temp;
    	char *pTemp;
    	if ( step != 0 ) {
    		nowTime -= lastTime;
    		Serial.printf( "[%6.2f M](%6.2f elap) Awaiting input %s loops left %d >", millis() / 60000.0, nowTime / 60000000.0, szInputs, loopLimit );
    		lastTime = micros();
    	}
    	else {
    		if ( !Serial.available() ) return;
    		nowTime -= lastTime;
    		Serial.printf( "[%6.2f M](%6.2f elap) Awaiting input %s loops left %d >", millis() / 60000.0, nowTime / 60000000.0, szInputs, loopLimit );
    		//Serial.printf( "[%6.2f] Awaiting input %s loops left %d >", millis() / 60000.0, szInputs, loopLimit );
    		lastTime = micros();
    		while ( Serial.available() ) {
    			temp = Serial.read( );
    			if ( (pTemp = strchr(szInputs, temp)) ) {
    				retVal = pTemp[0];
    				parseCmd( retVal );
    			}
    		}
    	}
    	while ( !Serial.available() );
    	while ( Serial.available() ) {
    		temp = Serial.read();
    		if ( (pTemp = strchr(szInputs, temp)) ) {
    			retVal = pTemp[0];
    			parseCmd( retVal );
    		}
    	}
    	Serial.print( '\n' );
    	if ( '?' == retVal ) checkInput( 1 ); // recurse on '?' to allow command show and response
    	return;
    }
    void parseCmd( char chIn ) { // pass chIn == '?' for help
    	switch (chIn ) {
    	case '?':
    		Serial.printf( "%s\n", " 0, 1-9 '#' passes continue loop before Pause\n\
     'R' Restart Teensy\n\
     'd' Directory of LittleFS\n\
     'c' Continuous Loop\n\
     'h' Hundred loops\n\
     'k' Thousand loops\n\
     'F' LittleFS_ Low Level Format Disk \n\
     'q' LittleFS_ Quick Format Disk \n\
     'v' Verbose All Dir Prints - TOGGLE\n\
     'p' Pause after all Dir prints - TOGGLE\n\
     'l' Show count of loop()'s, Bytes Read,Written\n\
     'm' Make ROOT dirs (needed after format !ROOTONLY)\n\
     'u' Update Filecount\n\
     'x' Directory filecount verify - TOGGLE\n\
     '?' Help list" );
    		break;
    	case 'R':
    		Serial.print(" RESTART Teensy ...");
    		delay(100);
    		SCB_AIRCR = 0x05FA0004;
    		break;
    	case '0':
    	case '1':
    	case '2':
    	case '3':
    	case '4':
    	case '5':
    	case '6':
    	case '7':
    	case '8':
    	case '9':
    		loopLimit = chIn - '0';
    		break;
    	case 'c':
    		loopLimit = -1;
    		break;
    	case 'd':
    		Serial.print( " d\n" );
    		printDirectory();
    		Serial.print( '\n' );
    		parseCmd( 'l' );
    		checkInput( 1 );
    		chIn = 0;
    		break;
    	case 'h':
    		loopLimit = 100;
    		break;
    	case 'k':
    		loopLimit = 1000;
    		break;
    	case 'F': // Low Level format
    		Serial.print( "\nFormatting Low Level:\n\t" );
    		myfs.lowLevelFormat('.');
    		Serial.print( "\n Done Formatting Low Level.\n" );
    		errsLFS = 0; // No Errors on new Format
    		parseCmd( 'u' );
    		break;
    	case 'q': // quick format
    		myfs.quickFormat();
    		errsLFS = 0; // No Errors on new Format
    		parseCmd( 'u' );
    		break;
    	case 'v': // verbose dir
    		showDir = !showDir;
    		showDir ? Serial.print(" Verbose on: ") : Serial.print(" Verbose off: ");
    		chIn = 0;
    		break;
    	case 'p': // pause on dirs
    		pauseDir = !pauseDir;
    		pauseDir ? Serial.print(" Pause on: ") : Serial.print(" Pause off: ");
    		chIn = 0;
    		break;
    	case 'x': // dir filecount Verify
    		bDirVerify = !bDirVerify;
    		bDirVerify ? Serial.print(" FileCnt on: ") : Serial.print(" FileCnt off: ");
    		dirVerify();
    		chIn = 0;
    		break;
    
    	case 'l': // Show Loop Count
    		Serial.printf("\n\t Loop Count: %u (#fileCycle=%u), Bytes read %u, written %u, #Files=%u\n", LoopCnt, lCnt, rdCnt, wrCnt, filecount );
    		if ( 0 != errsLFS )
    			Serial.printf("\t ERROR COUNT =%u\n", errsLFS );
    		dirVerify();
    		chIn = 0;
    		break;
    	case 'm':
    		Serial.printf("m \n\t Making Root Dirs\n" );
    		makeRootDirs();
    		parseCmd( 'd' );
    		chIn = 0;
    		break;
    	case 'u': // Show Loop Count
    		filecount = printDirectoryFilecount( myfs.open("/") );
    		Serial.printf("u \n\t Updated filecount %u\n", filecount );
    		chIn = 0;
    		break;
    
    	default:
    		Serial.println( chIn ); // never see without unhandled char in szInputs[]
    		break;
    	}
    	if ( 0 != chIn ) Serial.print( chIn );
    }
    
    uint32_t fTot, totSize;
    void printDirectory() {
    	fTot = 0, totSize = 0;
    	Serial.printf("printDirectory %s\n--------------\n", szDiskMem);
    	printDirectory(myfs.open("/"), 0);
    	Serial.printf(" %Total %u files of Size %u Bytes\n", fTot, totSize);
    	Serial.printf("Bytes Used: %llu, Bytes Total:%llu\n", myfs.usedSize(), myfs.totalSize());
    }
    
    unsigned int printDirectoryFilecount(File dir) {
    	unsigned int filecnt = 0;
    	while (true) {
    		File entry =  dir.openNextFile();
    		if (! entry) {
    			// no more files
    			break;
    		}
    		if (entry.isDirectory()) {
    			filecnt += printDirectoryFilecount(entry);
    		} else {
    			filecnt++;
    		}
    		entry.close();
    	}
    	return filecnt;
    }
    
    void printDirectory(File dir, int numTabs) {
    	//dir.whoami();
    	uint32_t fSize = 0, dCnt = 0, fCnt = 0;
    	if ( 0 == dir ) {
    		Serial.printf( "\t>>>\t>>>>> No Dir\n" );
    		return;
    	}
    	while (true) {
    		File entry =  dir.openNextFile();
    		if (! entry) {
    			// no more files
    			Serial.printf("\n %u dirs with %u files of Size %u Bytes\n", dCnt, fCnt, fSize);
    			fTot += fCnt;
    			totSize += fSize;
    			break;
    		}
    		for (uint8_t i = 0; i < numTabs; i++) {
    			Serial.print('\t');
    		}
    
    		if (entry.isDirectory()) {
    			Serial.print("DIR\t");
    			dCnt++;
    		} else {
    			Serial.print("FILE\t");
    			fCnt++;
    			fSize += entry.size();
    		}
    		Serial.print(entry.name());
    		if (entry.isDirectory()) {
    			Serial.println(" / ");
    			printDirectory(entry, numTabs + 1);
    		} else {
    			// files have sizes, directories do not
    			Serial.print("\t\t");
    			Serial.println(entry.size(), DEC);
    		}
    		entry.close();
    		//Serial.flush();
    	}
    }
    
    uint32_t cCnt = 0;
    uint32_t fileCycle(const char *dir) {
    	static char szFile[] = "_file.txt";
    	char szPath[150];
    	int ii;
    	lCnt++;
    	byte nNum = lCnt % MAXNUM;
    	char chNow = 'A' + lCnt % MAXNUM;
    	lfs_ssize_t resW = 1;
    
    	if ( dir[1] == 0 )	// catch root
    		sprintf( szPath, "/%c%s", chNow, szFile );
    	else
    		sprintf( szPath, "%s/%c%s", dir, chNow, szFile );
    	if ( cCnt >= 3 && myfs.exists(szPath) ) { // DELETE ALL KNOWN FILES
    		if ( nNum == 1 ) {
    			Serial.print( "\n == == ==   DELETE PASS START  == == == = \n");
    			if ( showDir ) {
    				printDirectory();
    				Serial.print( " == == ==   DELETE PASS START  == == == = \n");
    			}
    			delayMicroseconds(DELDELAY);
    		}
    	}
    	Serial.printf( ":: %s ", szPath );
    	if ( cCnt >= 3 && myfs.exists(szPath) ) { // DELETE ALL KNOWN FILES
    		readVerify( szPath, chNow );
    		myfs.remove(szPath);
    		filecount--;
    		Serial.printf(" %s ----DEL----", szDiskMem);
    		Serial.printf(" -- %c", chNow);
    		if ( showDir ) {
    			Serial.print("\n");
    			printDirectory(myfs.open(dir), 1);
    		}
    		if ( pauseDir ) checkInput( 1 );
    		Serial.println();
    	}
    	else {
    		if ( nNum == 0 ) {
    			nNum = 10;
    			cCnt++;
    			if ( cCnt >= 5 ) cCnt = 0;
    		}
    		file3 = myfs.open(szPath, FILE_WRITE);
    		if ( 0 == file3 ) {
    			Serial.printf( "\tXXX\tXXX\tXXX\tXXX\tFail File open {mkdir?}\n" );
    			delayMicroseconds(300000);
    			checkInput( 1 );	// PAUSE on CmdLine
    		}
    		else {
    			delayMicroseconds(ADDDELAY);
    			char mm = chNow + lowOffset;
    			uint32_t jj = file3.size() + 1;
    			for ( ii = 0; ii < (nNum * SUBADD + BIGADD ) && resW > 0; ii++ ) {
    				if ( 0 == ((ii + jj) / lowShift) % 2 )
    					resW = file3.write( &mm , 1 );
    				else
    					resW = file3.write( &chNow , 1 );
    				wrCnt++;
    				// if ( lCnt%100 == 50 ) mm='x'; // GENERATE ERROR to detect on DELETE read verify
    			}
    			file3.close();
    			Serial.printf(" %s +++ Add +++ [sz %u add %u]", szDiskMem, jj - 1, ii);
    			if (resW < 0) {
    				Serial.printf( "\n\twrite fail %i\n", resW );
    				parseCmd( '0' );
    				errsLFS++;
    				checkInput( 1 );	// PAUSE on CmdLine
    			}
    			else if ( jj == 1 ) filecount++; // File Added
    			Serial.printf(" ++ %c ", chNow);
    			readVerify( szPath, chNow );
    			if ( showDir ) {
    				Serial.print("\n");
    				printDirectory(myfs.open(dir), 1);
    			}
    		}
    		if ( pauseDir ) checkInput( 1 );
    		Serial.print("\n");
    		delayMicroseconds(ADDDELAY);
    	}
    	checkInput( 0 ); // user stop request?
    	if ( bDirVerify ) dirVerify();
    	return cCnt;
    }
    
    void dirVerify() {
    	if ( filecount != printDirectoryFilecount( myfs.open("/") ) ) {
    		Serial.printf( "\tFilecount mismatch %u != %u\n", filecount, printDirectoryFilecount( myfs.open("/") ) );
    		parseCmd( '0' );
    		errsLFS++;
    		checkInput( 1 );	// PAUSE on CmdLine
    	}
    }
    
    void readVerify( char szPath[], char chNow ) {
    	file3 = myfs.open(szPath);
    	if ( 0 == file3 ) {
    		Serial.printf( "\tV\t Fail File open %s\n", szPath );
    		parseCmd( '0' );
    		errsLFS++;
    		checkInput( 1 );
    	}
    	char mm;
    	char chNow2 = chNow + lowOffset;
    	uint32_t ii = 0;
    	while ( file3.available() ) {
    		file3.read( &mm , 1 );
    		rdCnt++;
    		//Serial.print( mm ); // show chars as read
    		ii++;
    		if ( 0 == (ii / lowShift) % 2 ) {
    			if ( chNow2 != mm ) {
    				Serial.printf( "<Bad Byte!  %c! = %c [0x%X] @%u\n", chNow2, mm, mm, ii );
    				parseCmd( '0' );
    				errsLFS++;
    				checkInput( 1 );
    				break;
    			}
    		}
    		else {
    			if ( chNow != mm ) {
    				Serial.printf( "<Bad Byte!  %c! = %c [0x%X] @%u\n", chNow, mm, mm, ii );
    				parseCmd( '0' );
    				errsLFS++;
    				checkInput( 1 );
    				break;
    			}
    		}
    	}
    	Serial.printf( "\tVerify %s bytes %u ", szPath, ii );
    	if (ii != file3.size()) {
    		Serial.printf( "\n\tRead Count fail! :: read %u != f.size %u", ii, file3.size() );
    		parseCmd( '0' );
    		errsLFS++;
    		checkInput( 1 );	// PAUSE on CmdLine
    	}
    	file3.close();
    }
    Last edited by defragster; 11-19-2020 at 12:55 AM. Reason: updated 'F' to show format : now w/progress dots :: Count/show when errs and auto '0' to stop

  12. #337
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    6,125
    Quote Originally Posted by KurtE View Post
    Quick update: I tried with different files and it looks like it hits up against some maximum time timeout or the like or maybe Teensy gets into some deadlock or???

    For the heck of I tried by using RAM (This one has an 8mb PSRAM installed). So edited to enable RAM, Changed the EXTRAM to allocate 4mb size buffer.

    I then tried to copy the larger files to this extram disk and it appeared to not have any problems doing so. However the file sizes look wrong!

    Click image for larger version. 

Name:	screenshot2.jpg 
Views:	18 
Size:	96.9 KB 
ID:	22507

    Hopefully you can see that for example the file I copied T4.1-cardlike.jpg on my main machine is 384kb, the resulting file on the RAM disk says it is 36.8kb... Maybe we just have some limitation on max file size on RAM disk or???

    Have a good Diner!
    First will do but was playing with the T3.5 and Littlefs. Sort of work. Still problem with that on pdf I have that does work on a SD card.

    Anyway, the jpg that I have from downloading from your post is only 214kb. That one works using RAM, SPI and on QSPI. When I put it on RAM it does show correct file size of 214 KB.

    Really have to dig into the code tomorrow to see what changed and see what it says - I am using a buffer of 40Kb. Getting same behavior even using the master branch. Maybe have to look over the spec.

  13. #338
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,203
    Posted code in #336 without actually testing a lowLevelFormat.

    Just did that on SPI disk and it completed. Added print notice before and after for user ...

    I saw a progress 'char' could be passed ... I tried that and it shows WAY TOO many ...

  14. #339
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,203
    Did a progress PR : Limit progressChar SPEW chars - assuming .block_count is a large number

    Tested SPI, QSPI, RAM , RAM is the fastest

    Tested LittleFS_Program - fast before use as formatted on upload - which may explain why TyCommander seemed to work once - it expect Teensy to come back online faster than it does when the onboard flash takes a long time to format first.
    Last edited by defragster; 11-18-2020 at 12:25 AM.

  15. #340
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    23,485
    I've added FILE_WRITE_BEGIN mode for FS open().

    https://github.com/PaulStoffregen/co...88fdfb3be48383

    https://github.com/PaulStoffregen/Li...57edbb0b0cd5ff

    https://github.com/PaulStoffregen/SD...17c19916be7e37

    This provides a more efficient way to open a file for writing (and read) where you intend to use random access with seek() or just overwrite the existing data. While you can just use seek(0) to go back to the beginning after open(filename, FILE_WRITE), this avoids the unnecessary overhead of the finding the end of the file.

  16. #341
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,203
    Did an LFSintegrity run to better fill 16M - then 'F'ormat for '.' timing. The dots were under 3 secs each with full llFormat in 96 seconds.

    Using:
    Code:
    #define NUMDIRS 28  // When not ROOTONLY must be 1 or more
    ...
    #define SUBADD 1000	// bytes added each pass (*times file number)
    #define BIGADD 1000	// bytes added each pass - bigger will quickly consume more space
    #define MAXNUM 26	// ALPHA A-Z is 26, less for fewer files
    This is 16MB QSPI:
    Code:
     Total 705 files of Size 13965000 Bytes
    Bytes Used: 15450112, Bytes Total:16777216
    	 Loop Count: 20 (#fileCycle=5925), Bytes read 149734100, written 58843000, #Files=705
    So that is enough slack space it seems as it was running at this point for a bit

    Doing Repro on SPI now ... took longer to fill - about 2X
    Code:
     Total 726 files of Size 15141000 Bytes
    Bytes Used: 16654336, Bytes Total:16777216
    
    	 Loop Count: 20 (#fileCycle=5176), Bytes read 116625000, written 51623000, #Files=726
    SPI format is longer too 5:48 or 348 seconds ... 8.7 seconds each for 40 dot chars. On the same chip ...

    Back to filling the disk - after SPI Format restarted and it filled to FULL:
    Code:
    :: /27_dir/X_file.txt  SPI_DISK +++ Add +++ [sz 0 add 24000] ++ X 	Verify /27_dir/X_file.txt bytes 24000 
    :: /27_dir/Y_file.txt  SPI_DISK +++ Add +++ [sz 0 add 257]	write fail -28[ 53.98 M] Awaiting input 0123456789RdchkFqvplmux? loops left 4 >0
     ++ Y 	Verify /27_dir/Y_file.txt bytes 0 
    [ 54.81 M] Awaiting input 0123456789RdchkFqvplmux? loops left 0 >
    [ 54.86 M] Awaiting input 0123456789RdchkFqvplmux? loops left 0 >
    	 Loop Count: 32 (#fileCycle=7148), Bytes read 162737000, written 73709257, #Files=457
    	Filecount mismatch 457 != 458
    
    ...
     Total 458 files of Size 15623000 Bytes
    Bytes Used: 16777216, Bytes Total:16777216
    
    	 Loop Count: 32 (#fileCycle=7148), Bytes read 162737000, written 73709257, #Files=457
    	Filecount mismatch 457 != 458
    Error -28 is :: LFS_ERR_NOSPC = -28, // No space left on device

    So it acted right up to then on 16M SPI - except I counted the file as added - when it wasn't

    Format of that took over 9 minutes ... ICK ????

  17. #342
    Senior Member
    Join Date
    Jul 2014
    Posts
    3,004
    Quote Originally Posted by PaulStoffregen View Post
    I want to be perfectly clear. This sort of code will not be supported in Teensyduino 1.54.
    As you realized later, this coding style WAS supported, at least for a while.
    Anyhow, my exercise was only to make the case that I suggested a coding style like (this is an example!)
    Code:
    File file = SD.open(filename,(O_EXCL | O_WRONLY | O_CREAT));
    if(file) {...}
    and not
    Code:
    File file;
    if(!SD.exists(filename))  file = SD.open(filename,FILE_WRITE); else { /* handle case */ }
    file.seek(0);
    (Yes, I know FILE_WRITE_BEGIN was added, so that code could be simplified)

    or the one I showed earlier, and you mistakenly assumed it is my suggestion.

    I do not care if the the suggested additional line is more general
    Code:
    File open(char *filename, uint8_t mode); // Arduino mode
    ....
    File open(char *filename, uint32_t mode); // 32 bit mode for more flexibility
    //or
    File open(char *filename, oflag_t mode);  // SDFat int mode
    Anyhow, I will close this argument and PRs I made

  18. #343
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,203
    Drive wasn't filled - format took 6.14 minutes (New ( 6.14 elap)sed time on command) this time:
    Code:
    Bytes Used: 15691776, Bytes Total:16777216
    [  0.26 M](  0.17 elap) Awaiting input 0123456789RdchkFqvplmux? loops left 0 >
    Formatting Low Level: .........................................................................................................................
    
     Done Formatting Low Level.
    u 
    	 Updated filecount 0
    F
    [  6.40 M](  6.14 elap)
    That is with the 120 chars printed as in current PR updated so dots aren't so long (6-9 secs) between - but on TyComm with non-prop fonts it can push under scroll bar to finish or narrow window.
    These are 3 secs between - and larger disks later may take even longer than 6-9 minutes - but better for now.

    So that's the lowLevelFormat 'progress' story ... the good news is a nearly FULL Q/SPI drive works up until all space is allocated - then will return an expected error.
    Wear leveling is limited to newly freed blocks - but with haphazard cycling by LPSintegrity test - they are probably well distributed.

  19. #344
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    6,125
    @KurtE

    Looks like MTP for storage areas isn't working so well after the last couple of updates. Still trying to figure out whats wrong. Going to take awhile I think. Need another diversion.

  20. #345
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    8,217
    @mjs513 - Let me know if there is another iteration of tests I should run on it...

    I might play a little with other diversion.

  21. #346
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    6,125
    Quote Originally Posted by KurtE View Post
    @mjs513 - Let me know if there is another iteration of tests I should run on it...

    I might play a little with other diversion.
    I will go on to other distractions. I just tested the updated MTPResponder with Builtin_SDCard and a breakout card reader and MTP for SD is failing reading anything but the Builtin_sdcard. So may be another issue. Going to post on the other thread now. Begining to get tired of MTP, think I need a diversion just to clear my head.

  22. #347
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,203
    Newest version of LFSintegrity.ino in post #336

    Updated with LittleFS constrained progress char print on LLformat.

    Now with elapsed time track on prompt line from time last command sent until prompt appears : like issuing 'F'ormat or other.

    <edit>: Updated again above - counts errors and does auto set of '0' to not continue right away on error.
    Last edited by defragster; 11-19-2020 at 01:00 AM.

  23. #348
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    6,125
    Quote Originally Posted by PaulStoffregen View Post
    One more detail is the definition of the "msd" pointer. If you're designing code like MTP which is meant to work with filesystems, you should probably define the pointer this way:

    Code:
      FS *msd = &SD;
    If you're write this sort of definition

    Code:
      SDClass *msd = &SD;
    then you're unnecessarily limiting yourself to only SD cards. The purpose of the FS class is to allow this sort of code to automatically work with LittleFS and SD, and other future filesystems we don't even imagine today.
    Evening Paul - been keeping this post in the back of my mind since you posted it and guess I am ready at this point to ask how does this really work. The reason I am asking is that @WMXZ came up with an interesting way to possibly incorporate everything in a single MTP library but now running into conflicts between littileFS and SDClass which you kind of allude to with the second format.

    Right now in the storage class I have a slight mod based on what I was working on to his existing coded so you could have multiple RAM storage devices along with his multiple SD Cards. But getting conflicts in SDClass:
    Code:
    In file included from D:\Users\Merli\Documents\Arduino\libraries\MTP_t4-master\src/Storage.h:32:0,
                     from D:\Users\Merli\Documents\Arduino\libraries\MTP_t4-master\src/MTP.h:36,
                     from D:\Users\Merli\Documents\Arduino\libraries\MTP_t4-master\examples\mtp-test\mtp-test.ino:6:
    F:\arduino-1.8.13-beta4\hardware\teensy\avr\libraries\SD\src/SD.h:159:49: error: 'FILE_READ' was not declared in this scope
      File open(const char *filepath, uint8_t mode = FILE_READ) {
                                                     ^
    F:\arduino-1.8.13-beta4\hardware\teensy\avr\libraries\SD\src/SD.h: In member function 'virtual File SDClass::open(const char*, uint8_t)':
    F:\arduino-1.8.13-beta4\hardware\teensy\avr\libraries\SD\src/SD.h:161:15: error: 'FILE_WRITE' was not declared in this scope
       if (mode == FILE_WRITE) flags = O_RDWR | O_CREAT | O_AT_END;
                   ^
    In file included from D:\Users\Merli\Documents\Arduino\libraries\MTP_t4-master\src/MTP.h:36:0,
                     from D:\Users\Merli\Documents\Arduino\libraries\MTP_t4-master\examples\mtp-test\mtp-test.ino:6:
    D:\Users\Merli\Documents\Arduino\libraries\MTP_t4-master\src/Storage.h: At global scope:
    D:\Users\Merli\Documents\Arduino\libraries\MTP_t4-master\src/Storage.h:195:52: error: 'FILE_READ' was not declared in this scope
       void OpenFileByIndex(uint32_t i, uint32_t mode = FILE_READ) ;
    believe it coming from this portion of the code;
    Code:
    #include "SD.h"
    #ifndef FILE_WRITE_BEGIN
      #define FILE_WRITE_BEGIN 2
    #endif
    
    // following is a device specific base class for storage classs
    extern SDClass sdx[];
    
    #ifdef USE_RAM
    #include "LittleFS.h"
    extern LittleFS_RAM ramfs[];
    
    class mSD_Base
    { 
      public:
      File sd_open(uint32_t store, const char *filename, uint32_t mode) 
      { if(!cs || (cs[store]<256)) return sdx[store].open(filename,mode); else return ramfs[store].open(filename,mode);
      }
      bool sd_mkdir(uint32_t store, char *filename) 
      { if(!cs || (cs[store]<256)) return sdx[store].mkdir(filename); else return ramfs[store].mkdir(filename);
      }
    
      bool sd_rename(uint32_t store, char *oldfilename, char *newfilename) 
      { if(!cs || (cs[store]<256)) return sdx[store].rename(oldfilename,newfilename); else return ramfs[store].rename(oldfilename,newfilename);
        
      }
      bool sd_remove(uint32_t store, const char *filename) 
      { if(!cs || (cs[store]<256)) return sdx[store].remove(filename); else return ramfs[store].remove(filename);
      }
      bool sd_rmdir(uint32_t store, char *filename) 
      { if(!cs || (cs[store]<256)) return sdx[store].rmdir(filename); else return ramfs[store].rmdir(filename);
      }
        
      uint32_t sd_totalClusterCount(uint32_t store) 
      { if(!cs || (cs[store]<256)) return sdx[store].sdfs.clusterCount(); else return ramfs[store].totalSize()/512;
      }
      uint32_t sd_freeClusterCount(uint32_t store)  
      { if(!cs || (cs[store]<256)) return sdx[store].sdfs.freeClusterCount(); else return (ramfs[store].totalSize()-ramfs.usedSize())/512;
      }
      uint32_t sd_sectorsPerCluster(uint32_t store) 
      { if(!cs || (cs[store]<256)) return sdx[store].sdfs.sectorsPerCluster(); else return 1;
      }
    
      bool setCs(const int *csx) { cs = csx; return true; }
    
      private:
      const int * cs = 0;
    };
    but I am lost on how to incorporate your recommendation. If you get a minute can you advise. If anyone else wants to jump to the discussion feel free. Stuck on getting everything to play nice together. Of course there are other issues but those I think I have a solution for.

  24. #349
    Senior Member
    Join Date
    Jul 2014
    Posts
    3,004
    Quote Originally Posted by PaulStoffregen View Post
    One more detail is the definition of the "msd" pointer. If you're designing code like MTP which is meant to work with filesystems, you should probably define the pointer this way:

    Code:
      FS *msd = &SD;
    If you're write this sort of definition

    Code:
      SDClass *msd = &SD;
    then you're unnecessarily limiting yourself to only SD cards. The purpose of the FS class is to allow this sort of code to automatically work with LittleFS and SD, and other future filesystems we don't even imagine today.
    I thought so (use of interfaces), but by extending lost this concept.
    Will revisit my MTP implementation in this light.

  25. #350
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    6,125
    Using the current version of MTP with mods for RAM and SPIFlash:
    Click image for larger version. 

Name:	Capture4.PNG 
Views:	7 
Size:	10.9 KB 
ID:	22533

    Right now this is a proof of concept. Extending now to QSPI should be easy.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •