LittleFS port to Teensy/SPIFlash

[defragster]

@defragster - I was looking at SdFat exsits() function and it looks like it only tests for files not sub directories.

Have to go to doctor and then work now. Will look at this more tonight after work.

Interesting - time was short took exist(dir) at face value - it gave expected results - because they don't exist when done either

Will see if I get back to it before your day gets there ...

 

[wwatson]

That's what I saw. Time to create a simple method to detect the existence of a directory I think if possible. Maybe check to see if object is a directory and if it is then compare the object name to the name we are trying to create. If the name matches then it exists else if it not a directory then create the directory?

My days are usually weekends

 

[defragster]

Odd situation - since it isn't working the obvious way in process - it exists in source media - and likely not in the dest media.

With recursion the processing may be going funny - but what works in SD ... if I have symmetric code ... isn't working on LFS.

... will poke around a bit ... doesn't seem I'm missing anything obvious ...

 

[defragster]

Figured it out - can't ask LFS mkdir to make a dir with a trailing "/" - works on SD but not LFS.

This change makes it work BOTH to SD and to LFS !!!!

BAD:

			strcat( szNewDir, entry.name());
			[B][COLOR="#FF0000"]strcat( szNewDir, "/");[/COLOR][/B]
			if ( destMedia == destLFS ) myfs.mkdir( szNewDir );
			else SD.mkdir( szNewDir );
			Serial.print("\n");
			errCnt += mediaTransfer(entry, szNewDir, destMedia);

WORKS:

			strcat( szNewDir, entry.name());
			if ( destMedia == destLFS ) myfs.mkdir( szNewDir );
			else SD.mkdir( szNewDir );
			Serial.print("\n");
			[B][COLOR="#FF0000"]strcat( szNewDir, "/");[/COLOR][/B]
			errCnt += mediaTransfer(entry, szNewDir, destMedia);

Will post usable code in a bit ...

 

[defragster]

LittleFS media data Save or Restore from SD card

Using one of these two commented xferSD() calls:

#define destLFS 1
#define destSD 2
	// xferSD( destSD ); // do MediaTransfer LFS TO SD
	// xferSD( destLFS ); // do MediaTransfer SD TO LFS

NOTE: In Any LittleFS/examples/Integrity place it in setup() before this line to prevent errors reported in filecount:

	filecount = printDirectoryFilecount( myfs.open("/") );  // Set base value of filecount for disk

And this piece of INO code { has to be INO to share the global "myfs" - unless it is made Media specific and extern in user code if not visibly global}.

Above placed in any of the LittleFS/examples/Integrity using this sketch code will 'recursively' copy as indicated between LFS and SD Media - all files and directories. Tested on PSRAM.ino and PROG.ino.
Removed most debug spew, only two cases of error displayed.
Example will use SD directory: "\LFS_CPY". SD Media of GB bigger than any LFS Media's MB's, so makes no sense to store/copy all of SD root. This SD directory name can be tested and changed programmatically {with change in xferSD()} to allow saving multiple runs of data if desired.

NOTE: This code has a DEBUG copy of makeRootDirsTest() that adds extra files and dirs to prove function when sending LFS data to SD. It is not needed in use.

For a quick test in Integrity examples put in setup - Fill the LFS Media using '3' to make some test files {perhaps "S" or "B"}, then restart to push to SD with destSD.
Then do a 'w' to Wipe all LFS Media files, and rebuild with destLFS and the target SD card that should hold the files. It will then restore to any LFS media - though it does not check for SIZE issues if the contents won't restore to Media in use.

#include <SD.h>

void xferSD( int copyType ) { // do MediaTransfer with SD
#define destLFS 1
#define destSD 2

	static bool initSD = true;
	Serial.print("Initializing SD card...");
	if ( initSD && !SD.begin(BUILTIN_SDCARD)) { // see if the card is present and can be initialized:
		Serial.println("\n\n SD Card failed, or not present - Cannot do Xfer\n");
	}
	else {
		initSD = false;
		Serial.println("card initialized.\n\n");
		if ( copyType == destSD ) {
			// char szSDdir[] = "/"; // COPY to SD ROOT
			char szSDdir[] = "LFS_CPY/"; // COPY to SD subdirectory
			if ( '/' != szSDdir[0] )
				SD.mkdir( szSDdir );
			[B]makeRootDirsTest(); // BUGBUG DEBUG make extra subdirs and files to show function[/B]
			Serial.println("\n STARTING :: LittleFS copy to SD card XFER ...\n\n");
			mediaTransfer( myfs.open("/"), szSDdir, destSD ); // TOO SD
			Serial.println("\n LittleFS copy to SD card XFER COMPLETE.\n\n");
		}
		else {
			char szLFSdir[] = "/";
			char szSDdir[] = "LFS_CPY";
			Serial.println("\n STARTING :: SD card copy to LittleFS XFER ...\n\n");
			mediaTransfer( SD.open(szSDdir), szLFSdir, destLFS );	// FROM SD
			Serial.println("\n SD card copy to LittleFS XFER COMPLETE.\n\n");
		}
	}
}

void mediaTransfer(File dir, char* szDir, int destMedia) {
	char szNewDir[36];
	while (true) {
		File entry =  dir.openNextFile();
		if (! entry) {
			break;
		}
		if (entry.isDirectory()) {
			strcpy( szNewDir, szDir);
			if ( destMedia == destLFS )
				myfs.mkdir( szNewDir );
			else
				SD.mkdir( szNewDir );
			strcat( szNewDir, entry.name());
			if ( destMedia == destLFS ) myfs.mkdir( szNewDir );
			else SD.mkdir( szNewDir );
			strcat( szNewDir, "/");
			mediaTransfer(entry, szNewDir, destMedia);
		} else {
			uint64_t fileSize, sizeCnt = 0, xfSize = 1;
			char mm[512];
			strcpy( szNewDir, szDir);
			strcat( szNewDir, entry.name() );
			File dataFile;
			if ( destMedia == destLFS ) {
				dataFile = myfs.open(szNewDir, FILE_WRITE_BEGIN);
			}
			else {
				dataFile = SD.open(szNewDir, FILE_WRITE_BEGIN);
			}
			if ( !dataFile )
				Serial.print("\td_FILE: NOT open\n");
			fileSize = entry.size();
			while ( entry.available() ) {
				if ( fileSize < sizeCnt ) break;
				if ( fileSize - sizeCnt >= sizeof(mm) ) xfSize = sizeof(mm);
				else xfSize = fileSize - sizeCnt;
				entry.read( &mm , xfSize );
				dataFile.write( &mm , xfSize );
				sizeCnt += xfSize;
			}
			if (fileSize != sizeCnt ) {
				Serial.print("\n File Size Error:: ");
				Serial.println( entry.name() );
			}
			dataFile.close();
		}
		entry.close();
	}
}

void makeRootDirsTest() {  // TEST DEBUG CODE
	char szDir[36];
	for ( uint32_t ii = 1; ii <= NUMDIRS; ii++ ) {
		sprintf( szDir, "/%lu_dir", ii );
		myfs.mkdir( szDir );
		sprintf( szDir, "/%lu_dir/aFile.txt", ii ); // BUGBUG DEBUG
		file3 = myfs.open(szDir, FILE_WRITE);
		file3.write( szDir , 12 );
		file3.close();

		sprintf( szDir, "/%lu_dir/TEST", ii ); // BUGBUG DEBUG
		myfs.mkdir( szDir ); // BUGBUG DEBUG
		sprintf( szDir, "/%lu_dir/TEST/bFile.txt", ii ); // BUGBUG DEBUG
		file3 = myfs.open(szDir, FILE_WRITE);
		file3.write( szDir , 12 );
		file3.close();
	}
	filecount = printDirectoryFilecount( myfs.open("/") );  // Set base value of filecount for disk
}

This function does all the work when properly called as desired: void mediaTransfer(File dir, char* szDir, int destMedia)

 

[wwatson]

Just tried it out. It works great I ran into that same problem with the forward slashes in my project. I kept forgetting which functions were in control of them.

 

[defragster]

Just tried it out. It works great I ran into that same problem with the forward slashes in my project. I kept forgetting which functions were in control of them.

Awesome- thx for testing/confirming!

... on returning to cleanup/test/upload I must have hit UNDO ... the "/" line went back ... unseen ... and I pulled out all the extra print()'s and .mkdir and .exist ... then it didn't work --- Arrgggh.

 

[wwatson]

Do you need a copy of your last working version? If you do here it is:

#include <SD.h>

void xferSD( int copyType ) { // do MediaTransfer with SD
#define destLFS 1
#define destSD 2

	static bool initSD = true;
	Serial.print("Initializing SD card...");
	if ( initSD && !SD.begin(BUILTIN_SDCARD)) { // see if the card is present and can be initialized:
		Serial.println("\n\n SD Card failed, or not present - Cannot do Xfer\n");
	}
	else {
		initSD = false;
		Serial.println("card initialized.\n\n");
		if ( copyType == destSD ) {
			// char szSDdir[] = "/"; // COPY to SD ROOT
			char szSDdir[] = "LFS_CPY/"; // COPY to SD subdirectory
			if ( '/' != szSDdir[0] )
				SD.mkdir( szSDdir );
			makeRootDirsTest(); // BUGBUG DEBUG make extra subdirs and files to show function
			Serial.println("\n STARTING :: LittleFS copy to SD card XFER ...\n\n");
			mediaTransfer( myfs.open("/"), szSDdir, destSD ); // TOO SD
			Serial.println("\n LittleFS copy to SD card XFER COMPLETE.\n\n");
		}
		else {
			char szLFSdir[] = "/";
			char szSDdir[] = "LFS_CPY";
			Serial.println("\n STARTING :: SD card copy to LittleFS XFER ...\n\n");
			mediaTransfer( SD.open(szSDdir), szLFSdir, destLFS );	// FROM SD
			Serial.println("\n SD card copy to LittleFS XFER COMPLETE.\n\n");
		}
	}
}

void mediaTransfer(File dir, char* szDir, int destMedia) {
	char szNewDir[36];
	while (true) {
		File entry =  dir.openNextFile();
		if (! entry) {
			break;
		}
		if (entry.isDirectory()) {
			strcpy( szNewDir, szDir);
			if ( destMedia == destLFS )
				myfs.mkdir( szNewDir );
			else
				SD.mkdir( szNewDir );
			strcat( szNewDir, entry.name());
			if ( destMedia == destLFS ) myfs.mkdir( szNewDir );
			else SD.mkdir( szNewDir );
			strcat( szNewDir, "/");
			mediaTransfer(entry, szNewDir, destMedia);
		} else {
			uint64_t fileSize, sizeCnt = 0, xfSize = 1;
			char mm[512];
			strcpy( szNewDir, szDir);
			strcat( szNewDir, entry.name() );
			File dataFile;
			if ( destMedia == destLFS ) {
				dataFile = myfs.open(szNewDir, FILE_WRITE_BEGIN);
			}
			else {
				dataFile = SD.open(szNewDir, FILE_WRITE_BEGIN);
			}
			if ( !dataFile )
				Serial.print("\td_FILE: NOT open\n");
			fileSize = entry.size();
			while ( entry.available() ) {
				if ( fileSize < sizeCnt ) break;
				if ( fileSize - sizeCnt >= sizeof(mm) ) xfSize = sizeof(mm);
				else xfSize = fileSize - sizeCnt;
				entry.read( &mm , xfSize );
				dataFile.write( &mm , xfSize );
				sizeCnt += xfSize;
			}
			if (fileSize != sizeCnt ) {
				Serial.print("\n File Size Error:: ");
				Serial.println( entry.name() );
			}
			dataFile.close();
		}
		entry.close();
	}
}

void makeRootDirsTest() {  // TEST DEBUG CODE
	char szDir[36];
	for ( uint32_t ii = 1; ii <= NUMDIRS; ii++ ) {
		sprintf( szDir, "/%lu_dir", ii );
		myfs.mkdir( szDir );
		sprintf( szDir, "/%lu_dir/aFile.txt", ii ); // BUGBUG DEBUG
		file3 = myfs.open(szDir, FILE_WRITE);
		file3.write( szDir , 12 );
		file3.close();

		sprintf( szDir, "/%lu_dir/TEST", ii ); // BUGBUG DEBUG
		myfs.mkdir( szDir ); // BUGBUG DEBUG
		sprintf( szDir, "/%lu_dir/TEST/bFile.txt", ii ); // BUGBUG DEBUG
		file3 = myfs.open(szDir, FILE_WRITE);
		file3.write( szDir , 12 );
		file3.close();
	}
	filecount = printDirectoryFilecount( myfs.open("/") );  // Set base value of filecount for disk
}

 

[defragster]

Do you need a copy of your last working version? If you do here it is:
...

No, Thanks, p#929 is after I got it sorted and that posted code was tested to work - just took a bit of extra time ... undoing all the 'cleanup' only then to see the misplaced "/" ... and redo all the cleanup and then test ...

 

[Sandro]

A practical example of LitteFS on Flash chip, with performance comparision.

From quoted linked post:

Hi all,
please let me know if I'd better start a new post instead of writing here...

I went on with my tests, and wish to share the (even partial) result:
- copying .raw audio files from SD to Flash chip: works fine
- creating an example for playing a .raw audio file from LittleFS: ok works fine
- comparing the read-time reading audio files, using traditional SerialFlash solution and LittleFS file system.

This is the sketch using LittleFS, main code:

#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SerialFlash.h>
#include "play_serialflash_raw_LittleFS.h"

AudioPlaySerialflashRaw_LittleFS  playFlash;
AudioOutputI2S           audio_out;
AudioConnection          patchCord1(playFlash, 0, audio_out, 0);
AudioConnection          patchCord2(playFlash, 0, audio_out, 1);
AudioControlSGTL5000     board;

void setup()
{
  board.enable();
  board.volume(0.2);
  AudioMemory(5);
}

void loop()
{
  playFlash.play("41.raw");
  delay(10000);
}

AudioPlaySerialflashRaw_LittleFS.h

#ifndef play_serialflash_raw_LittleFS_h_
#define play_serialflash_raw_LittleFS_h_

#include "Arduino.h"
#include <AudioStream.h>
#include "LittleFS.h"

class AudioPlaySerialflashRaw_LittleFS : public AudioStream
{
public:
	AudioPlaySerialflashRaw_LittleFS(void) : AudioStream(0, NULL)
	{
	    begin();
    }
	void begin(void);
	bool play(const char *filename);
	void stop(void);
	bool isPlaying(void) { return playing; }
	uint32_t positionMillis(void);
	uint32_t lengthMillis(void);
	virtual void update(void);
private:
	LittleFS_SPIFlash myfs; // ha i metodi di LittleFS che ha i metodi di FS
	File rawfile; // SerialFlashFile rawfile;
	uint32_t file_size;
	volatile uint32_t file_offset;
	volatile bool playing;
};

#endif

AudioPlaySerialflashRaw_LittleFS.cpp

#include <Arduino.h>
#include "play_serialflash_raw_LittleFS.h"
#include "spi_interrupt.h"

void AudioPlaySerialflashRaw_LittleFS::begin(void)
{
    playing = false;
    file_offset = 0;
    file_size = 0;
}


bool AudioPlaySerialflashRaw_LittleFS::play(const char *filename)
{
    stop();
    AudioStartUsingSPI();
    // rawfile = SerialFlash.open(filename);
    // if (!rawfile)
    if(!myfs.begin(6))
    {
        Serial.println("unable to access Flash");
        AudioStopUsingSPI();
        return false;
    }
    rawfile = myfs.open(filename);
    if(!rawfile)
    {
        Serial.println("unable to access file");
        AudioStopUsingSPI();
        return false;
    }
    file_size = rawfile.size();
    file_offset = 0;
    //Serial.println("able to open file");
    playing = true;
    return true;
}

void AudioPlaySerialflashRaw_LittleFS::stop(void)
{
    __disable_irq();
    if (playing)
    {
        playing = false;
        __enable_irq();
        rawfile.close();
        AudioStopUsingSPI();
    }
    else
    {
        __enable_irq();
    }
}


void AudioPlaySerialflashRaw_LittleFS::update(void)
{
    unsigned int i, n;
    audio_block_t *block;
    unsigned long T0;
    int time_lap;

    // only update if we're playing
    if (!playing)
        return;

    // allocate the audio blocks to transmit
    block = allocate();
    if (block == NULL) return;

    if (rawfile.available())
    {
        T0 = micros();
        n = rawfile.read(block->data, AUDIO_BLOCK_SAMPLES*2);
        time_lap=micros()-T0;
        file_offset += n;
        for (i=n/2; i < AUDIO_BLOCK_SAMPLES; i++)
        {
            block->data[i] = 0;
        }
        transmit(block);
        Serial.println(time_lap);
    }
    else
    {
        rawfile.close();
        AudioStopUsingSPI();
        playing = false;
    }
    release(block);
}

#define B2M (uint32_t)((double)4294967296000.0 / AUDIO_SAMPLE_RATE_EXACT / 2.0) // 97352592

uint32_t AudioPlaySerialflashRaw_LittleFS::positionMillis(void)
{
    return ((uint64_t)file_offset * B2M) >> 32;
}

uint32_t AudioPlaySerialflashRaw_LittleFS::lengthMillis(void)
{
    return ((uint64_t)file_size * B2M) >> 32;
}

where this part of code allows to measure the reading performance:

        time_lap=micros()-T0;
        file_offset += n;
        for (i=n/2; i < AUDIO_BLOCK_SAMPLES; i++)
        {
            block->data[i] = 0;
        }
        transmit(block);
        Serial.println(time_lap);

Now, using a T4.1@600MHz, and AUDIO_BLOCK_SAMPLES = 128, if I compare the reading performance of the previous code, with a similar example which uses AudioPlaySerialflashRaw:

#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SerialFlash.h>

AudioPlaySerialflashRaw  playFlash;
AudioOutputI2S           audio_out;
AudioConnection          patchCord1(playFlash, 0, audio_out, 0);
AudioConnection          patchCord2(playFlash, 0, audio_out, 1);
AudioControlSGTL5000     board;

void setup()
{
  board.enable();
  board.volume(0.2);
  AudioMemory(5);
  SerialFlash.begin(6);
}

void loop()
{
  playFlash.play("41.raw");
  delay(10000);
}

the update() time in microseconds is:
AudioPlaySerialflashRaw example:

50
50
50
50
50
50
50
50
50
50
50
...
... and so on

AudioPlaySerialflashRaw_LittleFS example:

77
78
696
78
77
78
77
78
77
78
77
78
77
78
77
78
77
78
773
78
77
78
77
78
77
78
77
78
78
78
78
78
78
78
696
78
78
78

This report shows that with LittleFS the reading time is 55% higher for 15 cycles, and after that (should be after 4KB of data) the next cycle is 10 times longer; for my application, where audio polyphony is up to 16, this slow reading time (maybe due to some "jumps" among sectors??) is incompatible

 

[defragster]

@Sandro - seems good to me to post that here - use case and perf data.
Did I miss what Teensy ... ah T_4.1 ... and what Flash media/interface?

 

[Sandro]

@Sandro - seems good to me to post that here - use case and perf data.
Did I miss what Teensy ... ah T_4.1 ... and what Flash media/interface?

Hi defragster, I'm using the Winbond 25Q128FVSG mounted on the Audio board.

 

[defragster]

The SerialFlash was specifically coded to avoid overhead - all blocks sequential and no FS overhead - just minimal linkage.

Not surprising on SPI it will suffer more overhead with LFS at slower speeds. Wonder if a NAND version would mount and read faster even as SPI? Certainly would expect it on QSPI soldered on PCB as it clocks faster - it may not run 4X faster - but it might being enough.

Does the Audio code just read a 256B block at a time? It might be a tweak to help it cache might make the diff - but the slow times would still probably show when missed as to read the data takes reading some other blocks in advance to find the data.

 

[Sandro]

The SerialFlash was specifically coded to avoid overhead - all blocks sequential and no FS overhead - just minimal linkage..

In my test, I prepared the flash chip with a deep format, than I copied from SD card just one file (41.raw, 1.18MB) ; maybe this file, even if alone, has been stored in not-adjacent blocks/sectors of the chip?

Not surprising on SPI it will suffer more overhead with LFS at slower speeds. Wonder if a NAND version would mount and read faster even as SPI? Certainly would expect it on QSPI soldered on PCB as it clocks faster - it may not run 4X faster - but it might being enough.

Right.. I could try with a W25N01GV mounted on the Teensy QSPI expansion (never used one of these nand chips) and check what happens...

Does the Audio codec just read a 256B block at a time? It might be a tweak to help it cache might make the diff - but the slow times would still probably show when missed as to read the data takes reading some other blocks in advance to find the data.

Yes, the Audio codec reads a 256B block each time slot of 2.9ms; but, depending on the desidered pitch, it could be needed to read (for example) only 1 byte ... or 2560 byte in the same slot 2.9ms. Caching audio data would be great but, considering all requiremts in my application (polyphony, number of contemporary voices, audio files dimensions, pitch...) it would occupy many (I mean 32-64) MB of fast/inner data RAM...

 

[defragster]

@Paul - just recalled a historical note that some idea for LFS to have a way to emulate 'perhaps' some 'SerialFlash' type storage files? i.e. direct access to sequential blocked files ???
> seems that is a real recollection ???
> of course it is a non-trivial extension - but what @Sandro is seeing suggests it could be the fix for FS overhead getting to file data for the Audio system in an _isr/DMA responsive time.

 

[Sandro]

@Paul - just recalled a historical note that some idea for LFS to have a way to emulate 'perhaps' some 'SerialFlash' type storage files? i.e. direct access to sequential blocked files ???
> seems that is a real recollection ???
> of course it is a non-trivial extension - but what @Sandro is seeing suggests it could be the fix for FS overhead getting to file data for the Audio system in an _isr/DMA responsive time.

@Paul, @defragster: going back to the origin of my tests around LittleFS is the fact that a SerialFlash-file can be written only once, as @Paul wrote on Github:

Several limitations apply to writing. Only previously unwritten portions of the file may be written. File sizes can never change. Writes may only be done within the file's original size.

and this is the big issue in my case; if this cannot be modified (i trust it cannot) and LittleFS brings too much overhead for audio operations, and optimizing LittleFS in this sense is a big/heavy/.../worthless job, than in my case a workaround can be create a sort of simple (and full of limitations, but I could be able to deploy it by myself with a little fortune..) "virtual file system" with SerialFlash, filling the SPI flash of small, fixed size and createErasable files ("snippet"), than splitting all files among snippets... and so on. This should allow to create/delete/move files without erasing all the Flash content.. In this sense I just tested that reading the content from 2 files (using SerialFlash) there is an almost neglectable delay (1-2 microseconds) when jumping from one file to another.

 

[Sandro]

@Paul, @defragster: going back to the origin of my tests around LittleFS is the fact that a SerialFlash-file can be written only once, as @Paul wrote on Github:

Sorry... I got confused... The issue is that a single SerialFlash-file can be deleted only by deleting all the flash chip.