That's what I saw. Time to create a simple method to detect the existence of a directory I think if possible. Maybe check to see if object is a directory and if it is then compare the object name to the name we are trying to create. If the name matches then it exists else if it not a directory then create the directory?
My days are usually weekends
Odd situation - since it isn't working the obvious way in process - it exists in source media - and likely not in the dest media.
With recursion the processing may be going funny - but what works in SD ... if I have symmetric code ... isn't working on LFS.
... will poke around a bit ... doesn't seem I'm missing anything obvious ...
Figured it out - can't ask LFS mkdir to make a dir with a trailing "/" - works on SD but not LFS.
This change makes it work BOTH to SD and to LFS !!!!
BAD:
WORKS:Code:strcat( szNewDir, entry.name()); strcat( szNewDir, "/"); if ( destMedia == destLFS ) myfs.mkdir( szNewDir ); else SD.mkdir( szNewDir ); Serial.print("\n"); errCnt += mediaTransfer(entry, szNewDir, destMedia);
Will post usable code in a bit ...Code:strcat( szNewDir, entry.name()); if ( destMedia == destLFS ) myfs.mkdir( szNewDir ); else SD.mkdir( szNewDir ); Serial.print("\n"); strcat( szNewDir, "/"); errCnt += mediaTransfer(entry, szNewDir, destMedia);
Using one of these two commented xferSD() calls:
NOTE: In Any LittleFS/examples/Integrity place it in setup() before this line to prevent errors reported in filecount:Code:#define destLFS 1 #define destSD 2 // xferSD( destSD ); // do MediaTransfer LFS TO SD // xferSD( destLFS ); // do MediaTransfer SD TO LFS
And this piece of INO code { has to be INO to share the global "myfs" - unless it is made Media specific and extern in user code if not visibly global}.Code:filecount = printDirectoryFilecount( myfs.open("/") ); // Set base value of filecount for disk
Above placed in any of the LittleFS/examples/Integrity using this sketch code will 'recursively' copy as indicated between LFS and SD Media - all files and directories. Tested on PSRAM.ino and PROG.ino.
Removed most debug spew, only two cases of error displayed.
Example will use SD directory: "\LFS_CPY". SD Media of GB bigger than any LFS Media's MB's, so makes no sense to store/copy all of SD root. This SD directory name can be tested and changed programmatically {with change in xferSD()} to allow saving multiple runs of data if desired.
NOTE: This code has a DEBUG copy of makeRootDirsTest() that adds extra files and dirs to prove function when sending LFS data to SD. It is not needed in use.
For a quick test in Integrity examples put in setup - Fill the LFS Media using '3' to make some test files {perhaps "S" or "B"}, then restart to push to SD with destSD.
Then do a 'w' to Wipe all LFS Media files, and rebuild with destLFS and the target SD card that should hold the files. It will then restore to any LFS media - though it does not check for SIZE issues if the contents won't restore to Media in use.
This function does all the work when properly called as desired: void mediaTransfer(File dir, char* szDir, int destMedia)Code:#include <SD.h> void xferSD( int copyType ) { // do MediaTransfer with SD #define destLFS 1 #define destSD 2 static bool initSD = true; Serial.print("Initializing SD card..."); if ( initSD && !SD.begin(BUILTIN_SDCARD)) { // see if the card is present and can be initialized: Serial.println("\n\n SD Card failed, or not present - Cannot do Xfer\n"); } else { initSD = false; Serial.println("card initialized.\n\n"); if ( copyType == destSD ) { // char szSDdir[] = "/"; // COPY to SD ROOT char szSDdir[] = "LFS_CPY/"; // COPY to SD subdirectory if ( '/' != szSDdir[0] ) SD.mkdir( szSDdir ); makeRootDirsTest(); // BUGBUG DEBUG make extra subdirs and files to show function Serial.println("\n STARTING :: LittleFS copy to SD card XFER ...\n\n"); mediaTransfer( myfs.open("/"), szSDdir, destSD ); // TOO SD Serial.println("\n LittleFS copy to SD card XFER COMPLETE.\n\n"); } else { char szLFSdir[] = "/"; char szSDdir[] = "LFS_CPY"; Serial.println("\n STARTING :: SD card copy to LittleFS XFER ...\n\n"); mediaTransfer( SD.open(szSDdir), szLFSdir, destLFS ); // FROM SD Serial.println("\n SD card copy to LittleFS XFER COMPLETE.\n\n"); } } } void mediaTransfer(File dir, char* szDir, int destMedia) { char szNewDir[36]; while (true) { File entry = dir.openNextFile(); if (! entry) { break; } if (entry.isDirectory()) { strcpy( szNewDir, szDir); if ( destMedia == destLFS ) myfs.mkdir( szNewDir ); else SD.mkdir( szNewDir ); strcat( szNewDir, entry.name()); if ( destMedia == destLFS ) myfs.mkdir( szNewDir ); else SD.mkdir( szNewDir ); strcat( szNewDir, "/"); mediaTransfer(entry, szNewDir, destMedia); } else { uint64_t fileSize, sizeCnt = 0, xfSize = 1; char mm[512]; strcpy( szNewDir, szDir); strcat( szNewDir, entry.name() ); File dataFile; if ( destMedia == destLFS ) { dataFile = myfs.open(szNewDir, FILE_WRITE_BEGIN); } else { dataFile = SD.open(szNewDir, FILE_WRITE_BEGIN); } if ( !dataFile ) Serial.print("\td_FILE: NOT open\n"); fileSize = entry.size(); while ( entry.available() ) { if ( fileSize < sizeCnt ) break; if ( fileSize - sizeCnt >= sizeof(mm) ) xfSize = sizeof(mm); else xfSize = fileSize - sizeCnt; entry.read( &mm , xfSize ); dataFile.write( &mm , xfSize ); sizeCnt += xfSize; } if (fileSize != sizeCnt ) { Serial.print("\n File Size Error:: "); Serial.println( entry.name() ); } dataFile.close(); } entry.close(); } } void makeRootDirsTest() { // TEST DEBUG CODE char szDir[36]; for ( uint32_t ii = 1; ii <= NUMDIRS; ii++ ) { sprintf( szDir, "/%lu_dir", ii ); myfs.mkdir( szDir ); sprintf( szDir, "/%lu_dir/aFile.txt", ii ); // BUGBUG DEBUG file3 = myfs.open(szDir, FILE_WRITE); file3.write( szDir , 12 ); file3.close(); sprintf( szDir, "/%lu_dir/TEST", ii ); // BUGBUG DEBUG myfs.mkdir( szDir ); // BUGBUG DEBUG sprintf( szDir, "/%lu_dir/TEST/bFile.txt", ii ); // BUGBUG DEBUG file3 = myfs.open(szDir, FILE_WRITE); file3.write( szDir , 12 ); file3.close(); } filecount = printDirectoryFilecount( myfs.open("/") ); // Set base value of filecount for disk }
Just tried it out. It works great
Originally Posted by wwatson
Just tried it out. It works greatAwesome- thx for testing/confirming!
... on returning to cleanup/test/upload I must have hit UNDO ... the "/" line went back ... unseen ... and I pulled out all the extra print()'s and .mkdir and .exist ... then it didn't work --- Arrgggh.
Do you need a copy of your last working version? If you do here it is:
Code:#include <SD.h> void xferSD( int copyType ) { // do MediaTransfer with SD #define destLFS 1 #define destSD 2 static bool initSD = true; Serial.print("Initializing SD card..."); if ( initSD && !SD.begin(BUILTIN_SDCARD)) { // see if the card is present and can be initialized: Serial.println("\n\n SD Card failed, or not present - Cannot do Xfer\n"); } else { initSD = false; Serial.println("card initialized.\n\n"); if ( copyType == destSD ) { // char szSDdir[] = "/"; // COPY to SD ROOT char szSDdir[] = "LFS_CPY/"; // COPY to SD subdirectory if ( '/' != szSDdir[0] ) SD.mkdir( szSDdir ); makeRootDirsTest(); // BUGBUG DEBUG make extra subdirs and files to show function Serial.println("\n STARTING :: LittleFS copy to SD card XFER ...\n\n"); mediaTransfer( myfs.open("/"), szSDdir, destSD ); // TOO SD Serial.println("\n LittleFS copy to SD card XFER COMPLETE.\n\n"); } else { char szLFSdir[] = "/"; char szSDdir[] = "LFS_CPY"; Serial.println("\n STARTING :: SD card copy to LittleFS XFER ...\n\n"); mediaTransfer( SD.open(szSDdir), szLFSdir, destLFS ); // FROM SD Serial.println("\n SD card copy to LittleFS XFER COMPLETE.\n\n"); } } } void mediaTransfer(File dir, char* szDir, int destMedia) { char szNewDir[36]; while (true) { File entry = dir.openNextFile(); if (! entry) { break; } if (entry.isDirectory()) { strcpy( szNewDir, szDir); if ( destMedia == destLFS ) myfs.mkdir( szNewDir ); else SD.mkdir( szNewDir ); strcat( szNewDir, entry.name()); if ( destMedia == destLFS ) myfs.mkdir( szNewDir ); else SD.mkdir( szNewDir ); strcat( szNewDir, "/"); mediaTransfer(entry, szNewDir, destMedia); } else { uint64_t fileSize, sizeCnt = 0, xfSize = 1; char mm[512]; strcpy( szNewDir, szDir); strcat( szNewDir, entry.name() ); File dataFile; if ( destMedia == destLFS ) { dataFile = myfs.open(szNewDir, FILE_WRITE_BEGIN); } else { dataFile = SD.open(szNewDir, FILE_WRITE_BEGIN); } if ( !dataFile ) Serial.print("\td_FILE: NOT open\n"); fileSize = entry.size(); while ( entry.available() ) { if ( fileSize < sizeCnt ) break; if ( fileSize - sizeCnt >= sizeof(mm) ) xfSize = sizeof(mm); else xfSize = fileSize - sizeCnt; entry.read( &mm , xfSize ); dataFile.write( &mm , xfSize ); sizeCnt += xfSize; } if (fileSize != sizeCnt ) { Serial.print("\n File Size Error:: "); Serial.println( entry.name() ); } dataFile.close(); } entry.close(); } } void makeRootDirsTest() { // TEST DEBUG CODE char szDir[36]; for ( uint32_t ii = 1; ii <= NUMDIRS; ii++ ) { sprintf( szDir, "/%lu_dir", ii ); myfs.mkdir( szDir ); sprintf( szDir, "/%lu_dir/aFile.txt", ii ); // BUGBUG DEBUG file3 = myfs.open(szDir, FILE_WRITE); file3.write( szDir , 12 ); file3.close(); sprintf( szDir, "/%lu_dir/TEST", ii ); // BUGBUG DEBUG myfs.mkdir( szDir ); // BUGBUG DEBUG sprintf( szDir, "/%lu_dir/TEST/bFile.txt", ii ); // BUGBUG DEBUG file3 = myfs.open(szDir, FILE_WRITE); file3.write( szDir , 12 ); file3.close(); } filecount = printDirectoryFilecount( myfs.open("/") ); // Set base value of filecount for disk }
No, Thanks, p#929 is after I got it sorted and that posted code was tested to work - just took a bit of extra time ... undoing all the 'cleanup' only then to see the misplaced "/" ... and redo all the cleanup and then test ...
Hi all,
please let me know if I'd better start a new post instead of writing here...
I went on with my tests, and wish to share the (even partial) result:
- copying .raw audio files from SD to Flash chip: works fine
- creating an example for playing a .raw audio file from LittleFS: ok works fine
- comparing the read-time reading audio files, using traditional SerialFlash solution and LittleFS file system.
This is the sketch using LittleFS, main code:
AudioPlaySerialflashRaw_LittleFS.hCode:#include <Audio.h> #include <Wire.h> #include <SPI.h> #include <SerialFlash.h> #include "play_serialflash_raw_LittleFS.h" AudioPlaySerialflashRaw_LittleFS playFlash; AudioOutputI2S audio_out; AudioConnection patchCord1(playFlash, 0, audio_out, 0); AudioConnection patchCord2(playFlash, 0, audio_out, 1); AudioControlSGTL5000 board; void setup() { board.enable(); board.volume(0.2); AudioMemory(5); } void loop() { playFlash.play("41.raw"); delay(10000); }AudioPlaySerialflashRaw_LittleFS.cppCode:#ifndef play_serialflash_raw_LittleFS_h_ #define play_serialflash_raw_LittleFS_h_ #include "Arduino.h" #include <AudioStream.h> #include "LittleFS.h" class AudioPlaySerialflashRaw_LittleFS : public AudioStream { public: AudioPlaySerialflashRaw_LittleFS(void) : AudioStream(0, NULL) { begin(); } void begin(void); bool play(const char *filename); void stop(void); bool isPlaying(void) { return playing; } uint32_t positionMillis(void); uint32_t lengthMillis(void); virtual void update(void); private: LittleFS_SPIFlash myfs; // ha i metodi di LittleFS che ha i metodi di FS File rawfile; // SerialFlashFile rawfile; uint32_t file_size; volatile uint32_t file_offset; volatile bool playing; }; #endif
Code:#include <Arduino.h> #include "play_serialflash_raw_LittleFS.h" #include "spi_interrupt.h" void AudioPlaySerialflashRaw_LittleFS::begin(void) { playing = false; file_offset = 0; file_size = 0; } bool AudioPlaySerialflashRaw_LittleFS::play(const char *filename) { stop(); AudioStartUsingSPI(); // rawfile = SerialFlash.open(filename); // if (!rawfile) if(!myfs.begin(6)) { Serial.println("unable to access Flash"); AudioStopUsingSPI(); return false; } rawfile = myfs.open(filename); if(!rawfile) { Serial.println("unable to access file"); AudioStopUsingSPI(); return false; } file_size = rawfile.size(); file_offset = 0; //Serial.println("able to open file"); playing = true; return true; } void AudioPlaySerialflashRaw_LittleFS::stop(void) { __disable_irq(); if (playing) { playing = false; __enable_irq(); rawfile.close(); AudioStopUsingSPI(); } else { __enable_irq(); } } void AudioPlaySerialflashRaw_LittleFS::update(void) { unsigned int i, n; audio_block_t *block; unsigned long T0; int time_lap; // only update if we're playing if (!playing) return; // allocate the audio blocks to transmit block = allocate(); if (block == NULL) return; if (rawfile.available()) { T0 = micros(); n = rawfile.read(block->data, AUDIO_BLOCK_SAMPLES*2); time_lap=micros()-T0; file_offset += n; for (i=n/2; i < AUDIO_BLOCK_SAMPLES; i++) { block->data[i] = 0; } transmit(block); Serial.println(time_lap); } else { rawfile.close(); AudioStopUsingSPI(); playing = false; } release(block); } #define B2M (uint32_t)((double)4294967296000.0 / AUDIO_SAMPLE_RATE_EXACT / 2.0) // 97352592 uint32_t AudioPlaySerialflashRaw_LittleFS::positionMillis(void) { return ((uint64_t)file_offset * B2M) >> 32; } uint32_t AudioPlaySerialflashRaw_LittleFS::lengthMillis(void) { return ((uint64_t)file_size * B2M) >> 32; }
where this part of code allows to measure the reading performance:
Now, using a T4.1@600MHz, and AUDIO_BLOCK_SAMPLES = 128, if I compare the reading performance of the previous code, with a similar example which uses AudioPlaySerialflashRaw:Code:time_lap=micros()-T0; file_offset += n; for (i=n/2; i < AUDIO_BLOCK_SAMPLES; i++) { block->data[i] = 0; } transmit(block); Serial.println(time_lap);
the update() time in microseconds is:Code:#include <Audio.h> #include <Wire.h> #include <SPI.h> #include <SerialFlash.h> AudioPlaySerialflashRaw playFlash; AudioOutputI2S audio_out; AudioConnection patchCord1(playFlash, 0, audio_out, 0); AudioConnection patchCord2(playFlash, 0, audio_out, 1); AudioControlSGTL5000 board; void setup() { board.enable(); board.volume(0.2); AudioMemory(5); SerialFlash.begin(6); } void loop() { playFlash.play("41.raw"); delay(10000); }
AudioPlaySerialflashRaw example:
AudioPlaySerialflashRaw_LittleFS example:Code:50 50 50 50 50 50 50 50 50 50 50 ... ... and so on
This report shows that with LittleFS the reading time is 55% higher for 15 cycles, and after that (should be after 4KB of data) the next cycle is 10 times longer; for my application, where audio polyphony is up to 16, this slow reading time (maybe due to some "jumps" among sectors??) is incompatibleCode:77 78 696 78 77 78 77 78 77 78 77 78 77 78 77 78 77 78 773 78 77 78 77 78 77 78 77 78 78 78 78 78 78 78 696 78 78 78
@Sandro - seems good to me to post that here - use case and perf data.
Did I miss what Teensy ... ah T_4.1 ... and what Flash media/interface?
Hi defragster, I'm using the Winbond 25Q128FVSG mounted on the Audio board.
The SerialFlash was specifically coded to avoid overhead - all blocks sequential and no FS overhead - just minimal linkage.
Not surprising on SPI it will suffer more overhead with LFS at slower speeds. Wonder if a NAND version would mount and read faster even as SPI? Certainly would expect it on QSPI soldered on PCB as it clocks faster - it may not run 4X faster - but it might being enough.
Does the Audio code just read a 256B block at a time? It might be a tweak to help it cache might make the diff - but the slow times would still probably show when missed as to read the data takes reading some other blocks in advance to find the data.
In my test, I prepared the flash chip with a deep format, than I copied from SD card just one file (41.raw, 1.18MB) ; maybe this file, even if alone, has been stored in not-adjacent blocks/sectors of the chip?
Right.. I could try with a W25N01GV mounted on the Teensy QSPI expansion (never used one of these nand chips) and check what happens...Not surprising on SPI it will suffer more overhead with LFS at slower speeds. Wonder if a NAND version would mount and read faster even as SPI? Certainly would expect it on QSPI soldered on PCB as it clocks faster - it may not run 4X faster - but it might being enough.
Yes, the Audio codec reads a 256B block each time slot of 2.9ms; but, depending on the desidered pitch, it could be needed to read (for example) only 1 byte ... or 2560 byte in the same slot 2.9ms. Caching audio data would be great but, considering all requiremts in my application (polyphony, number of contemporary voices, audio files dimensions, pitch...) it would occupy many (I mean 32-64) MB of fast/inner data RAM...Does the Audio codec just read a 256B block at a time? It might be a tweak to help it cache might make the diff - but the slow times would still probably show when missed as to read the data takes reading some other blocks in advance to find the data.
Last edited by Sandro; 08-03-2021 at 08:33 AM.
@Paul - just recalled a historical note that some idea for LFS to have a way to emulate 'perhaps' some 'SerialFlash' type storage files? i.e. direct access to sequential blocked files ???
> seems that is a real recollection ???
> of course it is a non-trivial extension - but what @Sandro is seeing suggests it could be the fix for FS overhead getting to file data for the Audio system in an _isr/DMA responsive time.
@Paul, @defragster: going back to the origin of my tests around LittleFS is the fact that a SerialFlash-file can be written only once, as @Paul wrote on Github:
and this is the big issue in my case; if this cannot be modified (i trust it cannot) and LittleFS brings too much overhead for audio operations, and optimizing LittleFS in this sense is a big/heavy/.../worthless job, than in my case a workaround can be create a sort of simple (and full of limitations, but I could be able to deploy it by myself with a little fortune..) "virtual file system" with SerialFlash, filling the SPI flash of small, fixed size and createErasable files ("snippet"), than splitting all files among snippets... and so on. This should allow to create/delete/move files without erasing all the Flash content.. In this sense I just tested that reading the content from 2 files (using SerialFlash) there is an almost neglectable delay (1-2 microseconds) when jumping from one file to another.Code:Several limitations apply to writing. Only previously unwritten portions of the file may be written. File sizes can never change. Writes may only be done within the file's original size.
Last edited by Sandro; 08-05-2021 at 10:49 AM.
Sorry... I got confused... The issue is that a single SerialFlash-file can be deleted only by deleting all the flash chip.
Reply With Quote
