Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 18 of 18

Thread: SerialFlash Library - Max Number of Files allowed?

  1. #1
    Junior Member
    Join Date
    Oct 2017
    Posts
    13

    SerialFlash Library - Max Number of Files allowed?

    Hi Everyone

    Project using prop shield + teensy 3.2 + SerialFlash Library

    I'm trying to write a program that logs data to Flash. I need it to run for ~ 2 hours, I am planning to adjust my sampling rate to max out the available storage.

    I had planned to write each data-frame to a file and then sequentially step through the files to read them out. I finally got it working, but I seem to use the SerialFlash library in a way that its not intended to be used.

    Using my one file per data-frame approach I am unable to create a file after I have stored 600 frames.

    I am not sure if this is because there is a problem with my code, or because I have wrong assumptions on how the library works or because there is a problem with the library. Could someone please check this for me?

    EDIT: I've confirmed that the file-size is not an issue. I have tested 4 byte files and I have tested 256 byte files, and 600 files is still the maximum.

    Here is my code for logging files, as you see the files are numbered based on the counter variable. Once my file-name hits 599, the next file I try to create fails ("File creation failed!!!")

    Any suggestions?

    Code:
     ///////////////////////////
    // Logs data to flash. Goes with readFiles.ino
    /////////////////
    
    
    #include <SerialFlash.h>
    #include <SPI.h>
    const int FlashChipSelect = 6; // digital pin for flash chip CS pin
    
    //for IMU ////////
    #include <NXPMotionSense.h>
    #include <Wire.h>
    #include <EEPROM.h>
    #include <util/crc16.h>
    
    NXPMotionSense imu;
    //////////////////
    
    int counter = 0;
    
    
    SerialFlashFile file; //Working file
    
    
    String dataAsString;
    
    
    void setup()
    {
      Serial.begin(9600); //open serial port
      while (!Serial); //wait until the serial port is open
      delay(100);
      Serial.println("Checking if it finds the Flash Chip...");
    
      if (!SerialFlash.begin(FlashChipSelect)) {
        Serial.println("");
        Serial.println("Unable to access SPI Flash chip");
      }
      Serial.println("");
      Serial.println("SPI Flash found");
      uint8_t id[5];
      SerialFlash.readID(id);
      Serial.println("ID:");
      Serial.print("0x");
      Serial.print(id[0], HEX);
      Serial.print(", 0x");
      Serial.print(id[1], HEX);
      Serial.print(", 0x");
      Serial.print(id[2], HEX);
      Serial.print(", 0x");
      Serial.println(id[3], HEX);
      Serial.println("... so far, so good");
      Serial.println("");
    
      imu.begin();
    
      
    
    
    
    }
    
    void loop() {
      //IMU data
      int ax, ay, az;
      int gx, gy, gz;
      int mx, my, mz;
    
      // get and print uncalibrated data
      if (imu.available()) {
        imu.readMotionSensor(ax, ay, az, gx, gy, gz, mx, my, mz);
     dataAsString = "" + String(ax) + "!!";
       // dataAsString = "Acc: " + String(ax) + ", " + String(ay) + ", " + String(az) + "!!";
        Serial.print("Testing the String: ");
        Serial.println(dataAsString);
        String currentTime = String(counter);
        createFile(dataAsString, currentTime);  //This creates a test file IF it doesn't exist
        counter++;
      }
    
    }
    
    
    
    ////////////////////////////////////////////////////////////////////////////
    //This function creates a file and writes the array specified above to it //
    ////////////////////////////////////////////////////////////////////////////
    
    void createFile( String dataString, String millisForFilename ) {
    
      //create the name (this should be cleaned up)
      String fileNameAsString = millisForFilename + ".dat" ;
      int nameLength = fileNameAsString.length() + 1;
      char filename[nameLength];
      fileNameAsString.toCharArray(filename, nameLength);
      
      String dataAsString = dataString;
      int  fileSize = dataAsString.length();
      char dataToWrite [fileSize + 1];
      dataAsString.toCharArray(dataToWrite, fileSize + 1);
      Serial.print("String length: ");
      Serial.println(fileSize);
      Serial.print("Char array length: ");
      Serial.println(sizeof(dataToWrite));
    
    
    
      Serial.println("");
      //Check if file exists, if not, create it.
      if (SerialFlash.exists(filename) == 0)
      {
        //File doesn't exist
        if (SerialFlash.create(filename, fileSize) == true)
        {
          Serial.println(filename);
        }
        else
        {
          Serial.println("File creation failed!!!");
        }
      }
      if (SerialFlash.exists(filename  ))
      {
        Serial.println("File Exists, trying to open...");
        file = SerialFlash.open(filename  );
        if (file)
        {
          // true if the file exists
          //      Serial.println(filename + ".dat opened");
          uint8_t buffer[fileSize];
          for (int i = 0; i < fileSize; i++) {
            buffer[i] = dataToWrite[i];
          }
    
    
          Serial.print("Writing Data to File ...");
          //IF YOU WANT TO WRITE TO ANOTHER POSITION IN THE FILE, USE "file.seek();"
          //IF YOU WANT TO CHECK YOUR CURRENT POSITION, USE "file.position()"
    
          file.write(buffer, fileSize);
          Serial.println("... completed");
          Serial.print("Position after writing: ");
          Serial.println(file.position());
        }
        else
        {
          //      Serial.println(filename + ".dat not opened!!!");
        }
      }
    }
    This is probably irrelevant, but for reference and to understand how I am thinking about this, here is the code I use for reading:


    Code:
    //////////////////////////////
    //This sketch sequentially opens every file in flash memory and prints its contents to the serial port.
    //////////////////////////////
    
    #include <SerialFlash.h>
    #include <SPI.h>
    SerialFlashFile file; //Working file
    const int FlashChipSelect = 6; // digital pin for flash chip CS pin
    //const int FlashChipSelect = 21; // Arduino 101 built-in SPI Flash
    
    void setup() {
    
      Serial.begin(9600);
    
      // wait for Arduino Serial Monitor (Just so we don't miss the display)
      while (!Serial) ;
      delay(100);
      Serial.println("All Data from all Files on SPI Flash chip:");
      Serial.println("");
    
      if (!SerialFlash.begin(FlashChipSelect)) {
        Serial.print("Unable to access SPI Flash chip");
        delay(2500);
      }
    
      SerialFlash.opendir();
      while (true) { //this lists by order of creation all files
        char filename[64]; //filename
        uint32_t filesize; //filesize (we will use this to read from them.
    
        if (SerialFlash.readdir(filename, sizeof(filename), filesize)) { //if it found a file
          file = SerialFlash.open(filename);  //open it
          char buffer2[filesize];             //create a buffer the size of the file
          file.read(buffer2, filesize);       //copy the file to the buffer
          Serial.print(filename);
          Serial.print(": ");
    
          for (int i = 0; i < filesize; i++) {
            Serial.print(buffer2[i]); //print the content of the buffer to the serial port
          }
          Serial.println(""); //new line at end of file
          file.close(); //close file
    
        } else { //if no more files are found
          Serial.println("");
          Serial.println("no more Data");
          break; // no more files
        }
      }
    }
    
    
    void loop() {
    }
    Last edited by fkeel; 10-28-2017 at 11:44 PM.

  2. #2
    If you open SerialFlashDirectory.cpp in the library, one of the first defines is

    #define DEFAULT_MAXFILES 600

    Not sure what happens if you increase it, but worth a try...

    D
    Last edited by Daniel-J; 10-29-2017 at 08:45 AM. Reason: fix url

  3. #3
    Junior Member
    Join Date
    Oct 2017
    Posts
    13
    Would be great if PaulStoffregen could weigh in on this. I can probably change things around to stay within the file limits, but that would make some other aspects of the project more complex.

    I'm super inexperienced with this, so changing code is a lot of work, but understanding unforeseen consequences of hardcoded variables in the library is also beyond my abilities.

    Thanks for finding this for me though!

  4. #4
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    22,122
    Yes, you can increase the limit. But you must do a full chip erase, because a portion of the chip is preallocated to storing the filenames.

    Increasing the limit will cause the chip erase to allocate more of the chip for filenames. Only then can you write more files.

  5. #5
    Junior Member
    Join Date
    Oct 2017
    Posts
    13
    Thanks for your answer and apologies for cross-posting.

    Is there a way to check how much space is being allocated for file-names?

  6. #6
    Senior Member+ manitou's Avatar
    Join Date
    Jan 2013
    Posts
    2,534
    Ref: https://github.com/PaulStoffregen/SerialFlash
    File names (including trailing null) storage is from a pool of 25560 bytes

    #define DEFAULT_MAXFILES 600
    #define DEFAULT_STRINGS_SIZE 25560

    The default meta-data area is 32768 bytes = 25560 + 8 + 600*2 +600*10
    but can be changed in SerialFlashDirectory.cpp The max strings size is 2**18 (262144)
    The rest of the flash is dedicated to file data, but each file is aligned on 256-byte page boundary, and erasable files are aligned on a chip-specific boundary.
    Last edited by manitou; 10-29-2017 at 07:50 PM.

  7. #7
    Junior Member
    Join Date
    Oct 2017
    Posts
    13
    Reporting back: Yes, you *can*, but its apparently not a good idea.

    Setting the file limit to 60k (which is what I had in mind) grinds to a halt (individual files take multiple seconds to write).

    Setting the file limit to 6k led to various errors (multiple files were not read correctly). Write times (though there is other stuff going on in my code as well, so assume that there is a constant added to all these numbers, were ~10ms initially, after about 700 files write files started moving to ~11ms. At 1000 files they were consistantly at or over 11ms and rising. At 1300 files, first errors started appearing and write times were 20ms.

    Setting the file limit to 3k led to the same behavior as 6k.

    When I say 'errors' I mean this type of thing: Name:  error.PNG
Views: 322
Size:  6.6 KB

    I only modified the DEFAULT_MAXFILES, didn't play with it any further, cause I don't really understand what I'm doing. The files I am writing are super short <25 bytes.

    Unless someone else has a smart idea, I'm gonna try and find some other solution.

    The reason I was going with tons of tiny files like this, was so that the teensy can access specific data bits at a later time. I am not sure if its even possible for the teensy to parse a 6mb file for a bit of information ...or is it?

    EDIT: Based on the documentation I assume I can move through the file using file.position() & file.seek(number); and then copy a subset of that file into a buffer. Gonna give that a shot now.

  8. #8
    Senior Member+ manitou's Avatar
    Join Date
    Jan 2013
    Posts
    2,534
    Things to watch out for. Even though you request 25 bytes per file, i think the minimum file boundary is 256 bytes, so you will be using a minimum of 256 bytes for each file. You can exhaust the file name string storage area (I don't think the library checks for overflow). Each file name plus null is rounded up to a multiple of 4-bytes. Your file names (using millis()) are say 12-bytes (or more), so 25560/12 is 2130 files before string area is overflowing. Finally, if you change the library parameters (number of files, string size, etc), you need to do a full erase of the SPI flash. FWIW, here is a function to print out the meta data for the simple file system

    Code:
    void meta() {
      //  paul's file system meta data
      // meta data: magic(4),maxfiles(2), strsize/4(2),  hashes*2
      //  dir: entry(10): addr(4),lth(4),str/4(2)
      int i, files, deleted, capacity, avail;
      uint32_t maxfiles, stringsize, straddr, buff, dir[3];
      uint16_t hash;
      uint8_t id[3];
    
      SerialFlash.read(4, &buff, 4);
      maxfiles = buff & 0xffff;
      stringsize = (buff & 0xFFFF0000) >> 14;
      Serial.print("maxfiles "); Serial.print(maxfiles);
      Serial.print("  stringsize "); Serial.println(stringsize);
    
      files = deleted = hash = 0;
      for (i = 0; i < maxfiles; i++) {
        SerialFlash.read(8 + 2 * i, &hash, 2);
        if (hash == 0xffff) break;
        if (hash) files++;
        else deleted++;
      }
      Serial.print("files "); Serial.print(files);
      Serial.print("   deleted "); Serial.println(deleted);
    
      if (i) {
        char str[80];
        SerialFlash.readID(id);
        sprintf(str, "ID %0x %0x %0x", id[0], id[1], id[2]);
        Serial.println(str);
        Serial.print("blockSize "); Serial.println(SerialFlash.blockSize());
        capacity = SerialFlash.capacity(id);
        Serial.print("Capacity "); Serial.print(capacity);
        // read last dir entry  i-1, could round up to 256
        dir[2] = 0;
        SerialFlash.read(8 + maxfiles * 2 + (i - 1) * 10, dir, 10);
        avail = capacity - (dir[0] + dir[1]);
        Serial.print("   free "); Serial.println(avail);
        straddr = 8 + maxfiles * 12;  // determine string usage
        straddr += dir[2] * 4;
        i = string_length(straddr) + dir[2] * 4;  //offset + last string
        Serial.print("string bytes used "); Serial.println(i);
      }
    }

  9. #9
    Senior Member+ manitou's Avatar
    Join Date
    Jan 2013
    Posts
    2,534
    I'm not sure of your ultimate goal, but if you just want to fill up the SPI flash with ax,ay,az samples, you could erase the flash and just use SerialFlash.write() and not bother with a file system. With 8 MB propshield flash, it only takes about 19 seconds to write all the 256-byte pages (32768), and only 4 seconds to read all the pages. To the write time you would need to add the IMU sample time, about 10 ms per sample. Denser records could be used, binary float values and binary millis()
    Code:
    #define PERPAGE 16
    struct imu_data {
      uint32_t ms;
      float ax, ay, az;
    } page[PERPAGE];   //  256 byte page 16x16
    I have a rough proof-of-concept sketch.
    Last edited by manitou; 11-01-2017 at 12:02 PM.

  10. #10
    Junior Member
    Join Date
    Oct 2017
    Posts
    13
    Thanks manitou. I will play with your example. What I need to do will be a little bit more involved, but if I can figure out what you're suggesting, that should help, cause the logging will be essentially equivalent.

    I'll be collecting information from more data-sources, maintaining specific fps of the recording, sending messages to other digital sensors and to an LED etc.: its for monitoring the behavior of a mechanical 'self-actuated' gripper. The gripper needs to complete a small task - reach out and hold on to something - while we do not have any access to it. I'm sending the teensy along to verify that its doing what it should be doing (collecting motion and force-related data). We're also sending a gopro along with it. In in the end I want to align the sampling rate with the framerate so we can play them back together comfortably.

    Regarding this:
    Even though you request 25 bytes per file, i think the minimum file boundary is 256 bytes, so you will be using a minimum of 256 bytes for each file.
    I was under the impression that the minimum file boundary only applied to erasable files. Is this incorrect? (I am referring to the 'Create New Files' section here: https://github.com/PaulStoffregen/SerialFlash)
    Last edited by fkeel; 10-31-2017 at 05:51 PM.

  11. #11
    Junior Member
    Join Date
    Oct 2017
    Posts
    13
    @manitou

    Sadly I am so spoiled by Arduino and Processing that I struggle with 'regular' C. If you could show me, in code, how you would use the struct that you wrote in your previous post, it would save me a potential large time experimenting. I'd be super grateful.

  12. #12
    Senior Member+ manitou's Avatar
    Join Date
    Jan 2013
    Posts
    2,534
    OK, here is raw logger to fill up flash.
    Code:
    // need to erase flash before starting
    #include <SerialFlash.h>
    #include <SPI.h>
    
    #define PERPAGE 16
    struct imu_data {
      uint32_t ms;
      float ax, ay, az;
    } page[PERPAGE];   //  256 byte page 16x16
    
    #define CS 6
    
    uint32_t capacity, pages, pages_written = 0;
    
    void write_data(float ax, float ay, float az) {
      static int rec = 0;
    
      page[rec].ms = millis();
      page[rec].ax = ax;
      page[rec].ay = ay;
      page[rec].az = az;
      rec++;
      if (rec < PERPAGE) return;
      rec = 0;
      SerialFlash.write(pages_written * 256, &page, sizeof(page));
      pages_written++;
    }
    
    void read_flash() {
      int page_cnt = 0;
    
      while (page_cnt < pages) {
        SerialFlash.read(page_cnt * 256, &page, sizeof(page));
        page_cnt++;
      }
    }
    
    void setup() {
      uint8_t id[3];
      uint32_t t, samples;
      float ax, ay, az;
    
      Serial.begin(9600);
      while (!Serial);
      SerialFlash.begin(CS);
      delay(3);
      SerialFlash.readID(id);
      Serial.printf( "ID %0x %0x %0x\n", id[0], id[1], id[2]);
      capacity = SerialFlash.capacity(id);
      pages = capacity / 256;
      Serial.printf("capacity %d  pages %d\n", capacity, pages);
    
    #if 1
      pages_written = 0;
      samples = 1;
      t = millis();
      while (pages_written < pages) {
        // get IMU data
        ax = samples; ay = -samples; az = samples & 1;
        write_data(ax, ay, az);
        samples++;
      }
      t = millis() - t;
      Serial.printf("%d samples in %d ms\n", samples - 1, t);
    #endif
    
      t = millis();
      read_flash();
      t = millis() - t;
      Serial.println(t);
      Serial.println(page[0].ms);
      Serial.println(page[1].ax);
    }
    
    void loop() {
    
    }
    ID ef 40 17
    capacity 8388608 pages 32768
    524288 samples in 19009 ms
    3338
    19923
    524274.00

    It doesn't do a real IMU read, you'd need to add if (imu.available()) { ...
    inside the pages_written while loop

    Erasable files have a big alignment, like 64k, but i'm pretty sure small files are aligned to 256 byte page. Look at the driver code, or print out ff.getFlashAddress() for a few opened files.
    Last edited by manitou; 11-01-2017 at 02:11 PM.

  13. #13
    Junior Member
    Join Date
    Oct 2017
    Posts
    13
    Cool. I will check this out later tonight. Thanks for all your help.

  14. #14
    Junior Member
    Join Date
    Oct 2017
    Posts
    13
    Hi Manitou

    I tested your code and I am getting behavior that I don't understand.

    I modified what you sent me a bit, to better understand whats going on:
    Code:
    #if 1
      pages_written = 0;
      samples = 1;
      t = millis();
      while (pages_written < pages) {
        //  get IMU data
        ax = 1; ay = 2; az = az + ax; <--------------CHANGED THIS LINE (I also tried other changes, none of which behaved the way I thought it would)
        write_data(ax, ay, az);
    
        samples++;
      }
      t = millis() - t;
      Serial.printf("%d samples in %d ms\n", samples - 1, t);
    #endif
    
      t = millis();
      read_flash();
      t = millis() - t;
      Serial.println(t);
      Serial.println(page[0].ax); <--------------CHANGED WHAT I'M WRITING TO THE SERIAL PORT
      Serial.println(page[1].ay);
      Serial.println(page[2].az);
      Serial.println(page[3].ax);
      Serial.println(page[4].ay);
      Serial.println(page[5].az);
      Serial.println(page[0].ms);
      Serial.println(page[10].ms);
      Serial.println(page[20].ms);
    }
    The output I would expect would be something like:
    Code:
    1
    2
    3
    1
    2
    6
    ~some value in ms
    ~some value in ms + a couple of ms
    ~some value in ms + a little more
    What I get is

    Code:
    ID ef 40 17
    capacity 8388608  pages 32768
    524288 samples in 19867 ms
    3344
    1.00
    2.00
    524277.06
    1.00
    2.00
    524280.06
    21950
    21950
    0
    What I *think* is happening, is that I am printing the first instance of ax, then the second instance of ay and then the third instance of az etc to the serial port. After that I think I am printing the first, 10th and 20th timestamp.
    Its behaving differently, not sure if my thinking is wrong, or my code is doing something stupid or if there is an actual bug.

    I run the 'EraseEverything' sketch (The example that came with the SerialFlash library) in between trials.

    Any suggestions?

  15. #15
    Senior Member+ manitou's Avatar
    Join Date
    Jan 2013
    Posts
    2,534
    You need to include the full sketch. I can't tell how az is initiliazed. my struct had only [16] elements, your are printing from [20]?

    my sketch is a little misleading, in that i am printing only the last "page" of data, because read_flash() reads ALL of the data inside that function, useful only for timing purposes. You would want to print values inside the read_flash() function as you read in each page of data ....

  16. #16
    Junior Member
    Join Date
    Oct 2017
    Posts
    13
    OK, I was thinking about it incorrectly. I just assumed that the array index of the 'page' variable was referring to the data-frame (or to what you refer to as "page"). I initially didn't quite follow your logic on how you were initializing things.
    I think your comment cleared things up. I'll be playing with this further, I think I understand it better now, thanks.

    Something that I'm not quite clear on: Why does your struct have 16 elements? Is that assuming it has 16 uint16 elements? Given that your sketch stores floats and uint32, is there any specific benefit or reason for the 16 elements?

    EDIT: I'm also somewhat surprised that I didn't get an array-index out of bounds error in my modified version of your sketch, as I did not change initialization.

    EDIT2: I'm obviously still confused about data-types and how you're using the struct. I guess it'll require some more playing on my side. Thanks so much for your help.
    Last edited by fkeel; 11-09-2017 at 10:38 PM.

  17. #17
    Senior Member+ manitou's Avatar
    Join Date
    Jan 2013
    Posts
    2,534
    It runs a bit faster if you write a page (256 bytes, or 16x16 for the struct) at a time. So a flash write is only happening after every 16 samples. You could just have a simple sample struct (no array), and write to the flash after each sample (you would need to adjust the counts to keep track of how much space is remaining on the flash). (3 floats and one uint32_t is 16 bytes)

    arduino/teensy C++ does not check for index-out-of-bounds. Sometimes you know because your program mysteriously fails.

    Note, i had edited the sketch in post #12 to count samples in ax

  18. #18
    Junior Member
    Join Date
    Oct 2017
    Posts
    13
    So 1 page stores 16 times 3 floats + 1 uint32.

    So I if I wanted to store 32 uint16, I would need 64 bytes, so I could have a page consisting of 4x64. (OK, I do really just need to play around with this and see it work in practice. Thinking out loud here now). I do think I have all the bits. I just need to figure out how they work together now. Thanks.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •