LittleFS port to Teensy/SPIFlash

mjs513 · Nov 7, 2020

Oh forgot one thing. Think @defragster mentioned this in another post but think we need an addition function like FORMAT for QSPIFlash and SPIFlash. Right now can't get there. I tried adding for QSPIFlash but no luck. Probably forgot somethin

KurtE · Nov 7, 2020

Morning all!

I just sunk up to stuff. Still running one of the first simple sketches just modified to which type of storage... Just updated it to QSPI version.

If I run it on my first T4.1 beta board with both RAM/FLASH it appears to run:

Code:

LittleFS Test

QSPI flash begin

Flash ID: EF 40 18
Flash size is 16.00 Mbyte
attemping to mount existing media

couldn't mount media, attemping to format

  waited 22266 us
  waited 119 us
  waited 22310 us
  waited 121 us
attemping to mount freshly formatted media

success
started
  waited 98 us
opened test.txt
wrote to file
  waited 101 us

read 0 bytes

done printing, now close the file

try writing more
opened test.txt again for writing
after write
  File    this=2006fdbc, f=20203650
  LittleFSFile this=20203650, refcount=1
  waited 116 us

opened file for read
read 34 bytes
This is a test....
another test

printDirectory
--------------
test.txt		34
...

The biggest bug I see

is my cut and paste, the spellchecker keep flagging: attemping

If I try to run on it on my one that that put the big 1MB chip, it errors out:

Code:

LittleFS Test
QSPI flash begin
Flash ID: 00 EF AA
Error starting ramdisk

It is always possible I screwed up on that chip? Anyone had any luck.

mjs513 · Nov 7, 2020

Morning @KurtE
That 1Mb Nand chip probably has a different LUTS for initialization and reads and formats from what I remember so doesn't surprise me it doesn't work.

Think Paul also mentioned that he was going to do a memory mapping for the NAND because it was too big?

PaulStoffregen · Nov 7, 2020

Might be a while until I work with the large NAND flash chip. Many details to consider, like checking the ECC status and bad block remapping.

mjs513 · Nov 7, 2020

PaulStoffregen said:
Might be a while until I work with the large NAND flash chip. Many details to consider, like checking the ECC status and bad block remapping.

Yeah - that NAND Flash seems to be a bit more complicated

KurtE · Nov 7, 2020

Understood: Right now I am playing with a creating sort of a random, maybe sort of typical usage case. Where I have IntervalTimer which when it runs,
it, checks if it should alternate between two log files. Right now I am doing it by how many writes, but could be by date it could be by size of file...

I figured playing with something like this helps me to see if I think most of the pieces are in place. Which will see more as I finish the program.

I have not very far yet, but for example the starting code for interval timer:

Code:

typedef struct {
  uint32_t sample_index;
  uint32_t sample_time;
} LogData_t;

void process_interval() {
  File f;
  LogData_t logdata;
  if (samples_in_file == SAMPLES_PER_FILE) {
    which_file++;
    samples_in_file = 0;
    if (which_file == (sizeof(file_names)/sizeof(file_names[0])) which_file = 0;
    f = myfs.open(file_names[which_file], FILE_WRITE);
    file.seek(0, SeekSet); 
  } else {
    f = myfs.open(file_names[which_file], FILE_WRITE);
  }
  logdate.sample_index = ++sample_index;
  logdata.sample_time = millis();
  f.write(&logdata, sizeof(logdata));
  f.close();  
  samples_in_file++;
}

Again warning I have even tried compile yet...

But questions to self include:
a) Will open file for Write always go to the end of the last write location? If so I put the code in for when I go back to using different file, I try to see to start...

b) file size - What controls file size? That is if I do the Seek back to 0, and close the file is it truncated to my new current position after the write? Or does it keep the old size.
Some systems, you may want to create a file that is large enough to hold all of it in the allocated areas and then seek back to the start... I am not sure if I am making sense here or not.
But simple test should probably test is something like:

Code:

File f= myfs.open("test", FILE_WRITE);
f.seek(1000, SeekSeT);
f.seek(0, SeekSet);
f.print("a");
f.close();

Again sometime unclear what the right thing is in some cases. Probably need some form of truncate

c) what does FILE_READ or FILE_WRITE mean. That is can I open a file read some stuff and write some stuff? Again example, maybe I want each file to hold the last time that any of 100 events happened and as such I open the file, seek to the position of eventX update the data, maybe seek to start of file to update some header data... and close.

KurtE · Nov 7, 2020

I thought before continue above, to try to answer some of my questions...
Here is a quick and dirty which is not working? ...

Code:

#include <LittleFS.h>
#include <IntervalTimer.h>
#include <SPI.h>

#define TEST_RAM
//#define TEST_SPI
//#define TEST_QSPI
#ifdef TEST_RAM
LittleFS_RAM myfs;
char buf[200000];
#elif defined(TEST_SPI)
LittleFS_SPIFlash myfs;
#else
LittleFS_QSPIFlash myfs;
#endif

void setup() {
  //  pinMode(13, OUTPUT);
  //  digitalWrite(13, HIGH);
  SPI.begin();
  while (!Serial) ; // wait
  Serial.println("LittleFS Test"); delay(5);

#ifdef TEST_RAM
  if (!myfs.begin(buf, sizeof(buf))) {
#elif defined(TEST_SPI)
  pinMode(7, OUTPUT);
  digitalWriteFast(7, LOW);
  SPI2.setMOSI(50);
  SPI2.setMISO(54);
  SPI2.setSCK(49);
  SPI2.begin();
  if (!myfs.begin(51, SPI2)) {
#else
    if (!myfs.begin()) {
#endif
    Serial.println("Error starting LittleFS Disk");
    while (1) ;
  }
  Serial.println("started");
  File f = myfs.open("test.txt", FILE_WRITE);
  Serial.println("opened test.txt");

  // Output 64 zeros...
  f.println("0000000000000000000000000000000000000000000000000000000000000000");
  Serial.println("wrote to file");
  Serial.printf("POS: % u Size: % u\n", f.position(), f.size());
  f.close();
  delay(10);
  printDirectory();
}
uint16_t loop_count = 0;
void loop() {
  delay(1000);
  uint8_t buffer[100];
  File f = myfs.open("test.txt", FILE_READ);
  int cbRead = f.read(buffer, sizeof(buffer));
  Serial.printf(" % u: % d - % s\n", loop_count, cbRead, buffer);
  uint8_t buf_pos = loop_count++ & 0x1f;
  f.seek(buf_pos, SeekSet);
  buffer[buf_pos] = (buffer[buf_pos] < '9') ? buffer[buf_pos]+1 : '0';
  f.write(buffer[buf_pos]);
  f.close();
  printDirectory();

}

void printDirectory() {
  Serial.println("printDirectory\n--------------");
  printDirectory(myfs.open(" / "), 0);
  Serial.println();
}

void printDirectory(File dir, int numTabs) {
  //dir.whoami();
  while (true) {

    static int count = 0;
    if (++count > 20) while (1) ;

    File entry =  dir.openNextFile();
    if (! entry) {
      // no more files
      //Serial.println("**nomorefiles**");
      break;
    }
    for (uint8_t i = 0; i < numTabs; i++) {
      Serial.print('\t');
    }
    Serial.print(entry.name());
    if (entry.isDirectory()) {
      Serial.println(" / ");
      printDirectory(entry, numTabs + 1);
    } else {
      // files have sizes, directories do not
      Serial.print("\t\t");
      Serial.println(entry.size(), DEC);
    }
    entry.close();
  }
}

Code:

LittleFS Test

started
opened test.txt
wrote to file
POS: 66 Size: 66
printDirectory
--------------

 0:  66 - 0000000000000000000000000000000000000000000000000000000000000000

printDirectory
--------------

 1:  66 - 0000000000000000000000000000000000000000000000000000000000000000

printDirectory
--------------

 2:  66 - 0000000000000000000000000000000000000000000000000000000000000000

printDirectory
--------------

I have not debugged my loop code yet. Again not sure if you can write to a one you opened for Read or vs versa... Write did not appear to work...
Also printDirectory is not printing anything...

EDIT: Fixed the above program.

mjs513 · Nov 7, 2020

Not home right now but you should also be Able to open a file for read/write. Not sure of the format... file_rdwr? Have to check when I get back

KurtE · Nov 7, 2020

Edit: Fixed the above program to at least be able to update ... Silly mistake

Code:

buffer[buf_pos] = (buffer[buf_pos] < '9') ? buffer[buf_pos]++ : '0';

should be:

Code:

buffer[buf_pos] = (buffer[buf_pos] < '9') ? buffer[buf_pos]+1 : '0';

But that implies I can write to a file I opened as Read...

As for open a file for READ/WRITE. The only two defines I see in FS.h are:

Code:

#define FILE_READ  0
#define FILE_WRITE 1

So obviously: does not good to do: FILE_READ|FILE_WRITE.

And obviously my seek to a position here and do a write did not truncate the file.

Also not sure yet why not showing anything in directory search.

defragster · Nov 7, 2020

PaulStoffregen said:
Oh, opps, sorry about that static count I left in the test code. ...

opps and .... argh ... I searched lib code for 20 and walked other dead ends ... and there it was in the sketch

... and every new sketch I made to test ...

Code:

void printDirectory(File dir, int numTabs) {
...
[B]    static int count = 0;   // BUGBUG remove this test code
    if (++count > 20) while (1) ;   // BUGBUG remove this test code
[/B]

@mjs513 crossposted some notes - but didn't see the context to look in the right spot.

KurtE · Nov 7, 2020

Another slight update: I changed from RAMDISK to SPI disk... and running on 4.1 with propshield again...

Fixed the print dir... function, something changed the "/" to " / " ... which did not find anything. I have been having some strange annoying Arduino IDE things... Yeah normally for bigger things use sublime...
<begin ignore>
Things like sometimes all of may Arduino windows decide to change font sizes. Sometimes when I compile fails, all of the windows change. Sometimes if I have the monitor window open on previous Teensy port, and then go to port menu and change ports...
Sometimes autoformat screws up
Somethings like I am on at the end of a line like: void loop() {
and hit CR it adds in a new line of }... Even though I already had one...
<End Ignore>

I found with running this sketch that it found the existing file: test.txt
and appended my string on to it... Which I suspected...

UPdated code, which removed the SPI2 stuff.

Code:

#include <LittleFS.h>
#include <IntervalTimer.h>
#include <SPI.h>

//#define TEST_RAM
#define TEST_SPI
//#define TEST_QSPI
#ifdef TEST_RAM
LittleFS_RAM myfs;
char buf[200000];
#elif defined(TEST_SPI)
LittleFS_SPIFlash myfs;
#else
LittleFS_QSPIFlash myfs;
#endif

void setup() {
  //  pinMode(13, OUTPUT);
  //  digitalWrite(13, HIGH);
  SPI.begin();
  while (!Serial) ; // wait
  Serial.println("LittleFS Test"); delay(5);

#ifdef TEST_RAM
  if (!myfs.begin(buf, sizeof(buf))) {
#elif defined(TEST_SPI)
  if (!myfs.begin(6)) {
#else
  if (!myfs.begin()) {
#endif
    Serial.println("Error starting LittleFS Disk");
    while (1) ;
  }
  Serial.println("started");
  File f = myfs.open("test.txt", FILE_WRITE);
  Serial.println("opened test.txt");

  // Output 64 zeros...
  f.println("0000000000000000000000000000000000000000000000000000000000000000");
  Serial.println("wrote to file");
  Serial.printf("POS: % u Size: % u\n", f.position(), f.size());
  f.close();
  delay(10);
  printDirectory();
}
uint16_t loop_count = 0;
void loop() {
  delay(1000);
  uint8_t buffer[100];
  File f = myfs.open("test.txt", FILE_READ);
  int cbRead = f.read(buffer, sizeof(buffer));
  Serial.printf(" % u: % d - % s\n", loop_count, cbRead, buffer);
  uint8_t buf_pos = loop_count++ & 0x1f;
  f.seek(buf_pos, SeekSet);
  uint8_t ch = buffer[buf_pos];
  ch = (ch < '9') ? ch + 1 : '0';
  buffer[buf_pos] = ch;
  f.write(buffer[buf_pos]);
  f.close();
  Serial.printf("%d:%s\n", buf_pos, buffer);
  printDirectory();

}

void printDirectory() {
  Serial.println("printDirectory\n--------------");
  printDirectory(myfs.open("/"), 0);
  Serial.println();
}

void printDirectory(File dir, int numTabs) {
  //dir.whoami();
  while (true) {

    static int count = 0;
    if (++count > 20) while (1) ;

    File entry =  dir.openNextFile();
    if (! entry) {
      // no more files
      //Serial.println("**nomorefiles**");
      break;
    }
    for (uint8_t i = 0; i < numTabs; i++) {
      Serial.print('\t');
    }
    Serial.print(entry.name());
    if (entry.isDirectory()) {
      Serial.println(" / ");
      printDirectory(entry, numTabs + 1);
    } else {
      // files have sizes, directories do not
      Serial.print("\t\t");
      Serial.println(entry.size(), DEC);
    }
    entry.close();
  }
}

Note: the code only runs the loop a couple of times at least for this run and hung

Code:

LittleFS Test

flash begin

Flash ID: EF 40 17
Flash size is 8.00 Mbyte
attemping to mount existing media

success

started

opened test.txt

wrote to file

POS: 200 Size: 200
  waited 364 us
printDirectory
--------------

myfolder / 

	morestuff.txt		78

test.txt		200



 0:  100 - 0000!00!0!0000////another test

This is a test....

another test

00000000000000000000000000000000
  waited 364 us
0:1000!00!0!0000////another test

This is a test....

another test

00000000000000000000000000000000
printDirectory
--------------

myfolder / 

	morestuff.txt		78

test.txt		200



 1:  100 - 1000!00!0!0000////another test

This is a test....

another test

00000000000000000000000000000000
  waited 364 us
1:1100!00!0!0000////another test

This is a test....

another test

00000000000000000000000000000000
printDirectory
--------------

myfolder / 

	morestuff.txt		78

test.txt		200



 2:  100 - 1100!00!0!0000////another test

This is a test....

another test

00000000000000000000000000000000
  waited 364 us
2:1110!00!0!0000////another test

This is a test....

another test

00000000000000000000000000000000
printDirectory
--------------

myfolder / 

	morestuff.txt		78

test.txt		200



 3:  100 - 1110!00!0!0000////another test

This is a test....

another test

00000000000000000000000000000000
  waited 50031 us
  waited 408 us
  waited 185 us
3:1111!00!0!0000////another test

This is a test....

another test

00000000000000000000000000000000
printDirectory
--------------

Will try again. Again wonder if there is some truncate function or do I need to maybe call remove first on the file name... ?

mjs513 · Nov 7, 2020

KurtE said:
Edit: Fixed the above program to at least be able to update ... Silly mistake

Code:

buffer[buf_pos] = (buffer[buf_pos] < '9') ? buffer[buf_pos]++ : '0';

should be:

Code:

buffer[buf_pos] = (buffer[buf_pos] < '9') ? buffer[buf_pos]+1 : '0';

But that implies I can write to a file I opened as Read...

As for open a file for READ/WRITE. The only two defines I see in FS.h are:

Code:

#define FILE_READ 0 #define FILE_WRITE 1

So obviously: does not good to do: FILE_READ|FILE_WRITE.

And obviously my seek to a position here and do a write did not truncate the file.

Also not sure yet why not showing anything in directory search.

Interesting, looking in lfs.h you have this:

Code:

// File open flags
enum lfs_open_flags {
    // open flags
    LFS_O_RDONLY = 1,         // Open a file as read only
    LFS_O_WRONLY = 2,         // Open a file as write only
    LFS_O_RDWR   = 3,         // Open a file as read and write
    LFS_O_CREAT  = 0x0100,    // Create a file if it does not exist
    LFS_O_EXCL   = 0x0200,    // Fail if a file already exists
    LFS_O_TRUNC  = 0x0400,    // Truncate the existing file to zero size
    LFS_O_APPEND = 0x0800,    // Move to end of file on every write

    // internally used flags
    LFS_F_DIRTY   = 0x010000, // File does not match storage
    LFS_F_WRITING = 0x020000, // File has been written since last flush
    LFS_F_READING = 0x040000, // File has been read since last flush
    LFS_F_ERRED   = 0x080000, // An error occured during write
    LFS_F_INLINE  = 0x100000, // Currently inlined in directory entry
    LFS_F_OPENED  = 0x200000, // File has been opened
};

which is what i was thinking since those are what I am use to seeing in elm-fat and what we have currently in SPIFS.

Since FS.h is wrapper for SDFAT_beta you see this in FsApiConstants.h

Code:

#if USE_FCNTL_H
#include <fcntl.h>
/* values for GNU Arm Embedded Toolchain.
[COLOR="#FF0000"] * O_RDONLY:   0x0
 * O_WRONLY:   0x1[/COLOR]
 * O_RDWR:     0x2
 * O_ACCMODE:  0x3
 * O_APPEND:   0x8
 * O_CREAT:    0x200
 * O_TRUNC:    0x400
 * O_EXCL:     0x800
 * O_SYNC:     0x2000
 * O_NONBLOCK: 0x4000
 */
/** Use O_NONBLOCK for open at EOF */
#define O_AT_END O_NONBLOCK  ///< Open at EOF.
typedef int oflag_t;
#else  // USE_FCNTL_H
[COLOR="#FF0000"]#define O_RDONLY  0X00  ///< Open for reading only.
#define O_WRONLY  0X01  ///< Open for writing only.[/COLOR]
#define O_RDWR    0X02  ///< Open for reading and writing.
#define O_AT_END  0X04  ///< Open at EOF.
#define O_APPEND  0X08  ///< Set append mode.
#define O_CREAT   0x10  ///< Create file if it does not exist.
#define O_TRUNC   0x20  ///< Truncate file to zero length.
#define O_EXCL    0x40  ///< Fail if the file exists.
#define O_SYNC    0x80  ///< Synchronized write I/O operations.

#define O_ACCMODE (O_RDONLY|O_WRONLY|O_RDWR)  ///< Mask for access mode.
typedef uint8_t oflag_t;
#endif  // USE_FCNTL_H

#define O_READ    O_RDONLY
#define O_WRITE   O_WRONLY

Maybe we need to suggest to Paul to add a few more for consistency.

defragster · Nov 7, 2020

KurtE said:
Another slight update: I changed from RAMDISK to SPI disk... and running on 4.1 with propshield again...

Fixed the print dir... function, something changed the "/" to " / " ... which did not find anything. I have been having some strange annoying Arduino IDE things... Yeah normally for bigger things use sublime...
...

Code:

... void printDirectory(File dir, int numTabs) { //dir.whoami(); while (true) { [B][COLOR="#FF0000"] static int count = 0; if (++count > 20) while (1) ; [/COLOR][/B] ...

Note: the code only runs the loop a couple of times at least for this run and hung

Code:

LittleFS Test [CODE]another test 00000000000000000000000000000000 printDirectory --------------

Will try again. Again wonder if there is some truncate function or do I need to maybe call remove first on the file name... ?

@KurtE (all) - in every LittleFS sketch with printDirectory() from original ... COMMENT out or REMOVE those two RED/BOLD lines. It was DEBUG to halt code for early testing after some 20 cycles inside that function.

See p#110 - after Paul's earlier note.

mjs513 · Nov 7, 2020

@KurtE
To fix your loop problem see post #110

KurtE · Nov 7, 2020

defragster said:
@KurtE (all) - in every LittleFS sketch with printDirectory() from original ... COMMENT out or REMOVE those two RED/BOLD lines. It was DEBUG to halt code for early testing after some 20 cycles inside that function.

See p#110 - after Paul's earlier note.

:0 - Forgot to remove it... Thanks... It is working.

@mjs513 - Yes I think we should have more defines. Also not sure how much they are or should be enforced. That is the above sketch, I open the file for READ and read it, but then seek to a position and do a write and that appears to go through.

mjs513 · Nov 7, 2020

Yep - running here as well on RAMDisk

had to try it. Think which defines you will select will depend on what you want to do. At they are there for compatibility.

UPDATE:
Added a few defines in FS.h:

Code:

#define FILE_RDWR    0X02  ///< Open for reading and writing.
#define FILE_AT_END  0X04  ///< Open at EOF.
#define FILE_APPEND  0X08  ///< Set append mode.
#define FILE_CREATE   0x10  ///< Create file if it does not exist.
#define FILE_TRUNC   0x20  ///< Truncate file to zero length.
#define FILE_EXCL    0x40  ///< Fail if the file exists.
#define FILE_SYNC    0x80  ///< Synchronized write I/O operations.

and if you want to test your modified sketch:

Code:

#include <LittleFS.h>
#include <IntervalTimer.h>
#include <SPI.h>

#define TEST_RAM
//#define TEST_SPI
//#define TEST_QSPI
#ifdef TEST_RAM
LittleFS_RAM myfs;
char buf[200000];
#elif defined(TEST_SPI)
LittleFS_SPIFlash myfs;
#else
LittleFS_QSPIFlash myfs;
#endif

File f;

void setup() {
  //  pinMode(13, OUTPUT);
  //  digitalWrite(13, HIGH);
  SPI.begin();
  while (!Serial) ; // wait
  Serial.println("LittleFS Test"); delay(5);

#ifdef TEST_RAM
  if (!myfs.begin(buf, sizeof(buf))) {
#elif defined(TEST_SPI)
  pinMode(6, INPUT_PULLUP);
  if (!myfs.begin(6)) {
#else
  if (!myfs.begin()) {
#endif
    Serial.println("Error starting LittleFS Disk");
    while (1) ;
  }
  Serial.println("started");
  f = myfs.open("test1.txt", FILE_CREATE | FILE_RDWR);
  Serial.println("opened test.txt");

  // Output 64 zeros...
  f.println("0000000000000000000000000000000000000000000000000000000000000000");
  Serial.println("wrote to file");
  Serial.printf("POS: % u Size: % u\n", f.position(), f.size());
  //f.close();
  f.seek(0);
  delay(10);
  printDirectory();
}
uint16_t loop_count = 0;
void loop() {
  delay(1000);
  uint8_t buffer[100];
  //File f = myfs.open("test.txt", FILE_READ);
  int cbRead = f.read(buffer, sizeof(buffer));
  Serial.printf(" % u: % d - % s\n", loop_count, cbRead, buffer);
  uint8_t buf_pos = loop_count++ & 0x1f;
  f.seek(buf_pos, SeekSet);
  uint8_t ch = buffer[buf_pos];
  ch = (ch < '9') ? ch + 1 : '0';
  buffer[buf_pos] = ch;
  f.write(buffer[buf_pos]);
  f.seek(0);
  //f.close();
  Serial.printf("%d:%s\n", buf_pos, buffer);
  printDirectory();

  if(loop_count == 100){
    f.close();
    Serial.println("Test Finished");
    printDirectory();
    while(1);
  }
}

void printDirectory() {
  Serial.println("printDirectory\n--------------");
  printDirectory(myfs.open("/"), 0);
  Serial.println();
}

void printDirectory(File dir, int numTabs) {
  //dir.whoami();
  while (true) {

    File entry =  dir.openNextFile();
    if (! entry) {
      // no more files
      //Serial.println("**nomorefiles**");
      break;
    }
    for (uint8_t i = 0; i < numTabs; i++) {
      Serial.print('\t');
    }
    Serial.print(entry.name());
    if (entry.isDirectory()) {
      Serial.println(" / ");
      printDirectory(entry, numTabs + 1);
    } else {
      // files have sizes, directories do not
      Serial.print("\t\t");
      Serial.println(entry.size(), DEC);
    }
    entry.close();
  }
}

KurtE · Nov 7, 2020

So far that is working. Also wondering about semantics and what things should or should not do...
For example:

Code:

  Serial.println("started");
  myfs.remove("test.txt");
  File f = myfs.open("test.txt", FILE_WRITE);
  Serial.println("opened test.txt");

  // Output 64 zeros...
  f.seek(100, SeekSet);  
  Serial.printf("POS: %u Size: %u\n", f.position(), f.size());
  f.seek(0, SeekSet);
  f.println("0000000000000000000000000000000000000000000000000000000000000000");
  Serial.println("wrote to file");
  Serial.printf("POS: %u Size: %u\n", f.position(), f.size());
  f.close();

I tell the file to Seek to 100, and then I seek back to zero and do write...
The output showed:

Code:

opened test.txt

POS: 100 Size: 0
wrote to file

POS: 66 Size: 66

Question is should the size be 100 after this or 66..
Also wondering about the 66? I thought I wrote 64 zeros plus trailing null character so expected 65.

Again not sure if the little fs knows or wants to know difference of how much space is allocated and how much space is used. Old Fat systems we wanted things like that to hopefully allocate space to keep from getting fragmented, such that we did not have to defrag it and faster random access if all contiguous.

PaulStoffregen · Nov 7, 2020

KurtE said:
c) what does FILE_READ or FILE_WRITE mean.

These are from Arduino's SD library API. Documentation is here:

https://www.arduino.cc/en/Reference/SDopen

While we may add some other stuff Arduino doesn't have, a major goal of mine is to keep compatibility with Arduino's API.

defragster · Nov 7, 2020

My A-Z files create/grow/remove with printDirs has been running since above posts on RamDisk with no issues. I should mix it up with directories doing the same in some fashion and known data beyond always using '@' perhaps. And a way to have large files in places to fill more of 16GB of space in the process.

@KurtE - did you scan the Design.MD file in the folder? I got partway (100 lines of 2,000!) into that when I came up with the 20 theory and the loop of just printDir - and discovered the halt ... but not the obvious cause. Then there is another Spec.MD (780 lines) with tech spec to read after.

It details the design of the FS and alternatives - choices and trade offs. To function on write only reformatted (garbage collect) - limit RAM use for structs - and logging/journaling like atomic operations in case of power loss.

Interesting details I didn't finish reading - but nothing like fast but fragile FAT. It records writes of data in blocks going forward able to move them at will when changed, leaving behind trashed blocks to garbage collect.

PaulStoffregen · Nov 7, 2020

KurtE said:
@mjs513 - Yes I think we should have more defines. Also not sure how much they are or should be enforced.

I agree, Arduino's File API is a bit too limiting. But my general approach to expanding / extending public APIs is usually slow and cautious. I want to avoid any incompatibility with Arduino's API, so anything add has to extend in ways that are compatible with programs using only Arduino's defs.

I do believe we need at least a way to open in write mode at the beginning of the file, since seeking to the end has quite a lot of overhead that's completely unnecessary if you're just going to seek back to the beginning. We probably also need a truncate function or while opening.

Realistically, we're probably only going to have 2 filesystems on Teensy in the foreseeable future: SdFat and LittleFS. These are just the logical filesystems. I imagine we'll have SdFat accessing SD cards and USB host MSC, and LittleFS accessing NOR & NAND flash on SPI & QSPI, and unused program memory, and of course the ramdisk. So that's a total of 8 supported disk types, 2 using SdFat and 6 using LittleFS. Are there others I'm forgetting?

defragster · Nov 7, 2020

@KurtE - also working in Sublime for IDE CMDline builds with TSET and TyCommander integrated [ or not to use pjrc loader ] works fine as SerMon long as DUAL_SERIAL is selected ( or triple ).

As noted I refined TSET to use PORTABLE build as I was afraid the LittleFS stuff might collide when the initial stuff was 'five' libs from github - but I gave that up and no issues doing Normal IDE Command Line Builds against github.com/PaulStoffregen/LittleFS in normal sketchbook\libraries.

defragster · Nov 7, 2020

PaulStoffregen said:
...
Realistically, we're probably only going to have 2 filesystems on Teensy in the foreseeable future: SdFat and LittleFS. These are just the logical filesystems. I imagine we'll have SdFat accessing SD cards and USB host MSC, and LittleFS accessing NOR & NAND flash on SPI & QSPI, and unused program memory, and of course the ramdisk. So that's a total of 8 supported disk types, 2 using SdFat and 6 using LittleFS. Are there others I'm forgetting?

What about LittleFS on PCB Boot Flash in a reserved area? T_4.0 might have 1MB, T_4.1 maybe 14 MB, and T_4.?? That is something the loader/bootloader would need to know about - and adaF_python would like that.

KurtE · Nov 7, 2020

@defragster - Thanks I have been avoiding doing dual serial as to avoid any incompatibilities.

@all I assume it is known, but the File system is case sensitive unlike fat file systems...
duplicated the write of the first file name:

Code:

  myfs.remove("test.txt");
  File f = myfs.open("test.txt", FILE_WRITE);
  Serial.println("opened test.txt");

  // Output 64 zeros...
  f.seek(100, SeekSet);  
  Serial.printf("POS: %u Size: %u\n", f.position(), f.size());
  f.seek(0, SeekSet);
  f.println("0000000000000000000000000000000000000000000000000000000000000000");
  Serial.println("wrote to file");
  Serial.printf("POS: %u Size: %u\n", f.position(), f.size());
  f.close();

  // wonderint case sensitive? 
  File f2 = myfs.open("TEST.txt", FILE_WRITE);
  f2.seek(0, SeekSet);
  f2.println("0000000000000000000000000000000000000000000000000000000000000000");
  f2.close();

And directory listing are showing:

Code:

printDirectory
--------------
TEST.txt		66
myfolder / 
	morestuff.txt		78
test.txt		66

defragster · Nov 7, 2020

I just cut my RamDisk to 20KB : char buf[20000];

I'm not watching failures - but they show and it continues - SLOWS DOWN - then gets back around to removing them and starting again.

Code:

.printDirectory RAMDISK
--------------
FILE	A_file.txt		0
FILE	B_file.txt		200
FILE	C_file.txt		400
FILE	D_file.txt		600
FILE	E_file.txt		400
FILE	F_file.txt		500
FILE	G_file.txt		600
FILE	H_file.txt		700
FILE	I_file.txt		800
FILE	J_file.txt		900
FILE	K_file.txt		1000
FILE	L_file.txt		1100
FILE	M_file.txt		1200
FILE	N_file.txt		1300
FILE	O_file.txt		1400
FILE	P_file.txt		1500
FILE	Q_file.txt		1600
FILE	R_file.txt		0
FILE	S_file.txt		0
FILE	T_file.txt		0
FILE	U_file.txt		0
FILE	V_file.txt		0
FILE	W_file.txt		0
FILE	X_file.txt		0
FILE	Y_file.txt		0
FILE	Z_file.txt		0

Bumping to 50KB it fits these files dynamically about 29KB (?) - but they are not growing larger as they should be. So 17KB of garbage and overhead it seems like in this case, :

Code:

.printDirectory RAMDISK
--------------
FILE	A_file.txt		1000
FILE	B_file.txt		100
FILE	C_file.txt		200
FILE	D_file.txt		300
FILE	E_file.txt		400
FILE	F_file.txt		500
FILE	G_file.txt		600
FILE	H_file.txt		700
FILE	I_file.txt		800
FILE	J_file.txt		900
FILE	K_file.txt		1000
FILE	L_file.txt		1100
FILE	M_file.txt		1200
FILE	N_file.txt		1300
FILE	O_file.txt		1400
FILE	P_file.txt		1500
FILE	Q_file.txt		1600
FILE	R_file.txt		1700
FILE	S_file.txt		1800
FILE	T_file.txt		1900
FILE	U_file.txt		2000
FILE	V_file.txt		2100
FILE	W_file.txt		2200
FILE	X_file.txt		2300
FILE	Y_file.txt		2400
FILE	Z_file.txt		2500

<edit>: Just remembered TABS are used so copy to Excel allows accurate size count [paste special, Text] - replaced above it 33.5KB

PaulStoffregen · Nov 7, 2020

defragster said:
What about LittleFS on PCB Boot Flash in a reserved area?

Short term, yes & no.

Yes, support for using part of program memory is next on my list of things to add. That might be Tuesday or later, as Robin & I are working all weekend on a large order for a special customer, not leaving much software dev time. This weekend is also Hackaday's virtual event. I won't be tuning in to much of it, but wanted to at least see a little later today.

Later, a lot later, like probably months or maybe late 2021 or farther, would be the time frame for the bootloader to be made aware of filesystem data stored in the program memory. Until then, uploading new code will (very slowly) wipe all the stored filesystem data. Best to add another chip on the bottom side of Teensy 4.1 or the audio shield or prop shield or other ways. A separate chip also avoids completely blocking all interrupts for the several milliseconds needed to erase flash sectors.

Speaking of those other ways, has anyone tried connecting a flash chip to SPI1 or SPI2? In theory it should work if you use something like this:

Code:

  myflash.begin(36, SPI2);

LittleFS port to Teensy/SPIFlash

Senior Member+

Senior Member+

Senior Member+

Well-known member

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Well-known member

Senior Member+

Well-known member

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Well-known member