LittleFS port to Teensy/SPIFlash

mjs513 · Dec 6, 2020

Have a question on how to do something with LittleFS and for that matter probably in general.

In lfs_util.h you can turn on different logging functions. Just cant't do a Serial.printf so how do I set it up to get them turned on.

KurtE · Dec 6, 2020

Good Morning all (or time in your time zone

)

Mike, I sort of looked at some of the debug stuff earlier and it looked like it was setup for a different system... I could look again.

Sorry I have playing around with MTP and the like. Although right now I am still wondering about why MTP dies when I copy a 300+KB file to QSPI...

So I have been playing around. Sorry some of this belongs probably in MTP thread, but part maybe here... As wondering some about how MTP does certain operations and how LittleFS works...

I have verified that system more or lies dies (drops the drive while still in bool MTPD::SendObject() {

Each time it dies first Windows opens up a progress window, which does not progress... So thoughts on this include:

a) Wonder if Windows is expecting us to send an event? Section 4.3.1 of the MTP manual talks some about asynchronous operations... Not sure if this applies but also did not see any appropriate events for progress? Also the progress end point is setup to come from Teensy to PC not the other way. Although probably could define it for both directions

b) Wonder if maybe Windows send another message asking for progress or not... So I quickly added some code to sendObject

Code:

  bool MTPD::SendObject() {
    uint32_t len = ReadMTPHeader();
    while (len) 
    { 
      //////////////////////////////////////////////////////////// 
      if(usb_mtp_available()) {
        digitalToggleFast(2);
        if(fetch_packet(rx_data_buffer)) {
           printContainer(); // to switch on set debug to 1 at beginning of file
        }
      }

Which I assumed would not work and does not get triggered, as I guessing is we only have one RX buffer on this RX Queue and we already received it... And we only give it back when we finish the processing of the event.

EDIT: Looks like this is using our queue of RX and the read operation automatically queues it back... So probably not this...

c) Now to what fits here. The current code minus my hacks:

Code:

  bool MTPD::SendObject() {
    uint32_t len = ReadMTPHeader();
    while (len) 
    { 
      receive_buffer();
      uint32_t to_copy = data_buffer_->len - data_buffer_->index;
      to_copy = min(to_copy, len);
      if(!storage_->write((char*)(data_buffer_->buf + data_buffer_->index), to_copy)) return false;
      data_buffer_->index += to_copy;
      len -= to_copy;
      if (data_buffer_->index == data_buffer_->len) 
      {
        usb_free(data_buffer_);
        data_buffer_ = NULL;
      }
    }
    storage_->close();
    return true;
  }

So it receives N packets of data... And then it tells the underlying FS to write the data. Which if I am guess correctly probably extends the data at that time...

Is it possible and would it speed up littleFS if at the start of this, we could do something like:

Code:

storage_->setFileSize(size

);

Where if it needs to allocate new stuff it can do all of the allocates and erasing up front?

Not sure if the FS class is setup for this? Example not sure if the file is newly created and I do something like: myfile.truncate(size);
Will it actually extend the file to that size?

Sorry if somewhat off topic.

WMXZ · Dec 6, 2020

mjs513 said:
Have a question on how to do something with LittleFS and for that matter probably in general.

In lfs_util.h you can turn on different logging functions. Just cant't do a Serial.printf so how do I set it up to get them turned on.

I do the following
in lfs_util.h

Code:

#define  LFS_YES_TRACE

void debug_print(const char *format, ...);

// Macros, may be replaced by system specific wrappers. Arguments to these
// macros must not have side-effects as the macros can be removed for a smaller
// code footprint

// Logging functions
#ifdef LFS_YES_TRACE
#define LFS_TRACE_(fmt, ...) \
    debug_print("%s:%d:trace: " fmt "%s\n", __FILE__, __LINE__, __VA_ARGS__)
......

and in the main .ino (or .cpp)

Code:

#include <stdarg.h>

extern "C" void debug_print(const char *format, ...)
{
  va_list args;

  va_start(args, format);
  for (; *format != 0; format++) { // no-frills stand-alone printf
    if (*format == '%') {
      ++format;
      if (*format == '%') goto out;
      if (*format == '-') format++; // ignore size
      while (*format >= '0' && *format <= '9') format++; // ignore size
      if (*format == 'l') format++; // ignore long
      if (*format == '\0') break;
      if (*format == 's') { debug_print((char *)va_arg(args, int));
      } else if (*format == 'd') { Serial.print(va_arg(args, int));
      } else if (*format == 'u') { Serial.print(va_arg(args, unsigned int));
      } else if (*format == 'x' || *format == 'X') {Serial.print(va_arg(args, int),HEX);
      } else if (*format == 'c' ) { Serial.print((char)va_arg(args, int));
      } else if (*format == 'p' ) { Serial.print(va_arg(args, int),HEX);}
    } else {
      out:
      if (*format == '\n') Serial.print('\r');
      Serial.print(*format);
    }
  }
  va_end(args);
}

I changed from printf to debug_print to avoid conflict in some system includes

mjs513 · Dec 6, 2020

@KurtE

Going to take me awhile to catch up on MTP again - way to focused on that Cypress chip

Anyway made some progress and think its a timing issue that I am stuck on. After looking at the data sheet I changed all the delayNanoseconds to 50nanosecond delays and changed the fram config to:

Code:

	{{0x03, 0x2E, 0xC2}, 24, 64, 256, 1048576, 10, 15},  //Cypress 8Mb FRAM

Now at least its mounting, unfortunately always wants to reformat but think thats timing. Following is not right but at least there is life:

Code:

D:\Users\Merli\Documents\Arduino\LittleFS tests\littlefs_teensy_test3\littlefs_teensy_test3.ino Dec  6 2020 12:03:37
LittleFS Timer Test
Flash ID: 03 2E C2
Flash size is 1.00 Mbyte
attempting to mount existing media
couldn't mount media, attemping to format
attempting to mount freshly formatted media
printDirectory
--------------

Disk Usuage:
Bytes Used: 512, Bytes Total:1048576
Using println and printf to printoutput file
opened PRINTOUTPUT1.txt
opened PRINTOUTPUT2.txt
printDirectory
--------------

-------------------------------
File1 contents:
-------------------------------
File1 size: 0

-------------------------------
File3 byte conversion test:
-------------------------------

Data_0: true
Data_1: 1.3574999571
Data_2: 314159
Data_3: 142
19
The Quick Brown Fox
Init Done - array loaded
...... ...... ......
create folder
  success


2nd Directory contents:
printDirectory
--------------


false, 0.0000000000, 0, 0, 

printDirectory
--------------

printDirectory
--------------
test1 / 
	file1.txt		16
test2 / 
	file2.txt		16
test3 / 

printDirectory
--------------
file20.txt		16
test1 / 
	file1.txt		16

mjs513 · Dec 6, 2020

Here's an example of it almost working. I rant MTP on the FRAM and copied a file from the desktop to FRAM, it worked kindof. It copied but when I went to open it only half the picture showed up. On this run when I went back to print directory:

Code:

LittleFS Test
Flash ID: 03 2E C2
Flash size is 1.00 Mbyte
attempting to mount existing media
started
printDirectory
--------------
FILE	T4.1-Cardlike.jpg		220021

printDirectory
--------------
FILE	mtpindex.dat		0

Disk Usuage:
Bytes Used: 512, Bytes Total:1048576

mjs513 · Dec 6, 2020

@WMXZ
Missed your post somehow and just noticed it - worked like a charm. At least I know what is causing the problem just don't know how to fix it yet. Still think its a timing issue on write/reads.

mjs513 · Dec 7, 2020

@Paul, @defragster, @KurtE, and all (i can add more)

Think I got that Cypress chip working with LittleFS now - was doing it the hard way - converted to using Paul's makeCommand-Address and it started working - by the way no wait time since there is no busy bit for the Cypress chips.

Code:

D:\Users\Merli\Documents\Arduino\LittleFS tests\littlefs_teensy_test3\littlefs_teensy_test3.ino Dec  7 2020 08:45:18
LittleFS Timer Test
Flash ID: 03 2E C2
Flash size is 1.00 Mbyte
attempting to mount existing media
printDirectory
--------------

Disk Usuage:
Bytes Used: 256, Bytes Total:1048576
Using println and printf to printoutput file
opened PRINTOUTPUT1.txt
opened PRINTOUTPUT2.txt
printDirectory
--------------
PRINTOUTPUT1.txt		628
PRINTOUTPUT2.txt		630

-------------------------------
File1 contents:
-------------------------------
File1 size: 628
abcdefghijklmnopqrstuvwxyz
Rec: 0, Float: 26.400000, Int: 98
abcdefghijklmnopqrstuvwxyz
Rec: 1, Float: 27.400000, Int: 99
abcdefghijklmnopqrstuvwxyz
Rec: 2, Float: 28.400000, Int: 100
abcdefghijklmnopqrstuvwxyz
Rec: 3, Float: 29.400000, Int: 101
abcde
ghijklmnopqrstuvwxyz
Rec: 4, Float: 30.400000, Int: 102
abcdefghijklmnopqrstuvwxyz
Rec: 5, Float: 31.400000, Int: 103
abcdefghijklmnopqrstuvwxyz
Rec: 6, Float: 32.400000, Int: 104
abcdefghijklmnopqrstuvwxyz
Rec: 7, Float: 33.400000, Int: 105
abcdefghi
klmnopqrstuvwxyz
Rec: 8, Float: 34.400000, Int: 106
abcdefghijklmnopqrstuvwxyz
Rec: 9, Float: 35.400000, Int: 107

-------------------------------
File3 byte conversion test:
-------------------------------

Data_0: true
Data_1: 1.3574999571
Data_2: 314159
Data_3: 142
19
The Quick Brown Fox
Init Done - array loaded
...... ...... ......
create folder
  success


2nd Directory contents:
printDirectory
--------------
PRINTOUTPUT1.txt		628
PRINTOUTPUT2.txt		630
structuredData / 
	logger.txt		480


true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox
true, 1.3574999571, 314159, 142, The Quick Brown Fox

printDirectory
--------------
PRINTOUTPUT1.txt		628
PRINTOUTPUT2.txt		630
file1.txt		32
file10.txt		16
file2.txt		32
file20.txt		16
file3.txt		32
file30.txt		16
structuredData / 
	logger.txt		480
test1 / 
test2 / 
test3 / 

printDirectory
--------------
PRINTOUTPUT1.txt		628
PRINTOUTPUT2.txt		630
file1.txt		32
file10.txt		16
file2.txt		32
file20.txt		16
file3.txt		32
file30.txt		16
structuredData / 
	logger.txt		480
test1 / 
	file1.txt		16
test2 / 
	file2.txt		16
test3 / 
	file3.txt		16

printDirectory
--------------
PRINTOUTPUT1.txt		628
PRINTOUTPUT2.txt		630
file1.txt		32
file10.txt		16
file2.txt		32
file20.txt		16
file3.txt		32
file30.txt		16
structuredData / 
	logger.txt		480
test1 / 
	file1.txt		16
test2 / 
	file2.txt		16
test3 / 
	file3.txt		16

Disk Usuage:
Bytes Used: 6272, Bytes Total:1048576

Here are the modified LittleFS files:

View attachment littlefs.zip

EDIT: Just tested it with MTP and it works - even created an extra 10 test files.

EDIT 2: Made one more change to the chip info line:

Code:

	{{0x03, 0x2E, 0xC2}, 24, 256, 256, 1048576, 250, 1200},  //Cypress 8Mb FRAM

This works - increasing seems to break it all.

KurtE · Dec 7, 2020

One of these days may have to get a Cyprus chip...

Still playing with the MTP, trying to see why/when it fails for larger files.

I thought I would first see if LittleFS might die on these larger files...
So have a screwy little test sketch which allows me to experiment with type in file size, buffer size and file name and try to create them...

Nothing special:

Code:

#include <LittleFS.h>
#include <SPI.h>
#include <SD.h>

//#define TEST_RAM
//#define TEST_SPI
#define TEST_QSPI
//#define TEST_SDCARD
#ifdef TEST_RAM
LittleFS_RAM myfs;
char buf[200000];
#elif defined(TEST_SPI)
LittleFS_SPIFlash myfs;
#elif defined(TEST_SDCARD)
SDClass myfs;
#else
LittleFS_QSPIFlash myfs;
#endif

void setup() {
#ifdef TEST_SPI
  SPI.begin();
#else
  pinMode(13, OUTPUT);
  digitalWrite(13, HIGH);
#endif
  while (!Serial && millis() < 5000) ; // wait
  Serial.read();
  delay(250);
  Serial.println("LittleFS Test"); delay(5);

#ifdef TEST_RAM
  if (!myfs.begin(buf, sizeof(buf))) {
#elif defined(TEST_SPI)
  if (!myfs.begin(6)) {
#elif defined(TEST_SDCARD)
  if (!myfs.begin(BUILTIN_SDCARD)) {
#else
  if (!myfs.begin()) {
#endif
    Serial.println("Error starting LittleFS Disk");
    while (1) ;
  }

  Serial.println("started");

  delay(10);
}
uint16_t loop_count = 0;
elapsedMillis em = 0;
uint8_t filename[256] = "test.txt";
uint8_t buffer[8 * 1024];

void loop() {
  printDirectory();
  Serial.println("Enter file size write size and name to create: ");
  while (!Serial.available()) ;
  uint32_t file_size = Serial.parseInt();
  uint16_t write_size = Serial.parseInt();
  uint8_t *psz = filename;
  int ch = Serial.read();

  while (ch == ' ') ch = Serial.read();
  while (ch > ' ') {  // not allowing spaces in test.
    *psz++ = ch;
    ch = Serial.read();
  }
  if (psz != filename) *psz = 0;  // null terminate
  if ((write_size > sizeof(buffer)) || (write_size == 0)) write_size = sizeof(buffer);

  Serial.printf("Create File:%s size:%u write size:%u\n", filename, file_size, write_size);
  elapsedMicros emTotal = 0;
  File f = myfs.open(filename, FILE_WRITE);
  Serial.println("*** opened ***");

  for (uint16_t i = 0; i < write_size; i++) buffer[i] = i & 0xff;
  while (file_size) {
    elapsedMicros em = 0;
    uint16_t output_count = (file_size > write_size) ? write_size : file_size;
    uint16_t bytes_output = f.write(buffer, output_count);
    if (output_count != output_count)
      Serial.printf("  %u - %u - %u\n", output_count, bytes_output, (uint32_t)em);
    file_size -= bytes_output;
  }
  f.close();
  Serial.printf("Closed: %u\n", (uint32_t)emTotal);
}

void printDirectory() {
  Serial.println("printDirectory\n--------------");
  printDirectory(myfs.open("/"), 0);
  Serial.println();
}

void printDirectory(File dir, int numTabs) {
  //dir.whoami();
  while (true) {

    static int count = 0;
    File entry =  dir.openNextFile();
    if (! entry) {
      // no more files
      //Serial.println("**nomorefiles**");
      break;
    }
    for (uint8_t i = 0; i < numTabs; i++) {
      Serial.print('\t');
    }
    Serial.print(entry.name());
    if (entry.isDirectory()) {
      Serial.println(" / ");
      printDirectory(entry, numTabs + 1);
    } else {
      // files have sizes, directories do not
      Serial.print("\t\t");
      Serial.println(entry.size(), DEC);
    }
    entry.close();
  }
}

Running on two T4.1s with TyCommander where it updates both chips at same time...

If I run with the original T4.1 the time for a large file, about 3mb...

Code:

Enter file size write size and name to create: 
Create File:bar.txt size:3355221 write size:8192
*** opened ***
[COLOR="#FF0000"]Closed: 26944362[/COLOR]
printDirectory
--------------
DSC03356.JPG		3355221
DSC03358.JPG		3280204
IMG_1259.jpg		368708
bar.txt		3355221
foo.txt		3355221

Enter file size write size and name to create:

Which I was earlier able to transfer that size file earlier from PC.

With my two newer T4.1s with the later QSPI flash (same chip later run)

Code:

Enter file size write size and name to create: 
Create File:bar.txt size:3355221 write size:8192
*** opened ***
[COLOR="#FF0000"]Closed: 41192928[/COLOR]
printDirectory
--------------
DSC03356.JPG		0
DXL-RS485.jpg		20702
IMG_0112-(002).jpg		19576
IMG_0119.jpg		0
IMG_0283.jpg		244650
IMG_0287.jpg		0
IMG_1121.jpg		79375
IMG_1132.jpg		47198
KeDeiTest.jpg		10142
bar.txt		3355221
foo.txt		3695221
phoenix-rotate-RJoy-LR.jpg		0
test.txt		0
usb3-board-back.jpg		47786

As you can see the earlier chip completes the run in about 27 seconds, and the new one in about 41 8) seconds.

Note: I did hack this up (and updated code above to allow to write using SD library to SDCard and assuming I did things right:

Code:

Enter file size write size and name to create: 
Create File:bar.txt size:3695221 write size:8192
*** opened ***
[COLOR="#FF0000"]Closed: 187088[/COLOR]
printDirectory
--------------
System Volume Information / 
	IndexerVolumeGuid		76
	WPSettings.dat		12
LaikH.JPG		2801
mtpindex.dat		0
test1.txt		21
hallow.JPG		21026
DSC03360.JPG		3161263
DRAKE.JPG		15775
foo.txt		3355221
bar.txt		3695221

Enter file size write size and name to create:

It took under .2 seconds! So needless to say LittleFS is no speed demon!

mjs513 · Dec 7, 2020

@KurtE - nope doesn't appear that way. But some of the things you might play with from within LittleFS are the chip settings, especially for progsize and progtime. Not sure how hard they are or if there is some wiggle room with the settings - haven't played with them yet - except for the Cypress chip. Right now they are all set at 256 byte transfers and about 3500 microseconds as a wait for not busy. Maybe pick a chip and try larger or smaller writes. Was wondering that myself as I was playing with the FRAM.

mjs513 · Dec 7, 2020

#KurtE
Got curious so ran a quick test using @defragster's BigFile test. Using the Winbond 128Mb flash as a test case and only changing the program time:
Default: 256/3500microseconds

Code:

Big write took 77.33 Sec for 8256000 Bytes : file3.size()=8256000
	[COLOR="#FF0000"]Big write KBytes per second 106.76 [/COLOR]
Bytes Used: 8335360, Bytes Total:16777216
-------------------------------------------------------------------------------

	{{0xEF, 0x40, 0x18}, 24, [COLOR="#FF0000"]256[/COLOR], 4096, 16777216, [COLOR="#FF0000"]2000[/COLOR], 400000}, // Winbond W25Q128JV*IQ/W25Q128FV
Big write took 54.91 Sec for 8280000 Bytes : file3.size()=8280000
	[COLOR="#FF0000"]Big write KBytes per second 150.79 [/COLOR]
Bytes Used: 8306688, Bytes Total:16777216


	{{0xEF, 0x40, 0x18}, 24, 128, 4096, 16777216, 2000, 400000}, // Winbond W25Q128JV*IQ/W25Q128FV
Big write took 73.69 Sec for 8280000 Bytes : file3.size()=8280000
	Big write KBytes per second 112.36 

	{{0xEF, 0x40, 0x18}, 24, 64, 4096, 16777216, 2000, 400000}, // Winbond W25Q128JV*IQ/W25Q128FV
Big write took 58.11 Sec for 8280000 Bytes : file3.size()=8280000
	Big write KBytes per second 142.49 
Bytes Used: 8306688, Bytes Total:16777216

	{{0xEF, 0x40, 0x18}, 24, 32, 4096, 16777216, 2000, 400000}, // Winbond W25Q128JV*IQ/W25Q128FV
Big write took 60.63 Sec for 8280000 Bytes : file3.size()=8280000
	Big write KBytes per second 136.56 
Bytes Used: 8306688, Bytes Total:16777216

So looks like 256 it is but it can prog faster at 2000microseconds

KurtE · Dec 7, 2020

@mjs513 - Not sure what that change does or should do?

That is, as far as I can tell, the maximum time it can wait for the program time to complete...

Code:

int LittleFS_SPIFlash::prog(lfs_block_t block, lfs_off_t offset, const void *buf, lfs_size_t size)
{
	if (!port) return LFS_ERR_IO;
	const uint32_t addr = block * config.block_size + offset;
	const uint8_t cmd = (addrbits == 24) ? 0x02 : 0x12; // page program
	uint8_t cmdaddr[5];
	make_command_and_address(cmdaddr, cmd, addr, addrbits);
	//printtbuf(cmdaddr, 1 + (addrbits >> 3));
	port->beginTransaction(SPICONFIG);
	digitalWrite(pin, LOW);
	port->transfer(0x06); // 0x06 = write enable
	digitalWrite(pin, HIGH);
	delayNanoseconds(250);
	digitalWrite(pin, LOW);
	port->transfer(cmdaddr, 1 + (addrbits >> 3));
	port->transfer(buf, nullptr, size);
	digitalWrite(pin, HIGH);
	port->endTransaction();
	//printtbuf(buf, 20);
	return wait(progtime);
}
...
int LittleFS_SPIFlash::wait(uint32_t microseconds)
{
	elapsedMicros usec = 0;
	while (1) {
		port->beginTransaction(SPICONFIG);
		digitalWrite(pin, LOW);
		uint16_t status = port->transfer16(0x0500); // 0x05 = get status
		digitalWrite(pin, HIGH);
		port->endTransaction();
		if (!(status & 1)) break;
		if (usec > microseconds) return LFS_ERR_IO; // timeout
		yield();
	}
	//Serial.printf("  waited %u us\n", (unsigned int)usec);
	return 0; // success
}

So if changing that helps. It implies you are now getting timeouts? Not sure if that is good?

mjs513 · Dec 7, 2020

KurtE said:
@mjs513 - Not sure what that change does or should do?

That is, as far as I can tell, the maximum time it can wait for the program time to complete...

.....So if changing that helps. It implies you are now getting timeouts? Not sure if that is good?

Thought about that after I posted and had some diner - also got the new toy with a PT8211 board so got distracted

. Will test more later. Funny thing is for the FRAM there is no time out use since there is no busy bit = at least according to the data sheet.

EDIT:
Just as a follow I reran at 2000us along with Error messages turned on since in the wait function there are 2 tests:

Code:

		if (!(status & 1)) break;
		if (usec > microseconds) return LFS_ERR_IO; // timeout

the first breaks the loop when the chip is no longer busy and the second throws an error if it timeouts.

I received no error messages so no timeouts. I also checked the data file contents via MTP and they all looked good. I also transferred the card like image and opened it no problem from PC to teensy.

defragster · Dec 7, 2020

@mjs513 - may not be relevant to Cypress - but it may be something common to the MRAM

I didn't read the PDF - but this line did pop out on my way to seeing the Pinout :: FOR THE ROHM FeRAM (opps been writing MeRAM for these)

>> Operating Frequency: 34 MHz (READ cycle) / 40 MHz (except for READ)

Given Freq at 30 MHz - there should be no need to wait? ... at least for the ROHM chips ...

KurtE · Dec 7, 2020

@Paul, @mjs513 and @defragster ... Sorry in advance if some of these are random questions or ...

Having a little fun with the test app, to allow me to type in file name sizes and write sizes...

It is helping me to understand or figure out I don't understand the proper semantics.

Example: Suppose I wish to to write to a file lets say foo.txt and supposed it is 1MB image. And the file may already exist or not exist. What is the best way to do this. Suppose foo.txt exists and is 2mb in size.

a) Suppose the code is something like:

Code:

    File f = myfs.open(filename, FILE_WRITE);
    for (uint16_t i = 0; i < write_size; i++) buffer[i] = i & 0xff;
    uint8_t write_counter = 0;
    while (file_size) {
      uint16_t output_count = (file_size > write_size) ? write_size : file_size;
      uint16_t bytes_output = f.write(buffer, output_count);
      file_size -= bytes_output;
      }
    }
    f.close();
  }

When this code completes the file will now be 3mb in size (It appends)

b) Like above, but I replace FILE_WRITE with FILE_WRITE_BEGIN, the file still ends up being 2mb in size with new stuff at start...

c) like b) but just before f.close() I do. f.truncate();
File ends up being 0 length... Defaults to 0 and not relative... (Yep I did this)

d) like b) but put the f.truncate() right after the myfs.open()... This worked and is what currently doing.

d2) could do like c) but instead maybe do: f.truncate(f.position());
I have not tried this, but I would think would work.

But which approach is better. Especially since sometimes through LittleFS other times to SDCards or ...

That is with some filesystems it is beneficial to have the storage preallocated and then fill in the data. Others maybe not? I am guessing maybe not in LittleFS case?

Related question: Suppose with something like MTP, I know the file that I going to write is lets say 3MB in size. Can LittleFS make use of this data.
It does look if I call f.truncate(3000000);
The underlying code will take the current size and if the truncate size is > current size, it will extend the file by writing 0s... My guess is that then if I try to write into this region it won't be able to directly write out the new data. That is will it have to clear out that region first again? If so does it make sense to have some mechanism for caller to FS to give you a hint on what size the file will likely be?

EDIT: I believe SDFat has a method: bool preAllocate(uint64_t length) ...

Probably enough rambling for one message

defragster · Dec 7, 2020

Just a test of course - normally if file write finds existing file it might want to delete content or refuse to open - unless just opening to append.

What about if ( file.size()> myX ) { file.seek(myX); file.close(); }

Yes, Truncate needs a value to not do 0 :: truncate(uint64_t size=0)

In the case of LittleFS_(flash) - any preAllocate() behavior might format - then end up doing a second write/alloc/format - not sure if it checks for re-use - though it might not actually ZERO write until close?. It would probably prealloc the Dir entries if a change is needed.

Code could add a command to truncate(3MB) and time it. Could then do the write and time that. And time the .close() (without or without the write()).

wwatson · Dec 7, 2020

KurtE said:
@Paul, @mjs513 and @defragster ... Sorry in advance if some of these are random questions or ...

Having a little fun with the test app, to allow me to type in file name sizes and write sizes...

It is helping me to understand or figure out I don't understand the proper semantics.

Example: Suppose I wish to to write to a file lets say foo.txt and supposed it is 1MB image. And the file may already exist or not exist. What is the best way to do this. Suppose foo.txt exists and is 2mb in size.

a) Suppose the code is something like:

Code:

File f = myfs.open(filename, FILE_WRITE); for (uint16_t i = 0; i < write_size; i++) buffer[i] = i & 0xff; uint8_t write_counter = 0; while (file_size) { uint16_t output_count = (file_size > write_size) ? write_size : file_size; uint16_t bytes_output = f.write(buffer, output_count); file_size -= bytes_output; } } f.close(); }

When this code completes the file will now be 3mb in size (It appends)

b) Like above, but I replace FILE_WRITE with FILE_WRITE_BEGIN, the file still ends up being 2mb in size with new stuff at start...

c) like b) but just before f.close() I do. f.truncate();
File ends up being 0 length... Defaults to 0 and not relative... (Yep I did this)

d) like b) but put the f.truncate() right after the myfs.open()... This worked and is what currently doing.

d2) could do like c) but instead maybe do: f.truncate(f.position());
I have not tried this, but I would think would work.

But which approach is better. Especially since sometimes through LittleFS other times to SDCards or ...

That is with some filesystems it is beneficial to have the storage preallocated and then fill in the data. Others maybe not? I am guessing maybe not in LittleFS case?

Related question: Suppose with something like MTP, I know the file that I going to write is lets say 3MB in size. Can LittleFS make use of this data.
It does look if I call f.truncate(3000000);
The underlying code will take the current size and if the truncate size is > current size, it will extend the file by writing 0s... My guess is that then if I try to write into this region it won't be able to directly write out the new data. That is will it have to clear out that region first again? If so does it make sense to have some mechanism for caller to FS to give you a hint on what size the file will likely be?

EDIT: I believe SDFat has a method: bool preAllocate(uint64_t length) ...

Probably enough rambling for one message

I ran into the same situation using SdFat with bench.ino. The size of the file was not consistent and could not determine the end of file. It would change. If I am not mistaken file size of the file and EOF should be the same after closing. Maybe I am wrong. This was using preAllocate(). There was unused space after the actual written data that was arbitrary. Using maybe fielded reads with a fixed number of records would not be a problem but let's say a text file without proper end of file marking would be. This what I ran into with bench.ino. The file size was not always reported as 5000000. Sometimes it was less. What should the file size be if the 5000000 is not completely used. The preAllocated size of 5000000 or the actual size of the data written? EOF character or size of data within the file. Truncate(sizeof(dataWitten) )then close?
.

KurtE · Dec 8, 2020

wwatson said:
I ran into the same situation using SdFat with bench.ino. The size of the file was not consistent and could not determine the end of file. It would change. If I am not mistaken file size of the file and EOF should be the same after closing. Maybe I am wrong. This was using preAllocate(). There was unused space after the actual written data that was arbitrary. Using maybe fielded reads with a fixed number of records would not be a problem but let's say a text file without proper end of file marking would be. This what I ran into with bench.ino. The file size was not always reported as 5000000. Sometimes it was less. What should the file size be if the 5000000 is not completely used. The preAllocated size of 5000000 or the actual size of the data written? EOF character or size of data within the file. Truncate(sizeof(dataWitten) )then close?
.

I could be wrong, I am not a file system expert, but...

I believe as far as the File System is concerned there really is no EOF character or the like. The File system simply has a couple of logical counters. The Current "Position" and "File Size". I believe right now when you close the file the position information is not used and the size stays the size. With some FileSystems I believe like FAT the file header may contain at least two sizes. I don't remember the actual names but lets say the actual size and maybe the allocated size. I also don't remember if the system remembers the preallocate size or not, but I believe it is mainly only used when you first create a file as a hint to system to allow it to try to grab as much of this space as possible to keep it unfragmeneted. And again I don't know LittleFS well enough to know if you could get a speed boost if the system knew how big the file will be, when something like MTP is going to transfer a file to it.

With our current FS.h limited options to the file open, there are very few hints we can pass through, to underlying system to know the users intent. So we probably need to maybe simply document some optional ways to perform different operations and maybe with the different File Systems the preferred ways may be different.

Example: I wish to create a log file. Lets call it log.txt and I wish to overwrite that file if it exists and the new contents to fully replace the old contents. Which way is best?
a) myfs.remove("log.txt"); File f = myfs.open("log.txt", FILE_WRITE);... myfs.close();
b) File f = myfs.open("log.txt", FILE_WRITE_BEGIN); f.truncate(); ... f.close();
c) File f = myfs.open("log.txt", FILE_WRITE_BEGIN); ... f.truncate(f.position())); ... f.close();
...
And it is the same answer for both SDFat as well as LittleFS?

There are probably lots of similar scenarios as well. Like creating a file that is logically N logical records where each record takes N bytes... Here you don't want to truncate when closing.

But right now I simply more curious on if we can get things fast enough to do simple things with things like MTP and copy files back and forth.

KurtE · Dec 8, 2020

@all - To somewhat answer some of my previous questions, wondering.

Doing something like: f.truncate(5000000);
Will probably not extend the file in a way that makes it good for me to then write new data in that space. What I believe this will do is to extend the file writing 0s from the current EOF to the specified location. And as Erasing sectors and the like writes all 0xff's, I am pretty sure that when we go back out, it will then have to erase these pages again, before writing out the real data.

So again wondering if it makes sense to add a method to FS.h interface to allow user code to call something like preAllocateSize(), which in SD version would call through.
Wondering if a reasonable performing version could be developed for LittleFS.

Sorry again if my mind is wondering too much.

But also wondering if the code could be setup that in certain cases it could erase larger chunks of flash.
That is we currently call the Sector Erase (4KB) command for each of these take 45ms but max could be 400ms... But there are also block erase commands
There is one version which erases 32KB block with typical 120ms max 1600 and likewise a 64KB block which typical 150 max 2000
Yes these are still real slow, and may not work in the case of MTP where we may hang too long. Although wondering if again if we can setup to maybe do stuff while this is in operation.

Also wondering with MTP and the like, when we are receiving an object, it retrieves data from the USB, and moves it into some buffer, until the buffer has some N bytes available, at which time it calls off to the underlying File system with that buffer. Currently that is by default setup for 8kb(8192) bytes. I believe this was chosen as this is probably a default sector size on SDFat code so maybe optimal for that one File system. Wondering if there should be a method on File System (FS.h) that code like MTP might query to find out what size writes are preferred....

Now back to playing.

defragster · Dec 8, 2020

KurtE said:
@all - To somewhat answer some of my previous questions, wondering.

Doing something like: f.truncate(5000000);
Will probably not extend the file in a way that makes it good for me to then write new data in that space. What I believe this will do is to extend the file writing 0s from the current EOF to the specified location. And as Erasing sectors and the like writes all 0xff's, I am pretty sure that when we go back out, it will then have to erase these pages again, before writing out the real data.
...

I see truncate() does see larger size and Zero write "// fill with zeros" - so the timing test mentioned posts back would show format at that time it seems as the file chain would be extended.

Whether LFS is smart enough to overwrite those 0's (0xff's) the next pass with user data is a question. The other note in that prior post suggested testing the timing again at that point.

@mjs513 - IIRC - found that NAND can't rewrite blocks though and when it does "// fill with zeros" that will alloc the blocks and apply the 'tag' info for each block, so even if the zero fill doesn't touch the block where format to 0xff is Zero's it would have to detect that for NOR and it would fail for NAND?

I didn't read enough to see how the block link list is stored? i.e. are directory entries filled with the block list - or do the tag'd block link to the next in the chain? In either case that would take new code in the 'lfs' core code to orchestrate that.

Back to what I saw as the best solution was the combined bit lists to 'know': LFS used, free unformatted, free formatted and keep the unformatted space minimized by user command or by background task ( not sure how that would work with active access to the 'disk' )

The bit list seemed promising on small NOR media of 8 and 16 MB - but that got ugly/slow testing the 64MB media - especially when the bit tracking needed to be 100% and those lists were transitory in RAM only. And who knows what that would look like when 128 and 256MB NAND shows up. It seems logically it could be orchestrated and maintained - with some overhead of memory, time and adding storage of those lists to FLASH perhaps. Bottom line is the slow Format speed needs to be done at some point. LFS by default always does that on the data/meta data write.

defragster · Dec 9, 2020

Experiment - worth lots of time spent otherwise ... see p#539 :: "Code could add a command to truncate(3MB) and time it. Could then do the write and time that. And time the .close() (without or without the write())."

Did a quick DUPE and edit of LFSintegrity bigfile() - change snippet below.

>> On TRUNCATE the blocks re preformatted and allocated :: took 68.27
-->> Assume this takes longer than write as the Dir_Meta blocks are formatted in advance - then just updated during write?
>> On Write blocks are ReFormatted and written :: took 67.87

SerMon output after QuickFormat, DIR then create TWO pre truncate() files, then DIR:

Code:

 0 dirs with 0 files of Size 0 Bytes
 Total 0 files of Size 0 Bytes
Bytes Used: 8192, Bytes Total:16777216

Start Big write of 8364032 Bytes with TRUNCATE ... patience ...
Big write [B]TRUNCATE /0_bigftrunc.txt took 68.27 Sec [/B]for 8364032 Bytes : file3.size()=0
......................................................................................................
Big [B]write /0_bigftrunc.txt took 67.87 Sec for 8361984 Bytes [/B]: file3.size()=8364032
	Big write KBytes per second 123.21 

Bytes Used: 8388608, Bytes Total:16777216

[  2.52 M](2.26913 M elap) Awaiting input 0123456789RdchkFqvplmusBbZztyYxf+-? loops left 0 >

Start Big write of 4173824 Bytes with TRUNCATE ... patience ...
Big write [B]TRUNCATE /1_bigftrunc.txt took 33.95 Sec [/B]for 4173824 Bytes : file3.size()=0
..................................................
Big [B]write /1_bigftrunc.txt took 33.29 Sec for 4171776 Bytes[/B] : file3.size()=4173824
	Big write KBytes per second 125.32 

Bytes Used: 12570624, Bytes Total:16777216

[  5.77 M](1.12101 M elap) Awaiting input 0123456789RdchkFqvplmusBbZztyYxf+-? loops left 0 > d
printDirectory QSPI_DISK
--------------
FILE	0_bigftrunc.txt		8364032
FILE	1_bigftrunc.txt		4173824

 0 dirs with 2 files of Size 12537856 Bytes
 Total 2 files of Size 12537856 Bytes
Bytes Used: 12570624, Bytes Total:16777216

Process in code snippet:

Code:

...
		file3 = [B]myfs.open(myFile, FILE_WRITE);[/B]
	} while ( fileID < '4' && file3.size() > 0);
	[B]file3.truncate( toWrite );
	file3.close();[/B]
	timeMe = micros() - timeMe;
	Serial.printf( "\nBig write TRUNCATE %s took %5.2f Sec for %lu Bytes : file3.size()=%llu\n", myFile , timeMe / 1000000.0, xx, file3.size() );
	memset( someData, fileID, 2048 );
	int hh = 0;
	timeMe = micros();
	[B]file3 = myfs.open(myFile, FILE_WRITE);
	file3.seek( 0 );[/B]
	while ( toWrite > 2048 && resW > 0 ) {
		[B]resW = file3.write( someData , 2048 );[/B]
...

NOTE: File sizes above are not equal the initial guess at size is rounded down to the last full write of 2048 bytes in this case - as shown in the last while()

Also I skipped the 'delete and verify' but added that - had to add a truncate() to the actual written size or it failed hitting the extra initial bytes:

Code:

Delete with read verify all #bigfile's
	Verify /0_bigftrunc.txt bytes 8361984 : ..................................................	GOOD! >>  bytes 8361984
	Big read&compare KBytes per second 2416.87 
	Verify /1_bigftrunc.txt bytes 4171776 : ..................................................	GOOD! >>  bytes 4171776
	Big read&compare KBytes per second 2442.15

defragster · Dec 9, 2020

Follow up to p#544. PreFormatted Media when bitmap says reutn without format returns these results.

It show indeed to write a file on HALF the disk (with above test) - the .truncate() used HALF the blocks, then the write voids those blocks and write the data to the other half of the disk blocks.

The PreFormat aware coding cuts the time way down - but Truncate is not a good answer, though PreFormat aware helps a great deal.

On the MTP thread FrankB mentioned formatting during use - Paul hinted at that with a background formatter? If that were coordinated and safe during use that would be cool. But only one format at a time - so having PreFormat information and having the blocks ready for LFS will keep it for asking for a format.

This is an example:

Code:

 checkFormat:: Pre-Formatted Blocks Map 


 checkUsed:: lfs Used Blocks Map: #used is 2 : lfs traverse return 0 (neg on err)  [COLOR="#FF0000"][B]// NO LFS DATA USED  (except the root 'bits')[/B][/COLOR]

0	0x3	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1280	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1600	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1920	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2240	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2560	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2880	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3200	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3520	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3840	0x0	0x0	0x0	0x0	0x0	0x0	0x0
 checkUsed:: lfs Used Blocks Map: #used is 2

 lfs Used&Format overlap errs?: {eol}

 checkFormat:: lfsUsed = checkused; [COLOR="#FF0000"][B]// ALL media BLOCKS FORMATTED (except the root 'bits')[/B][/COLOR]

0	0xfffffffc	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
320	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
640	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
960	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
1280	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
1600	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
1920	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2240	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2560	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2880	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3200	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3520	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3840	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
 Done myfs.checkFormat() in 581803 us.

 CheckFormat On: 
[  7.64 M](0.00000 M elap) Awaiting input 0123456789RdchkFqvplmusSBbZztyYxf+-? loops left 0 >

Start Big write of 8364032 Bytes with TRUNCATE ... patience ...  [COLOR="#FF0000"][B]// TIMES 15 and 13 seconds NOT 68 and 67 seconds for 8MB write[/B][/COLOR]
Big write [B]TRUNCATE /0_bigftrunc.txt took 15.36 Sec[/B] for 8364032 Bytes : file3.size()=0
......................................................................................................
Big [B]write /0_bigftrunc.txt took 13.70 Sec for 8361984 Bytes[/B] : file3.size()=8361984
	Big write KBytes per second 610.30 

Bytes Used: 8388608, Bytes Total:16777216

[  8.27 M](0.48445 M elap) Awaiting input 0123456789RdchkFqvplmusSBbZztyYxf+-? loops left 0 >

 checkFormat:: Pre-Formatted Blocks Map 

 lfschange missed used 2 != 2048

 checkUsed:: lfs Used Blocks Map: #used is 2048 : lfs traverse return 0 (neg on err)[COLOR="#FF0000"][B]// HALF LFS DATA USED [/B][/COLOR]

0	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xfffffff	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1280	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1600	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1920	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2240	0xf0000000	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2560	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
2880	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3200	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3520	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
3840	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff	0xffffffff
 checkUsed:: lfs Used Blocks Map: #used is 2048

 lfs Used&Format overlap errs?: {eol}

 checkFormat:: lfsUsed = checkused; [COLOR="#FF0000"][B]// ONLY TWO media BLOCKS FORMATTED [/B][/COLOR]

0	0x0	0x0	0x0	0x0	0x0	0x0	0x30000000	0x0	0x0	0x0
320	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
640	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
960	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1280	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1600	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
1920	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2240	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2560	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
2880	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3200	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3520	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0	0x0
3840	0x0	0x0	0x0	0x0	0x0	0x0	0x0
 Done myfs.checkFormat() in 23205 us.

KurtE · Dec 11, 2020

@all...

Not sure at times if I should keep all of the data like this on MTP thread or LittleFS thread...

But thought I would do a slight follow on to the last post I did yesterday on the MTP thread: https://forum.pjrc.com/threads/43050-MTP-Responder-Contribution?p=262907&viewfull=1#post262907

Where I did three transfers from PC over MTP to the QSPI Flash (the Slower build date...)...

So I was curious what all LittleFS did when the MTP code is looping through receiving data from the PC and then doing 8KB writes to the LittleFS, so I instrumented the QSPI LittleFS calls, where I output an 'R' when the LittleFS_QSPIFlash::readfunction is called. 'P' when Prog is called and 'E' when Erase was called.

I wanted to start from scratch so just before this run, I run sketch that did a Low Level Format of the CHIP...

Code:

MTP_test
SDIO Storage 0 254 sdio 264289280 58577408
SD Storage 1 10 sd1 failed or missing
RAM Storage 0 RAM1 1999872 512
RAM Storage 1 RAM2 4000000 512
Program Storage 0 PROGM 983040 8192
RRRRRRRRQSPI Storage 0 QSPI 16777216 8192
SPIFlash Storage 0 3 nand1 failed or missing
SPIFlash Storage 1 4 nand2 failed or missing
SPIFlash Storage 2 5 nand3 failed or missing
SPIFlash Storage 3 6 nand4 8388608 651264


Setup done
MTPD::SendObject: len:69245
EV-At..+..+..+..+..+..+..+..+..+..+..+..+..+..+..+..+RRRRRRRE.+.+.+.+.+.+.+PRPRPRPRPRPRPRP.+RPRPRPRPRPRPRPRPRE.+.+.+.+.+.+.+PRP.+RPRPRPRPRPRPRPRPRPRPRPRPRPRP.+RE.+.+.+.+.+.+.+RWR 8192 8192 48265 0 145050
PRPRPRPRPRPRPRPRPRPRPRPRP.+RPRPRPRE.+.+.+.+.+.+.+PRPRPRPRPRPRP.+RPRPRPRPRPRPRPRPRPRE.+.+.+.+.+.+.+RRWR 8192 8192 40073 8192 [COLOR="#FF0000"]100505[/COLOR]
PRPRPRPRPRPRPRPRPRPRPRPRPRP.+RPRPRE.+.+.+.+.+.+.+PRPRPRPRPRPRPRPRP.+RPRPRPRPRPRPRPRE.+.+.+.+.+.+.+RWR 8192 8192 31881 16384 [COLOR="#FF0000"]100800[/COLOR]
PRPRPRPRPRPRPRPRPRPRPRPRPRPRP.+RPRE.+.+.+.+.+.+.+PRPRPRPRPRPRPRPRPRPRPRPRPRPRP.+RPRE.+.+.+.+.+.+.+RRRWR 8192 8192 23689 24576 [COLOR="#FF0000"]98679[/COLOR]
PRPRPRPRPRPRPRPRPRPRPRPRP.+RPRPRPRE.+.+.+.+.+.+.+PRPRPRPRPRPRP.+RPRPRPRPRPRPRPRPRPRE.+.+.+.+.+.+.+.+RWR 8192 8192 14985 32768 102678
PRPRPRPRPRPRPRPRPRPRPRPRPRP.+RPRPRE.+.+.+.+.+.+.+PRPRPRPRPRPRPRP.+RPRPRPRPRPRPRPRPRE.+.+.+.+.+.+.+RRWR 8192 8192 6793 40960 97992
PRPRPRPRPRPRPRPRPRPRPRPRPRP.+RPRPRE.+.+.+.+.+.+.+PRPRPRPRPRPRPRPRPRPRPRPRPRPRP.+RPRE.+.+.+.+.+$RWR 8192 8192 0 49152 100044
PRPRPRPRPRPRPRPRPRPRPRPRPRPRPRPREPRPRPRPRPRPRPRPRPRPRPRPRPRPRPRPRERRRRWR 8192 8192 0 57344 95932
PRPRPRPRPRPRPRPRPRPRPRPRPRPRWR 3709 0 65536 5689
PRRPRCL 69245

Again as mentioned in other Thread the '.' in these lines are when my yield event code call back is begin called and my delay between processing of these calls has happened. Currently set to 6ms.
The + is when there is a USB buffer ready to be read.

So it looks like for every 8KB write I do, it does 16 (Prog calls followed by Read calls) then Erase then repeat ... Looks like on average the Erase calls take > 40ms

Again question is does it always have to do an erase of a block? I just did a full low level format!

Note the last number on most of these rows is how many microSeconds that write operation took. So pretty close to 100ms per 8KB write.
Which is consistent with the file of about 330KB which would be about 42 8kb block taking over 4 seconds to copy...

Now compare this to the time to write the same file to the SDIO SD Card

Code:

EV-At..+..+..+..+..+..+..+..+..+..+..+..+..+..+..+..+WR 8192 8192 60553 0 [COLOR="#FF0000"]2086[/COLOR]
.+.+.+.+....+..+..+..+..+..+..+..+..+..+..+..+WR 8192 8192 52361 8192 [COLOR="#FF0000"]371[/COLOR]
.+.+.+.+..+..+..+..+..+..+..+..+..+..+..+..+.+WR 8192 8192 43657 16384 [COLOR="#FF0000"]8991[/COLOR]
.+.+.+.+..+..+..+..+..+..+..+..+..+..+..+WR 8192 8192 35977 24576 373
.+.+.+.+...+..+..+..+..+..+..+..+..+..+..+..+WR 8192 8192 27785 32768 844
.+.+.+.+....+..+..+..+..+..+..+..+..+..+..+..+WR 8192 8192 19593 40960 372
.+.+.+.+..+..+..+..+..+..+..+..+..+..+..+..+WR 8192 8192 11401 49152 381
.+.+.+.+..+..+..+..+..+..+..+..+..+..+..+..+WR 8192 8192 3209 57344 373
.+.+.+.+....+..+..+WR 3709 0 65536 163
CL 69245

I don't have the RPE calls or the like instrumented here, but again if you look at the last number on these lines, there is a real big difference in speed!

defragster · Dec 11, 2020

KurtE said:
@all...
...
So it looks like for every 8KB write I do, it does 16 (Prog calls followed by Read calls) then Erase then repeat ... Looks like on average the Erase calls take > 40ms

Again question is does it always have to do an erase of a block? I just did a full low level format!

...

Many pages/posts back :

11-21-2020, 06:51 PM#378

11-28-2020, 01:47 AM#462

seems to be the early posts for noting and code for skipping the Always Erase/Formatting before write resulting in the net 5X faster write throughput. Paul has implemented the blockIsBlank() code to read the block and skip formatting when it is already 0xff's at the time cost of us's not ms's - perhaps that isn't working as expected as it should when Media start with full LLformat already done for First Use?

... been an ongoing thing I've evolved some - but it is an invasive set of changes that needs to evolve/invade more for total integration if it is to work.

It can work for ALL the media so far : NOR Flash { SPI and QSPI } even RAM {not useful except testing} , one part not yet done is _Program as it only has STATIC funcs()'s so far that can't access Class data, and some of the funcs() added to test have local static memory - so it won't work with multiple mounted drives as it stands without putting some 'static' data into the Class storage. Question is when NAND Flash evolves will it have the same blockIsBlank().

@Paul asked some good questions on those posts weeks back ... more work was done before taking the last week+ off ...

I found LFS core FUNC():Traverse() that reports all LFS_used blocks on the media. This is useful/critical as those blocks don't need format tested, and must not be formatted { Traverse() time is worse with fuller disk, FORMATTED test is worse on larger/unused disk }

As far as I can see on those media it needs one bit per block - in perhaps 3 bitmaps to track in some fashion: char bitmap[blocks/8][3] = { 'FORMATTED', 'LFS_used PRIOR', 'LFS_used NOW' }

That was well and good in testing {with 2 bitmaps} up to 16MB Flash with blocks/8 using 512 bytes and not taking too long to create on the fly, but having 16K blocks on 64MB media takes longer to scan and larger bitmaps for tracking and needs Smarter complete tracking to avoid extra scans that take so long. And ideally a stored 'static' to the Media copy of at least one of those bitmaps in the 'current state'. { lying to LFS about media size and reserving some end blocks to maintain this would be needed }, and Media would need a Full Low Level Format when the Disk is created to trust the Media - or at least a Full blockIsBlank() scan of the media. Then tracking of the two types for Format [this is the one new needed invasive addition]: {LFS format before use making it Dirty/USED, PJRC code FORMAT leaving it Formatted and Write Ready}

So with this properly implemented an LFS_Format_Write proceeds after a quick bit test. And with background Formatting all media blocks would always be pre-formatted and ready for use - all LFS would ever see it Read and Write with no format waits.

The question is if this is worth more effort for a 5X speedup? And will NAND offer the same interface and ability for the tracking with acceptably sized bitmaps? In fact some of those 'RPRPRP' strings noted in post #546 may not work if the 'P'rogram relies on rewriting blocks that NAND cannot support, but they may work if the current LFS_CODE has been designed with this in mind - being NAND aware was the subject of some github notes @mjs513 and @defragster found.

The other question is if background non-blocking format to Media can be done during active read/write? Given a known list of free dirty blocks to format for next use - relying on the bitmaps[3].

So that has been up in the air and now in the background assuming it won't get done before TD 1.54 is released in current initial SLOW state.

mjs513 · Dec 11, 2020

@KurtE
Know this may sound funny but the default program size is 256bytes for all the flash chips in LittleFS with an erasesize of 4096 bytes respectively yet we are interfacing MTP at 8192bytes since that is essentially what SD cards with FAT is expecting. Wonder if changing some of these settings can be changed.

Is it possible to change the numbers of bytes transferred by MTP to a smaller size like 4096?

KurtE · Dec 11, 2020

mjs513 said:
@KurtE
Know this may sound funny but the default program size is 256bytes for all the flash chips in LittleFS with an erasesize of 4096 bytes respectively yet we are interfacing MTP at 8192bytes since that is essentially what SD cards with FAT is expecting. Wonder if changing some of these settings can be changed.

Is it possible to change the numbers of bytes transferred by MTP to a smaller size like 4096?

I think I asked something about this in some of my earlier ramblings. Back in https://forum.pjrc.com/threads/58033-LittleFS-port-to-Teensy-SPIFlash?p=262644&viewfull=1#post262644

Wondering if the FS interface should have some method that one can ask it what is the preferred write size...

Alternatively one could maybe build this into MTP, code, like potentially in something like:

Code:

storage.addFilesystem(progmfs[ii], lfs_progm_str[ii]);

Code:

storage.addFilesystem(progmfs[ii], lfs_progm_str[ii], 4096);

LittleFS port to Teensy/SPIFlash

Senior Member+

Senior Member+

Well-known member

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Well-known member

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+

Senior Member+