memFile

Status
Not open for further replies.

Frank B

Senior Member
I'd like to propose a new minimal "Filesystem" which provides a "file" that is internally just a memorylocation. A Pointer. A read() or write() would just be a memcpy.
This way, everything that uses a file can be used for raw data in flash, ram, psram.
For example, we wou'dnt need distinct sourcecodes anymore for some things i.e. like "playMem" from the audio library.
 
We sort-of have this now, with LittleFS_RAM. Only minor issue is it incurs all the LittleFS overhead meant for non-volatile storage, so performance isn't as good as it could be, but still a lot better than using it on flash.

But yeah, a new library could be written to very directly implement file access on top of ordinary memory. Are you going to do it? Or is the proposal for me or someone else to do all the work?

If the access is directly to memory, one design issue would be how to handle allocation. If it's static, then a special API is probably needed to "create" the file and give it the memory region, and then disallow growing the file size beyond the amount of memory.
 
The idea is that it's just a pointer. Not a file system.

Paul, are you in a bad mood somehow? :)

No, of course I would do "all the work". I just wanted to hear if others besides me would find this somehow useful.
Anything that takes a "file" now can then work directly with memory. Without modifications.
 
Here is a first working version: https://github.com/FrankBoesing/MemFile

How to use:
Code:
/*
  MEMFile file dump
  This example code is in the public domain.
*/

#include <MEMFILE.h>

char testData[] = {'1', '2', '3', '4', '5', '6', '7', '8'};

MemFS  myfs;

void setup()
{
  // open the file.
  File dataFile = myfs.open((char*) &testData, sizeof(testData), FILE_WRITE);

  if (dataFile) {
    while (dataFile.available()) {
      Serial.write(dataFile.read());
    }
    Serial.println();
    dataFile.seek(1);
    dataFile.write('!');
    dataFile.seek(0);
    while (dataFile.available()) {
      Serial.write(dataFile.read());
    }
    dataFile.close();
  }
}

void loop()
{
}
(almost as any file - but you need the size)

There are no filenames, no directory, no paths. Just the adress and the size. The size is needed for compatibility, so that available() can work.

Attention: The size parameter is mandatory.
 
Last edited:
@Paul: You can copy it to core or add it as a library, or neither. Whatever you like. I will add an example, probably.

Useful for audio, graphics, etc...
 
Last edited:
Nice work Frank. - Sample needed this to see output :) :: Serial.begin(1492);
Code:
/*
  MEMFile file dump
  This example code is in the public domain.
*/

#include "MEMFILE.h"
char testData[24] = {'1', '2', '3', '4', '5', '6', '7', '8'};
MemFS  myfs;

void setup()
{
  // open the file.
  File dataFile = myfs.open((char*) &testData, sizeof(testData), FILE_WRITE);
  Serial.begin(1492);
  if (dataFile) {
    Serial.printf( "Filesize = %lu bytes\n", [B]dataFile.size()[/B] );
    dataFile.seek(8);
    dataFile.write("[B]ABCDEFGHIJKLMNOP[/B][U]QRSTUVWXYZ[/U]");
    dataFile.seek(0);
    Serial.println("Current File");
    while (dataFile.available()) {
      Serial.write(dataFile.read());
    }
    dataFile.seek(1);
    dataFile.write('!');
    dataFile.seek( [B]dataFile.position()[/B]+5);
    dataFile.write('!');
    dataFile.seek(0);
    Serial.println("\nModified File");
    while (dataFile.available()) {
      Serial.write(dataFile.read());
    }
    dataFile.close();
  }
}

void loop()
{
}

Output:
Code:
Filesize = 24 bytes
Current File
12345678ABCDEFGHIJKLMNOP
Modified File
1!34567!ABCDEFGHIJKLMNOP
 
Works with PROGMEM, too (of course no writes, then - that's intended - it's just a pointer)
Have not tested PSRAM.
 
Works with PROGMEM, too (of course no writes, then - that's intended - it's just a pointer)
Have not tested PSRAM.

Yes - static data from ProgMem as a file seems like a nice feature.

Can you make it STREAM ready - would be handy to store startup data before Serial is online:
Code:
  if (dataFile) {
    CrashReport( dataFile );

Code:
MemFileStream:16: error: no match for call to '(CrashReportClass) (File&)'
     CrashReport( dataFile );
 
I asked first :)

Paul says there is a Stream Helper Class in the install?

Not sure where that is just now ???
T:\Arduino_1.8.15td155\hardware\teensy\avr\cores\teensy4\Stream.cpp
 
One issue I see is the library doesn't track how much data has been written to the file. The file's size seems to always be the buffer length, even before any data has been written.

For example:

Code:
#include <MEMFILE.h>

char testData[1000];

MemFS  myfs;

void setup()
{
  while (!Serial);
  memset(testData, '*', sizeof(testData));
  
  //File dataFile = myfs.open((char*) &testData, sizeof(testData), FILE_WRITE_BEGIN);
  File dataFile = myfs.open("name", FILE_WRITE_BEGIN);

  if (dataFile) {
    Serial.printf("file size before printing: %llu\n", dataFile.size());
    dataFile.print(CrashReport);
    Serial.printf("file size after printing: %llu\n", dataFile.size());
    dataFile.seek(0);
    while (dataFile.available()) {
      Serial.write(dataFile.read());
    }
    dataFile.close();
  }
}

void loop()
{
}

This should print the CrashReport data. Indeed it does. But then is spews a bunch of extra garbage.

screenshot.png

The available() function is supposed to be aware of the actual amount of data the file holds. Returning the maximum possible size rather than the actual data which has been stored in the file breaks this sort of usage.
 
Yes. Its memory and has no real "length". It's intended to be used for existing memory data (memory area) which by definition has a fixed length - if it was written or not. RAM, program flash, etc.
If you wanted, you could use it to dump a set of registers (where the hardware allows that).


You can not "create" that file, too. It's already there. per definition.

The main use case is, for sure , access to const "PROGMEM" data. Can be a audio file, some graphics, etc.
 
Its memory and has no real "length".

Trouble is, the model of File does have a notion of data length. If you don't implement length, especially for this case

Code:
  File dataFile = myfs.open("anyname", FILE_WRITE_BEGIN);

then your lack of keeping track of the actual data length breaks the usage model. Or you only support certain usage and your library which claims to give a File type will be confusing to anyone who expects to be able to generally use it as File.
 
Explain how it could use a variable length, please..

will be confusing to anyone who expects to be able to generally use it as File.
No. The only way to open it, is to use a size. A user who don't understand this can't use an array, too.

Of course, you can continue, and write sepcial code to access memory.
But the whole play memory object (aduio lib) wouldn't be needed with this kind of file. And many more duplicate code, too.
 
Explain how it could use a variable length, please..

Before I write more explanation, could you please run this?

Code:
#include <MEMFILE.h>

MemFS  myfs;

void setup()
{
  while (!Serial);
  File dataFile = myfs.open("name", FILE_WRITE_BEGIN);
  if (dataFile) {
    dataFile.print(CrashReport);
    dataFile.seek(0);
    while (dataFile.available()) {
      Serial.write(dataFile.read());
    }
    dataFile.close();
  }
}

void loop()
{
}
 
Your example from above uses available() for a string , which wrong in so many way... do you use available() for strings in other cases?
Ignoring terminating zero?
 
Ahk, ok... you don't use the size parameter... (undocumented ;) should have known this. Ok, extend the File class please, so that this version is not needed.
 
Your example from above uses available() for a string

No Frank, it's a "File", not a string.

Code:
  [B][COLOR="#D22222"]File[/COLOR][/B] dataFile = myfs.open("name", FILE_WRITE_BEGIN);


which wrong in so many way

When we talk of what is correct versus wrong, the File class comes with pretty well expected behavior for these functions.
 
Ok, i just modifed it to
Code:
  File open(const char *ptr, uint8_t mode = FILE_READ) {    
    return File();
  }
So you just can't access it. That's all I can do, without knowing the size - everything else needs a patch to FS.
 
This is fixed with the latest version. It does not work anymore. I knew that was a mistake also to allow "no size" (hackish...) You found it right away.


If you want this problem to be fixed - please make a constructive suggestion.
If you don't want all this - just don't use it. The library is still available for anyone else who finds it useful and don't wants to write extra code for PROGMEM etc.
 
I'll add this case to the readme i'll write and to my example post above, so that it is documented.

2021-08-29 22_27_05-Window.png
part #1. Done.
 
Status
Not open for further replies.
Back
Top