Forum Rule: Always post complete source code & details to reproduce any issue!
Page 2 of 2 FirstFirst 1 2
Results 26 to 44 of 44

Thread: memFile

  1. #26
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    24,769
    Quote Originally Posted by Frank B View Post
    please make a constructive suggestion.
    You should add a length variable to your MemFile class. For the non-standard constructor (create with an array, not part of the FS class) set the length to the array size. For the standard constructor (inherited from FS.h) set it to zero, so the file starts with no data.

    In most of the functions, use the length instead of "sz". In write() update the length, but of course still use "sz" so you don't allow writing beyond the buffer.

    File class comes with a well documented API. A more faithful implementation of the API requires 4 more bytes to store the data length, and a small amount of extra code. Not difficult. Not needed for the usage cases you have envisioned, but when you publish a library using a well known API, odds are strong other people will use it according that API.

  2. #27
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    9,054
    I could add it, but you just need to wait for the first overflow, if the lib does not know the size of i.e. an array in ram that gets written.
    Not sure if this is better.. i could remove support for "write", too. That would be save. But would hinder a number of use cases.
    Or i could just set "readonly" when no size is given.

    I'll think about it.

  3. #28
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,854
    I see I missed a bunch more following posts - but in p#8 I found :: dataFile.position()
    to show the current location in the file? For a 'new' file that would be 'length' during writes.

    Quote Originally Posted by PaulStoffregen View Post
    You did. But your question uses the wrong way to print CrashReport (with any file or stream)

    Code:
        CrashReport( dataFile );

    The correct usage is

    Code:
        dataFile.print(CrashReport);
    Just back - yes - I see misuse:

    It works using that code: {opps - that code even in open Sketch window beside this )
    Code:
    Filesize = 512 bytes
    Current File
    No Crash Data To Report
      Hopefully all is well, but certain types of crashes can't be reported:
    	stuck in an infinite loop (technically, hardware still running properly)
    	remaining in a low power sleep mode
    	access to certain peripherals without their clock enabled (eg, FlexIO)
    	change of CPU or bus clock speed without use of glitchless mux
    with:
    Code:
    #include "MEMFILE.h"
    char testData[512] = {};
    MemFS  myfs;
    
    void setup()
    {
      // open the file.
      File dataFile = myfs.open((char*) &testData, sizeof(testData), FILE_WRITE);
      Serial.begin(1492);
      if (dataFile) {
        dataFile.print(CrashReport);
    
    ...

  4. #29
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    9,054
    Addition: There more I think about it, the more "not compatible to the documentation" is it anyway - just because there is no filename

    I'll ignore that.

    Edit: Too late here, on this side of the earth, now.
    Edit2: doesn't it work to init the aray with zeroes and check for \0 in the loop - as you would do it in other cases, too? (just for your special issue with CrashReport)

  5. #30
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    9,054
    Works perfectly if the string is handled correctly:
    Code:
    #include <MEMFILE.h>
    
    char testData[1000];
    
    MemFS  myfs;
    
    void setup()
    {
      while (!Serial);
      memset(testData, 0, sizeof(testData));
      
      File dataFile = myfs.open((char*) &testData, sizeof(testData), FILE_WRITE_BEGIN);
      //File dataFile = myfs.open("name", FILE_WRITE_BEGIN);
    
      if (dataFile) {
        Serial.printf("file size before printing: %llu\n", dataFile.size());
        dataFile.print(CrashReport);
        Serial.printf("file size after printing: %llu\n", dataFile.size());
        dataFile.seek(0);
        char c;
        while (dataFile.available() &&( (c = dataFile.read()) != 0)) {          
          Serial.write(c);
        }
        dataFile.close();
      }
    }
    
    void loop()
    {
    }
    Good night.

    Edit: You can also use strlen, then - as it is the correct way.

  6. #31
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,854
    This file is like a 'PreAllocated' file - so the size/'sz' IS the size/.size()?

    It did give me pause - that's why I found : dataFile.position()

    In FS.h these are supported: { T:\Arduino_1.8.15td155\hardware\teensy\avr\cores\t eensy4\FS.h }
    Code:
        Serial.print("dataFile.usedSize = ");
        Serial.println(dataFile.usedSize);
        Serial.print("dataFile.totalSize = ");
        Serial.println(dataFile.totalSize);
    Code:
    MemFileStream:25: error: 'class File' has no member named 'usedSize'
         Serial.println(dataFile.usedSize);
                                 ^
    MemFileStream:27: error: 'class File' has no member named 'totalSize'
         Serial.println(dataFile.totalSize);
    For #include "MEMFILE.h":
    > Perhaps track dataFile.usedSize as the biggest value that ofs/Offset ever gets during write?

    Of course that initialize would be ZERO - perhaps when opened ReadOnly dataFile.usedSize would be set to SIZE rather than 0?

  7. #32
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,854
    @Paul: Is T:\Arduino_1.8.15td155\hardware\teensy\avr\cores\t eensy4\FS.h the 'spec' for what should be supported for a "File" of FS class?

    It could also incorporate: virtual bool truncate(uint64_t size=0) { return (f) ? f->truncate(size) : false; }

    To set the desired .size() value for 'ofs', while .totalSize() would maintain and report that as allocated size/'sz'

  8. #33
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    9,054
    its usedSize() and totalSize() (-> functions ) - same as in littleFS. I copied that from littlFS - if thats wrong, it is wrong in lttlefFS, too?

  9. #34
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,854
    Quote Originally Posted by Frank B View Post
    its usedSize() and totalSize() (-> functions ) - same as in littleFS. I copied that from littlFS - if thats wrong, it is wrong in lttlefFS, too?
    opps - I forgot parens() ... maybe I need to update ...

  10. #35
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,854
    Apparently I need more coffee ... code again was ALL wrong ...

    Indeed this works:
    Code:
        Serial.print("myfs.usedSize = ");
        Serial.println(myfs.usedSize());
        Serial.print("myfs.totalSize = ");
        Serial.println(myfs.totalSize());
    But the point was usedSize could be - biggest ever 'ofs' starting at zero - unless ReadOnly?

    Also need to add .truncate() support to reset .size() - and limit 'ofs' when bigger than truncate value?

    <edit>
    And I suppose .size() == .usedSize() where only .totalSize() returns full size of Pre-Allocated file?

  11. #36
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    9,054
    @Tim: Just take a look at the sourcecode - i'm happy about bug reports.

    @Paul I tend to leave it as it is (now). For the following reasons:

    - No known size means, there will be buffer overflows without any chance to prevent them
    - There is no filename, and thus it is "incompatible" anyway, and will always be.
    - Now, it just does not work with the default : open("name", FILE_WRITE_BEGIN) - every user notices this early, when he first runs the code. No surprises (and overflows), later.

    So, (of course) it's your decision if you want to add it to TD, or not. It will be available on Github anyway. Or, feel free to copy & modify it, and maybe you find a better solution.

    Edit: the *all* capital letters where not intended.. just had "caps lock" enabled Can rename it, if wanted.
    Last edited by Frank B; 08-29-2021 at 09:56 PM.

  12. #37
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    24,769
    Quote Originally Posted by defragster View Post
    @Paul: Is T:\Arduino_1.8.15td155\hardware\teensy\avr\cores\t eensy4\FS.h the 'spec' for what should be supported for a "File" of FS class?
    Yes, more or less. But it's far from the sort of formal definition you'd expect from standards organizations like IEEE, IETF, etc. For many of the functions, the required functionality comes from Arduino's SD library documentation.

    Also to be honest, FS.h is going to become a sort of moving target with version 1.56 and maybe 1.57. More APIs will be added in the near future, requiring updates to SD / SdFat, LittleFS, and other other libraries like Frank's new MEMFILE.

  13. #38
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    24,769
    Quote Originally Posted by Frank B View Post
    So, (of course) it's your decision if you want to add it to TD, or not.
    Let's wait until either FS.h is unlikely to get more changes, or until a lot of people are using MEMFILE.

  14. #39
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,854
    @Frank - it seems to be a good and useful piece of code.

    Name could be supported? Change this to .begin() ? :: myfs.open((char*) &testData, sizeof(testData), FILE_WRITE);
    >> myfs.begin( (char*) &testData, sizeof(testData));


    Then with data space for 'name' it could work with : myfs.open("name", FILE_WRITE_BEGIN)
    >> it would reset some things as a 'new file'
    >> if name doesn't strcmp() on .open() - then the old file is wiped, .close() would have to be non-destructive.

    @Paul : yes, FS set to change ... but that sounds like the functions set 'today' that need "emulation"/support?

    Don't have any history or feelings about FS.h ... looked at it more today than prior AFAIK.

  15. #40
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    24,769
    Quote Originally Posted by defragster View Post
    but that sounds like the functions set 'today' that need "emulation"/support?
    With any shared API, if a library implements only a subset of the API, it will be incompatible with programs using those parts of the API it didn't implement. It's not just File. We've seen it before with Stream peek() or other lesser used functions, where most programs work but then someone uses it with code using those other functions and it fails (sometimes without a compiler error or warning - just doesn't work). I believe there's an open issue where Teensy's String class isn't implementing functions Arduino added to theirs some time ago. It's on my list to look at before 1.55-beta2...

    In this particular case, I'm pretty sure those issues will come up when (if) many people use the library, and also as we gradually add more ways to use files. For example, the audio library lacks any File-based recording. If we someday add raw format recording (on my lower priority to-do list), the number of samples recorded is known only as the file length, since raw format doesn't have a header like WAV.

    My intention isn't to keep arguing on this point.

    But I do want to again mention pretty major changes to FS.h are planned, which will let people use files and filesystems in some very convenient ways, but those API changes will impose a lot of work onto libraries which inherit the File & FS classes.

  16. #41
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,409
    Quote Originally Posted by PaulStoffregen View Post
    But I do want to again mention pretty major changes to FS.h are planned, which will let people use files and filesystems in some very convenient ways, but those API changes will impose a lot of work onto libraries which inherit the File & FS classes.
    Paul,

    as I mentioned in another thread (https://forum.pjrc.com/threads/68061...ot-overwritten) . If an implementation does not provide an override function for every virtual member in FS (either existing or new ones we add) and the FS is using the usage counting wrapper file like SD library does. It will build just fine and it will fault (typically trying to call through a pointer with value of 1...

    Sorry about name of other thread... Also funny thing was I figured out what was happening right at end of typing it in...

    That is current code for default implementations for each member is something like:
    Code:
    	virtual bool getAccessDateTime(uint16_t* pdate, uint16_t* ptime) {
      		return (f) ? f->getAccessDateTime(pdate, ptime) : false;
      	}
    So if you call the through the wrapped File f is not null so it goes through the vtable to the actual object...
    Which if it uses this method we assume f is NULL and returns false...

    Problem is that the code currently unions the f with the usage count:
    Code:
    union {
    		// instances of base File class use this pointer
    		File *f;
    		// instances of derived classes (which actually access media)
    		// use this reference count which is managed by the base class
    		unsigned int refcount;
    	};
    Problem is when the file is wrapped it increments the reference count so now 1. so when it calls through it says is it NULL no... So then tries to call using PTR of 1...

    In the thread I showed two possible ways to fix. The one I am doing locally is to remove the union, which then required me to update the constructor to initialize the refCount...

    Question is, should I do a PR of this for the next beta? Note: I would need to do it again as my local copy is actually from a different branch where we are adding Dates and Times...

  17. #42
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    9,054
    Quote Originally Posted by PaulStoffregen View Post
    I believe there's an open issue where Teensy's String class isn't implementing functions Arduino added to theirs some time ago. It's on my list to look at before 1.55-beta2...
    My PR regarding this was confirmed to work by both involved guys.
    Edit: If you plan to work on the string class, make some checks if it is thread safe. For 1170.

    Quote Originally Posted by PaulStoffregen View Post
    For example, the audio library lacks any File-based recording.
    Yes I mentioned that in the waveplayer thread.
    Had planned to write it, but I think I have lost the desire now.

    Quote Originally Posted by PaulStoffregen View Post
    If we someday add raw format recording (on my lower priority to-do list), the number of samples recorded is known only as the file length, since raw format doesn't have a header like WAV.
    If you record a file you never know the length. If you record wav or raw.
    It can be both done by the same code - for wav you just need to add a header (and please, if you do it, consider more than two channels)
    As in every other case: When you write to an array, you need to know the size of the array. No way to do that without this information if you don't want to risk buffer overflows.

    If you want to add recording, you have to write parts of the code twice - (without something like the MemFS). Fun


    Quote Originally Posted by PaulStoffregen View Post
    My intention isn't to keep arguing on this point.
    Yup, same for me ;-)
    Last edited by Frank B; 08-30-2021 at 09:15 AM.

  18. #43
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    14,854
    End of day here Frank - I didn't touch the code as today wasn't a good day for that and too many other new issues beyond the couple I saw and perhaps suggested working paths for.

    Makes sense to be able to mark 'end of file' size known to be written when writable - and full size when read only. And perhaps reserving the number of bytes needed for 'name' might be handy.

    Then having truncate if the memFile will be re-used - and not closed()'d. Or a pair of Ping-Pong buffers where one written while the other read then marked empty. Perhaps an _isr() usage case to store data until loop can empty while the 'other' buffer is getting filled - and then knowing the length of data written independent of the totalSize of the space available. There may be better ways to handle that - but given it is such fast memcpy() code it would be more _isr() friendly - like the Audio reads that may have led to this?

  19. #44
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    9,054
    Tim.. dunno.. I think doing any more than now is against the intention of memfile.
    Any more sophisticated code would mean taking away the basis of the whole thing. Real file systems, like littleFS can do all that better. But it should not be a full-fledged file system. On purpose.
    Its just for mem accesses. I can't imagine were someone wants to write data to it when the size of both is unknown*. That a jobs for a real FS.
    Perhaps that's a point I wasn't clear enough about...

    Done here.
    Last edited by Frank B; 08-30-2021 at 11:39 AM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •