Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 25 of 25

Thread: exFAT SD card append-write fails after 256 kb when writing to 2 files

  1. #1
    Junior Member
    Join Date
    Jan 2022
    Location
    Netherlands
    Posts
    12

    exFAT SD card append-write fails after 256 kb when writing to 2 files

    Hi,

    I have a Teensy 4.1 that writes to exFAT card. Using the Jan 2022 Teensyduino which I believe the most recent. It's for a datalogger. I want two log files. I write to the files by appending to them in loop(), once every 100 ms, one file after the other.
    What's happening is that after 256 kB has been appended with 8 kB blocks without any issues, the file(s) on the SD card no longer wants to be written to.
    But if I only append to one file, then all works fine.

    When I write to both files successively, then after crossing the 256kB size limit and only when writing to two files instead of just one, then the file no longer grows in size. Write attempts simply fail.

    The strange thing is that the write-failed file on that SD card then remains 'broken'. Also when I reboot the Teensy, or power cycle it. Only reformatting the SD card (I use my Windows 11 PC for that) fixes it. Or starting over with a new filename. But then that new one also only does the first 256 kB, and then nothing can be added.

    Looks to me like a bug in the exFAT SD card libraries.

    I'm using (SdioConfig(FIFO_SDIO)). Could that be it?

    Archived example project attached. Only 55 lines. Reduced to the bare minimum size just to isolate and highlight the issue.
    To replicate that it does work ok with just one file, comment out line 53 (write_to_log_file (filename_B);) , erase the card (or change the file names), and then the A file does grow beyond 256 k.

    Anyone any ideas? exFAT directory tables get corrupted?

    For info, this is what I use as SD card:
    SD card Manufacturer ID: 0x1b OEM ID:0x534d (SM)
    Product: ED4QT
    Version: 3.0
    Serial number: 0x0e534a33 = 240339507
    Manufacturing date: 2/2011
    SD card formatted for exFATSD: ok
    Attached Files Attached Files

  2. #2
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    16,240
    Was any other SD media tried? What size is the Media?
    Was a chkdsk/scandisk done on the Media before PC reformatting?

    The code in ZIP is short so here it is:
    Code:
    #include <SdFat.h>
    #include <SdFatConfig.h>
    
    #define WRITE_TO_SD_CHUNK_SIZE 8192
    
    SdFs sd;
    uint8_t st[WRITE_TO_SD_CHUNK_SIZE];
    
    void setup() {
    // USB serial port
        Serial.begin(115200);
        int retry = 0;    // Serial is the USB port, but if no PC connected/active, then proceed anyway
        while ((!Serial) && (retry++ <100))
            delay(10);
    
        Serial.println("test exFAT logger");
     
        if (sd.begin(SdioConfig(FIFO_SDIO))) 
            Serial.println ("sd.begin() done");
    
    // init char buffer with arbitrary readable data
        for (int i=0; i<WRITE_TO_SD_CHUNK_SIZE; i++)
            st[i] = ' ' + i & 0x3f;
    
    }
    
    void write_to_log_file (char *log_filename)
    {
        FsFile F;
        int written;
        fspos_t pos;
                        
        F = sd.open(log_filename, O_WRONLY | O_CREAT | O_APPEND);
        if (F)
        {
            written = F.write (st, WRITE_TO_SD_CHUNK_SIZE);
            F.fgetpos(&pos);
            Serial.printf ("file %s, ", log_filename);
            Serial.printf ("pos=%d, ", pos);
            Serial.printf ("adding %d bytes, written %d\n", WRITE_TO_SD_CHUNK_SIZE, written);
        }
        
        F.close();  
    }
    
    void loop ()
    {
        char filename_A[40] = "FILE_A.txt";
        char filename_B[40] = "FILE_B.txt";
    
        write_to_log_file (filename_A);
        write_to_log_file (filename_B); // To replicate that it does work ok with just one file, comment out line 53 (write_to_log_file (filename_B);)
    
        delay(100);
    }
    Last edited by defragster; 04-25-2022 at 05:24 PM. Reason: is not id

  3. #3
    Senior Member
    Join Date
    Jul 2014
    Posts
    3,453
    I tried the example
    it fails here with
    Code:
    FFFFFFFF EB3A9
    DBG_FAIL: ExFatPartition.cpp.195
    DBG_FAIL: ExFatFileWrite.cpp.108
    DBG_FAIL: ExFatFileWrite.cpp.663
    file FILE_A.txt, pos=65536, adding 8192 bytes, written 0
    The fist lines are the parameters from
    Code:
    bool ExFatPartition::fatPut(uint32_t cluster, uint32_t value)
    so it called with cluster=0xFFFFFFFF

    the SD card was 32 GB formatted as exFAT wit 64 kB cluster size.

    My hunch is a Bug in exFAT, may you could open an issue with Bill Greiman, using this example program

  4. #4
    Senior Member
    Join Date
    Jul 2014
    Posts
    3,453
    I remover the two files and restarted program
    Code:
    test exFAT logger
    sd.begin() done
    file FILE_A.txt, pos=8192, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=8192, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=16384, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=16384, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=24576, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=24576, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=32768, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=32768, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=40960, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=40960, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=49152, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=49152, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=57344, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=57344, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=65536, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=65536, adding 8192 bytes, written 8192
    DBG_FAIL: ExFatFileWrite.cpp.75
    DBG_FAIL: ExFatFileWrite.cpp.663
    file FILE_A.txt, pos=65536, adding 8192 bytes, written 0
    DBG_FAIL: ExFatFileWrite.cpp.75
    DBG_FAIL: ExFatFileWrite.cpp.663
    file FILE_B.txt, pos=65536, adding 8192 bytes, written 0
    interesting debug gives now different write error
    Previous observation was with existing files, so difference may be possible

    actual error happens in
    Code:
    bool ExFatFile::addCluster() {
      uint32_t find = m_vol->bitmapFind(m_curCluster ?  m_curCluster + 1 : 0, 1);
      if (find < 2) {
        DBG_FAIL_MACRO;
        goto fail;
      }

  5. #5
    Junior Member
    Join Date
    Jan 2022
    Location
    Netherlands
    Posts
    12
    Thank you for prompt responses and confirmation that it's not specific to just my SD card, confirmation that it it replicates and that there's thus something of a bug.
    I can confirm that if I change the 'Allocation unit size' in the Windows 11 Quick Format that I used, that the file position where the error occurs scales up linearly. For example, I initially had 128k as 'Allocation unit size' and the problem was triggered after 256k was written, but when I changed to 1 MB, the problem occurs after 2 MB has been appended to each file.

    If I had the debugger working, and the exFAT spec at hand (no, I don't have it...) then I would now set breakpoints and single step into the library sources. But I don't have that luxury. So keeping fingers crossed now that others will soon manage to isolate and fix.

  6. #6
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    16,240
    P#1 noted: I'm using (SdioConfig(FIFO_SDIO)). Could that be it?

    Are there options that work on that media? That might indicate generic or specific issue.

    Perhaps rewrite using only SD.h style calls to SD (from one of /examples) ... which still maps to SdFat. If that works: it is a work around for now - and indicates a problematic code path versus total fail on exFat formatted media.

  7. #7
    Senior Member
    Join Date
    Jul 2014
    Posts
    3,453
    @sicco
    You can easily switch on the internal debug information of SdFat by uncommenting
    setting
    Code:
    #define USE_DBG_MACROS 2
    in
    Code:
    ...\hardware\teensy\avr\libraries\SdFat\src\common\DebugMacros.h
    It gives you a traceback of failing

    This is not as good as a debugger, but better than nothing

  8. #8
    Junior Member
    Join Date
    Jan 2022
    Location
    Netherlands
    Posts
    12
    Quote Originally Posted by WMXZ View Post
    I tried the example
    it fails here with
    Code:
    FFFFFFFF EB3A9
    DBG_FAIL: ExFatPartition.cpp.195
    DBG_FAIL: ExFatFileWrite.cpp.108
    DBG_FAIL: ExFatFileWrite.cpp.663
    file FILE_A.txt, pos=65536, adding 8192 bytes, written 0
    The fist lines are the parameters from
    Code:
    bool ExFatPartition::fatPut(uint32_t cluster, uint32_t value)
    so it called with cluster=0xFFFFFFFF

    the SD card was 32 GB formatted as exFAT wit 64 kB cluster size.

    My hunch is a Bug in exFAT, may you could open an issue with Bill Greiman, using this example program
    OK. Stupid question maybe, but who is this Mr Bill Greiman and how do I open an issue with that person?

  9. #9
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    16,240
    Mr Greiman is the author of SdFat - forum search will find his posts/threads and link to his github presence where an issue could be posted.

    The ...\hardware\teensy\avr\libraries\SdFat\library.pr operties file provides this info:
    ...
    url=https://github.com/greiman/SdFat
    repository=https://github.com/greiman/SdFat.git
    ...

  10. #10
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    26,569
    Before you report the issue to Bill, please download and test with his latest version from github.

    https://github.com/greiman/SdFat

    The copy we ship with Teensyduino is slightly older and has several minor changes. Best to double check the bug really does happen with his latest version, just in case the problem is something we did (messed up) in modifications for Teensy.

    This is the place to report the issue.

    https://github.com/greiman/SdFat/issues/new

    Also best to include complete code to reproduce the problem.

  11. #11
    Senior Member
    Join Date
    Jul 2014
    Posts
    3,453
    Just downloaded actual version and without further editing of SdConfig.h and activating build-in debug messages, I get
    Code:
    test exFAT logger
    sd.begin() done
    file FILE_A.txt, pos=8192, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=8192, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=16384, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=16384, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=24576, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=24576, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=32768, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=32768, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=40960, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=40960, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=49152, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=49152, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=57344, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=57344, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=65536, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=65536, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=65536, adding 8192 bytes, written 0
    file FILE_B.txt, pos=65536, adding 8192 bytes, written 0
    file FILE_A.txt, pos=65536, adding 8192 bytes, written 0
    file FILE_B.txt, pos=65536, adding 8192 bytes, written 0
    Edit: I opened an issue on Billīs github
    Last edited by WMXZ; 04-27-2022 at 06:16 AM.

  12. #12
    Senior Member
    Join Date
    Nov 2012
    Posts
    335
    I posted a reply on github.

    I tested with SdFat 2.1.2 and 2.1.4-beta.3
    I uses 32GB FAT, 64GB exFAT, 512GB exFAT on a Teensy 4.1

    I used Teensyduino 1.56 with Arduino 1.8.19

    Seems to work on all. I used Samsung SD cards.

    With 32GB FAT
    test exFAT logger
    2.1.4-beta.3
    sd.begin() done
    file FILE_A.txt, pos=8192, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=8192, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=16384, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=16384, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=24576, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=24576, adding 8192 bytes, written 8192
    ....
    file FILE_B.txt, pos=7004160, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=7012352, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=7012352, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=7020544, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=7020544, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=7028736, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=7028736, adding 8192 bytes, written 8192

    With 64GB exFAT
    test exFAT logger
    2.1.4-beta.3
    sd.begin() done
    file FILE_A.txt, pos=8192, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=8192, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=16384, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=16384, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=24576, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=24576, adding 8192 bytes, written 8192
    ...
    file FILE_A.txt, pos=6774784, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=6774784, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=6782976, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=6782976, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=6791168, adding 8192 bytes, written 8192

    With 512GB exFAT
    est exFAT logger
    2.1.4-beta.3
    sd.begin() done
    file FILE_A.txt, pos=8192, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=8192, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=16384, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=16384, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=24576, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=24576, adding 8192 bytes, written 8192
    ...
    file FILE_A.txt, pos=7315456, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=7315456, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=7323648, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=7323648, adding 8192 bytes, written 8192
    file FILE_A.txt, pos=7331840, adding 8192 bytes, written 8192
    file FILE_B.txt, pos=7331840, adding 8192 bytes, written 8192

  13. #13
    Senior Member
    Join Date
    Jul 2014
    Posts
    3,453
    Thanks Bill,
    I tried with new disk and it worked also here.
    sane configuration (SdFat 2.1.2, Teensyduino 1.56 with Arduino 1.8.19)
    must have screwed up my earlier testing.
    Last edited by WMXZ; 04-27-2022 at 03:59 PM.

  14. #14
    Junior Member
    Join Date
    Jan 2022
    Location
    Netherlands
    Posts
    12
    Quote Originally Posted by WMXZ View Post
    Thanks Bill,
    I tried with new disk and it worked also here.
    sane configuration (SdFat 2.1.2, Teensyduino 1.56 with Arduino 1.8.19)
    must have screwed up my earlier testing.
    I updated my SdFat library from 2.1.0 to 2.1.2 and that seems to have fixed the bug.
    Thank you all!

  15. #15
    Senior Member BriComp's Avatar
    Join Date
    Apr 2014
    Location
    Cheltenham, UK
    Posts
    812
    Quote Originally Posted by PaulStoffregen View Post
    Before you report the issue to Bill, please download and test with his latest version from github.

    https://github.com/greiman/SdFat

    The copy we ship with Teensyduino is slightly older and has several minor changes. Best to double check the bug really does happen with his latest version, just in case the problem is something we did (messed up) in modifications for Teensy.

    This is the place to report the issue.

    https://github.com/greiman/SdFat/issues/new

    Also best to include complete code to reproduce the problem.
    Is this version modified for Teensy?

  16. #16
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    16,240
    Quote Originally Posted by BriComp View Post
    Is this version modified for Teensy?
    Expect not until Paul integrates it for next Beta ... the SD.h edits. AFAIK they are still on PJRC's task list.

  17. #17
    Senior Member BriComp's Avatar
    Join Date
    Apr 2014
    Location
    Cheltenham, UK
    Posts
    812
    Quote Originally Posted by defragster View Post
    Expect not until Paul integrates it for next Beta ... the SD.h edits. AFAIK they are still on PJRC's task list.
    Se we have to continue to use the defective code?

  18. #18
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    26,569
    Can confirm, I'm able to reproduce the bug with original SdFat 2.1.0 and Teensyduino's copy of SdFat, which is based on 2.1.0.

    SdFat 2.1.2 fixes the problem.


    Quote Originally Posted by defragster View Post
    Expect not until Paul integrates it for next Beta ... the SD.h edits. AFAIK they are still on PJRC's task list.
    Maybe I should start 1.57-beta1 sooner.

    Would also be nice to have an installer with the Wire library supporting I2C slave mode.


    Quote Originally Posted by BriComp View Post
    Se we have to continue to use the defective code?
    Well, you can install SdFat 2.1.2 right now.

    Or you can wait for 1.57-beta1.

  19. #19
    Senior Member BriComp's Avatar
    Join Date
    Apr 2014
    Location
    Cheltenham, UK
    Posts
    812
    Well, you can install SdFat 2.1.2 right now.
    Has that been modified for Teensy?
    If not what sort of problems can ensue?

  20. #20
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    26,569
    I've updated the Teensy specific copy to SdFat 2.1.2.

    https://github.com/PaulStoffregen/SdFat

  21. #21
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    16,240
    Quote Originally Posted by PaulStoffregen View Post
    I've updated the Teensy specific copy to SdFat 2.1.2.

    https://github.com/PaulStoffregen/SdFat
    Lucky it was just a .point release (or two) behind , I scrolled through the small half of the diffs ...

  22. #22
    Quote Originally Posted by PaulStoffregen View Post
    I've updated the Teensy specific copy to SdFat 2.1.2.

    https://github.com/PaulStoffregen/SdFat
    May I ask what is the "correct" way to update a TeensyDuino library in a case like this, i.e. between releases?

  23. #23
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    16,240
    Quote Originally Posted by joepasquariello View Post
    May I ask what is the "correct" way to update a TeensyDuino library in a case like this, i.e. between releases?
    The github link provides a 'Code' / ZIP download of the updated code, download that.

    That 'folder of code' can be placed in one of two places so the IDE will use that updated code:

    > {sketchbook}\libraries\SdFat
    --> this will be used until removed and be used before any SdFat, so would later need to be manually removed

    > Remove and replace the folder in the Teensy install: {local install}\hardware\teensy\avr\libraries\SdFat
    --> This will work until newer TeensyDuino installer is used.

  24. #24
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    26,569
    Usually you can just download the library from github, as a ZIP file or using the git software (if you have it on your computer and know how to use it). Then look at the verbose output Arduino prints for the full pathname where the library is located. Delete all the files in that location and replace with the new copy.

    But sometimes you will also need to update the core library or other libraries, if there are dependencies between them. Typically you'll find out because the new version won't compile, though the compiler's error messages usually tell you the name of something specific which isn't defined, but not which library you need to update to get that specific thing. You can probably guess which other libraries to update, or maybe search on github for the name of specific undefined thing, or just ask here on this forum. Usually I or others involved in library development will be able to answer.

    A 1.57-beta1 installer will be coming in a matter of days with the fix for this issue and a lot of other improvements.

  25. #25
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    26,569
    Teensyduino 1.57-beta1 is now available, with SdFat 2.1.2 which fixes this bug.

    https://forum.pjrc.com/threads/70196...no-1-57-Beta-1

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •