exFAT SD card append-write fails after 256 kb when writing to 2 files

**sicco** · 04-25-2022, 03:24 PM

Hi,

I have a Teensy 4.1 that writes to exFAT card. Using the Jan 2022 Teensyduino which I believe the most recent. It's for a datalogger. I want two log files. I write to the files by appending to them in loop(), once every 100 ms, one file after the other.
What's happening is that after 256 kB has been appended with 8 kB blocks without any issues, the file(s) on the SD card no longer wants to be written to.
But if I only append to one file, then all works fine.

When I write to both files successively, then after crossing the 256kB size limit and only when writing to two files instead of just one, then the file no longer grows in size. Write attempts simply fail.

The strange thing is that the write-failed file on that SD card then remains 'broken'. Also when I reboot the Teensy, or power cycle it. Only reformatting the SD card (I use my Windows 11 PC for that) fixes it. Or starting over with a new filename. But then that new one also only does the first 256 kB, and then nothing can be added.

Looks to me like a bug in the exFAT SD card libraries.

I'm using (SdioConfig(FIFO_SDIO)). Could that be it?

Archived example project attached. Only 55 lines. Reduced to the bare minimum size just to isolate and highlight the issue.
To replicate that it does work ok with just one file, comment out line 53 (write_to_log_file (filename_B);) , erase the card (or change the file names), and then the A file does grow beyond 256 k.

Anyone any ideas? exFAT directory tables get corrupted?

For info, this is what I use as SD card:
SD card Manufacturer ID: 0x1b OEM ID:0x534d (SM)
Product: ED4QT
Version: 3.0
Serial number: 0x0e534a33 = 240339507
Manufacturing date: 2/2011
SD card formatted for exFATSD: ok

**defragster** · 04-25-2022, 04:11 PM

Was any other SD media tried? What size is the Media?
Was a chkdsk/scandisk done on the Media before PC reformatting?

The code in ZIP is short so here it is:

Code:

#include <SdFat.h>
#include <SdFatConfig.h>

#define WRITE_TO_SD_CHUNK_SIZE 8192

SdFs sd;
uint8_t st[WRITE_TO_SD_CHUNK_SIZE];

void setup() {
// USB serial port
    Serial.begin(115200);
    int retry = 0;    // Serial is the USB port, but if no PC connected/active, then proceed anyway
    while ((!Serial) && (retry++ <100))
        delay(10);

    Serial.println("test exFAT logger");
 
    if (sd.begin(SdioConfig(FIFO_SDIO))) 
        Serial.println ("sd.begin() done");

// init char buffer with arbitrary readable data
    for (int i=0; i<WRITE_TO_SD_CHUNK_SIZE; i++)
        st[i] = ' ' + i & 0x3f;

}

void write_to_log_file (char *log_filename)
{
    FsFile F;
    int written;
    fspos_t pos;
                    
    F = sd.open(log_filename, O_WRONLY | O_CREAT | O_APPEND);
    if (F)
    {
        written = F.write (st, WRITE_TO_SD_CHUNK_SIZE);
        F.fgetpos(&pos);
        Serial.printf ("file %s, ", log_filename);
        Serial.printf ("pos=%d, ", pos);
        Serial.printf ("adding %d bytes, written %d\n", WRITE_TO_SD_CHUNK_SIZE, written);
    }
    
    F.close();  
}

void loop ()
{
    char filename_A[40] = "FILE_A.txt";
    char filename_B[40] = "FILE_B.txt";

    write_to_log_file (filename_A);
    write_to_log_file (filename_B); // To replicate that it does work ok with just one file, comment out line 53 (write_to_log_file (filename_B);)

    delay(100);
}

**WMXZ** · 04-25-2022, 05:15 PM

I tried the example
it fails here with

Code:

FFFFFFFF EB3A9
DBG_FAIL: ExFatPartition.cpp.195
DBG_FAIL: ExFatFileWrite.cpp.108
DBG_FAIL: ExFatFileWrite.cpp.663
file FILE_A.txt, pos=65536, adding 8192 bytes, written 0

The fist lines are the parameters from

Code:

bool ExFatPartition::fatPut(uint32_t cluster, uint32_t value)

so it called with cluster=0xFFFFFFFF

the SD card was 32 GB formatted as exFAT wit 64 kB cluster size.

My hunch is a Bug in exFAT, may you could open an issue with Bill Greiman, using this example program

**WMXZ** · 04-25-2022, 05:47 PM

I remover the two files and restarted program

Code:

test exFAT logger
sd.begin() done
file FILE_A.txt, pos=8192, adding 8192 bytes, written 8192
file FILE_B.txt, pos=8192, adding 8192 bytes, written 8192
file FILE_A.txt, pos=16384, adding 8192 bytes, written 8192
file FILE_B.txt, pos=16384, adding 8192 bytes, written 8192
file FILE_A.txt, pos=24576, adding 8192 bytes, written 8192
file FILE_B.txt, pos=24576, adding 8192 bytes, written 8192
file FILE_A.txt, pos=32768, adding 8192 bytes, written 8192
file FILE_B.txt, pos=32768, adding 8192 bytes, written 8192
file FILE_A.txt, pos=40960, adding 8192 bytes, written 8192
file FILE_B.txt, pos=40960, adding 8192 bytes, written 8192
file FILE_A.txt, pos=49152, adding 8192 bytes, written 8192
file FILE_B.txt, pos=49152, adding 8192 bytes, written 8192
file FILE_A.txt, pos=57344, adding 8192 bytes, written 8192
file FILE_B.txt, pos=57344, adding 8192 bytes, written 8192
file FILE_A.txt, pos=65536, adding 8192 bytes, written 8192
file FILE_B.txt, pos=65536, adding 8192 bytes, written 8192
DBG_FAIL: ExFatFileWrite.cpp.75
DBG_FAIL: ExFatFileWrite.cpp.663
file FILE_A.txt, pos=65536, adding 8192 bytes, written 0
DBG_FAIL: ExFatFileWrite.cpp.75
DBG_FAIL: ExFatFileWrite.cpp.663
file FILE_B.txt, pos=65536, adding 8192 bytes, written 0

interesting debug gives now different write error
Previous observation was with existing files, so difference may be possible

actual error happens in

Code:

bool ExFatFile::addCluster() {
  uint32_t find = m_vol->bitmapFind(m_curCluster ?  m_curCluster + 1 : 0, 1);
  if (find < 2) {
    DBG_FAIL_MACRO;
    goto fail;
  }

**sicco** · 04-25-2022, 07:14 PM

Thank you for prompt responses and confirmation that it's not specific to just my SD card, confirmation that it it replicates and that there's thus something of a bug.
I can confirm that if I change the 'Allocation unit size' in the Windows 11 Quick Format that I used, that the file position where the error occurs scales up linearly. For example, I initially had 128k as 'Allocation unit size' and the problem was triggered after 256k was written, but when I changed to 1 MB, the problem occurs after 2 MB has been appended to each file.

If I had the debugger working, and the exFAT spec at hand (no, I don't have it...) then I would now set breakpoints and single step into the library sources. But I don't have that luxury. So keeping fingers crossed now that others will soon manage to isolate and fix.

**defragster** · 04-25-2022, 08:42 PM

P#1 noted: I'm using (SdioConfig(FIFO_SDIO)). Could that be it?

Are there options that work on that media? That might indicate generic or specific issue.

Perhaps rewrite using only SD.h style calls to SD (from one of /examples) ... which still maps to SdFat. If that works: it is a work around for now - and indicates a problematic code path versus total fail on exFat formatted media.

**WMXZ** · 04-26-2022, 03:32 PM

@sicco
You can easily switch on the internal debug information of SdFat by uncommenting
setting

Code:

#define USE_DBG_MACROS 2

in

Code:

...\hardware\teensy\avr\libraries\SdFat\src\common\DebugMacros.h

It gives you a traceback of failing

This is not as good as a debugger, but better than nothing

**sicco** · 04-26-2022, 06:09 PM

Originally Posted by WMXZ

I tried the example
it fails here with

Code:

FFFFFFFF EB3A9
DBG_FAIL: ExFatPartition.cpp.195
DBG_FAIL: ExFatFileWrite.cpp.108
DBG_FAIL: ExFatFileWrite.cpp.663
file FILE_A.txt, pos=65536, adding 8192 bytes, written 0

The fist lines are the parameters from

Code:

bool ExFatPartition::fatPut(uint32_t cluster, uint32_t value)

so it called with cluster=0xFFFFFFFF

the SD card was 32 GB formatted as exFAT wit 64 kB cluster size.

My hunch is a Bug in exFAT, may you could open an issue with Bill Greiman, using this example program

OK. Stupid question maybe, but who is this Mr Bill Greiman and how do I open an issue with that person?

**defragster** · 04-26-2022, 06:28 PM

Mr Greiman is the author of SdFat - forum search will find his posts/threads and link to his github presence where an issue could be posted.

The ...\hardware\teensy\avr\libraries\SdFat\library.pr operties file provides this info:
...
url=https://github.com/greiman/SdFat
repository=https://github.com/greiman/SdFat.git
...

**PaulStoffregen** · 04-26-2022, 06:32 PM

Before you report the issue to Bill, please download and test with his latest version from github.

https://github.com/greiman/SdFat

The copy we ship with Teensyduino is slightly older and has several minor changes. Best to double check the bug really does happen with his latest version, just in case the problem is something we did (messed up) in modifications for Teensy.

This is the place to report the issue.

https://github.com/greiman/SdFat/issues/new

Also best to include complete code to reproduce the problem.

**WMXZ** · 04-27-2022, 06:00 AM

Just downloaded actual version and without further editing of SdConfig.h and activating build-in debug messages, I get

Code:

test exFAT logger
sd.begin() done
file FILE_A.txt, pos=8192, adding 8192 bytes, written 8192
file FILE_B.txt, pos=8192, adding 8192 bytes, written 8192
file FILE_A.txt, pos=16384, adding 8192 bytes, written 8192
file FILE_B.txt, pos=16384, adding 8192 bytes, written 8192
file FILE_A.txt, pos=24576, adding 8192 bytes, written 8192
file FILE_B.txt, pos=24576, adding 8192 bytes, written 8192
file FILE_A.txt, pos=32768, adding 8192 bytes, written 8192
file FILE_B.txt, pos=32768, adding 8192 bytes, written 8192
file FILE_A.txt, pos=40960, adding 8192 bytes, written 8192
file FILE_B.txt, pos=40960, adding 8192 bytes, written 8192
file FILE_A.txt, pos=49152, adding 8192 bytes, written 8192
file FILE_B.txt, pos=49152, adding 8192 bytes, written 8192
file FILE_A.txt, pos=57344, adding 8192 bytes, written 8192
file FILE_B.txt, pos=57344, adding 8192 bytes, written 8192
file FILE_A.txt, pos=65536, adding 8192 bytes, written 8192
file FILE_B.txt, pos=65536, adding 8192 bytes, written 8192
file FILE_A.txt, pos=65536, adding 8192 bytes, written 0
file FILE_B.txt, pos=65536, adding 8192 bytes, written 0
file FILE_A.txt, pos=65536, adding 8192 bytes, written 0
file FILE_B.txt, pos=65536, adding 8192 bytes, written 0

Edit: I opened an issue on Bill´s github

**Bill Greiman** · 04-27-2022, 01:35 PM

I posted a reply on github.

I tested with SdFat 2.1.2 and 2.1.4-beta.3
I uses 32GB FAT, 64GB exFAT, 512GB exFAT on a Teensy 4.1

I used Teensyduino 1.56 with Arduino 1.8.19

Seems to work on all. I used Samsung SD cards.

With 32GB FAT
test exFAT logger
2.1.4-beta.3
sd.begin() done
file FILE_A.txt, pos=8192, adding 8192 bytes, written 8192
file FILE_B.txt, pos=8192, adding 8192 bytes, written 8192
file FILE_A.txt, pos=16384, adding 8192 bytes, written 8192
file FILE_B.txt, pos=16384, adding 8192 bytes, written 8192
file FILE_A.txt, pos=24576, adding 8192 bytes, written 8192
file FILE_B.txt, pos=24576, adding 8192 bytes, written 8192
....
file FILE_B.txt, pos=7004160, adding 8192 bytes, written 8192
file FILE_A.txt, pos=7012352, adding 8192 bytes, written 8192
file FILE_B.txt, pos=7012352, adding 8192 bytes, written 8192
file FILE_A.txt, pos=7020544, adding 8192 bytes, written 8192
file FILE_B.txt, pos=7020544, adding 8192 bytes, written 8192
file FILE_A.txt, pos=7028736, adding 8192 bytes, written 8192
file FILE_B.txt, pos=7028736, adding 8192 bytes, written 8192

With 64GB exFAT
test exFAT logger
2.1.4-beta.3
sd.begin() done
file FILE_A.txt, pos=8192, adding 8192 bytes, written 8192
file FILE_B.txt, pos=8192, adding 8192 bytes, written 8192
file FILE_A.txt, pos=16384, adding 8192 bytes, written 8192
file FILE_B.txt, pos=16384, adding 8192 bytes, written 8192
file FILE_A.txt, pos=24576, adding 8192 bytes, written 8192
file FILE_B.txt, pos=24576, adding 8192 bytes, written 8192
...
file FILE_A.txt, pos=6774784, adding 8192 bytes, written 8192
file FILE_B.txt, pos=6774784, adding 8192 bytes, written 8192
file FILE_A.txt, pos=6782976, adding 8192 bytes, written 8192
file FILE_B.txt, pos=6782976, adding 8192 bytes, written 8192
file FILE_A.txt, pos=6791168, adding 8192 bytes, written 8192

With 512GB exFAT
est exFAT logger
2.1.4-beta.3
sd.begin() done
file FILE_A.txt, pos=8192, adding 8192 bytes, written 8192
file FILE_B.txt, pos=8192, adding 8192 bytes, written 8192
file FILE_A.txt, pos=16384, adding 8192 bytes, written 8192
file FILE_B.txt, pos=16384, adding 8192 bytes, written 8192
file FILE_A.txt, pos=24576, adding 8192 bytes, written 8192
file FILE_B.txt, pos=24576, adding 8192 bytes, written 8192
...
file FILE_A.txt, pos=7315456, adding 8192 bytes, written 8192
file FILE_B.txt, pos=7315456, adding 8192 bytes, written 8192
file FILE_A.txt, pos=7323648, adding 8192 bytes, written 8192
file FILE_B.txt, pos=7323648, adding 8192 bytes, written 8192
file FILE_A.txt, pos=7331840, adding 8192 bytes, written 8192
file FILE_B.txt, pos=7331840, adding 8192 bytes, written 8192

**WMXZ** · 04-27-2022, 03:28 PM

Thanks Bill,
I tried with new disk and it worked also here.
sane configuration (SdFat 2.1.2, Teensyduino 1.56 with Arduino 1.8.19)
must have screwed up my earlier testing.

**sicco** · 04-28-2022, 07:15 AM

Originally Posted by WMXZ

Thanks Bill,
I tried with new disk and it worked also here.
sane configuration (SdFat 2.1.2, Teensyduino 1.56 with Arduino 1.8.19)
must have screwed up my earlier testing.

I updated my SdFat library from 2.1.0 to 2.1.2 and that seems to have fixed the bug.
Thank you all!

**BriComp** · 04-28-2022, 09:09 AM

Originally Posted by PaulStoffregen

Before you report the issue to Bill, please download and test with his latest version from github.

https://github.com/greiman/SdFat

The copy we ship with Teensyduino is slightly older and has several minor changes. Best to double check the bug really does happen with his latest version, just in case the problem is something we did (messed up) in modifications for Teensy.

This is the place to report the issue.

https://github.com/greiman/SdFat/issues/new

Also best to include complete code to reproduce the problem.

Is this version modified for Teensy?

**defragster** · 04-28-2022, 09:22 AM

Originally Posted by BriComp

Is this version modified for Teensy?

Expect not until Paul integrates it for next Beta ... the SD.h edits. AFAIK they are still on PJRC's task list.

**BriComp** · 04-28-2022, 09:37 AM

Originally Posted by defragster

Expect not until Paul integrates it for next Beta ... the SD.h edits. AFAIK they are still on PJRC's task list.

Se we have to continue to use the defective code?

**PaulStoffregen** · 04-28-2022, 11:12 AM

Can confirm, I'm able to reproduce the bug with original SdFat 2.1.0 and Teensyduino's copy of SdFat, which is based on 2.1.0.

SdFat 2.1.2 fixes the problem.

Originally Posted by defragster

Expect not until Paul integrates it for next Beta ... the SD.h edits. AFAIK they are still on PJRC's task list.

Maybe I should start 1.57-beta1 sooner.

Would also be nice to have an installer with the Wire library supporting I2C slave mode.

Originally Posted by BriComp

Se we have to continue to use the defective code?

Well, you can install SdFat 2.1.2 right now.

Or you can wait for 1.57-beta1.

**BriComp** · 04-28-2022, 01:04 PM

Well, you can install SdFat 2.1.2 right now.

Has that been modified for Teensy?
If not what sort of problems can ensue?

**PaulStoffregen** · 04-28-2022, 02:23 PM

I've updated the Teensy specific copy to SdFat 2.1.2.

https://github.com/PaulStoffregen/SdFat

**defragster** · 04-28-2022, 05:45 PM

Originally Posted by PaulStoffregen

I've updated the Teensy specific copy to SdFat 2.1.2.

https://github.com/PaulStoffregen/SdFat

Lucky it was just a .point release (or two) behind

, I scrolled through the small half of the diffs ...

**joepasquariello** · 04-30-2022, 04:56 PM

Originally Posted by PaulStoffregen

I've updated the Teensy specific copy to SdFat 2.1.2.

https://github.com/PaulStoffregen/SdFat

May I ask what is the "correct" way to update a TeensyDuino library in a case like this, i.e. between releases?

**defragster** · 04-30-2022, 05:37 PM

Originally Posted by joepasquariello

May I ask what is the "correct" way to update a TeensyDuino library in a case like this, i.e. between releases?

The github link provides a 'Code' / ZIP download of the updated code, download that.

That 'folder of code' can be placed in one of two places so the IDE will use that updated code:

> {sketchbook}\libraries\SdFat
--> this will be used until removed and be used before any SdFat, so would later need to be manually removed

> Remove and replace the folder in the Teensy install: {local install}\hardware\teensy\avr\libraries\SdFat
--> This will work until newer TeensyDuino installer is used.

**PaulStoffregen** · 04-30-2022, 06:40 PM

Usually you can just download the library from github, as a ZIP file or using the git software (if you have it on your computer and know how to use it). Then look at the verbose output Arduino prints for the full pathname where the library is located. Delete all the files in that location and replace with the new copy.

But sometimes you will also need to update the core library or other libraries, if there are dependencies between them. Typically you'll find out because the new version won't compile, though the compiler's error messages usually tell you the name of something specific which isn't defined, but not which library you need to update to get that specific thing. You can probably guess which other libraries to update, or maybe search on github for the name of specific undefined thing, or just ask here on this forum. Usually I or others involved in library development will be able to answer.

A 1.57-beta1 installer will be coming in a matter of days with the fix for this issue and a lot of other improvements.

**PaulStoffregen** · 05-07-2022, 12:47 PM

Teensyduino 1.57-beta1 is now available, with SdFat 2.1.2 which fixes this bug.

https://forum.pjrc.com/threads/70196...no-1-57-Beta-1

Thread: exFAT SD card append-write fails after 256 kb when writing to 2 files

Thread Tools

Search Thread

Display

exFAT SD card append-write fails after 256 kb when writing to 2 files

Posting Permissions