Progress on a new exFAT SD library.

Status
Not open for further replies.

Bill Greiman

Well-known member
I am working on a new SD library for exFAT formatted SD cards. This library supports exFAT but not FAT32/FAT16 since I want the library to run on an ATmega328.

There is now a 1TB SanDisk card so an Uno could write a 1 TB file in about 40 days. This assumes a write speed of 300 KB/sec. A Teensy 3.6 could do it in less than a day.

I spent a lot of time analyzing existing exFAT implementations and developing faster smaller algorithms for this library.

I have the basic functions implemented in prototype form and its size/speed is promising. I expect the size to increase as I add error checks and reliability features. I hope this sketch will be under 16 KB on an Uno with ASCII file names.

This version has these major functions:
Code:
open()
close()
remove()
mkdir()
rmdir()
read()
write()
seek()
sync()

Here is a test sketch:

Code:
// Example exFAT test sketch
#include "SdFs.h"
SdSpiAltDriver spi;
SdSpiCard card;
FsVolume sd(&card);
File file;

const uint8_t CS_PIN = 10;

#define error(s) {Serial.println(F(s));return;}

void setup() {
  Serial.begin(9600);
  Serial.println(F("Type any character to begin"));
  while (!Serial.available()) {yield();}
  if (!card.begin(&spi, CS_PIN, SD_SCK_MHZ(50))) error("card begin");
  if (!sd.begin()) error("sd.begin");
  if (!file.open(&sd, "Test.txt", O_READ | O_WRITE | O_CREAT)) error("sd.open");
  file.println(F("Hello World!"));
  if (!file.close()) error("close");
  Serial.println(F("Done"));
}
void loop() {}

Code:
With ASCII file names.

Sketch uses 13,848 bytes (42%) of program storage space. Maximum is 32,256 bytes.
Global variables use 865 bytes (42%) of dynamic memory, leaving 1,183 bytes for local variables. Maximum is 2,048 bytes.

Code:
With Unicode file names.

Sketch uses 14,800 bytes (45%) of program storage space. Maximum is 32,256 bytes.
Global variables use 865 bytes (42%) of dynamic memory, leaving 1,183 bytes for local variables. Maximum is 2,048 bytes.

Eventually I will refactor the SdFat code like this library and introduce a Linux like virtual file level so I won't have the nasty problem of combining FAT16/FAT32 code with exFAT code in a single library.
 
I am now testing performance. Here are some results comparing the new exFAT code with SdFat on a Teensy 3.6 at 240 MHz.

Code:
240MHz 32GB Samsung PRO Select

exFAT - optimized multi-block transfers
size,write,read
bytes,KB/sec,KB/sec
512,20494.06,20325.18
1024,20901.03,20466.31
2048,21119.46,20612.96
4096,21163.42,20756.79
8192,20918.07,20729.20
16384,21228.12,20798.73
32768,21225.97,20775.14

FAT32 - optimized multi-block transfers
size,write,read
bytes,KB/sec,KB/sec
512,19351.64,20038.57
1024,19622.93,20151.31
2048,19781.84,20267.48
4096,19713.73,20406.22
8192,19703.59,20272.52
16384,19935.85,20421.07
32768,19764.08,20561.98


exFAT - DMA buffer size transfers
size,write,read
bytes,KB/sec,KB/sec
512,635.83,2078.10
1024,1230.60,3094.75
2048,2258.98,5175.83
4096,4218.59,7439.68
8192,4347.95,9290.08
16384,7750.22,13930.35
32768,13531.13,15825.08


FAT32 - DMA buffer size transfers
size,write,read
bytes,KB/sec,KB/sec
512,625.86,2161.35
1024,1180.08,3116.54
2048,2234.67,5428.18
4096,4005.58,7786.67
8192,4292.35,9313.15
16384,7737.33,13764.86
32768,13082.89,15767.35

exFAT is almost always faster for write. The exFat file is contiguous so there is no overhead for access to the FAT for exFAT.

FAT32 and exFAT read performance is very close.

Unfortunately DMA performance is poor for small transfers on both exFAT and FAT32. The DMA controller errata prevents optimized multi-block transfers.
 
nice work... 20+MB/sec read *or* write to uSD seems very impressive for such a teensy-priced processor

A Teensy 3.6 could [write 1TB] in less than a day.

to think I could use my T3.6 and put my whole life on a microSD so quicky ... bonus!
 
sounds really neat, as fat32 can only handle up to 32gb sd cards. keep it up! :)
At least 2TB work with Windows, it's just that the built-in format utility is limited to 32GB. 2TB partitions formatted with Linux work fine. SDFormater can also be used.
 
I've been told some versions of Windows have a (perhaps severe) bug in scandisk with FAT32 volumes larger than 32GB. It's been suggested that's the (real) reason all of Microsoft's programs stubbornly refuse to format FAT32 larger than 32GB.

FAT32 allows up to 2^28 clusters. Why they didn't use all 32 bits is a mystery to me, but the spec clearly says the top 4 bits must be ignored. On SD cards with 512 byte sectors, the maximum cluster size is 64K. I believe that works out to be 16TB maximum size.

I've seen guidance that 32K (64 sectors) is the maximum recommended cluster size for compatibility, even though the format allows 128 sector clusters.

Windows supposedly will allow FAT32 to specify up to 4096 bytes per sector. But nearly all devices that access SD cards expect the sector size to be fixed at 512 bytes.
 
I've been told some versions of Windows have a (perhaps severe) bug in scandisk with FAT32 volumes larger than 32GB. It's been suggested that's the (real) reason all of Microsoft's programs stubbornly refuse to format FAT32 larger than 32GB.

FAT32 allows up to 2^28 clusters. Why they didn't use all 32 bits is a mystery to me, but the spec clearly says the top 4 bits must be ignored. On SD cards with 512 byte sectors, the maximum cluster size is 64K. I believe that works out to be 16TB maximum size.

I've seen guidance that 32K (64 sectors) is the maximum recommended cluster size for compatibility, even though the format allows 128 sector clusters.

Windows supposedly will allow FAT32 to specify up to 4096 bytes per sector. But nearly all devices that access SD cards expect the sector size to be fixed at 512 bytes.

FAT32 volumes can be very large. The SdFat SdFormatter example will format an SDXC card FAT32. I use 64KB clusters. I did a library for USB mass storage using the core FatLib from SdFat and tested it with large hard drives formatted FAT32.

I don't have a beta on github. I have a huge amount of work yet. I want to integrate FAT32/FAT16 in to a new library. I want it to be easy to write a block driver for new devices and integrate the device.

Here is a prototype sketch that can write to eXFAT, FAT32, FAT16, and FAT12. You could replace the card object with any block driver class that conforms to the BlockDevInterface.

Code:
// Test library with exFAT, FAT32, FAT16, and FAT12 support
#include "SdFs.h"
#include "FreeStack.h"
SdSpiAltDriver spi;
SdSpiCard card;
FsVolume sd(&card);
FsFile file;

#define CS_PIN SS

#define error(s) {Serial.println(F(s));return;}

void setup() {
  Serial.begin(9600);
  if (!card.begin(&spi, CS_PIN, SD_SCK_MHZ(50))) error("card begin");
  if (!sd.begin()) error("sd begin");
  if (!file.open(&sd, "test.txt", O_READ|O_WRITE|O_CREAT)) error("open");
  if (!file.write("Hello World!", strlen("Hello World!"))) error("write");
  printFreeStack();
  file.close();
  sd.end();
  Serial.println("Done");
  printFreeStack();
}
void printFreeStack() {
  Serial.print(F("FreeStack: "));
  Serial.println(FreeStack());
}

void loop() {}
I tested on an Uno. Here is the flash size. It uses dynamic RAM.
Code:
Sketch uses 21,808 bytes (67%) of program storage space. Maximum is 32,256 bytes.
Global variables use 291 bytes (14%) of dynamic memory, leaving 1,757 bytes for local variables. Maximum is 2,048 bytes.
Here is RAM use. The first value is when max RAM is allocated and the second value is after deleting dynamic objects.
Code:
exFat SD
FreeStack: 1044
Done
FreeStack: 1699


FAT32 SD
FreeStack: 1080
Done
FreeStack: 1699

Here is the block device interface class.
Code:
class BlockDevInterface {
 public:
   virtual bool begin() = 0;
   virtual bool end() = 0;
    /**
   * Read a 512 byte block..
   *
   * \param[in] block Logical block to be read.
   * \param[out] dst Pointer to the location that will receive the data.
   * \return The value true is returned for success and
   * the value false is returned for failure.
   */  
  virtual bool readBlock(uint32_t block, uint8_t* dst) = 0;
  /** End multi-block transfer and go to idle state.
   * \return The value true is returned for success and
   * the value false is returned for failure.
   */     
  virtual bool syncBlocks() = 0;
  /**
   * Writes a 512 byte block.
   *
   * \param[in] block Logical block to be written.
   * \param[in] src Pointer to the location of the data to be written.
   * \return The value true is returned for success and
   * the value false is returned for failure.
   */    
  virtual bool writeBlock(uint32_t block, const uint8_t* src) = 0;
  /**
   * Read multiple 512 byte blocks.
   *
   * \param[in] block Logical block to be read.
   * \param[in] nb Number of blocks to be read.
   * \param[out] dst Pointer to the location that will receive the data.
   * \return The value true is returned for success and
   * the value false is returned for failure.
   */  
  virtual bool readBlocks(uint32_t block, uint8_t* dst, size_t nb) = 0;
  /**
   * Write multiple 512 byte blocks.
   *
   * \param[in] block Logical block to be written.
   * \param[in] nb Number of blocks to be written.
   * \param[in] src Pointer to the location of the data to be written.
   * \return The value true is returned for success and
   * the value false is returned for failure.
   */    
  virtual bool writeBlocks(uint32_t block, const uint8_t* src, size_t nb) = 0;
};

Here is the SPI interface class for use with the SdSpiCard class.
Code:
class SdSpiBaseDriver {
 public:
  /** Start transaction and set SPI options.
   *
   */
  virtual void activate() = 0;
  /** Initialize the SPI bus.
   *
   * \param[in] chipSelectPin SD card chip select pin.
   */
  virtual void begin(uint8_t chipSelectPin) = 0;
  /**
   * End SPI transaction.
   */
  virtual void deactivate() = 0;
  /** Receive a byte.
   *
   * \return The byte.
   */
  virtual uint8_t receive() = 0;
  /** Receive multiple bytes.
   *
   * \param[out] buf Buffer to receive the data.
   * \param[in] n Number of bytes to receive.
   *
   * \return Zero for no error or nonzero error code.
   */
  virtual uint8_t receive(uint8_t* buf, size_t n) = 0;
  /** Send a byte.
   *
   * \param[in] data Byte to send
   */
  virtual void send(uint8_t data) = 0;
  /** Send multiple bytes.
  *
  * \param[in] buf Buffer for data to be sent.
  * \param[in] n Number of bytes to send.
  */
  virtual void send(const uint8_t* buf, size_t n) = 0;
  /** Set CS low. */
  virtual void select() = 0;
  /** Save SPI settings.
   * \param[in] spiSettings SPI speed, mode, and bit order.
   */
  virtual void setSpiSettings(SPISettings spiSettings) = 0;
  /** Set CS high. */
  virtual void unselect() = 0;
};
 
Last edited:
Last edited:
I now have C++ iostream working with ExFat. I have the fstream, ifstream, ofstream classes. I have most of the stream manipulators and format flags.

I found more code size optimizations so it is practical on an Uno.

Sample sketch:
Code:
// Example of iostream with exFAT.
#include "SdFs.h"

SdExFat sd;

const uint8_t CS_PIN = 10;

#define error(s) {Serial.println(F(s));return;}

void setup() {
  char c;
  Serial.begin(9600);
  Serial.println(F("Type any character to begin"));
  while (!Serial.available()) {yield();}

  // Initialize SD
  if (!sd.begin(CS_PIN)) error("sd.begin");
  
  //Create and open fstream file.
  ofstream os("ExFatTest.txt");

  // Write Line to file.
  os << "Line to ExFat file." << endl;

  // Close File.
  os.close();
  
  Serial.println(F("Done"));
}
void loop() {}

Size on Uno:
Code:
Sketch uses 13,772 bytes (42%) of program storage space. Maximum is 32,256 bytes.
Global variables use 913 bytes (44%) of dynamic memory, leaving 1,135 bytes for local variables. Maximum is 2,048 bytes.

I am also using Paul's convention of a special pin number for SDIO. The above sketch will run with the Teensy 3.6 on-board microSD with this change.

Code:
//  if (!sd.begin(CS_PIN)) error("sd.begin");

  if (!sd.begin(SDIO_PIN)) error("sd.begin");

The size added by having drivers for SDIO and SPI cards loaded on Teensy 3.6 is not significant. Here is the size of the above sketch on Teensy 3.6.
Code:
Sketch uses 35,028 bytes (3%) of program storage space. Maximum is 1,048,576 bytes.
Global variables use 5,848 bytes (2%) of dynamic memory, leaving 256,296 bytes for local variables. Maximum is 262,144

This sketch that prints a line takes almost 24K.
Code:
void setup() {
  Serial.begin(9600);
  Serial.println("Done");
}
void loop() {}
Code:
Sketch uses 23,836 bytes (2%) of program storage space. Maximum is 1,048,576 bytes.
Global variables use 5,072 bytes (1%) of dynamic memory, leaving 257,072 bytes for local variables. Maximum is 262,144 bytes.
 
Last edited:
Bill,

Any progress on the ExFat library? I have a project that needs to talk to an ExSd card created by a dashcam.

Greg
 
Status
Not open for further replies.
Back
Top