SdFat_Usage example: datalog.bin output file contains garbage

Status
Not open for further replies.

thecomfychair

Active member
I am familiarising myself with using an SD card on a Teensy 4.1, currently investigating reading and writing text to files. The SdFat_Usage example compiles and runs just fine, and appears to be very useful and informative. However, when opening the 'datalog.bin' file in Notepad on my PC, I find that it contains a vast array of nonsensical characters after the expected "Just some test data written to the file (by SdFat functions)".

Using Arduino 1.8.16, TD 1.55, on a Teensy 4.1.
SD card already has some files on from previous experiments creating a file browser and audio player.

Here are the first few lines of the file (the whole file is some 300,000 lines):
Code:
Just some test data written to the file (by SdFat functions)    S y s t e  rm   V o l u   m e SYSTEM~1    eQeQ  eQ     AM u s i c  ®B e e   ÿÿÿÿ  ÿÿÿÿMUSICBEE    &EeQeQ  FeQ     DATALOG BIN    !RJS  õcJSÂ1  €TEST    TXT    !R!R    !R”"   C1 . m 3 u  ù  ÿÿÿÿÿÿÿÿÿÿ  ÿÿÿÿl a y l i  ùs t s T e s   t   M u s i c  ùB e e t e s   t P MUSICB~1M3U     !R!R    !R      Cu   ÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿ  ÿÿÿÿi s t s T  e s t   1 .   m 3 M u s i c  B e e P l a   y l MUS~3E69M3U     !R!R    !R      pinf   frmamp4a   schm    itun       schi   userVîÇ   cert      XrighveID  gplat   aver tranÉ$¤åsongA toolP419medi   mode                      nameSebastiao Mota                                                                                                                                                                                                                                                    €chtbhuXwÝéÇhU›wâ–y&?ZFœ>ó×äe4ž#¨¶D?Ã@¹6D
æa²P~º+e*úGzЬ
jlÃUo¡ëýŒàÓÏ'‘{¶`"3HʆÁDH×NÔšyVzþ?OtëP„Ç·t ›)-5bæg[gXë@AÁ<{@lÎ¼Í ¸vÌݶpv…€®aíhBá\
Å*0ßâÅ#1å7ô]¸¹ë, ó'é s–ç‚eZ{:Ä‹a—‘Ú0'ý«i?‹93p<k:Ì•Ò‡œÔÂE$’Àå÷|ÁDùØ!KA?dÒQŠìD˜•]‰Ñ´ý*Ù@ÛŽ5œbò0?eä
à[Ý*ž
q¯•ð¿7H—`y×UEnàWb?ˆÎ,wNw^ÞÐIn#?êøéá|¤yârüóû8N*Á^š7[]í7ô[å¹áªm.eí®tç?IÙ±Ôâðnâ¤!œ¸ûO¢œŽ·îæ¾ñá¿ph¸¼Ó„[“ßÊ€‚"Ô©f6]‚ÚåNdã?ì?	*ë8ƒ8“Õb*Û?R˜?q£!èá‚è~˜*%ß0dC¯$ÐþJD¬ã,î×3ˆJ/v®]Gs5¾¬>A*l¶œ÷-›Ù\ý¿"ÃnbIhHyû£ÄØPò §Rqe?Dß^Y~'Ÿ&þö˜¢ø°9&?þµ²)ö‹<&ðn3T¬yz$|á_
C¾ZwRCe%ñÔ¹Q}—djÖeGpýzçž-1ç„„†èÜ¡Ãå?CO£0mÃþUÜë•>¾^û xbìú©]¨þäk¼,Pg!Â?”ü…óÈ\?îÇýÕ¬éð˜Å   ˆsignœöÎ@YJ¨'
Ëј¨©:î|Œc*YN–Ìâض`7tn€‰¹½l9`oD¨pÎOçž›RåqçŸZwv÷”*?Áû@µ"/m“?#³æ?܈“‹æ(	Çs¼ŠùÂÎ	¯=ói°F·¥2^ûú<“Y‚"85”FFkå*2¥}÷~  {Pfree

I'm presuming this is the rest of the data on the SD card, further down it has things like:
Code:
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>asset-info</key>
	<dict>
		<key>file-size</key>
		<integer>15540266</integer>
		<key>flavor</key>
		<string>2:256</string>
	</dict>
</dict>
</plist>
Which must be somehow related to the music files that are also on the card.

This is the example sketch as I am using it:
Code:
/*
  SdFat usage from SD library

  Starting with Teensyduino 1.54, the SD library is a thin wrapper for SdFat.

  You can access the main SdFat filesystem with "SD.sdfs".  You may wish to
  use SD.sdfs.begin() to cause SdFat to access the SD card using faster
  drivers than the default.  You may also wish to open files as FsFile to
  gain access to SdFat's special file functions which are not available using
  the simpler SD File.

  This example shows some of the ways to select optimized SdFat drivers and
  how to use special file truncation and pre-allocation for optimized data
  logging.

  This example code is in the public domain.
*/
#include <SD.h>

void setup()
{
  //Uncomment these lines for Teensy 3.x Audio Shield (Rev C)
  //SPI.setMOSI(7);  // Audio shield has MOSI on pin 7
  //SPI.setSCK(14);  // Audio shield has SCK on pin 14

  Serial.begin(9600);
  while (!Serial); // wait for Arduino Serial Monitor

  Serial.print("Initializing SD card...");
  bool ok;
  const int chipSelect = BUILTIN_SDCARD;

  // Instead of the usual SD.begin(pin), you can access the underlying
  // SdFat library for much more control over how the SD card is
  // accessed.  Uncomment one of these, or craft your own if you wish
  // to use SdFat's many special features.

  // Faster SPI frequency.  16 MHz is default for longer / messy wiring.
//  ok = SD.sdfs.begin(SdSpiConfig(chipSelect, SHARED_SPI, SD_SCK_MHZ(24)));

  // Very slow SPI frequency.  May be useful for hardware with slow buffers.
  //ok = SD.sdfs.begin(SdSpiConfig(chipSelect, SHARED_SPI, SD_SCK_MHZ(4)));

  // SdFat offers DEDICATED_SPI optimation when no other SPI chips are
  // connected.  More CPU time is used and results may vary depending on
  // interrupts, but for many cases speed is much faster.
  //ok = SD.sdfs.begin(SdSpiConfig(chipSelect, DEDICATED_SPI, SD_SCK_MHZ(16)));

  // Access the built in SD card on Teensy 3.5, 3.6, 4.1 using FIFO
  ok = SD.sdfs.begin(SdioConfig(FIFO_SDIO));

  // Access the built in SD card on Teensy 3.5, 3.6, 4.1 using DMA (maybe faster)
  //ok = SD.sdfs.begin(SdioConfig(DMA_SDIO));

  if (!ok) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
  Serial.println();

  // After the SD card is initialized, you can access is using the ordinary
  // SD library functions, regardless of whether it was initialized by
  // SD library SD.begin() or SdFat library SD.sdfs.begin().
  //
  Serial.println("Print directory using SD functions");
  File root = SD.open("/");
  while (true) {
    File entry = root.openNextFile();
    if (!entry) break; // no more files
    Serial.print(entry.name());
    if (entry.isDirectory()) {
      Serial.println("/");
    } else {
      printSpaces(40 - strlen(entry.name()));
      Serial.print("  ");
      Serial.println(entry.size(), DEC);
    }
    entry.close();
  }

  // You can also access the SD card with SdFat's functions
  //
  Serial.println();
  Serial.println("Print directory using SdFat ls() function");
  SD.sdfs.ls();

  // You can access files using SdFat which uses "FsFile" for open files
  // FsFile offers more capability than regular SD "File".  As shown in this
  // example, you can truncate tiles.  You can also pre-allocate a file on
  // the SD card (if it does not yet have any data, the reason we truncate
  // first).  Pre-allocation impoves the speed of writes within the already
  // allocated space while data logging or performing other small writes.
  //
  Serial.println();
  Serial.println("Writing to datalog.bin using SdFat functions");
  FsFile myfile = SD.sdfs.open("datalog.bin", O_WRITE | O_CREAT);
  unsigned int len = myfile.fileSize();
  Serial.print("datalog.bin started with ");
  Serial.print(len);
  Serial.println(" bytes");
  if (len > 0) {
    // reduce the file to zero if it already had data
    myfile.truncate();
  }
  if (myfile.preAllocate(40*1024*1024)) {
    Serial.print("  Allocate 40 megabytes for datalog.bin");
  } else {
    Serial.print("  unable to preallocate this file");
  }
  myfile.print("Just some test data written to the file (by SdFat functions)");
  myfile.write(0);
  myfile.close();

  // You can also use regular SD functions, even to access the same file.  Just
  // remember to close the SdFat FsFile before opening as a regular SD File.
  //
  Serial.println();
  Serial.println("Reading to datalog.bin using SD functions");
  File f = SD.open("datalog.bin");
  if (f) {
    char mybuffer[100];
    int index = 0;
    while (f.available()) {
      char c = f.read();
      mybuffer[index] = c;
      if (c == 0) break;  // end of string
      index = index + 1;
      if (index == 99) break; // buffer full
    }
    mybuffer[index] = 0;
    Serial.print("  Read from file: ");
    Serial.println(mybuffer);
  } else {
    Serial.println("unable to open datalog.bin :(");
  }
  f.close();

  // When mixing SD and SdFat file access, remember for writing that
  // SD defaults to appending if you open with FILE_WRITE.  You must
  // use FILE_WRITE_BEGIN if you wish to overwrite the file from the
  // start.  With SdFat, O_WRITE or O_RDWR starts overwriting from the
  // beginning.  You must add O_AT_END if you wish to appead.
}

void loop()
{
  // nothing happens after setup finishes.
}


void printSpaces(int num) {
  for (int i = 0; i < num; i++) {
    Serial.print(" ");
  }
}

Is this a bug, or is there some problem with my implementation of this example?
 
Last edited:
@thecomfychair

Just ran your example sketch on a T4.1 using the builtin SD Card per you example but it seems to working fine However if I open the file on windows I am seeing the same gibberish that you see. Be honest never really checked before what was in the file. My guess is that its due to preallocating 40mb of space. So lets try deleting the code for pre-allocating space on the SD card. So your test sketch will look like:

Code:
/*
  SdFat usage from SD library

  Starting with Teensyduino 1.54, the SD library is a thin wrapper for SdFat.

  You can access the main SdFat filesystem with "SD.sdfs".  You may wish to
  use SD.sdfs.begin() to cause SdFat to access the SD card using faster
  drivers than the default.  You may also wish to open files as FsFile to
  gain access to SdFat's special file functions which are not available using
  the simpler SD File.

  This example shows some of the ways to select optimized SdFat drivers and
  how to use special file truncation and pre-allocation for optimized data
  logging.

  This example code is in the public domain.
*/
#include <SD.h>

void setup()
{
  //Uncomment these lines for Teensy 3.x Audio Shield (Rev C)
  //SPI.setMOSI(7);  // Audio shield has MOSI on pin 7
  //SPI.setSCK(14);  // Audio shield has SCK on pin 14

  Serial.begin(9600);
  while (!Serial); // wait for Arduino Serial Monitor

  Serial.print("Initializing SD card...");
  bool ok;
  const int chipSelect = BUILTIN_SDCARD;

  // Instead of the usual SD.begin(pin), you can access the underlying
  // SdFat library for much more control over how the SD card is
  // accessed.  Uncomment one of these, or craft your own if you wish
  // to use SdFat's many special features.

  // Faster SPI frequency.  16 MHz is default for longer / messy wiring.
//  ok = SD.sdfs.begin(SdSpiConfig(chipSelect, SHARED_SPI, SD_SCK_MHZ(24)));

  // Very slow SPI frequency.  May be useful for hardware with slow buffers.
  //ok = SD.sdfs.begin(SdSpiConfig(chipSelect, SHARED_SPI, SD_SCK_MHZ(4)));

  // SdFat offers DEDICATED_SPI optimation when no other SPI chips are
  // connected.  More CPU time is used and results may vary depending on
  // interrupts, but for many cases speed is much faster.
  //ok = SD.sdfs.begin(SdSpiConfig(chipSelect, DEDICATED_SPI, SD_SCK_MHZ(16)));

  // Access the built in SD card on Teensy 3.5, 3.6, 4.1 using FIFO
  ok = SD.sdfs.begin(SdioConfig(FIFO_SDIO));

  // Access the built in SD card on Teensy 3.5, 3.6, 4.1 using DMA (maybe faster)
  //ok = SD.sdfs.begin(SdioConfig(DMA_SDIO));

  if (!ok) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
  Serial.println();

  // After the SD card is initialized, you can access is using the ordinary
  // SD library functions, regardless of whether it was initialized by
  // SD library SD.begin() or SdFat library SD.sdfs.begin().
  //
  Serial.println("Print directory using SD functions");
  File root = SD.open("/");
  while (true) {
    File entry = root.openNextFile();
    if (!entry) break; // no more files
    Serial.print(entry.name());
    if (entry.isDirectory()) {
      Serial.println("/");
    } else {
      printSpaces(40 - strlen(entry.name()));
      Serial.print("  ");
      Serial.println(entry.size(), DEC);
    }
    entry.close();
  }

  // You can also access the SD card with SdFat's functions
  //
  Serial.println();
  Serial.println("Print directory using SdFat ls() function");
  SD.sdfs.ls();

  // You can access files using SdFat which uses "FsFile" for open files
  // FsFile offers more capability than regular SD "File".  As shown in this
  // example, you can truncate tiles.  You can also pre-allocate a file on
  // the SD card (if it does not yet have any data, the reason we truncate
  // first).  Pre-allocation impoves the speed of writes within the already
  // allocated space while data logging or performing other small writes.
  //
  Serial.println();
  Serial.println("Writing to datalog.bin using SdFat functions");
  FsFile myfile = SD.sdfs.open("datalog.bin", O_WRITE | O_CREAT);
  unsigned int len = myfile.fileSize();
  Serial.print("datalog.bin started with ");
  Serial.print(len);
  Serial.println(" bytes");
  if (len > 0) {
    // reduce the file to zero if it already had data
    myfile.truncate();
  }

  myfile.print("Just some test data written to the file (by SdFat functions)");
  myfile.write(0);
  myfile.close();

  // You can also use regular SD functions, even to access the same file.  Just
  // remember to close the SdFat FsFile before opening as a regular SD File.
  //
  Serial.println();
  Serial.println("Reading to datalog.bin using SD functions");
  File f = SD.open("datalog.bin");
  if (f) {
    char mybuffer[100];
    int index = 0;
    while (f.available()) {
      char c = f.read();
      mybuffer[index] = c;
      if (c == 0) break;  // end of string
      index = index + 1;
      if (index == 99) break; // buffer full
    }
    mybuffer[index] = 0;
    Serial.print("  Read from file: ");
    Serial.println(mybuffer);
  } else {
    Serial.println("unable to open datalog.bin :(");
  }
  f.close();

  // When mixing SD and SdFat file access, remember for writing that
  // SD defaults to appending if you open with FILE_WRITE.  You must
  // use FILE_WRITE_BEGIN if you wish to overwrite the file from the
  // start.  With SdFat, O_WRITE or O_RDWR starts overwriting from the
  // beginning.  You must add O_AT_END if you wish to appead.
}

void loop()
{
  // nothing happens after setup finishes.
}


void printSpaces(int num) {
  for (int i = 0; i < num; i++) {
    Serial.print(" ");
  }
}
and now if you open the file in Notepad or in notepad++ in my case:
Code:
Just some test data written to the file (by SdFat functions) null

By definition pre-allocation:
Allocate contiguous clusters to an empty file.
so basically it allocates 40mb of space on the sd card for the data file whether you have valid data in it our - it unitialized data so what ever was previously on the card (if you didn't do a format) you are going to be seeing besides the data you wrote.

Hope that explains it.
 
When the file is closed in the example, it is still 40MB long. You can reduce the file's length to whatever has been written so far by using the truncate() function.
Changing this:
Code:
  myfile.write(0);
  myfile.close();
to this:
Code:
  myfile.write(0);
  myfile.truncate();
  myfile.close();
will result in a file that is less than 1kB long instead of being 40MB.

Pete
 
My guess is that its due to preallocating 40mb of space. So lets try deleting the code for pre-allocating space on the SD card.

You are absolutely correct, I see the same result. Thank you very much for the clarification!

You can reduce the file's length to whatever has been written so far by using the truncate() function.

This is another good solution that I may use later, thank you!
 
Last edited:
Status
Not open for further replies.
Back
Top