File abstraction and SdFat integration

@KrisKasprak
Maybe you are mixing up file-explorer and file-system?
IFAIK, each file has mainly the following attributes
-Filename
-Attributes (hidden, read only etc)
-file size
-create time
-modify time
-storage pointers
which are stored in the directory structure of the FAT
what you see in powerpoint file, pictures, wav files, etc are in header of file and it should be the task of the file-explorer to read and display this info.
for your data logger you could use additional fields in wav header, or picture format to take advantage of existing file-explorers
 
The files i was using as a goal are files that use Microsoft’s “structured storage” document format, PPT files show number of slides for example. Installs of those products have some methodology for the OS to leverage the document properties. This capability is way past the structure of a simple text file.

Thanks for chiming in.
 
I encountered the following 'reality' while trying to integrate MTP responder to new SdFat Integration

So far, MTP responder uses an index file that is stored on disk but relies on data being kept in memory
SdFat-beta allows this, while interfacing to SdFat-beta via SD this does not work as can be seen from following test program

Code:
#define TEST 0
#if TEST==0
  #include "SdFat.h"
  SdFs SD;
  FsFile file;
#else
  #include "SD.h"
  File file;
#endif

void setup() {
  // put your setup code here, to run once:
  uint32_t index=123456;
  while(!Serial);
  #if TEST==0
    SD.begin(SdioConfig(FIFO_SDIO));
  #else
    SD.begin();
  #endif
  SD.remove("test1.dat");
  file=SD.open("test1.dat",FILE_WRITE);

  Serial.println(index);
  file.write(&index,4);
  [COLOR="#FFA500"]file.flush();[/COLOR]

  index=0;
  file.seek(0);
  file.read(&index,4);
  Serial.println(index);

  file.close();
}

void loop() {
  // put your main code here, to run repeatedly:

}

While I can implement a work around, it would be interesting what generated the difference behaviour.
Obviously if there is a update in SD /Fs, this would be great.

Edit: inserting file.flush(); does not help
 
Last edited:
While I can implement a work around, it would be interesting what generated the difference behaviour.

Maybe you meant to write this?

Code:
  #if TEST==0
    SD.begin(SdioConfig(FIFO_SDIO));
  #else
    SD.begin([COLOR="#4B0082"]BUILTIN_SDCARD[/COLOR]);
  #endif

If you call SD.begin() with no input, it defaults to a SPI connection with CS on pin 10.
 
Maybe you meant to write this?

Code:
  #if TEST==0
    SD.begin(SdioConfig(FIFO_SDIO));
  #else
    SD.begin([COLOR="#4B0082"]BUILTIN_SDCARD[/COLOR]);
  #endif

If you call SD.begin() with no input, it defaults to a SPI connection with CS on pin 10.

I changed that, but still no good
 
@WMXZ
Have been looking at porting MTP to handle Flash as well so decided to give your sketch a shot using QSPIFlash:
Code:
#include <LittleFS.h>

LittleFS_QSPIFlash myfs;
char buf[200000];
File file1;

void setup() {
  pinMode(13, OUTPUT);
  digitalWrite(13, HIGH);
  while (!Serial) ; // wait
  Serial.println("LittleFS Test"); delay(5);
  if (!myfs.begin()) {
    Serial.println("Error starting ramdisk");
    while (1) ;
  }
  Serial.println("started");

  printDirectory();


}


void loop() {
}
Looks like it at least works for QSPIFlash:
Code:
LittleFS Test

QSPI flash begin

Flash ID: EF 40 18
Flash size is 16.00 Mbyte
attemping to mount existing media
success
started

printDirectory
--------------
FILE	9px_0001.bmp		307254
FILE	9px_0003.bmp		307254
FILE	PRINTOUTPUT2.txt		3780
FILE	file1		176
FILE	file2		224
FILE	file3		224

123456
123456

printDirectory
--------------
FILE	9px_0001.bmp		307254
FILE	9px_0002.bmp		307254
FILE	9px_0003.bmp		307254
FILE	PRINTOUTPUT1.txt		3768
FILE	file10		144
FILE	file1		176
FILE	file20		128
FILE	file2		224
FILE	file30		112
FILE	file3		224
DIR	structuredData / 
	FILE	logger.txt		4320
FILE	test1.dat		4
 
Hi!
This does not work:

Code:
FD = SD.open(Path, FILE_WRITE_BEGIN);

but these are OK:
Code:
 FD = SD.open(Path, FILE_WRITE);
 FD = SD.open(Path, FILE_READ);

Is it only a small glitch or according to design?

Regards,
Göran
 
This does not work:

Normally I ask for a complete program to reproduce the problem. But this seemed so trivial that I just wrote a little program to test.

Code:
#include <SD.h>

const int chipSelect = 10;

void setup() {
  while (!Serial) ; // wait for serial port to connect.

  Serial.print("Initializing SD card...");
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    while (1) {
    }
  }
  Serial.println("card initialized.");

  const char *path = "myfile.txt";
  File f;

  f = SD.open(path, FILE_WRITE);
  int len = f.size();
  Serial.print("file is ");
  Serial.print(len);
  Serial.println(" bytes");
  if (len < 80) {
    Serial.println("adding to file");
    f.print("more text");
  }
  f.close();

  Serial.println("writing at beginning of file");
  f = SD.open(path, FILE_WRITE_BEGIN);
  f.print("BeginWrite");
  f.close();

  Serial.println("reading file");
  f = SD.open(path, FILE_READ);
  if (f) {
    char buf[100];
    int len = f.read(buf, sizeof(buf) - 1);
    buf[len] = 0;
    Serial.println(buf);
  }

}

void loop() {
}

After I run this 4 times, the file does have the "BeginWrite" as its first bytes and the effect of the file growing each time I run it can be seen.

sc.png

Seems to work fine. I tested with 1.54-beta6.

If you still believe there is a bug here, please show a complete program which reproduces the problem.
 
Hi!
This does not work:

Code:
FD = SD.open(Path, FILE_WRITE_BEGIN);

but these are OK:
Code:
 FD = SD.open(Path, FILE_WRITE);
 FD = SD.open(Path, FILE_READ);

Is it only a small glitch or according to design?

Regards,
Göran

Sorry not sure what the glitch is? Does not start writing at start? When you close file does not remove data after last write? Probably by design. If you want to truncate you probably have to explicitly call member methods to do so.
 
Hi again!

It seems to be a limit on the length of a directory name to be less than 10 characters.

This code:

Code:
#include <strings.h>
#include <SD.h>
#include <SPI.h>

bool CreateDir(char *Path)
{
  Serial.print("Creating directory: ");  
  Serial.print(Path);   
  Serial.print("     PathLen: ");  
  Serial.print(strlen(Path), DEC); 
      
  if(SD.mkdir(Path))
    Serial.println(" --> Success!");  
  else
    Serial.println(" --> No success!");
  
}

void setup() 
{
    Serial.begin(115200);
    delay(100);
    
    while(!Serial);
    delay(100); 
    
    SD.begin(BUILTIN_SDCARD);
    delay(100);
     

   CreateDir("/ABCDEFGH");
   CreateDir("/ABCDEFGH/12345678");      
   CreateDir("/ABCDEFGH/12345678/abcdefgh");  

   CreateDir("/ABCDEF");
   CreateDir("/ABCDEF/1234567890");      
   CreateDir("/ABCDEF/1234567890/abcdefgh");         

   CreateDir("/ABCDEFGHI");
   CreateDir("/ABCDEFGHI/12345678");      
   CreateDir("/ABCDEFGHI/12345678/abcdefgh");               

}

void loop() {
  // put your main code here, to run repeatedly:

}

produces this output:

Code:
Creating directory: /ABCDEFGH     PathLen: 9 --> Success!                                                
Creating directory: /ABCDEFGH/12345678     PathLen: 18 --> Success!                                      
Creating directory: /ABCDEFGH/12345678/abcdefgh     PathLen: 27 --> Success!                             
Creating directory: /ABCDEF     PathLen: 7 --> Success!                                                  
Creating directory: /ABCDEF/1234567890     PathLen: 18 --> No success!                                   
Creating directory: /ABCDEF/1234567890/abcdefgh     PathLen: 27 --> No success!                          
Creating directory: /ABCDEFGHI     PathLen: 10 --> No success!                                           
Creating directory: /ABCDEFGHI/12345678     PathLen: 19 --> No success!                                  
Creating directory: /ABCDEFGHI/12345678/abcdefgh     PathLen: 28 --> No success!


Regards,
Göran
 
Yes, I didn't properly state the problem in the previous message.
However, the sketch don't compile when 'FILE_WRITE_BEGIN' is referenced.

Please see the included 'Datalogger' example:

Code:
/*
  SD card datalogger
 
 This example shows how to log data from three analog sensors
 to an SD card using the SD library.
 	
 The circuit:
 * analog sensors on analog ins 0, 1, and 2
 * SD card attached to SPI bus as follows:
 ** MOSI - pin 11, pin 7 on Teensy with audio board
 ** MISO - pin 12
 ** CLK - pin 13, pin 14 on Teensy with audio board
 ** CS - pin 4,  pin 10 on Teensy with audio board
 
 created  24 Nov 2010
 modified 9 Apr 2012
 by Tom Igoe
 
 This example code is in the public domain.
 */

#include <SD.h>
// #include <SPI.h>

// On the Ethernet Shield, CS is pin 4. Note that even if it's not
// used as the CS pin, the hardware CS pin (10 on most Arduino boards,
// 53 on the Mega) must be left as an output or the SD library
// functions will not work.

// change this to match your SD shield or module;
// Arduino Ethernet shield: pin 4
// Adafruit SD shields and modules: pin 10
// Sparkfun SD shield: pin 8
// Teensy audio board: pin 10
// Teensy 3.5 & 3.6 & 4.1 on-board: BUILTIN_SDCARD
// Wiz820+SD board: pin 4
// Teensy 2.0: pin 0
// Teensy++ 2.0: pin 20
const int chipSelect = 4,
          justTest   =  FILE_WRITE_BEGIN;   // Will not compile.

void setup()
{
  //UNCOMMENT THESE TWO LINES FOR TEENSY AUDIO BOARD:
  //SPI.setMOSI(7);  // Audio shield has MOSI on pin 7
  //SPI.setSCK(14);  // Audio shield has SCK on pin 14

  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect.
  }


  Serial.print("Initializing SD card...");

  // see if the card is present and can be initialized:
  if (!SD.begin(BUILTIN_SDCARD)) {
    Serial.println("Card failed, or not present");
    while (1) {
      // No SD card, so don't do anything more - stay stuck here
    }
  }
  Serial.println("card initialized.");
}

void loop()
{
  // make a string for assembling the data to log:
  String dataString = "";

  // read three sensors and append to the string:
  for (int analogPin = 0; analogPin < 3; analogPin++) {
    int sensor = analogRead(analogPin);
    dataString += String(sensor);
    if (analogPin < 2) {
      dataString += ",";
    }
  }

  // open the file.
  File dataFile = SD.open("datalog.txt", FILE_WRITE);

  // if the file is available, write to it:
  if (dataFile) {
    dataFile.println(dataString);
    dataFile.close();
    // print to the serial port too:
    Serial.println(dataString);
  } else {
    // if the file isn't open, pop up an error:
    Serial.println("error opening datalog.txt");
  }
  delay(100); // run at a reasonable not-too-fast speed
}

A listing from the failed compilation is attached.

I've installed the 'SD-Juse_Use_SdFat'-library, but it seems not to be used.

Are there any better library alternatives?

Regards,
Göran
 

Attachments

  • CompOut.txt
    20.5 KB · Views: 67
the symbol is defined in cores/teensy(3/4)/FS.h
can you check its existence, or if you have a second FS.h on your system?
maybe in "SD-Juse_Use_SfFat"?
 
From your error log on msg #63:

Code:
/home/linuxgoran/Downloads/arduino-1.8.13-linux64/arduino-1.8.13/hardware/teensy/../tools/arm/bin/arm-none-eabi-g++ -c
 -O1 -g -Wall -ffunction-sections -fdata-sections -nostdlib -MMD -std=gnu++14 -fno-exceptions -fpermissive -fno-rtti
-fno-threadsafe-statics -felide-constructors -Wno-error=narrowing -mthumb -mcpu=cortex-m7 -mfloat-abi=hard -mfpu=fpv5-d16
 -D__IMXRT1062__ [B][COLOR="#B22222"]-DTEENSYDUINO=153[/COLOR][/B] -DARDUINO=10813 -DARDUINO_TEENSY41 -DF_CPU=450000000
 -DUSB_SERIAL -DLAYOUT_US_ENGLISH -I/tmp/arduino_build_638032/pch
-I/home/linuxgoran/Downloads/arduino-1.8.13-linux64/arduino-1.8.13/hardware/teensy/avr/cores/teensy4
-I/home/linuxgoran/Downloads/arduino-1.8.13-linux64/arduino-1.8.13/hardware/teensy/avr/libraries/SD
-I/home/linuxgoran/Downloads/arduino-1.8.13-linux64/arduino-1.8.13/hardware/teensy/avr/libraries/SPI
/tmp/arduino_build_638032/sketch/Datalogger.ino.cpp -o /tmp/arduino_build_638032/sketch/Datalogger.ino.cpp.o
Datalogger:40: error: 'FILE_WRITE_BEGIN' was not declared in this scope
           justTest =  FILE_WRITE_BEGIN;   // Will not compile.
                       ^

Looks like you have Teensyduino 1.53 installed.

For FILE_WRITE_BEGIN and all the other new SD / SdFat stuff, you need to install 1.54-beta6.

https://forum.pjrc.com/threads/66182-Teensyduino-1-54-Beta-6

Just installing the new SD lib and SdFat is not enough. You also need the updated FS.h in the core library. Please, do yourself a favor and install 1.54-beta6 so you have the complete set of files. If you try to install it all piecemeal from github, odds of getting everything exactly right are slim. We have the installer to make getting the complete and coherent set of libraries as easy as possible.
 
Back
Top