SD card - setting file pointer for read

Bodger62

Member
I have a file on an SD card with the following format, file can be up to 2Gbytes

16:44:04:5748 Rx 1 0x00200001 x 8 fc ff 00 00 01 00 00 00
16:44:04:5750 Rx 1 0x00260001 x 8 24 00 d6 ff e0 fc 00 00
16:44:04:5767 Rx 1 0x00800002 x 8 68 c5 68 c5 68 c5 68 c5
.
.
.

which I need to convert to a different format before saving it
into a different file on the SD card again.

When I tried to write the data to the second file when the first file was open
it wouldn't work, as no data was written.

I plan to read 64 lines into a ram buffer and then store it
to the SD card.

The problem I have is that when I open the SD card to read more
data the file pointer is at the start of the file.

Is there any way to set the file pointer at the last position I read
the data from, without reading in every line up to that position.

I need something like

// open the file.
myFile = SD.open(fn, FILE_READ);

fseek(myFile, (64 * 32), 0); //64 lines each line is 32 bytes

The code currently reads one line before writing it and it
is too slow to use.

Example: To read data from line 65 I have to read in the preceding
64 lines and then read in the data I need to convert.
 

Attachments

  • Alternative_Test.ino
    7.6 KB · Views: 12
I have a file on an SD card with the following format, file can be up to 2Gbytes

16:44:04:5748 Rx 1 0x00200001 x 8 fc ff 00 00 01 00 00 00
16:44:04:5750 Rx 1 0x00260001 x 8 24 00 d6 ff e0 fc 00 00
16:44:04:5767 Rx 1 0x00800002 x 8 68 c5 68 c5 68 c5 68 c5
.
.
.

which I need to convert to a different format before saving it
into a different file on the SD card again.

When I tried to write the data to the second file when the first file was open
it wouldn't work, as no data was written.
This post bothered me in that I was pretty sure that I have been able to have one file open for reading and another open for writing and was able to do a read-modify-write operation. I couldn't find that code, so I wrote a test program that demonstrates that reading from one file writing to another while both are open works as expected. I did find that reading 32 bytes, modifying them and writing the new 32 bytes goes very slowly! There is apparently a lot of overhead when SDFat switches from one file structure to another and sets up the new SD hardware register values. To keep the update time more reasonable, I did the read-modify-write operation on blocks of 1000 32-byte lines. Even with 32,000-byte blocks, read-modify-write takes about 4 times longer than just writing the original data.

Here's the test program:

Code:
// Test to check for bugs reading from one file and
// writing to a second file.
// MJB 8/28/2024
#include <Arduino.h>
#include <SD.h>
#include <MTP_Teensy.h>
#include "TimeLib.h"

const char compileTime[] = " Compiled on " __DATE__ " " __TIME__;
void setup() {
   elapsedMillis Leptontimer;
  // Wait for Serial connection--but only for 1/2 second
  Serial.begin(9600);
  //wait 2 seconds for serial to start up
  delay(2000);

  Serial.printf("\n\nT4.1 File Read/Write Test %s\n", compileTime);
  Serial.println("Type '?' to list commands.");
  InitSD();  // Starts up SD Card and MTP
  delay(1000);

}

void loop() {
  char ch;
  if(Serial.available()){
    ch = Serial.read();
    switch(ch){
      case '?' :
        Serial.println("1: write the original test file (TFile1.txt).\n");
        Serial.println("2: Read TFile1, modify data and write to TFile2.\n");
        Serial.println("3: Show the first 20 lines of TFile2.\n");
        Serial.println("d: Show SD Card directory.\n");
        Serial.println("m: Update MTP so you can see new files on your PC.\n");
        Serial.println("r: Remove TFile1.txt and TFile2.txt if they exist.\n");
      break;
      case '1' : WriteFile1();
      break;
      case '2' : Read1Write2();
      break;
      case '3' : Show20();
      break;     
      case 'd' : SDDir();
      break;
      case 'm' : MTPReset();
      break; 
      case 'r' : RemoveFiles();
      break;
      default: Serial.println("Oops, Invalid command.  Type '?' to see valid commands.\n");

    }
  }
  MTP.loop();
}

// Write a million 32-character lines to file TFile1.txt
void WriteFile1(void){
FsFile tfile1;  // Used FsFile so I can pre-allocate space
uint32_t stlen;
uint32_t i,ltime;
elapsedMillis ftimer;
// String is 32-characters.  Terminating 0x00 isn't sent.
// String ends with line feed to display nicely.
const char *string1 = {"OLD: ABCDEFGHIJKLMNOPQRSTUVWXYZ\n"};
  stlen = strlen(string1);
  Serial.printf("Writing 1,000,000 copies of: string1 with length %lu bytes\n", stlen);
  tfile1 = SD.sdfs.open("TFile1.txt", O_WRITE | O_CREAT);
  if(tfile1.preAllocate(stlen * 1000000)){
    Serial.println("Preallocation successful");
  } else Serial.println("Preallocation failed!");
  delay(1);
  // This simple write routine is NOT optimized for speed
  for(i=0; i< 1000000; i++){
    tfile1.write(string1, stlen);
    if((i%100000) == 0){ // show progress ever 100,000 lines to keep user happy
      ltime = ftimer;
      Serial.printf("Line %6lu  at %6.3f\n", i,(float)ltime/1000.0);
    }
  }
  tfile1.truncate(); //Truncate in case we don't write all we preallocated.
  tfile1.close();
}

// Remove files TFile1.txt and TFile2.txt if they exist.
// Writing to an existing file is much slower, as pre-allocation fails.
void RemoveFiles(void){
  if(SD.sdfs.exists("TFile1.txt")) SD.sdfs.remove("TFile1.txt");
  if(SD.sdfs.exists("TFile2.txt")) SD.sdfs.remove("TFile2.txt");
  Serial.println("TFile1.txt and TFile2.txt removed if they existed\n");
}

// Read a million 32-character lines from file TFile1.txt
// Modify the data and write to TFile2.txt
#define NUMLINES 1000 // Number of lines to read and write each time
char block1[32 * NUMLINES];

void Read1Write2(void){
FsFile tfile1, tfile2;  // Used FsFile so I can pre-allocate space
uint32_t stlen, blklen;
uint32_t i,j,lnstart,ltime;

elapsedMillis ftimer;

  stlen = 32;
  blklen = stlen * NUMLINES;

  Serial.println("reading 1,000,000 copies string1, modifying, and writing to TFile2.txt\n");
  Serial.printf("Reading and writing in blocks of %lu lines.\n", NUMLINES);
  tfile1 = SD.sdfs.open("TFile1.txt", O_READ);
  tfile2 = SD.sdfs.open("TFile2.txt", O_WRITE | O_CREAT);
  if(tfile2.preAllocate(stlen * 1000000)){
    Serial.println("Preallocation of TFile2.txt successful");
  } else Serial.println("Preallocation failed!");
  delay(1);
 
  for(i=0; i< 1000000/NUMLINES; i++){
    tfile1.read(block1, blklen); // read the whole block
    // change first three letters from 'OLD' to 'NEW' for each string in block
    for(j= 0; j< NUMLINES; j++ ){
      lnstart = j * stlen;
      block1[lnstart] = 'N'; block1[lnstart+1] = 'E'; block1[lnstart+2] = 'W';
    }
    delayMicroseconds(2);
    tfile2.write(block1, blklen);

    if((i%100) == 0){ // show progress every 100000 lines to keep user happy
      ltime = ftimer;
      Serial.printf("Line %6lu  at %6.3f\n", i*NUMLINES,(float)ltime/1000.0);
    }
  }
  tfile1.close();
  tfile2.truncate(); //Truncate in case we don't write all we preallocated.
  tfile2.close();
}


// Show the first 20 lines of TFile2.txt
void Show20(void){
  FsFile tfile2; 
uint32_t stlen;
uint32_t i;
// String is 32-characters.  Terminating 0x00 isn't sent.
// String ends with line feed to display nicely.
char string1[32];
  stlen = 32;
  Serial.println("Showing first 20 lines of TFile2.txt\n");
  tfile2 = SD.sdfs.open("TFile2.txt", O_READ);
  for(i= 0; i< 20; i++){
    tfile2.read(string1, stlen);
    Serial.println(string1);
  }
  Serial.println();
}

time_t get_TeensyTime(void) {
  return Teensy3Clock.get();
}

/* ------------------------------------------------------------------------------
 User-provided date time callback function.
 See SdFile::dateTimeCallback() for usage.
------------------------------------------------------------------------------*/
void dateTime(uint16_t *date, uint16_t *time) {
  // use the year(), month() day() etc. functions from timelib

  // return date using FAT_DATE macro to format fields
  *date = FAT_DATE(year(), month(), day());

  // return time using FAT_TIME macro to format fields
  *time = FAT_TIME(hour(), minute(), second());
}


void InitSD(void){ 
  if (SD.begin(BUILTIN_SDCARD)) {
    setSyncProvider(get_TeensyTime);  // helps put time into file directory data
    // set date time callback function
    SdFile::dateTimeCallback(dateTime);
    MTP.addFilesystem(SD, "SD Card");
    Serial.println("Added SD card using built in SDIO");
  } else {
    Serial.println("No SD Card");
  }
  // mandatory to begin the MTP session.
  MTP.begin();
}

void MTPReset(void){
  MTP.send_DeviceResetEvent();
  Serial.println("Requested PC to reset MTP Directory");
}

void SDDir(void){
  Serial.println("SD Card Directory");
  SD.sdfs.ls(LS_DATE | LS_SIZE);
  Serial.println();
}


Here are the results of a test run. Times are in seconds.

Code:
T4.1 File Read/Write Test  Compiled on Aug 28 2024 17:19:38
Type '?' to list commands.
Add **SDClass** file system
Added SD card using built in SDIO
1: write the original test file (TFile1.txt).
2: Read TFile1, modify data and write to TFile2.
3: Show the first 20 lines of TFile2.
d: Show SD Card directory.
m: Update MTP so you can see new files on your PC.
r: Remove TFile1.txt and TFile2.txt if they exist.
<r>
TFile1.txt and TFile2.txt removed if they existed
<1>
Writing 1,000,000 copies of: string1 with length 32 bytes
Preallocation successful
Line      0  at  0.014
Line 100000  at  0.172
Line 200000  at  0.336
Line 300000  at  0.501
Line 400000  at  0.643
Line 500000  at  0.783
Line 600000  at  0.935
Line 700000  at  1.077
Line 800000  at  1.219
Line 900000  at  1.359
<2>
reading 1,000,000 copies string1, modifying, and writing to TFile2.txt
Reading and writing in blocks of 1000 lines.
Preallocation of TFile2.txt successful
Line      0  at  0.017
Line 100000  at  0.617
Line 200000  at  1.156
Line 300000  at  1.671
Line 400000  at  2.178
Line 500000  at  3.095
Line 600000  at  3.694
Line 700000  at  4.204
Line 800000  at  4.717
Line 900000  at  5.233
<3>
Showing first 20 lines of TFile2.txt
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ
NEW: ABCDEFGHIJKLMNOPQRSTUVWXYZ

SD Card Directory
2024-08-28 17:15            0 mtpindex.dat
2024-08-28 17:20     32000000 TFile1.txt
2024-08-28 17:20     32000000 TFile2.txt

Note that the only difference between the original and new data is that the original starts with 'OLD'.
 
Yes the SD library only caches one sector at a time (512 bytes), so if you switch back and forth between concurrent files it improves performance a lot to keep file writes/reads at a multiple of that size.
 
Yes the SD library only caches one sector at a time (512 bytes), so if you switch back and forth between concurrent files it improves performance a lot to keep file writes/reads at a multiple of that size.
Is that one sector per file, or one sector for all open files?
 
Back
Top