Teensy Slowing Down as Code Progresses

[IWantYourCookies]

Hi, so I am currently trying to write code for my teensy 4.1 that reads all the lines of csv file and outputs them into another csv file. I realised that the csv files I am trying to use are too big for the teensy to store all at once so I made code to go through the csv and read chunks of lines at a time (currently set to 2000). I have written the read part of the code and tried it with smaller files (a few thousand lines) and it works fine but when I try it with the the size file I need (over a million lines) it gets progressively slower and slower at reading the file with every chunk it processes, to the point where it will probably never get through the whole file. I assumed it was a memory issue, which doesn't make sense to me because the only variables being stored are like 4 ints and a string of the data in the chunk of 2000 lines and I replace the data in that string with the new file data every time I process a new 2000 lines, so the memory useage shouldn't be increasing. I added some code from this forum to check my RAM usage and it looks stable but the code still slows down. I am not sure what is happening any help is greatly appreciated, I can't attach csv files here so instead I put the data in a txt file that is attached, and this is not all the data, the real file is just over a million line long but it's just more of what is in the txt file. My code is the following:

#include <SD.h>
#include <SPI.h>
#include <iostream>
#include <tuple>
#include <vector>

extern unsigned long _heap_start;
extern unsigned long _heap_end;
extern char *__brkval;

uint32_t freeRAMTeensy() {
    return (char *)&_heap_end - __brkval;
}


File myFile;

const int SDCard = BUILTIN_SDCARD;

int fileBuf = 2000;



std::tuple<int, int, String> fileRead(int lin, int buf){

  myFile = SD.open("data.csv");
  //Serial.println("Stage 1.");

  String tempString;
  int lineStrt = lin;
  int lines = 0;

  int go = 0;
  if(lin == 0){
    go = 1;
  }
  if (myFile) {
    while (myFile.available() && (lin-lineStrt)<buf){
      String temptempStr = myFile.read();
      if(go == 1){
        tempString += temptempStr;
        if(temptempStr == "10"){
          lin++;
          //Serial.println(lin);
        }
      }
      if(temptempStr == "10" && go == 0){
        lines++;
        //Serial.print("lines is: ");
        //Serial.println(lines);
        if(lines == lin){go = 1;}
      }
    }
    // close the file:
    myFile.close();
  } else {
      // if the file didn't open, print an error:
    Serial.println("error opening doc.");
  }
  return  std::make_tuple(lin, lineStrt, tempString);
  //Serial.println("Stage 3.");
}



void setup(){
 // Open serial communications and wait for port to open:
  Serial.begin(9600);
   while (!Serial) {
    ; // wait for serial port to connect.
  }


  Serial.print("Initializing SD card...");

  if (!SD.begin(SDCard)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");

  if (SD.exists("NewNewdata.csv")) {
    // The SD library writes new data to the end of the
    // file, so to start a new recording, the old file
    // must be deleted before new data is written.
    SD.remove("NewNewdata.csv");
  }
 
}

void loop(){

  int line = 0;
  int StrtLn;
 
  int go = 1;
  while(go == 1){
    String fileString;
    for(int i =0; i <1; i++){
        auto[r1, r2, r3] = fileRead(line, fileBuf);
        line = r1;
        StrtLn = r2;
        fileString = r3;
    }
    if((line-StrtLn) != fileBuf){
      go = 0;
      //Serial.println("Stage 5.");
    }
    //Serial.println("Stage 4.");
    Serial.println(line);
    Serial.println(fileString.length());
    //Serial.println(fileString);
    Serial.print("RAM is: ");
    Serial.println(freeRAMTeensy());

  }

  Serial.println(StrtLn);
  Serial.println(line);
  //Serial.println(fileString);
  Serial.println("All done.");
  while(1) {}
}

 

[jmarsh]

You keep mentioning "2000 lines" in your post but the code doesn't reference anything like that. What is the code that you posted meant to demonstrate exactly?

 

[IWantYourCookies]

You keep mentioning "2000 lines" in your post but the code doesn't reference anything like that. What is the code that you posted meant to demonstrate exactly?

Sorry that was the wrong code, I updated it, thanks for pointing it out. The 2000 lines is the fileBuf int.

 

[Rezo]

You’re opening and closing the file each time you call readFile - not best practice, and possibly the root cause of your app’s slowdown as it progresses

Open the file once and read/write to it in your loop. When you are finally done, close it.

 

[MarkT]

The time taken to open a file depends on the logarithm of the file size with FAT if I remember correctly, small files live in one block, large files have a hierarchy of blocks to trace through.