IWantYourCookies
Member
Hi, so I am currently trying to write code for my teensy 4.1 that reads all the lines of csv file and outputs them into another csv file. I realised that the csv files I am trying to use are too big for the teensy to store all at once so I made code to go through the csv and read chunks of lines at a time (currently set to 2000). I have written the read part of the code and tried it with smaller files (a few thousand lines) and it works fine but when I try it with the the size file I need (over a million lines) it gets progressively slower and slower at reading the file with every chunk it processes, to the point where it will probably never get through the whole file. I assumed it was a memory issue, which doesn't make sense to me because the only variables being stored are like 4 ints and a string of the data in the chunk of 2000 lines and I replace the data in that string with the new file data every time I process a new 2000 lines, so the memory useage shouldn't be increasing. I added some code from this forum to check my RAM usage and it looks stable but the code still slows down. I am not sure what is happening any help is greatly appreciated, I can't attach csv files here so instead I put the data in a txt file that is attached, and this is not all the data, the real file is just over a million line long but it's just more of what is in the txt file. My code is the following:
Code:
#include <SD.h>
#include <SPI.h>
#include <iostream>
#include <tuple>
#include <vector>
extern unsigned long _heap_start;
extern unsigned long _heap_end;
extern char *__brkval;
uint32_t freeRAMTeensy() {
return (char *)&_heap_end - __brkval;
}
File myFile;
const int SDCard = BUILTIN_SDCARD;
int fileBuf = 2000;
std::tuple<int, int, String> fileRead(int lin, int buf){
myFile = SD.open("data.csv");
//Serial.println("Stage 1.");
String tempString;
int lineStrt = lin;
int lines = 0;
int go = 0;
if(lin == 0){
go = 1;
}
if (myFile) {
while (myFile.available() && (lin-lineStrt)<buf){
String temptempStr = myFile.read();
if(go == 1){
tempString += temptempStr;
if(temptempStr == "10"){
lin++;
//Serial.println(lin);
}
}
if(temptempStr == "10" && go == 0){
lines++;
//Serial.print("lines is: ");
//Serial.println(lines);
if(lines == lin){go = 1;}
}
}
// close the file:
myFile.close();
} else {
// if the file didn't open, print an error:
Serial.println("error opening doc.");
}
return std::make_tuple(lin, lineStrt, tempString);
//Serial.println("Stage 3.");
}
void setup(){
// Open serial communications and wait for port to open:
Serial.begin(9600);
while (!Serial) {
; // wait for serial port to connect.
}
Serial.print("Initializing SD card...");
if (!SD.begin(SDCard)) {
Serial.println("initialization failed!");
return;
}
Serial.println("initialization done.");
if (SD.exists("NewNewdata.csv")) {
// The SD library writes new data to the end of the
// file, so to start a new recording, the old file
// must be deleted before new data is written.
SD.remove("NewNewdata.csv");
}
}
void loop(){
int line = 0;
int StrtLn;
int go = 1;
while(go == 1){
String fileString;
for(int i =0; i <1; i++){
auto[r1, r2, r3] = fileRead(line, fileBuf);
line = r1;
StrtLn = r2;
fileString = r3;
}
if((line-StrtLn) != fileBuf){
go = 0;
//Serial.println("Stage 5.");
}
//Serial.println("Stage 4.");
Serial.println(line);
Serial.println(fileString.length());
//Serial.println(fileString);
Serial.print("RAM is: ");
Serial.println(freeRAMTeensy());
}
Serial.println(StrtLn);
Serial.println(line);
//Serial.println(fileString);
Serial.println("All done.");
while(1) {}
}
Attachments
Last edited: