painfully slow SD file copy code / any suggestions?

Status
Not open for further replies.

martianredskies

Well-known member
i've incorporated this handy commandline library into my teensy 3.6 project as a means of sending serial commands.

https://create.arduino.cc/projecthub/mikefarr/simple-command-line-interface-4f0a3f

To begin, I just wanted to give myself basic file system control of the built-in SD; delete, rename, mkdir, rmdir, copy, read etc.
It works, however copying a file from the SD card to a new file/filename is painfully slow, well over 30seconds for a 32kb file.

The code i am using is;

Code:
//copy file
    if (strcmp(result, copyID) == 0){
      File myFileIn = SD.open(secondWord, FILE_READ);
      File myFileOut = SD.open(thirdWord, FILE_WRITE);
      
      print2("copy file> ", secondWord);
      print2("to file> ", thirdWord);
      
      while (myFileIn.available()) {
        myFileOut.write(myFileIn.read());
      }
      myFileIn.close();
      myFileOut.close();
      commandLineSwitch = 2;
    }

and example of the sequence i would send to copy a file : abcdefg (*some header) copy (*command) filenameA.dat (*secondWord) filenameB.dat (*thirdWord);

Any help would be appreciated.
 
Try using this Serial.readBytes(buffer, length)

something like : myFileOut.write(myFileIn.readBytes(buffer, myFileIn.available()) );

That may need to end with :: , myFileIn.available());

That will do a transfer of a buffer of data not a byte at a time.

It may need a delay(1); in the while(myFileIn.available()) to make sure any pending packets arrive before exiting.
 
Instead of reading/writing a single byte at a time, try reading/writing a 512 byte block

Indeed if p#2 quick change helps measurably - then reading and buffering into a buffer of 512bytes to write blocks would be the next step
 
Try using this Serial.readBytes(buffer, length)

something like : myFileOut.write(myFileIn.readBytes(buffer, myFileIn.available()) );

That may need to end with :: , myFileIn.available());

That will do a transfer of a buffer of data not a byte at a time.

It may need a delay(1); in the while(myFileIn.available()) to make sure any pending packets arrive before exiting.

Apparently reading 1st file into a buffer and then writing from buffer to the 2nd file is monumentally faster. With the code below its much faster, would your approach make more sense? I'm a coding amateur so always want to learn the right way to do things.

Code:
//copy file
    if (strcmp(result, copyID) == 0){
      File myFileIn = SD.open(secondWord, FILE_READ);
      int filesize = myFileIn.size();
      Serial.println(filesize,"bytes");
      uint16_t fileBuffer[filesize];
      File myFileOut = SD.open(thirdWord, O_WRITE | O_CREAT| O_APPEND);
      //delay(500);
      
      //print2("copy file> ", secondWord);
      //print2("to file> ", thirdWord);

      for (int i = 0; i < filesize; i++){
        fileBuffer[i] = myFileIn.read();
        Serial.println(i);
      }

      for (int i = 0; i < filesize; i++){
        myFileOut.write(fileBuffer[i]);
        Serial.println(i);
      }

      myFileIn.close();
      myFileOut.close();
      commandLineSwitch = 2;
    }
 
Indeed if that is trying to stack allocate the size of the full file that is not the best plan.

Creating a 512 byte buffer - or maybe 4096 bytes - and filling and writing it in turn would be best.

Also doing readBytes() as in p#2 would be more efficient than 1 byte reads.
 
Indeed if that is trying to stack allocate the size of the full file that is not the best plan.

Creating a 512 byte buffer - or maybe 4096 bytes - and filling and writing it in turn would be best.

Also doing readBytes() as in p#2 would be more efficient than 1 byte reads.

I can't get it to copy with that method. This is how i tried to implement;

Code:
//copy file
    if (strcmp(result, copyID) == 0){
      File myFileIn = SD.open(secondWord, FILE_READ);
      int filesize = myFileIn.size();
      Serial.println(filesize,"bytes");
      byte fileBuffer[512];
      File myFileOut = SD.open(thirdWord, O_WRITE | O_CREAT| O_APPEND);
      //delay(500);
      
      print2("copy file> ", secondWord);
      print2("to file> ", thirdWord);
      delay(1000);

      while (myFileIn.available()) {
        myFileOut.write(myFileIn.readBytes(fileBuffer, filesize) );
        //myFileOut.write(myFileIn.read());
      }

      myFileIn.close();
      myFileOut.close();
      commandLineSwitch = 2;
    }
 
loop should be perhaps:
Code:
      while (myFileIn.available()) {
        myFileOut.write(myFileIn.readBytes(fileBuffer, [B][U]myFileIn.available()[/U][/B]) );
        //myFileOut.write(myFileIn.read());
      }
 
forgive my ignorance, could you alter the example code to demonstrate this?

Posting a useable sketch is difficult, i've got everything in tabs in arduinoIDE.
 
first step - correcting post #2

Code:
//copy file
    if (strcmp(result, copyID) == 0){
      File myFileIn = SD.open(secondWord, FILE_READ);
      int filesize = myFileIn.size();
      Serial.println(filesize,"bytes");
#define FILE_BUFFER 512
      byte fileBuffer[FILE_BUFFER];
      File myFileOut = SD.open(thirdWord, O_WRITE | O_CREAT| O_APPEND);
      //delay(500);
      
      print2("copy file> ", secondWord);
      print2("to file> ", thirdWord);
      delay(1000);

      while (myFileIn.available()) {
        int len = myFileIn.available();
        if ( len > FILE_BUFFER ) len = FILE_BUFFER;
        readBytes(fileBuffer, len)
        myFileOut.write( fileBuffer, len );
        delay(1); // maybe needed to allow USB buffer to arrive
      }

      myFileIn.close();
      myFileOut.close();
      commandLineSwitch = 2;
    }
 
It works! It's aliveeeeeeee!

just corrected;

Code:
readBytes(fileBuffer, len)

to
Code:
myFileIn.readBytes(fileBuffer, len);

speed is light years better! Thanks for your help!
 
Awesome.

glad to help - especially after the half wrong answer in post #2 and #9 before the lights turned on :)

It could be improved with the 512B chunking writes to SD - but that would take more overhead filling that buffer to 512 and writing that between reads. Likely the USB would come in even 64B chunks - but that would be assuming. And if moved to a T_4.0 the faster USB would come in 512B chunks.

That might make the SD writes go faster if exploration was something that seemed fun.
 
Awesome.

glad to help - especially after the half wrong answer in post #2 and #9 before the lights turned on :)

It could be improved with the 512B chunking writes to SD - but that would take more overhead filling that buffer to 512 and writing that between reads. Likely the USB would come in even 64B chunks - but that would be assuming. And if moved to a T_4.0 the faster USB would come in 512B chunks.

That might make the SD writes go faster if exploration was something that seemed fun.

Right now i think it's "good enough" to move forward. The largest files i'll be dealing with are probably 1mb max, so speed isn't crucial for any internal file managment. Anything going over serial will be sent via character arrays and the raw data streamed straight into a SD card file with code like;

Code:
myFile.println(msb_lsb, HEX); //write value to SD card

i was happy with the speeds.

You've been a great help, thanks for your patience with me.
 
Status
Not open for further replies.
Back
Top