Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 17 of 17

Thread: painfully slow SD file copy code / any suggestions?

  1. #1
    Member
    Join Date
    May 2019
    Location
    Brisbane, QLD
    Posts
    71

    painfully slow SD file copy code / any suggestions?

    i've incorporated this handy commandline library into my teensy 3.6 project as a means of sending serial commands.

    https://create.arduino.cc/projecthub...terface-4f0a3f

    To begin, I just wanted to give myself basic file system control of the built-in SD; delete, rename, mkdir, rmdir, copy, read etc.
    It works, however copying a file from the SD card to a new file/filename is painfully slow, well over 30seconds for a 32kb file.

    The code i am using is;

    Code:
    //copy file
        if (strcmp(result, copyID) == 0){
          File myFileIn = SD.open(secondWord, FILE_READ);
          File myFileOut = SD.open(thirdWord, FILE_WRITE);
          
          print2("copy file> ", secondWord);
          print2("to file> ", thirdWord);
          
          while (myFileIn.available()) {
            myFileOut.write(myFileIn.read());
          }
          myFileIn.close();
          myFileOut.close();
          commandLineSwitch = 2;
        }
    and example of the sequence i would send to copy a file : abcdefg (*some header) copy (*command) filenameA.dat (*secondWord) filenameB.dat (*thirdWord);

    Any help would be appreciated.

  2. #2
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    10,946
    Try using this Serial.readBytes(buffer, length)

    something like : myFileOut.write(myFileIn.readBytes(buffer, myFileIn.available()) );

    That may need to end with :: , myFileIn.available());

    That will do a transfer of a buffer of data not a byte at a time.

    It may need a delay(1); in the while(myFileIn.available()) to make sure any pending packets arrive before exiting.

  3. #3
    Junior Member
    Join Date
    Aug 2018
    Location
    Brisbane, Australia
    Posts
    13
    Instead of reading/writing a single byte at a time, try reading/writing a 512 byte block

  4. #4
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    10,946
    Quote Originally Posted by thebigg View Post
    Instead of reading/writing a single byte at a time, try reading/writing a 512 byte block
    Indeed if p#2 quick change helps measurably - then reading and buffering into a buffer of 512bytes to write blocks would be the next step

  5. #5
    Member
    Join Date
    May 2019
    Location
    Brisbane, QLD
    Posts
    71
    Quote Originally Posted by defragster View Post
    Try using this Serial.readBytes(buffer, length)

    something like : myFileOut.write(myFileIn.readBytes(buffer, myFileIn.available()) );

    That may need to end with :: , myFileIn.available());

    That will do a transfer of a buffer of data not a byte at a time.

    It may need a delay(1); in the while(myFileIn.available()) to make sure any pending packets arrive before exiting.
    Apparently reading 1st file into a buffer and then writing from buffer to the 2nd file is monumentally faster. With the code below its much faster, would your approach make more sense? I'm a coding amateur so always want to learn the right way to do things.

    Code:
    //copy file
        if (strcmp(result, copyID) == 0){
          File myFileIn = SD.open(secondWord, FILE_READ);
          int filesize = myFileIn.size();
          Serial.println(filesize,"bytes");
          uint16_t fileBuffer[filesize];
          File myFileOut = SD.open(thirdWord, O_WRITE | O_CREAT| O_APPEND);
          //delay(500);
          
          //print2("copy file> ", secondWord);
          //print2("to file> ", thirdWord);
    
          for (int i = 0; i < filesize; i++){
            fileBuffer[i] = myFileIn.read();
            Serial.println(i);
          }
    
          for (int i = 0; i < filesize; i++){
            myFileOut.write(fileBuffer[i]);
            Serial.println(i);
          }
    
          myFileIn.close();
          myFileOut.close();
          commandLineSwitch = 2;
        }

  6. #6
    Member
    Join Date
    May 2019
    Location
    Brisbane, QLD
    Posts
    71
    Well, it worked good for a 32kb file, for a 128kb file it crashes the teensy. I guess i'm grabbing more memory than available.

  7. #7
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    10,946
    Indeed if that is trying to stack allocate the size of the full file that is not the best plan.

    Creating a 512 byte buffer - or maybe 4096 bytes - and filling and writing it in turn would be best.

    Also doing readBytes() as in p#2 would be more efficient than 1 byte reads.

  8. #8
    Member
    Join Date
    May 2019
    Location
    Brisbane, QLD
    Posts
    71
    Quote Originally Posted by defragster View Post
    Indeed if that is trying to stack allocate the size of the full file that is not the best plan.

    Creating a 512 byte buffer - or maybe 4096 bytes - and filling and writing it in turn would be best.

    Also doing readBytes() as in p#2 would be more efficient than 1 byte reads.
    I can't get it to copy with that method. This is how i tried to implement;

    Code:
    //copy file
        if (strcmp(result, copyID) == 0){
          File myFileIn = SD.open(secondWord, FILE_READ);
          int filesize = myFileIn.size();
          Serial.println(filesize,"bytes");
          byte fileBuffer[512];
          File myFileOut = SD.open(thirdWord, O_WRITE | O_CREAT| O_APPEND);
          //delay(500);
          
          print2("copy file> ", secondWord);
          print2("to file> ", thirdWord);
          delay(1000);
    
          while (myFileIn.available()) {
            myFileOut.write(myFileIn.readBytes(fileBuffer, filesize) );
            //myFileOut.write(myFileIn.read());
          }
    
          myFileIn.close();
          myFileOut.close();
          commandLineSwitch = 2;
        }

  9. #9
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    10,946
    loop should be perhaps:
    Code:
          while (myFileIn.available()) {
            myFileOut.write(myFileIn.readBytes(fileBuffer, myFileIn.available()) );
            //myFileOut.write(myFileIn.read());
          }

  10. #10
    Member
    Join Date
    May 2019
    Location
    Brisbane, QLD
    Posts
    71
    Same result unfortunately.

  11. #11
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    10,946
    Can you post usable sketch? It shouldn't be 'filebuffer'

  12. #12
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    10,946
    my bad on that - it does need a buffer and the buffer should be in the write not the read

  13. #13
    Member
    Join Date
    May 2019
    Location
    Brisbane, QLD
    Posts
    71
    forgive my ignorance, could you alter the example code to demonstrate this?

    Posting a useable sketch is difficult, i've got everything in tabs in arduinoIDE.

  14. #14
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    10,946
    first step - correcting post #2

    Code:
    //copy file
        if (strcmp(result, copyID) == 0){
          File myFileIn = SD.open(secondWord, FILE_READ);
          int filesize = myFileIn.size();
          Serial.println(filesize,"bytes");
    #define FILE_BUFFER 512
          byte fileBuffer[FILE_BUFFER];
          File myFileOut = SD.open(thirdWord, O_WRITE | O_CREAT| O_APPEND);
          //delay(500);
          
          print2("copy file> ", secondWord);
          print2("to file> ", thirdWord);
          delay(1000);
    
          while (myFileIn.available()) {
            int len = myFileIn.available();
            if ( len > FILE_BUFFER ) len = FILE_BUFFER;
            readBytes(fileBuffer, len)
            myFileOut.write( fileBuffer, len );
            delay(1); // maybe needed to allow USB buffer to arrive
          }
    
          myFileIn.close();
          myFileOut.close();
          commandLineSwitch = 2;
        }

  15. #15
    Member
    Join Date
    May 2019
    Location
    Brisbane, QLD
    Posts
    71
    It works! It's aliveeeeeeee!

    just corrected;

    Code:
    readBytes(fileBuffer, len)
    to
    Code:
    myFileIn.readBytes(fileBuffer, len);
    speed is light years better! Thanks for your help!

  16. #16
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    10,946
    Awesome.

    glad to help - especially after the half wrong answer in post #2 and #9 before the lights turned on

    It could be improved with the 512B chunking writes to SD - but that would take more overhead filling that buffer to 512 and writing that between reads. Likely the USB would come in even 64B chunks - but that would be assuming. And if moved to a T_4.0 the faster USB would come in 512B chunks.

    That might make the SD writes go faster if exploration was something that seemed fun.

  17. #17
    Member
    Join Date
    May 2019
    Location
    Brisbane, QLD
    Posts
    71
    Quote Originally Posted by defragster View Post
    Awesome.

    glad to help - especially after the half wrong answer in post #2 and #9 before the lights turned on

    It could be improved with the 512B chunking writes to SD - but that would take more overhead filling that buffer to 512 and writing that between reads. Likely the USB would come in even 64B chunks - but that would be assuming. And if moved to a T_4.0 the faster USB would come in 512B chunks.

    That might make the SD writes go faster if exploration was something that seemed fun.
    Right now i think it's "good enough" to move forward. The largest files i'll be dealing with are probably 1mb max, so speed isn't crucial for any internal file managment. Anything going over serial will be sent via character arrays and the raw data streamed straight into a SD card file with code like;

    Code:
    myFile.println(msb_lsb, HEX); //write value to SD card
    i was happy with the speeds.

    You've been a great help, thanks for your patience with me.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •