Uploading Teensy files to PC

[mborgerson]

As part of an in-progress data logging program, I've been working on optimizing the uploading of files from the Teensy SD card to a host program that I've written for the PC. After several days work, I've come up with an algorithm seems to transfer files at consistent speeds and without errors. The transfer function transfers 4KB blocks and waits for a block prompt character from the PC. I found that it is possible to lose bytes in the transfer if the PC steps away from reading the incoming USB Serial port long enough the transfer to time out on the Teensy end. Prompting the Teensy for each block seems to eliminate that problem. I've run my transfer program on a T3.6 and a T 4.0 (using SDFat 2.0 and Teensyduino 1.48 in both cases.) The upload rates are as follows:


T3.6 180mHz, fastest 1.06MBytes/second
T4.0 600MHz fastest 6.89MBytes/second SDFat Configured for SDIO and FIFO mode

USB-connected SD Card reader ~24MBytes/second

The upload speed on the T4.0 doesn't seem completely tied to CPU speed---the transfer rate is about the same at 396MHz clock and drops to about 4MBytes/second at 150MHz. Changing the upload block size to 8KB or 16KB doesn't seem to improve the upload rate. I think that's because the 4KB SD read is just the write size to allow it to be sent by the USB driver just in time for the PC to receive it and prompt for the next block without much delay.

My host program is written in C++ on RadStudio and compiles to X86 code. During the upload there is an upload counter that is update about once per MByte. I'm writing to an NVME SSD which has excellent write speeds, but I will also test it with a spinning HD.


Here's an excerpt of the Teensy code showing the algorithm:

wridx = 0;
  fsize = upfile.fileSize();

  uptotal = 0;
    // tell the PC what size blocks to expect
    Serial.write((unsigned char *)&upsize, 2);
    delayMicroseconds(1200);
    
  startmicro = micros();
  do {
    bytesleft = fsize - uptotal;
    if(bytesleft > upsize){
      num2read = upsize;
    } else {
      num2read = bytesleft;
    }
    // read from file.  After first block, USB is probably 
    // still sending previous block.
    //
    numread = upfile.read(upbuffer, num2read);
    WaitForREQ();  // wait for block request from PC
    startmicro = micros();
    uptotal += numread;
    LEDON
    Serial.write(upbuffer, numread);
    LEDOFF
  } while(uptotal < fsize); // end of do loop
  LEDON  
      // now send a zero length to indicate end of transfer
  numread = 0;
  WaitForREQ();
  Serial.write((unsigned char *)&numread, 2);
  Serial.write((unsigned char *)&numread, 2);

  delay(10);
  LEDOFF
  delay(100);

This algorithm relies on error-free block transfers via USB, so there are no checksums.

Are there other algorithms with better or equal speeds that I should consider?

In another thread, Paul hinted that future optimizations to USB serial may increase the transfer speed. I wonder how that might affect my algorithm.