In Need of Teensy 4.1 Data Transfer Debug Help

Hi there,

I have a Teensy 4.1 set up as a simple data logger based on mborgerson's excellent "Generic data logger object". On top of this I've added some serial data transfer capabilities so I can just plug in the logger to my computer (Macbook) and transfer the binary data to a file to some directory there. Then I can decode this file and have data pulled into python.

I had this firmware/software stably working for some time, but now, especially on longer files (longer than 1 minute), I get hangup in my code. It gets hung up in the while loop below, where it seems there are no bytes waiting to be read, I get the following when I have my print statements uncommented:

ser.in_waiting: 0
Sd_buff_idx: 3060
sd_buff_size: 4864

I have had the code work stably for months, and then run into this issue. I replaced the usb cable with a new one and that solved the problem for a couple more months (as I read on some forum when they get old/super used they can skip bits??). Anyways the problem is back.

I'm looking for any advice on how to solve this issue and make my data transfer more robust. My code is pretty scrappy and my knowledge limited, so bear with me :).

I am thinking that I should have some error handling that if I am stuck in that while loop (while sd_buff_idx < sd_buff_size) for over a second or something, I should send a command to the teensy to reset the file position to the current position minus my sd_buff_idx that I have successfully read (using seek()) and then try transferring the block again?

I figured I would make this post as I am working on debugging to hopefully get some input from smart people instead of banging my head against the wall all week on this! Code below!!

Python side:

Code:
def read_file(filename, file_size):
    ser = establish_serial()

    # read file with single blocks
    print("Filename is: ", filename)  # DEBUG
    filename_term = filename + "x"
    filename_term = bytes(filename_term, "utf-8")

    sd_buff_size = 4864
    # usbBuffSize = 64
    num_buffs = file_size / sd_buff_size
    print("Original num buffs = ", num_buffs)

    if file_size % sd_buff_size != 0:
        print("chopping last buffer")
        num_buffs = file_size // sd_buff_size  # skip last sd_buffer
        print("New num buffs = ", num_buffs)
        # TODO: need to also chop expected file size down here too?
    num_buffs = int(num_buffs)
    bytes_received = []  # ATTN maybe prealocate with file size

    # open target file
    ser.write(b"o")
    ser.write(filename_term)
    # time.sleep(.1)

    """ #test filename worked
    # terminator = b'x'
    # # Read until terminator is reached
    # data = ser.read_until(terminator)
    # # Convert bytes to string
    # data_str = data.decode('utf-8')
    # # Print string
    # print("Filename confirmation: ", data_str)
    #"""

    sd_buff = bytes()
    sd_buff_idx = 0

    for i in range(num_buffs):
        ser.write(b"t")

        while sd_buff_idx < sd_buff_size:
            # TODO DEBUG
            # print("Sd_buff_idx: {}".format(sd_buff_idx))
            # print("sd_buff_size: {}".format(sd_buff_size))
            # print("ser.in_waiting: {}".format(ser.in_waiting))
            num_read = min(int(ser.in_waiting), sd_buff_size - sd_buff_idx)
            if num_read > 0:
                bytesIn = ser.read(num_read)
                sd_buff_idx += num_read
                sd_buff += bytesIn
            # print("sd_buffIdx = {}".format(sd_buff_idx)) # DEBUG

        # print("we have the bytes") # DEBUG
        bytes_received.extend(sd_buff)
        sd_buff = bytes()
        sd_buff_idx = 0

    # close target file
    ser.write(b"c")
    ser.close()
    return bytes_received


Firmware side:

Code:
void openFile()
{
  String myFileName = Serial.readStringUntil('x', 20);
  // check for correct file
  mydl.dataFile = mydl.sdx.open(myFileName, O_READ);

  // // DEBUG filename confirmation
  // String fileNameTerm = myFileName + "x";
  // Serial.write(fileNameTerm.c_str(), fileNameTerm.length());

  // Check if the file opened successfully
  if (!mydl.dataFile)
  {
    Serial.println("In openFile(): ");
    Serial.println("Error opening file!");
    Serial.print("File name was: ");
    Serial.print(myFileName);
    fastBlink();
    return;
  }
  mydl.dataFile = mydl.sdx.open(myFileName, O_READ);
  mydl.dataFile.rewind();
}

void transferBlock()
{
  // Allocate a buffer to hold the data from the SD card
  uint8_t sdBuffer[SD_BUFFER_SIZE];
  // Allocate a buffer to hold the data for USB transmission
  uint8_t usbBuffer[USB_BUFFER_SIZE];
  uint8_t usbBuffperSDBuff = SD_BUFFER_SIZE / USB_BUFFER_SIZE;
  uint32_t writtenBytes = 0;
  mydl.dataFile.read(sdBuffer, SD_BUFFER_SIZE);
  for (int i = 0; i < usbBuffperSDBuff; i++)
  {
    // seems like memmove might be more robust than memcpy:
    // https://stackoverflow.com/questions/4415910/memcpy-vs-memmove
    memmove(usbBuffer, &sdBuffer[writtenBytes], USB_BUFFER_SIZE);
    Serial.write(&sdBuffer[writtenBytes], USB_BUFFER_SIZE);
    writtenBytes += USB_BUFFER_SIZE;
  }
  if (ledOn)
  {
    LEDOFF;
  }
  else
  {
    LEDOFF;
  }
}
 
On the firmware side, have you tried adding Serial.flush() after your call to Serial.write()? Your USB_BUFFER_SIZE is 64, which is small for USB, and you may have data sitting in USB buffers on the Teensy side. If that helps, I would also increasing USB_BUFFER_SIZE for better efficiency. I also notice that transferBlock() moves bytes from sdBuffer to usbBuffer, but then sdBuffer is passed to Serial.write(), so usbBuffer is never actually used and you could eliminate that variable and the memmove().
 
I did two things differently when I was uploading files to a Python script a few years ago.
1. I set up the T4.1 to use Dual Serial USB then used Serial1 exclusively for the binary transfers.
2. I used a much larger buffer, 51200 bytes (100 512-byte file blocks).

For what it's worth, here is the binary upload function-edited from the original to
remove some stuff related to a command parser:

Code:
void CMBU(const char *fnameptr) {
#define UPSIZE 51200
  uint8_t upbuffer[UPSIZE];
  uint32_t  numread;
  static  SdFile upfile;
  uint16_t bcount;
  bool upopen;
  uint32_t uptotal;

  
   Serial.printf("Sending %s to host.\n", fnameptr);
  upopen = upfile.open(fnameptr, FILE_READ);
  if (!upopen) {
    delay(1000);
    Serial.println("Could not open the file.");
    return;
  }

  bcount = 0;
  uptotal = 0;
  delayMicroseconds(1000);  // wait for Python host to get ready

  do {
    numread = upfile.read(upbuffer, UPSIZE);

    uptotal += numread;
    SerialUSB1.write(upbuffer, numread);
    //  I found that my python host program needed an occasional small
    //  break in the data stream to keep up with the Teensy
    if (bcount++ > 16) {
      delayMicroseconds(50);
      bcount = 0;
    }
  } while (numread > 0);  // Stops at the end of the file
  Serial.printf("Upload complete. %lu bytes sent.\n", uptotal);
  upfile.close();
}

I think my Python host had a timeout that closed the file if no new data appeared at the Serial1 input
in a few hundred milliseconds.
 
I should add that I quit using binary uploads a year or two ago when the MTP library matured and was much better integrated into TeensyDuino. A key feature is the MTPReset function, which updates the Teensy internal MTP directory so that your PC can see the most recent additions to your SD Card files. Now, when I've collected some data, I call MTPReset, then open the Teensy device and the SDCard folder. I can then transfer files to the Matlab Work folder on my PC with drag and drop. Unfortunately, MTP is not so well integrated into MacOS. I use an app called Commander One PRO for MTP access on my Mac Mini, but it sometimes refuses to find the Teensy after an error---forcing me to restart the MAC.
 
I did two things differently when I was uploading files to a Python script a few years ago.
1. I set up the T4.1 to use Dual Serial USB then used Serial1 exclusively for the binary transfers.
2. I used a much larger buffer, 51200 bytes (100 512-byte file blocks).

For what it's worth, here is the binary upload function-edited from the original to
remove some stuff related to a command parser:

Code:
void CMBU(const char *fnameptr) {
#define UPSIZE 51200
  uint8_t upbuffer[UPSIZE];
  uint32_t  numread;
  static  SdFile upfile;
  uint16_t bcount;
  bool upopen;
  uint32_t uptotal;

 
   Serial.printf("Sending %s to host.\n", fnameptr);
  upopen = upfile.open(fnameptr, FILE_READ);
  if (!upopen) {
    delay(1000);
    Serial.println("Could not open the file.");
    return;
  }

  bcount = 0;
  uptotal = 0;
  delayMicroseconds(1000);  // wait for Python host to get ready

  do {
    numread = upfile.read(upbuffer, UPSIZE);

    uptotal += numread;
    SerialUSB1.write(upbuffer, numread);
    //  I found that my python host program needed an occasional small
    //  break in the data stream to keep up with the Teensy
    if (bcount++ > 16) {
      delayMicroseconds(50);
      bcount = 0;
    }
  } while (numread > 0);  // Stops at the end of the file
  Serial.printf("Upload complete. %lu bytes sent.\n", uptotal);
  upfile.close();
}

I think my Python host had a timeout that closed the file if no new data appeared at the Serial1 input
in a few hundred milliseconds.


Interesting, I had tried something similar in the past with larger writes and just waiting til we get all the bytes, but had some robustness issues. I tried to replicate the functionality you pasted here to fit with my code structure, and when I test it, it appears to work about 80% of the time. I am testing data transfer for 7 files that range from 15 seconds to a couple of minutes. Most of the data looks good, but occasionally it is all garbled up. I'm plotting some accelerometer data as an example. Sometimes it looks correct (on the left plot below), and others it is all messed up (right below), likely because my python code is skipping/missing bytes so everything is offset when I try to decode?? (Note I have not yet implemented your suggestion to use dual serial on the Teensy, which is probably a good idea).
proper_plot.png
error_plot.png



Here's the python code:

Code:
def get_serial_port():
    ports = serial.tools.list_ports.comports()
    # TODO: windows compatability (does not have usbmodem in name)
    logger_ports = [port.device for port in ports if "usbmodem" in port.device]
    if len(logger_ports) != 1:
        print("In cass_commands.get_serial_port()")
        print("Error! More than one teensy port found!")
        return ""
    else:
        return logger_ports[0]

def establish_serial(baud_rate=9600):
    serial_port = get_serial_port()
    ser = serial.Serial(serial_port, baud_rate)
    if not ser.is_open:
        ser.open()
    ser.reset_input_buffer()
    ser.reset_output_buffer()
    return ser

def download_all():
    # ser = establish_serial()
    my_filenames = list_files()
    my_file_sizes = file_sizes()
    if not len(my_filenames):
        # print("No Files")
        return []
    dir_name = "tmp_{}".format(int(time.time()))

    filepaths = [
        bytes_to_file(read_full_file(filename, file_size), filename, dir_name)
        for filename, file_size in zip(my_filenames, my_file_sizes)
    ]

    return filepaths[-1]


def bytes_to_file(my_bytes, filename, filepath="tmp_{}".format(int(time.time()))):
    if not os.path.exists(filepath):
        os.mkdir(filepath)

    # Write the bytes to a file in the "tmp" directory
    full_filepath = Path(filepath, filename)
    with open(full_filepath, "wb") as f:
        f.write(bytes(my_bytes))
    return filepath

def read_full_file(filename):
    ser = establish_serial()
    print("Filename is: ", filename)  # DEBUG
    filename_term = filename + "x"
    filename_term = bytes(filename_term, "utf-8")

    # open target file
    ser.write(b"o")  # initiate file transfer (calls FullFileTransfer())
    ser.write(filename_term)

    bytes_received = []  # ATTN maybe prealocate with file size
    timeout = 0.1
    start_time = time.time()

    ser.write(b"e")

    while True:
        if ser.in_waiting > 0:
            bytes_received += ser.read(ser.in_waiting)
            start_time = time.time()  # Reset the timeout timer
        elif time.time() - start_time > timeout:
            print("TIMEOUT BREAK!")
            break

    ser.write(b"c")
    ser.flush()
    ser.close()
    return bytes_received

And the relevant firmware:

Code:
void openFile()
{
  String myFileName = Serial.readStringUntil('x', 20);
  // check for correct file
  mydl.dataFile = mydl.sdx.open(myFileName, O_READ);

  // // DEBUG filename confirmation
  // String fileNameTerm = myFileName + "x";
  // Serial.write(fileNameTerm.c_str(), fileNameTerm.length());

  // Check if the file opened successfully
  if (!mydl.dataFile)
  {
    Serial.println("In openFile(): ");
    Serial.println("Error opening file!");
    Serial.print("File name was: ");
    Serial.print(myFileName);
    fastBlink();
    return;
  }
  mydl.dataFile = mydl.sdx.open(myFileName, O_READ);
  mydl.dataFile.rewind();
}

void closeFile()
{
  // String myFileName = Serial.readStringUntil('x',20);
  // check for correct file
  bool fileClosed = mydl.dataFile.close();

  if (!fileClosed)
  {
    Serial.println("Error closing file!");
    fastBlink();
    return;
  }
}

void FullFileTransfer() {
    bcount = 0;
    uptotal = 0;
    delayMicroseconds(1000);  // wait for Python host to get ready

    do {
      numread = mydl.dataFile.read(upbuffer, UPSIZE);

      uptotal += numread;
      Serial.write(upbuffer, numread);
      //  I found that my python host program needed an occasional small
      //  break in the data stream to keep up with the Teensy
      if (bcount++ > 2) {
        delayMicroseconds(50);
        bcount = 0;
      }
    } while (numread > 0);  // Stops at the end of the file
}




I switched over to much smaller chunks that I send one chunk at a time with the idea that if I have any missing bytes, I can potentially return to the start of that chunk and try to write it again...That is what I was working on implementing this morning before you replied. I added a function to say, hey if we are stuck in the while loop where ser.in_waiting is 0 but our buffer isn't full, then reset our position in the file to the current position minus the bytes written in that buffer. Then try again. Maybe a combination of this plus the suggestion for much bigger buffers would make it more robust???

Python:
Code:
def read_file(filename, file_size):
    ser = establish_serial()

    # read file with single blocks
    print("Filename is: ", filename)  # DEBUG
    filename_term = filename + "x"
    filename_term = bytes(filename_term, "utf-8")

    sd_buff_size = 4864
    # usbBuffSize = 64
    num_buffs = file_size / sd_buff_size
    print("Original num buffs = ", num_buffs)

    if file_size % sd_buff_size != 0:
        print("chopping last buffer")
        num_buffs = file_size // sd_buff_size  # skip last sd_buffer
        print("New num buffs = ", num_buffs)
        # TODO: need to also chop expected file size down here too?
    num_buffs = int(num_buffs)
    bytes_received = []  # ATTN maybe prealocate with file size

    # open target file
    ser.write(b"o")
    ser.write(filename_term)
    # time.sleep(.1)

    """ #test filename worked
    # terminator = b'x'
    # # Read until terminator is reached
    # data = ser.read_until(terminator)
    # # Convert bytes to string
    # data_str = data.decode('utf-8')
    # # Print string
    # print("Filename confirmation: ", data_str)
    #"""

    sd_buff = bytes()
    sd_buff_idx = 0

    for i in range(num_buffs):
        ser.write(b"t")
        timer = time.monotonic()

        while sd_buff_idx < sd_buff_size:
            # TODO DEBUG
            num_read = min(int(ser.in_waiting), sd_buff_size - sd_buff_idx)
            if num_read > 0:
                bytesIn = ser.read(num_read)
                sd_buff_idx += num_read
                sd_buff += bytesIn
                timer = time.monotonic()
            elif num_read == 0 and time.monotonic() - timer > 1.0:
                # reset the position in the file to curr_position - sd_butt_idx
                print("In elif !!!")
                print("Sd_buff_idx: {}".format(sd_buff_idx))
                print("sd_buff_size: {}".format(sd_buff_size))
                print("ser.in_waiting: {}".format(ser.in_waiting))
                buff_success = _reset_buff(ser, sd_buff_idx)
                print("buff_success = {}".format(buff_success))
                i -= 1
                break

            # print("sd_buffIdx = {}".format(sd_buff_idx)) # DEBUG

        # print("we have the bytes") # DEBUG
        bytes_received.extend(sd_buff)
        sd_buff = bytes()
        sd_buff_idx = 0

    # close target file
    ser.write(b"c")
    ser.close()
    return bytes_received


def _reset_buff(ser, bytes_to_reset):
    ser.close()
    ser.open()
    # start _reset_buff method
    ser.write(b"n")

    # send bytes to reverse by
    bytes_to_reset = str(bytes_to_reset)
    bytes_to_reset += "x"
    print("_reset_buff: bytes with term char = {}".format(bytes_to_reset))
    ser.write(bytes(bytes_to_reset, "utf-8"))

    while ser.in_waiting < 1:
        pass

    confirmation_char = ser.read()
    print("confirmation_char: {}".format(confirmation_char))
    if confirmation_char == b"x":  # Compare with bytes literal
        return True
    else:
        return False

Firmware:

Code:
void resetFilePos()

{

  // Serial.flush();

  char terminator = 'x';

  Serial.setTimeout(5000);

  String bytesToMoveStr = Serial.readStringUntil(terminator);

  int bytesToMove = bytesToMoveStr.toInt();

  mydl.dataFile.seekCur(-bytesToMove);

  Serial.print('x');

}
 
Last edited:
On the firmware side, have you tried adding Serial.flush() after your call to Serial.write()? Your USB_BUFFER_SIZE is 64, which is small for USB, and you may have data sitting in USB buffers on the Teensy side. If that helps, I would also increasing USB_BUFFER_SIZE for better efficiency. I also notice that transferBlock() moves bytes from sdBuffer to usbBuffer, but then sdBuffer is passed to Serial.write(), so usbBuffer is never actually used and you could eliminate that variable and the memmove().
Good points. I tried adding Serial.flush() but no luck. I will clean up that unused code too. You can probably tell my c++ knowledge is basically nil. I made a more detailed reply to mborgerson above that may give you some more ideas/insights into my problem.

It is hard to debug because it is hard to replicate. It happens randomly and sometimes not at all, maybe about 30-40% of the time? But it is often enough that it makes my serial data transfer basically unusable at the moment...
 
Back
Top