Teensy4.0 slows down while logging data to SD card

[Ruturaj]

I’m reaching out for some advice on an issue that’s been troubling me for the past three weeks. I’ve designed a custom breakout PCB for a Teensy 4.0 that logs basic sensor data (accelerometer, gyroscope, and barometer), with a Kalman filter to a microSD card. The setup logs data at a rate of 5 samples per second, and it is stored in a .dat file. To monitor the main loop, I have a non-blocking blink function that ensures the loop is running without delays.

The Problem: After around 20 minutes of continuous operation, the main loop slows down significantly, making the logging process almost unusable. This issue only occurs when the logging function is active. If I comment out the function responsible for data logging, the loop runs smoothly for extended periods without any slowdown.

The microSD card is connected to the Teensy via SPI mode. I’ve included an image of the schematic for reference.

What I’ve Tried:

• Monitoring the system performance without the logging function—no slowdown occurs.
• Adjusting the sampling rate and other minor tweaks with no success.
• Adding a heatsink on the Teensy processor to dissipate heat, but didn't make a difference.

I need the system to reliably log data for at least 2 hours without encountering this problem. Any insights or suggestions on what could be causing this issue would be greatly appreciated!

 

[PaulStoffregen]

Any insights or suggestions on what could be causing this issue would be greatly appreciated!

Quick blind guess (without seeing your code) perhaps you're closing and re-opening the file? With FAT filesystems, seeking to the end of a file becomes more expensive as the file size grows. Usually the solution is to limit how much data you write to any 1 file, and automatically create a new file after the last file becomes "full".

An alternative might be to make sure the card is formatted as FAT64 (aka exfat). Then use the SdFat access to preallocate the entire file. I believe Bill (author of SdFat) put some way into SdFat to allocate the entire file as contiguous sectors, which ought to bypass the need to traverse FAT entries to seek within it. FWIW, I personally haven't tried this so can't confirm from 1st hand experience. But the question has some up before and I recall Bill mentioning this option. It's only possible on FAT64. I believe all cards larger than 32GB are supposed to come formatted that way, but if you do have a card with FAT32 of FAT16 format, it definitely will not work.

 

[Angelo]

If I remember, writing the file at the root level, not in any sub-directory, will also avoid slowing down. But can't remember if it is valid for all FATxx formats.

 

[nopayne]

Have you experimented with batching your writes to the file? I wonder what the impact of reducing the number of writes you are doing from 5x a second to maybe 1x every 2 seconds. This would only work if you are okay with losing a few readings in the case of a failure.

Rotating the logs is probably a better approach overall but batching might be easier to test out.

 

[PaulStoffregen]

While throwing out random ideas, might also be worth mentioning always writing 512 bytes or 4096 bytes at a time (1 or 8 sectors) tends to have much better performance. But that's (probably) unrelated to a problem where performance gets worse over time.

 

[Ruturaj]

Thank you all for your valuable feedback! Following PaulStoffregen's suggestion, I formatted my SD card to exFAT with a 256KB allocation unit size, which allowed me to successfully log data for around 45 minutes.

To give you an overview of my current code setup:

• Initialization: At startup, the Teensy reads a .TXT file on the SD card to get the current file number, increments it, and then creates a new .dat file with that number. This file is opened and kept open for data logging.
• Main Loop: In the main loop, sensor data is logged to the open .dat file at a rate of 5 times per second.

The function called during setup:

void generate_file_dat(void) {
   if (!SD.begin(chipSelect)) {
    Serial.println("SD card initialization failed!");
    while (true) {
      digitalWrite(led_pin, HIGH);
      delay(200);
      digitalWrite(led_pin,LOW);
      delay(200);
    }
  }

  delay(200);

  myFile = SD.open("Filename.TXT");

  delay(50);

  if (myFile) {
    while(myFile.available())
      file_number = myFile.readString();
  }
 
  delay(200);
  myFile.close();

  for (size_t i = 0; i < file_number.length(); i++) {
        file_number_int = file_number_int * 10 + (file_number[i] - '0');
      }

  //Serial.println(file_number_int);
  file_number_int += 1;
  //Serial.println(file_number_int);

  delay(200);

  SD.remove(file_name.c_str());
  delay(200);
  myFile = SD.open(file_name.c_str(), FILE_WRITE);

  if(myFile) {
    myFile.print(file_number_int);
    myFile.flush();
  }
  delay(200);
  myFile.close();
  delay(200);
 
  String file_number_string = String(file_number_int) + ".dat";
  Serial.println(file_number_string);

  myFile = SD.open(file_number_string.c_str(), FILE_WRITE);
  delay(200);
}

The function which logs values in loop:

void save_values_dat() {
  if(current_time - save_counter>200000){
    save_counter = micros();
    struct datastore mydata;
    if (myFile) {
      mydata.timestamp_sd = current_time;
      mydata.AltitudeBarometer_sd = AltitudeBarometer/100;
      mydata.kalman_altitude_sd = AltitudeKalman/100;
      mydata.roll_imu_sd = roll_IMU;
      mydata.pitch_imu_sd = pitch_IMU;
      mydata.roll_kalman_sd = KalmanAngleRoll;
      mydata.pitch_kalman_sd = KalmanAnglePitch;
      mydata.AccZInertial_sd = AccZInertial/981.0;
      mydata.crashed = 'N';
      myFile.write((const uint8_t *)&mydata, sizeof(mydata));
      myFile.flush();
    }
  }
}

The data structure looks like this:

struct datastore {
  unsigned long timestamp_sd;
  float AltitudeBarometer_sd;
  float kalman_altitude_sd;
  float roll_imu_sd;
  float pitch_imu_sd;
  float roll_kalman_sd;
  float pitch_kalman_sd;
  float AccZInertial_sd;
  char crashed;
};

 

[aleglakov]

delays are slowing down your code, sir.

 

[Ruturaj]

delays are slowing down your code, sir.

Only the setup function has delays, not in the main loop.

 

[Pio]

What is the type of the current_time and save_counter ?
One way to test it deeper would be to grab the micros() value around the write and flush to see how much time it takes to execute these functions and which one is causing problems.
A reliable data logging with less frequent SD write could use an external SPI SRAM chip (ie 24lc256), or even better - a small non volatile F-RAM to store a pack of readings and a status byte telling if the data has been already written to the SD. Even if the Teensy crashes, it could check the status byte on boot and resume logging. With F-RAM data would survive the power cycle. Could use the same SPI bus as the SD card using only 1 gpio for the chip select.

--edit--
Actually no, it would be better if the RAM is on a different bus, because the SD write might be still in progress when the new data has to be written to the RAM.
Just looked at the LCSC, cheapest 2k I2C F-RAM $0.4531

MB85RC16PNF-G-JNERE1 FUJITSU | Memory - LCSC Electronics

MB85RC16PNF-G-JNERE1 FUJITSU $0.4531 - SOIC-8 FRAM ROHS datasheet, price, inventory C8739

www.lcsc.com

 

[AndyA]

Can your system count on the log file getting closed cleanly at the end or does it need to be able to cope with things stopping unexpectedly?

If you can ensure that the file will be closed cleanly at the end then skip the flush() command. That will allow the filesystem to only physically write to the card in neat blocks rather than having to update the card with arbitrary sized amounts of data each time.

Also one nit-pick that's unrelated to the issue at hand - your data structure that you are writing should really be defined as packed. Without that specification the compiler is free to apply padding and change the data alignment as it sees fit. There is no way to ensure that the compiler for whatever is reading the log file will pick the same alignment and padding options. By specifying that the structure is packed you force the compiler to store the data in exactly the structure you define. Since all your values are 32 bits other than the char at the end this probably isn't going to make any difference in this specific case but that may change and it's a good habit to have.

 

[UhClem]

I prefer to write my data logging programs differently.
Data collection is driven by interrupts and the data gets packed off to a data buffer. Multiple buffers some power of two times 512 bytes in size. When a buffer is full, the foreground task notices and begins the write operation. This insulates the data collection from most delays in the writing process. Which can be very long especially once you add in FAT driven delays.

The first time I did this, I wrote my own minimal FAT16 code. When the file was opened it would find a large continuous block of free space and use that. Delaying all of the FAT updates till the end.

 

[mborgerson]

The data structure looks like this:

struct datastore {
  unsigned long timestamp_sd;
  float AltitudeBarometer_sd;
  float kalman_altitude_sd;
  float roll_imu_sd;
  float pitch_imu_sd;
  float roll_kalman_sd;
  float pitch_kalman_sd;
  float AccZInertial_sd;
  char crashed;
};

One problem that I see is that your data structure is either 33, 34, or 36 bytes in length---depending on whether you've done anything to adjust the structure packing size. None of those sizes will fit evenly into a 512- byte SD sector. At some point, writing a structure to the file will require writing a sector with some portion of the structure, then getting a new sector address and moving the rest of the structure into that new sector buffer.

For maximum write efficiency, you could use a larger secondary buffer, accumulate your data in the buffer and write that larger buffer all at once. You could try an intermediate buffer of 8704 bytes, (which is 512 * 17) which would work nicely with 2-byte packing of the structures. Note that, unless you pre-allocate a file, it may sometimes take more than 200mSec to do a file write if the SD card has to erase a new storage block to finish the write.

Also, flushing, or syncing the file has to update the directory sectors, so you might try doing that only once per 5 seconds, or so---depending on your tolerance for missed data at the end of the file on an unplanned shutdown.

 

[UhClem]

it may sometimes take more than 200mSec to do a file write

The specification has something about this. 4.13.1.7.2 (version 8 of the spec) says that the maximum FAT write time shall be less than 750ms. A little later in 4.13.1.8.2 it has a table for various speed classes. All of which show an average time of 100ms but keep that maximum of 750ms.

Plus of course what the file system code is up to. I recall looking at one library many years ago that in order to allocate a new cluster would traverse a files FAT chain. Twice.

 

[jmarsh]

"Write time" doesn't necessarily include erase time.

 

[mborgerson]

"Write time" doesn't necessarily include erase time.

For the OP's case, erase time should not be an issue. I think that writing 33 bytes five times per second should not show any major delays due to block erase time. The block erases, when required. should be covered by the internal RAM buffers managed by the SD Card.

Long write times are much more likely to interfere with very high data storage rates--like writing a 32-byte structure 100,000 times per second.

Another factor to note is that the OP mentioned running the SD in SPI mode, as is done with the MicroMod data logging carrier board. Some testing I did a few years ago shows that the SPI-mode read transfers are about 8 times slower than with the SDIO interface on the T4.1. Write transfers show about the same speeds for both SPI and SDIO for small block writes For 32-byte writes, both interfaces show about 300KB/second. At that speed, it's not surprising that the OP ran into problems when the file system code started requiring long FAT traversals after extended logging. With the SDIO interface the speed jumps by about two orders of magnitude when the write block size is 512 bytes or larger. The SPI interface speeds up more gradually, topping out at about 2.4MB/second.

https://forum.pjrc.com/threads/7148...-won-t-open-SD?p=315781&viewfull=1#post315781