Teensy 4 Data Logger Project

Hi all,

I am starting a personal project and want to post the high level idea to see if it seems feasible, and to see if anyone has tips/suggestions.

I am looking to use the Teensy 4.1 platform to make a data logger with the following functionality:
  • On-board 9-dof IMU (using the: ICM-20948)
  • 15 analog channels (for various sensors)
  • minimum 500 Hz sampling rate
  • Wifi data transfer to computer
  • No SD card (so stores data files on flash/ram)
  • An hour of record time possible for each file
  • Able to store at least a total 5 hours on the device

I plan to make a custom PCB based on the teensy 4.1. I'll fabricate a nice waterproof housing for it, and develop a simple suite of (mostly) analog sensors to go with it (accels, gyros, potentiometers, pressure sensors, etc). The use case is for data acquisition for mountain bikes.

A few years back, for work, I made a shield for the Teensy 3.6 that did this, though I stored data on SD, and transferred data via serial to the computer (I wrote a little matlab package to do this). I based the firmware off an old SDFat example, which can be found here .

I'm having trouble uploading photos here for some reason, but if you follow this link, you can see a summary of what I did.

I want to build a similar device but improved upon for myself. Using and example from Paul's littleFS wrapper, I got data from an ICM-20948 breakout board to a text file from a Teensy 4.0. The code is below.

Code:
#include <sensors.h>

// Declare custom functions
void logData();
void stopLogging();
void dumpLog();
void menu();
void listFiles();
void eraseFiles();
void printDirectory(FS &fs);
void printDirectory(File dir, int numSpaces);
void printSpaces(int num);
void printRawAGMT(ICM_20948_AGMT_t agmt);
void printFormattedFloat(float val, uint8_t leading, uint8_t decimals);

// Instance of LittleFS_Program derived class
LittleFS_Program myfs;

// NOTE: This option is only available on the Teensy 4.0, Teensy 4.1 and Teensy Micromod boards.
// With the additonal option for security on the T4 the maximum flash available for a 
// program disk with LittleFS is 960 blocks of 1024 bytes
#define PROG_FLASH_SIZE 1024 * 900 // Specify size to use of onboard Teensy Program Flash chip
                                  // This creates a LittleFS drive in Teensy PCB FLash. 
#define SERIAL_PORT Serial
#define WIRE_PORT Wire // Your desired Wire port.
// The value of the last bit of the I2C address.
// On the SparkFun 9DoF IMU breakout the default is 1, and when the ADR jumper is closed the value becomes 0
#define AD0_VAL 1

// create vars
ICM_20948_I2C myICM; // Otherwise create an ICM_20948_I2C object
File dataFile;  // Specifes that dataFile is of File type
int record_count = 0;
bool write_data = false;
uint32_t diskSize;

void setup()
{
  // Open serial communications and wait for port to open:
  Serial.begin(115200);
  while (!Serial) {
    // wait for serial port to connect.
  }

  //delay to open serial moniter
  delay(1000);

  // Initialize ICM
  WIRE_PORT.begin();
  WIRE_PORT.setClock(400000);
  //myICM.enableDebugging(); // Uncomment this line to enable helpful debug messages on Serial
  bool initialized = false;
  while (!initialized)
  {
  myICM.begin(WIRE_PORT, AD0_VAL);

    SERIAL_PORT.print(F("Initialization of the sensor returned: "));
    SERIAL_PORT.println(myICM.statusString());
    if (myICM.status != ICM_20948_Stat_Ok)
    {
      SERIAL_PORT.println("Trying again...");
      delay(500);
    }
    else
    {
      initialized = true;
    }
  }
  // Initialize LittleFS
  Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);

  Serial.println("Initializing LittleFS ...");

    diskSize = PROG_FLASH_SIZE;

  // checks that the LittFS program has started with the disk size specified
  if (!myfs.begin(diskSize)) {
    Serial.printf("Error starting %s\n", "PROGRAM FLASH DISK");
    while (1) {
      // Error, so don't do anything more - stay stuck here
    }
  }
  Serial.println("LittleFS initialized.");
  
  menu();
  
}

void loop()
{ 
  if ( Serial.available() ) {
    char rr;
    rr = Serial.read();
    switch (rr) {
      case 'l': listFiles(); break;
      case 'e': eraseFiles(); break;
      case 's':
        {
          Serial.println("\nLogging Data!!!");
          write_data = true;   // sets flag to continue to write data until new command is received
          // opens a file or creates a file if not present,  FILE_WRITE will append data to
          // to the file created.
          dataFile = myfs.open("datalog.txt", FILE_WRITE);
          logData();
        }
        break;
      case 'x': stopLogging(); break;
      case 'd': dumpLog(); break;
      case '\r':
      case '\n':
      case 'h': menu(); break;
    }
    while (Serial.read() != -1) ; // remove rest of characters. 
  } 

  if(write_data) logData();
}

// =========== FOR ICM =======================================================================================
void printPaddedInt16b(int16_t val)
{
  if (val > 0)
  {
    SERIAL_PORT.print(" ");
    if (val < 10000)
    {
      SERIAL_PORT.print("0");
    }
    if (val < 1000)
    {
      SERIAL_PORT.print("0");
    }
    if (val < 100)
    {
      SERIAL_PORT.print("0");
    }
    if (val < 10)
    {
      SERIAL_PORT.print("0");
    }
  }
  else
  {
    SERIAL_PORT.print("-");
    if (abs(val) < 10000)
    {
      SERIAL_PORT.print("0");
    }
    if (abs(val) < 1000)
    {
      SERIAL_PORT.print("0");
    }
    if (abs(val) < 100)
    {
      SERIAL_PORT.print("0");
    }
    if (abs(val) < 10)
    {
      SERIAL_PORT.print("0");
    }
  }
  SERIAL_PORT.print(abs(val));
}

void printRawAGMT(ICM_20948_AGMT_t agmt)
{
  SERIAL_PORT.print("RAW. Acc [ ");
  printPaddedInt16b(agmt.acc.axes.x);
  SERIAL_PORT.print(", ");
  printPaddedInt16b(agmt.acc.axes.y);
  SERIAL_PORT.print(", ");
  printPaddedInt16b(agmt.acc.axes.z);
  SERIAL_PORT.print(" ], Gyr [ ");
  printPaddedInt16b(agmt.gyr.axes.x);
  SERIAL_PORT.print(", ");
  printPaddedInt16b(agmt.gyr.axes.y);
  SERIAL_PORT.print(", ");
  printPaddedInt16b(agmt.gyr.axes.z);
  SERIAL_PORT.print(" ], Mag [ ");
  printPaddedInt16b(agmt.mag.axes.x);
  SERIAL_PORT.print(", ");
  printPaddedInt16b(agmt.mag.axes.y);
  SERIAL_PORT.print(", ");
  printPaddedInt16b(agmt.mag.axes.z);
  SERIAL_PORT.print(" ], Tmp [ ");
  printPaddedInt16b(agmt.tmp.val);
  SERIAL_PORT.print(" ]");
  SERIAL_PORT.println();
}

void printFormattedFloat(float val, uint8_t leading, uint8_t decimals)
{
  float aval = abs(val);
  if (val < 0)
  {
    SERIAL_PORT.print("-");
  }
  else
  {
    SERIAL_PORT.print(" ");
  }
  for (uint8_t indi = 0; indi < leading; indi++)
  {
    uint32_t tenpow = 0;
    if (indi < (leading - 1))
    {
      tenpow = 1;
    }
    for (uint8_t c = 0; c < (leading - 1 - indi); c++)
    {
      tenpow *= 10;
    }
    if (aval < tenpow)
    {
      SERIAL_PORT.print("0");
    }
    else
    {
      break;
    }
  }
  if (val < 0)
  {
    SERIAL_PORT.print(-val, decimals);
  }
  else
  {
    SERIAL_PORT.print(val, decimals);
  }
}

void printScaledAGMT(ICM_20948_I2C *sensor)
{
  SERIAL_PORT.print("Scaled. Acc (mg) [ ");
  printFormattedFloat(sensor->accX(), 5, 2);
  SERIAL_PORT.print(", ");
  printFormattedFloat(sensor->accY(), 5, 2);
  SERIAL_PORT.print(", ");
  printFormattedFloat(sensor->accZ(), 5, 2);
  SERIAL_PORT.print(" ], Gyr (DPS) [ ");
  printFormattedFloat(sensor->gyrX(), 5, 2);
  SERIAL_PORT.print(", ");
  printFormattedFloat(sensor->gyrY(), 5, 2);
  SERIAL_PORT.print(", ");
  printFormattedFloat(sensor->gyrZ(), 5, 2);
  SERIAL_PORT.print(" ], Mag (uT) [ ");
  printFormattedFloat(sensor->magX(), 5, 2);
  SERIAL_PORT.print(", ");
  printFormattedFloat(sensor->magY(), 5, 2);
  SERIAL_PORT.print(", ");
  printFormattedFloat(sensor->magZ(), 5, 2);
  SERIAL_PORT.print(" ], Tmp (C) [ ");
  printFormattedFloat(sensor->temp(), 5, 2);
  SERIAL_PORT.print(" ]");
  SERIAL_PORT.println();
}

// =========== FOR littleFS ===================================================================================

void logData()
{
    // make a string for assembling the data to log:
    String dataString = "";

    // read three sensors and append to the string:
    if (myICM.dataReady()) {
      myICM.getAGMT();
      float gx = myICM.accX();
      float gy = myICM.accY();
      float gz = myICM.accZ();
      dataString += "Accel x: ";
      dataString += String(gx);
      dataString += ", Accel y: ";
      dataString += String(gy);
      dataString += ", Accel z: ";
      dataString += String(gz);
    }
  
    // if the file is available, write to it:
    if (dataFile) {
      dataFile.println(dataString);
      // print to the serial port too:
      Serial.println(dataString);
      record_count += 1;
    } else {
      // if the file isn't open, pop up an error:
      Serial.println("error opening datalog.txt");
    }
    delay(50); // run at a reasonable not-too-fast speed for testing
}

void stopLogging()
{
  Serial.println("\nStopped Logging Data!!!");
  write_data = false;
  // Closes the data file.
  dataFile.close();
  Serial.printf("Records written = %d\n", record_count);
}


void dumpLog()
{
  Serial.println("\nDumping Log!!!");
  // open the file.
  dataFile = myfs.open("datalog.txt");

  // if the file is available, write to it:
  if (dataFile) {
    while (dataFile.available()) {
      Serial.write(dataFile.read());
    }
    dataFile.close();
  }  
  // if the file isn't open, pop up an error:
  else {
    Serial.println("error opening datalog.txt");
  } 
}

void menu()
{
  Serial.println();
  Serial.println("Menu Options:");
  Serial.println("\tl - List files on disk");
  Serial.println("\te - Erase files on disk");
  Serial.println("\ts - Start Logging data (Restarting logger will append records to existing log)");
  Serial.println("\tx - Stop Logging data");
  Serial.println("\td - Dump Log");
  Serial.println("\th - Menu");
  Serial.println();
}

void listFiles()
{
  Serial.print("\n Space Used = ");
  Serial.println(myfs.usedSize());
  Serial.print("Filesystem Size = ");
  Serial.println(myfs.totalSize());

  printDirectory(myfs);
}

void eraseFiles()
{
  myfs.quickFormat();  // performs a quick format of the created di
  Serial.println("\nFiles erased !");
}

void printDirectory(FS &fs) {
  Serial.println("Directory\n---------");
  printDirectory(fs.open("/"), 0);
  Serial.println();
}

void printDirectory(File dir, int numSpaces) {
   while(true) {
     File entry = dir.openNextFile();
     if (! entry) {
       //Serial.println("** no more files **");
       break;
     }
     printSpaces(numSpaces);
     Serial.print(entry.name());
     if (entry.isDirectory()) {
       Serial.println("/");
       printDirectory(entry, numSpaces+2);
     } else {
       // files have sizes, directories do not
       printSpaces(36 - numSpaces - strlen(entry.name()));
       Serial.print("  ");
       Serial.println(entry.size(), DEC);
     }
     entry.close();
   }
}

void printSpaces(int num) {
  for (int i=0; i < num; i++) {
    Serial.print(" ");
  }
}

And I have an h file with not much in it, but I plan to populate it more. Now it basically just contains a struct that I plan to use for data formatting and logging:

Code:
/*
Header file for all things sensor setup
*/

// Include statements
# include <Arduino.h>
#include <LittleFS.h> // file system
#include <ICM_20948.h> // ICM

struct data {
    uint32_t t_millis;
    uint32_t t_micros;
    
    // accelerometer output
    float gx;
    float gy;
    float gz;

    // gyroscope output
    float wx;
    float wy;
    float wz;

    // magnetometer output
    float Tx;
    float Ty;
    float Tz;

};

For reference, I am mainly a mechanical engineer, who has learned some really scrappy EE and software stuff as needed (I designed the shield for the teensy 3.6, designed and fabricated the housing, made the firmware for it based on the lowlatency example, and built some software for analysis). I just took a year off all engineering projects, so I'm pretty darn rusty -- bear with me!

I have a few questions and general musings that I'd welcome comments on:
  1. Logging firmware options: I tried to retrofit the SDFat lowLatencyLogger example to work with the LittleFS file system, but am running into some issues with the sdFat and SDBaseFile classes having a bunch of differences to littleFS and File classes respectively. I'm not sure what would be better, working off the littleFS example code that I have attached, and building out that functionality, or trying to adapt the lowLatencyLogger example from SDFat to work with littleFS. I'd like to store this data to a binary file to keep file sizes as small as possible. And I want to make sure I enforce a consistent sampling frequency, of at least 500 Hz. I'll be logging somewhere in the ballpark of 18 measurements at any given time (~15 analog outputs, 9 from the IMU which is I2C and likely the sampling rate bottleneck, and time from the teensy's RTC, I think the 4's have rtc's on them now which is cool).
  2. Storing data on Teensy: Does anyone have a good place to understand the difference between program memory (I think this is flash?), RAM, QSPI, and SPI that are listed in Paul's littleFS readme? It seems the program memory has the most available storage so I should be using that, but I would like to better understand what's going on here. Does using the memory on the Teensy seem like a good possible alternative to what I was previously doing with a microSD card on the Teensy 3.6?
  3. Data transfer: once my files are stored on the device, I'll be transferring them to the computer and storing them in some database. I used serial transfer for the old device I made, but am thinking about using WiFi in order to not have to use a cord, and to hopefully get faster transfer rates. I'll use an esp32 chip for this with SPI communication. Anything I need to watch out for, or any reason I should choose Serial or Bluetooth over wifi for data transfer?
  4. PCB: For the PCB, I plan to use the Teensy 4.1 bootloader chip and make my own board entirely, unlike last time where I just designed the shield (that had an IMU, charging circuit, leds, analog inputs that interface with connectors, etc on it) and dropped a Teensy 3.6 breakout onto it. I plan to use USB-c instead of micro because I think it's cleaner. I found a good example of someone making a board around the teensy 4 on github. I think this seems pretty straightforward, but we'll see. I've gotten the boards made with all components placed on them before, but I will probably need to drop the teensy boot loader chip on myself unless I can ship it out to whatever company is building the boards. Any tips for soldering a little guy like that on myself?

I'd welcome any feedback, suggestions, or questions you guys have on the project. Happy to share any more info needed to clarify project details. I wanted to ask you guys for input before I get too deep. Thanks a ton!

-Matt
 
Having no SD seems to make this project a non-starter. Assuming that you compress your IMU data to 2 bytes per channel (i.e. 16 bits to store each axis of accel data instead of a 32 bit float) and 2 bytes for each analog channel, that's 48 bytes per frame (18 bytes for the IMU data, 30 bytes for the analog data). At 500 Hz, you're talking 86.4 MB of storage required.

I regularly build data acquisition systems with Teensy, but in all of the cases I buffer the data and write to SD.
 
Having no SD seems to make this project a non-starter. Assuming that you compress your IMU data to 2 bytes per channel (i.e. 16 bits to store each axis of accel data instead of a 32 bit float) and 2 bytes for each analog channel, that's 48 bytes per frame (18 bytes for the IMU data, 30 bytes for the analog data). At 500 Hz, you're talking 86.4 MB of storage required.

Good point -- I should have calculated this first thing haha. I guess the extra memory may allow me to store bigger buffers and then write larger chunks to SD? But I'm not sure how important optimizing that stuff is since usually my I2C devices are what bottleneck logging speeds.

Do you usually use the Teensy breakouts, or do you have any experience with making custom boards with the bootloader chips?
 
Good point -- I should have calculated this first thing haha. I guess the extra memory may allow me to store bigger buffers and then write larger chunks to SD? But I'm not sure how important optimizing that stuff is since usually my I2C devices are what bottleneck logging speeds.

Do you usually use the Teensy breakouts, or do you have any experience with making custom boards with the bootloader chips?

Yeah, I would use SPI to communicate with that IMU. I typically use a large circular buffer to buffer the data and then write it to SD in 512 byte chunks. I tend to use the IMU data ready interrupt to trigger data collection, so pseudo code would look like:

Code:
void daq() {
  // Read IMU
  // Read analog channels
  // Any filtering, state estimation, or telemetry
  // Add to data log buffer
}

void setup() {
  // Init sensors
  // Init datalog
  // Attach daq to IMU data ready interrupt
}

void loop() {
  // write data log buffer to SD
}

I do both Teensy breakouts or custom boards depending on the timeframe, budget, and other requirements like temperature ranges. Definitely faster, cheaper, and easier to just design a PCB that a Teensy 4.1 solders onto if it meets your requirements.
 
Back
Top