teensy 3.5 freezes when data logging

Status
Not open for further replies.
F

florisvb

Member
I am using a teensy 3.5 to log data from two serial devices. After an hour or two of successful logging, the teensy appears to freeze. If I unplug/replug it starts working again. I'm hoping someone might have some advice on what I can change about the code to make the system more robust.

Below is the full source code. Essentially it loops through the operations of reading serial data and saving to the SD card, and there's some diagnostic code to flash the LED so I can see what state it is in. It should be writing a new line of data at about 10-11 Hz. When the file it is writing to gets larger than 100 kb it creates a new unique filename to write to.

The two things I think could be happening are:
1. The way I'm opening / writing to / closing the file might not be the "right" way.. advice on how to improve that would be great
2. Perhaps the serial communication is freezing - is that possible, and if so, is there something I can do to skip a read if it is holding things up for too long?


Code: 
#include <Chrono.h> //Sketch > Include Library, and add the file downloaded from here: https://www.arduinolibraries.info/libraries/chrono
#include <TinyGPS.h>
#include <SoftwareSerial.h>
#include <SD.h>
#include <HardwareSerial.h>
#include <Arduino.h>
File myFile;// Creates "myFile" as the SD card related commands.
TinyGPS gps;
Chrono GPSTimer;
Chrono LEDLoop;
Chrono LEDGPS;
int good = 13; // LED pin on Teensy
String wind; // variable to hold wind data

float flat, flon;
unsigned long fix_age, time, date; //Date can be added in very easily.
unsigned long age;

unsigned long time_wind_read = 0;
unsigned int LEDstatus = 0;
unsigned int LEDperiod = 100;
unsigned int max_file_size = 100000;
char date_char[6];
char time_char[8];
char fileNameGPS[33];

void setup(){
  
  pinMode(good, OUTPUT); // sets pin 13 as LED OUTPUT pin
  
  Serial3.begin(115200);//, SERIAL_8N1, RX, TX); //BAUD rate of Trisonica mini.
  Serial4.begin(9600);// Baud rate of GPS (GP20U7)

  while (!SD.begin(BUILTIN_SDCARD)){ //Checks if there is an SD card.
        delay(2000);
        digitalWrite(good, HIGH);
        delay(200);// Blinks and waits until SD card is connected.
        digitalWrite(good, LOW);
  }

  
  strcpy(date_char, "unknown");
  strcpy(time_char, "unknown");
  strcpy(fileNameGPS, "unknown.TXT");
  
}

void loop(){

    // LED flashing for diagnostics
    if (LEDLoop.hasPassed(LEDperiod)) {
      LEDLoop.restart();
      if (LEDstatus==0) {
        LEDstatus = 1;
        digitalWrite(good, HIGH);
      }
      else {
        LEDstatus = 0;
        digitalWrite(good, LOW);
      }
    }

    // open file to write
    myFile = SD.open(fileNameGPS, FILE_WRITE); //Open most recently created file.
  
    // save data from wind sensor serial device
    if (Serial3.available()){
          wind = Serial3.readStringUntil('\n');//updates wind data when new data is available
          myFile.print("WIND:: ");
          myFile.print(" millis: ");
          myFile.print(millis());
          myFile.print(" " + wind); 
          myFile.print("\n");
          time_wind_read = millis();
          
          }
          
      //save GPS data with Teensy time-stamp
      if (Serial4.available()){
         char c = Serial4.read();
         gps.encode(c); //GPS parser.
         if (GPSTimer.hasPassed(500)){
              GPSTimer.restart();
              gps.f_get_position(&flat, &flon, &age);
              // time in hhmmsscc, date in ddmmyy
              gps.get_datetime(&date, &time, &fix_age);
              myFile.print("GPS:: ");
              myFile.print(" millis: ");
              myFile.print(millis());
              myFile.print(" UTCTime: ");
              myFile.print(time);//UTC Time stamp 7 hours ahead of Reno time.
              myFile.print(" UTCDate: ");
              myFile.print(date);
              myFile.print(" Latitude: ");
              myFile.print(flat, 6);//Latitude
              myFile.print(" Longitude: ");
              myFile.print(flon, 6);//Longitude
              myFile.print(" Satellite: ");
              myFile.print(gps.satellites() == TinyGPS::GPS_INVALID_SATELLITES ? 0 : gps.satellites());
              myFile.print("\n");

              // LED flashing for diagnostics
              if ((millis() - time_wind_read) < 100) {
                if (date!=0) {
                  LEDperiod = 25; // GPS and wind are writing
                }
                else {
                  LEDperiod = 100; // wind is writing, but not GPS
                }
              }
              else {
                if (date!=0) {
                  LEDperiod = 200; // GPS writing, but not wind
                }
                else {
                  LEDperiod = 1000; // nothing is writing
                }
              }
        }
        }

       // as soon as GPS date available switch to new file
       if (strcmp(fileNameGPS, "unknown.TXT")==0) {
         if (date!=0) {
          ltoa(time/100, time_char, 10);
          strcpy(fileNameGPS, "W"); // can only have 8 characters in filename!!!
          strcat(fileNameGPS, time_char);
          strcat(fileNameGPS, ".TXT");
         }
       }

       // every time file too big, make new file
       else {
         if (myFile.size() > max_file_size){
            ltoa(time/100, time_char, 10);
            strcpy(fileNameGPS, "W"); // can only have 8 characters in filename!!!
            strcat(fileNameGPS, time_char);
            strcat(fileNameGPS, ".TXT");
         }
       }

       //Save the data to the file. 
       myFile.close(); 

       
    }
 
I think I may have resolved this issue by moving the file open and close commands to right before/after the print commands. This ensures that the file is only being accessed if there is in fact data to write. As the code was written before, every loop would result in a file open/close command, even when it was not necessary!

That said, I would love to hear any suggestions for improving writing the data to the SD card, if there is a better way.
 
Open and close each file exactly once. If you want to make sure data is being written, use flush. Not too often as it can be really slow.
 
You do not need to open and close your file each time you write to it. Your code should look more like this:

Code: 
file myFile;
void setup(){
// generate initial file name, start timers, etc.
     myFile = SD.open(fileNameGPS, FILE_WRITE); //Open most recently created file.

}

void loop(){  
    //  If wind data available,  write it to myFile
    //  if  GPS data available,  write it to myFile
    if (myFile.size() > max_file_size){
        myFile.close();
        // make a new file name
        myFile = SD.open(fileNameGPS, FILE_WRITE); //Open with new file name
      }
      //  to make sure the file directory gets updated regularly call myFile.sync()
      //  about every 10 seconds or so
}  // end of loop()

Another thing you have to worry about is that GPS units can have very 'bursty' output--a lot of NMEA messages come out at once.
If the process of opening and closing the file or a file write takes too long, the GPS output can overflow the standard 64-byte serial receive buffer and you will lose some data. Your options then are to use an intervalTimer to collect GPS data into a larger secondary buffer, then pull the data from that buffer, parse it and write to the file in your loop() function.

Note that SD Card writes can take up to several hundred milliseconds when a file write has to erase a new flash block on the SD Card. A GPS communicating at 9600 Baud can overflow the serial buffer in about 66 milliseconds if it sends NMEA packets one right after the other. If your trisonics wind sensor sends more than 64 bytes at a time, it could be even more problematic with its higher baud rate.

NOTE: these are examples of the kind of problems I solved with my generic data logger object.
 
Thanks for the explanation! I don't think my GPS is overloading the buffer, but the wind sensor probably is, as every once in a while some data seems to be missing. It's not a big enough problem that I've been worried about it, but would be nice to fix that.

Where can I find your generic data logger object and some examples on how to incorporate it into my code?
 
The SD specification shows a maximum FAT write time which can be up to 750ms. This is the time to update both copies of the FAT and the directory entry. Each of which probably requires a read/modify/write process.

Things can get worse if allocating a new cluster requires scanning a lot of sectors.

The best way to deal with this is not compatible with the serial library code as it requires processing the serial data in the ISR, storing it into a buffer, and having the foreground task handle the job of writing full buffers to the SD card. That is the way I structure my data logging code.
 
florisvb said:
Do you have an example you can share?
Click to expand...

A couple but they might confuse more than they help. The first is something I wrote long ago for the SparkFun Logomatic. Straight up C and Arduino libraries nowhere to be found. My effort for the Teensy 3.6 was written in Forth.

I really need to find a new home for my web pages since Earthlink dropped that service.
 
florisvb said:
Thanks for the explanation! I don't think my GPS is overloading the buffer, but the wind sensor probably is, as every once in a while some data seems to be missing. It's not a big enough problem that I've been worried about it, but would be nice to fix that.

Where can I find your generic data logger object and some examples on how to incorporate it into my code?
Click to expand...

The generic logger code and examples were posted in the "Project Guidance: forum a month or two ago. You can search for "generic data logger:.

Here is a mod of your code that I've simplified a bit and added a secondary queue for the GPS serial input. It's only been running for about 20 minutes, so far.
I'll do a longer test later this evening.

Code: 
/******************************************************************************
 * Simplified GPS logger with 2KB secondary GPS serial buffer
 * M. Borgerson   8/4/2020
 * Adapted from code posted by florisvb  on 7/30/2020
 * 
 * Tested on T3.6 at 48MHz with GP20U7 GPS module
 * 
 */

#include <TinyGPS.h>
#include <SD.h>

File myFile;// Creates "myFile" as the SD card related commands.
TinyGPS gps;

const int ledpin = 13; // LED pin on Teensy
String wind; // variable to hold wind data

float flat, flon;

// changed time and date names to gpstime and gpsdate
uint32_t fix_age, gpstime, gpsdate; //Date can be added in very easily.
uint32_t age;

uint32_t max_file_size = 4000;   // Note small size for testing! new file about every 18 seconds
char date_char[6];
char time_char[8];
char fileNameGPS[33];

IntervalTimer GPSTimer;


/**************************************************************************
*  there are many ways to implement a serial data queue.
*  This version uses simple C code.
* **********************************************************************/
#define GPSQMAX 2048
char gpsQdata[GPSQMAX];
volatile uint16_t gpsQlen = 0;   // must be volatile because they are changed in interrupt handler
volatile uint16_t gpsQhead = 0;
uint16_t gpsQtail = 0;


// this function is called by the interval timer interrupt handler
void GPSQPut(char ch) {
  if (gpsQlen < GPSQMAX) { // queue not full, so add data
    gpsQdata[gpsQhead] = ch;
    gpsQhead++;  // move to next buffer location
    if (gpsQhead >= GPSQMAX) gpsQhead = 0; // circle back to beginning
    gpsQlen++;
  }
  // if queue is full, nothing is saved
}

// This function is called from loop() to retrieve buffered characters
// it is only called when gpsQlen > 0 ;
char GPSQGet() {
  char ch;
  ch = gpsQdata[gpsQtail];
  gpsQtail++;
  if (gpsQtail >= GPSQMAX) gpsQtail = 0; // circle back to beginning
  noInterrupts();  // stop interrupts while decrementing Q length
  gpsQlen--; // decrement length
  interrupts();

  return ch;
}

// this is the handler for the GPSTimer interrupt
// it checks for input data on Serial4 and puts incoming
// data in the gpsQ
void GPSHandler(void) {
  char ch;
  while (Serial4.available()) {
  ch = Serial4.read();
    GPSQPut(ch);
  }
}

void setup() {

  pinMode(ledpin, OUTPUT); // sets pin 13 as LED OUTPUT pin

  Serial3.begin(115200);//, SERIAL_8N1, RX, TX); //BAUD rate of Trisonica mini.
  Serial4.begin(9600);// Baud rate of GPS (GP20U7)

  while (!SD.begin(BUILTIN_SDCARD)) { //Checks if there is an SD card.
    delay(2000);
    digitalWrite(ledpin, HIGH);
    delay(200);// Blinks and waits until SD card is connected.
    digitalWrite(ledpin, LOW);
  }
  // Now start the interval timer to fill the queue
  GPSTimer.begin(&GPSHandler, 10000); // triggers each 10,000uSec or at 100Hz

}

void loop() {
  bool goodSentence;
  uint16_t gpsSats;
  char gpsOutput[120];
  // removed LED flashing for diagnostics

  // removed wind sensor stuff

  //save GPS data with Teensy time-stamp
  if (gpsQlen > 0) {
    digitalWrite(ledpin, HIGH);
    char c = GPSQGet();

    digitalWrite(ledpin, LOW);
    goodSentence = gps.encode(c); //GPS parser returns true good packet decoded
    // NOTE:  Since more than one type of sentence is received, you will see multiple lines of 
    //        output each second. Only the millis value will be different.
    if (goodSentence) {
      gps.f_get_position(&flat, &flon, &age);
      // time in hhmmsscc, date in ddmmyy
      gps.get_datetime(&gpsdate, &gpstime, &fix_age);
      // get number of valid satellites   Use zero when none valid
      gpsSats = gps.satellites() == TinyGPS::GPS_INVALID_SATELLITES ? 0 : gps.satellites();
      // keep the same verbose output as original example
      snprintf(gpsOutput,119, "GPS:: millis: %6lu  UTCTime:%8lu  UTCDate:%6lu Latitude: %9.6f Longitude: %9.6f Satellites: %u\n",
               millis(), gpstime, gpsdate, flat, flon, gpsSats);

      // the file object will return true if the file is open
      if (!myFile) { // try to open a new file
        // open file to write
        MakeGPSFileName(fileNameGPS); // create name with embedded time
        Serial.printf("Opening File: % s\n", fileNameGPS);
        myFile = SD.open(fileNameGPS, FILE_WRITE); //Open most recently created file.
      }
      if (myFile) { // file is open, write the data
        myFile.print(gpsOutput);
        Serial.print(gpsOutput);   // comment this out if you don't want to see every line
        Serial.send_now();
      } else {
        Serial.println("Could not open GPS File!");
      }

    }  // end of if(goodSentence)...

  } // end of if(gpsQlen > 0)...

  if (myFile.size() > max_file_size) {
    myFile.close();
    Serial.println("File Closed");
    // with next good GPS sentence, code will note that file is not open and start a new file
  }


}   // end of loop()...


// create a new file name at location pointed to by fnptr
// should only be called after GPS has locked and gpstime is valid
void  MakeGPSFileName(char *fnptr) {
  // this could be put inline
  snprintf(fnptr,12, "W%06lu.txt", gpstime / 100); // 8.3 file name is 12 chars max
}
 
Last edited:
I ran the posted example code overnight and it collected good data for about 16 hours.

Once you get the GPS collection and file open/close stuff under control, you can start to work on the really interesting part: How to efficiently merge and store the data from different sensors collecting at different rates. Over a few decades of collecting oceanographic data, I've settled on the "Collect it all and let Matlab sort it out" method. That usually means defining a binary record structure containing all the data. For an application with a wind sensor at 4Hz and 1Hz GPS it might look something like this:

struct WindGPS {
struct windbinary[4];
struct GPSbinary;
}

Note that I've skipped all the internal structure stuff and the typedefs needed to get it to compile. If florisvb's application involves collecting wind data from a moving platform, the GPS binary data should include the speed and course data to allow separating true wind from the apparent wind generated by the platform speed.
 
I'm glad you got something working. Your "Collect during yield" approach should work OK, since the SD routines will keep checking for data during the longer SD card write intervals.

Note: If you are using SDFat 2.0B, it supports long file names, so you can get more expressive with file names. I generally uses something like Prefix_YYMMDDHH.NNN, where the file start time is encoded in the name and the extension is the unit serial number. When you do that, the files in a directory listing show up in chronological order and the extensions keep the instruments separated. That makes it easier to select time ranges and instruments for later processing. For the oceanographic loggers I worked with, Prfefix was usually a cruise or mooring identifier. A lot of the instruments were moored for 6 to 9 months and, with a file every day, you ended up with a LOT of files. The scientists often wrote very complex Matlab programs to process the data.

I'll check out your Python script. I've been meaning to switch to Python for my PC host programs for quite a while. I've spent over a decade writing host programs in versions of Borland/Embarcadero C++ Builder. You end up with a nice GUI application, but it is difficult to distribute the .exe files as most email systems block .exe files---even if they are zipped.
 
Status
Not open for further replies.
Back
Top