Need help saving serial data to SD card

Ash

Member
I am having two Teensy 4.1 (lets call them T1 and T2) setup in such a way that T1 reads data from the sensors and sends them at 100 Hz to T2 via Serial 5. Because I have control over the string sent by T1, I also send out a '\r' in the serial to let T2 know that a full string has been received. This is working fine as I want it to.

The problem comes when I am trying to store the data received by T2 into the SD card. I am also naming the file based on the date provided by my RTC module and also index the file based on the number of lines I have stored in the file. What happens is that now the serial data I am receiving gets garbled and T2 keeps restarting. In addition, the file names in the SD card are garbled as well.
Here is a screen shot of the serial monitor
Serial Output.JPG

Here is a screen shot of the file name in the SD card
SD file_names.JPG

I have even tried storing the serial data with a constant file name instead, and even that isn't changing anything. If I comment out the parts involving SD card, the received serial data in T2 is exactly as sent by T1.

Would anyone know what is causing my problems or if there is a better way to the SD write? Below is the code that i am using for T2:
Code:
#include <SD.h>
#include <RTClibw1.h>
#include <SPI.h>
#include <Wire.h>

 //SD CARD
const int chipSelect = BUILTIN_SDCARD;
File dataFile ;
boolean file_w = false;
uint8_t file_idx=0;
static boolean write_flag=false;
char file_name[13];

//Serial read
char rx_arr[90];
static uint8_t ndex=0;
static uint16_t line_count = 0;


//Time for SD File
RTC_DS3231 rtc;
static uint8_t rtcYear, rtcMonth, rtcDay;



#define chip1 Serial5

void setup()
{
  Serial.begin(115200);
  chip1.begin(115200);
  
  String sktch = __FILE__;
  sktch = (sktch.substring((sktch.indexOf("\r")), (sktch.lastIndexOf("\\")) + 1));
  Serial.print("Sketch Name: ");
  Serial.println(sktch);
  Serial.print("Flashed On : ");
  Serial.println(F(__DATE__ " " __TIME__));

  delay(500);

  Serial.print("Initializing SD card...");
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect))
  {
    Serial.println("Card failed, or not present");
    while (1) ;
  }
  Serial.println("SD Card Found.");
  
  
  if (! rtc.begin()) {
    Serial.println("Couldn't find RTC");
    while (1);
  }
  rtc.adjust(DateTime(__DATE__, __TIME__));
  
  DateTime now = rtc.now();
  rtcDay = now.day();
  rtcMonth = now.month();
  rtcYear = now.year()%100;
  
  delay(3000);
}

void loop()
{
   receive_data();
   if (write_flag == true)
      record_data();
}

void receive_data()
{
  while(chip1.available())
  {
    char datachar = chip1.read();
    rx_arr[ndex] = datachar;

    if(datachar=='\r')
      {
        rx_arr[ndex]='\0';
        ++line_count;

        Serial.println(rx_arr);
        write_flag = true;

        ndex=0;
        if (line_count == 0)
          {
            file_idx = file_idx+1;
          }
        break;
      }
     else
       write_flag = false;
      
    ndex = ndex + 1;
  }
}

void record_data()
{

  DateTime now = rtc.now();

    // change file name & index if on a different date
    if((rtcDay != now.day()))
    {
      rtcYear = now.year()%100;
      rtcMonth = now.month();
      rtcDay = now.day();
      file_idx = 0;
    }
  sprintf(file_name, "%02u%02u%02u_F%03u.txt", rtcYear, rtcMonth, rtcDay, file_idx);
  dataFile  = SD.open(file_name, FILE_WRITE);
  dataFile.println(rx_arr);
  dataFile.close();
  write_flag=false;
}
 
Code:
char file_name[13];
You are printing 15 characters into the filename. The array should be at least 16 chars long to hold the NULL as well.

Pete
 
Thanks for pointing that out Pete. I did change the size to test out the code with a constant file name and forgot to change it back. I have now changed it to 16 and tested it again just to make sure. All the problems with writing to SD card is still present.
 
It seems your rx_arr is too short and overflows. There are more than 90 characters in the last line printed in the serial monitor
 
The character string sent from my T1 will have a maximum of 80 characters (including the string termination character). I have tested the rx_arr without the SD card code and there is no problems in that. What you are seeing in the image of the serial monitor is the garbled one where portions of two lines have been printed together.
 
The character string sent from my T1 will have a maximum of 80 characters (including the string termination character). I have tested the rx_arr without the SD card code and there is no problems in that. What you are seeing in the image of the serial monitor is the garbled one where portions of two lines have been printed together.

I see three possible issues:
1. if the /r char does not come every 80 chars, then your ndex will point to bytes outside the limits of the rxchar[] buffer. Try to stop incrementing ndex if it is >= 79.
2. the file.println may not be compatible with 0 terminated char arrays. (not sure)
3. SD card access is ‘blocking’ code. With this coding approach, the cpu may not be given the option to swallow all the incoming serial data. It will thus miss some chars. Possibly the \r. And the your ndex overflows…

The serial buffer sizes are small in the default T4 serial ports. And some ports get smaller buffers than others. Buffer sizes can be increased though.

A better approach would be to do the SD card actions from loop() and check the serial input in a timer interrupt service routine. With a decent size (a few kB) elastic buffer in between. That way the SD card access gets a lower priority than the task of reading and swallowing the serial input. So if the SD card needs say >100 ms for its writing actions, nothing gets lost or overflows.
 
I am confident that /r being sent in the Serial to my T2. Else, I wouldn't be able to see the rx_arr printed on my Serial monitor.

Did some further digging into the code by commenting out all the SD related lines and keeping only the Serial read related lines. I printed out the states of the variables on the Serial monitor just to monitor that everything is working as intended.

I am surprised to see that that line_count variable is not showing the correct value. I am seeing that every time I disconnect and reconnect my T2 it starts at a random value as shown below
Serial Output_new.JPG

What surprises me even more is that when I close the serial monitor and open it back again, the line_count variable starts again from some value in the ~12000 range
line_count bug.jpg
 

Attachments

  • Serial Output_new.JPG
    Serial Output_new.JPG
    27.5 KB · Views: 81
Last edited:
I am confident that /r being sent in the Serial to my T2. Else, I wouldn't be able to see the rx_arr printed on my Serial monitor.

Did some further digging into the code by commenting out all the SD related lines and keeping only the Serial read related lines. I printed out the states of the variables on the Serial monitor just to monitor that everything is working as intended.

I am surprised to see that that line_count variable is not showing the correct value. I am seeing that every time I disconnect and reconnect my T2 it starts at a random value as shown below
View attachment 31900

What surprises me even more is that when I close the serial monitor and open it back again, the line_count variable starts again from some value in the ~12000 range
View attachment 31899

Ash, your global variables are defined like this:
char rx_arr[90];
static uint8_t ndex=0;
static uint16_t line_count = 0;

So in the Teensy RAM memory the locations ndex and line_count are effectively where rx_arr[91…93] are also.
With line_count showing up as random numbers, i’d argue that you have been writing to rx_arr[89++].

That’s possible because you do not check if ndex goes above 89…

If your code ever misses just one /r, then this can happen.
If the other Teensy that sends the serial data ever sends out a longer string than this already happens. Say a welcome message at startup. Or a power down.

Solution for now: change the line with
ndex++;

into
if (ndex<89) ndex++;

and change

if(datachar=='\r')

into

if((datachar=='\r') || (ndex > 88))


Also be aware that /r and /n can mean different things for different computers…
 
Last edited:
Solution for now: change the line with
ndex++;

into
if (ndex<89) ndex++;

and change

if(datachar=='\r')

into

if((datachar=='\r') || (ndex > 88))

implemented this change and now the line_count variable is displaying the right values. Also noted the file indexing on the SD card seems to work with this change. Just curious why it would miss the \r character when reading when Serial.read() acquires one character every iteration of the loop?

Also the incoming serial data is now garbled every once in a while, where some data on the current line is being left out, next few incoming lines are skipped and merged with the next set of data.

SDwrite_problems.JPG
 
implemented this change and now the line_count variable is displaying the right values. Also noted the file indexing on the SD card seems to work with this change. Just curious why it would miss the \r character when reading when Serial.read() acquires one character every iteration of the loop?

Also the incoming serial data is now garbled every once in a while, where some data on the current line is being left out, next few incoming lines are skipped and merged with the next set of data.

View attachment 31951
Explained in #6 above.
I think you are using Serial5. That has by default only 64 bytes of ‘elastic’ FIFO buffer. Your messages have more bytes already.
Try adding serial buffer memory before chip1.begin()
with chip1. addMemoryForRead(void *buffer, size_t length), with a static buffer of say 1024 bytes.
 
Explained in #6 above.
I think you are using Serial5. That has by default only 64 bytes of ‘elastic’ FIFO buffer. Your messages have more bytes already.
Try adding serial buffer memory before chip1.begin()
with chip1. addMemoryForRead(void *buffer, size_t length), with a static buffer of say 1024 bytes.

Just to explain why you need a larger RX buffer. In your loop() function, when record_data() is called, it will usually be pretty fast, but every once in a while, the SD card must do some "housekeeping", and when it does, record_data() may not return for as much as 40-50 ms. During those long delays, if bytes are coming in, the RX buffer will fill up and overflow, so by the time your program gets back to calling chip1.available(), it will already have missed data, and what is in the buffer may not be a valid message, or could be missing the '\r' character, etc. Increasing the size of the RX buffer using addMemoryForRead() will ensure that when record_data() takes a long time, there will be enough space in the RX buffer for all of the bytes that arrive during that delay.

Code:
void loop()
{
   receive_data();
   if (write_flag == true)
      record_data();
}
 
Just to explain why you need a larger RX buffer. In your loop() function, when record_data() is called, it will usually be pretty fast, but every once in a while, the SD card must do some "housekeeping", and when it does, record_data() may not return for as much as 40-50 ms. During those long delays, if bytes are coming in, the RX buffer will fill up and overflow, so by the time your program gets back to calling chip1.available(), it will already have missed data, and what is in the buffer may not be a valid message, or could be missing the '\r' character, etc. Increasing the size of the RX buffer using addMemoryForRead() will ensure that when record_data() takes a long time, there will be enough space in the RX buffer for all of the bytes that arrive during that delay.

Thanks for explaining for explaining this. Based on the above replies in #10 and #11, I have declared a global serial buffer of type char - char serial_buffer[1024] and then chip1.addMemoryForRead(serial_buffer, sizeof(serial_buffer)) before chip1.begin().

Still seeing the scrambling of serial data just as before, may be a little less frequent.

Also tried something that popped up in my mind, which was to increase the serial baud rate at the teensy module (T2) receiving the data. I changed it to 230400 just to reduce the dependency on the serial buffer but all I see is square boxes and some random characters. Only 115200 seems to work. (I am sending the data from my first module (T1) at 115200)
 
The T1 and T2 baud rate must match and be set to the same value - either both 115200 or 230400.
Confirm both T1 and T2 share a common Ground pin as well as the Rx and Tx wires.

Not seeing updated code since p#1 - and assuming there are no places where interrupts are disabled?
Just to eliminate some chances for overrun and allow function try making:
Code:
//Serial read
char rx_arr[90];
static uint8_t ndex=0;

into:
Code:
//Serial read
char rx_arr[[B]512[/B]];
static [B]uint16_t [/B]ndex=0;
 
The T1 and T2 baud rate must match and be set to the same value - either both 115200 or 230400.
Confirm both T1 and T2 share a common Ground pin as well as the Rx and Tx wires.

Both T1 and T2 are running at 115200. One ground pin on T1 is connected to a ground pin on T2 and only the T1-TX pin is connected to T2-RX, as I am doing a one-way data transfer.
And I am not using any interrupts in my code.

With the above mentioned changes to the variable sizes and types, still no luck. The updated code is below:
Code:
#include <SD.h>
#include <RTClibw1.h>
#include <SPI.h>
#include <Wire.h>

 //SD CARD
const int chipSelect = BUILTIN_SDCARD;
File dataFile ;
boolean file_w = false;
uint8_t file_idx=0;
static boolean write_flag=false;
char file_name[16];

//Serial read
char rx_arr[512];
static uint16_t ndex=0;
static uint16_t line_count = 0;
char serial_buffer[1024];

//Time for SD File
RTC_DS3231 rtc;
static uint8_t rtcYear, rtcMonth, rtcDay;



#define chip1 Serial5

void setup()
{
  Serial.begin(115200);
  chip1.addMemoryForRead(serial_buffer, sizeof(serial_buffer));
  chip1.begin(115200);
  
  String sktch = __FILE__;
  sktch = (sktch.substring((sktch.indexOf("\r")), (sktch.lastIndexOf("\\")) + 1));
  Serial.print("Sketch Name: ");
  Serial.println(sktch);
  Serial.print("Flashed On : ");
  Serial.println(F(__DATE__ " " __TIME__));

  delay(500);

  Serial.print("Initializing SD card...");
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect))
  {
    Serial.println("Card failed, or not present");
    while (1) {
      // No SD card, so don't do anything more - stay stuck here
    }
  }
  Serial.println("SD Card Found.");
  
  
  if (! rtc.begin()) {
    Serial.println("Couldn't find RTC");
    while (1);
  }
  rtc.adjust(DateTime(__DATE__, __TIME__));
  
  DateTime now = rtc.now();
  rtcDay = now.day();
  rtcMonth = now.month();
  rtcYear = now.year()%100;

  delay(3000);
}

void loop()
{
   receive_data();
   if (write_flag == true)
      record_data();
}

void receive_data()
{
  while(chip1.available())
  {
    char datachar = chip1.read();
    rx_arr[ndex] = datachar;
    if((datachar=='\r') || (ndex > 88))
      {
        rx_arr[ndex]='\0';
        line_count = line_count + 1;
        Serial.println(rx_arr);
        write_flag = true;

        ndex=0;
        if (line_count == 0)
          {
            file_idx = file_idx+1;
          }
        break;
      }
     else
       {
        write_flag = false;
        //ndex = ndex + 1;
       }
      
    if(ndex<89)
      ndex = ndex + 1;
  }
}

void record_data()
{

  DateTime now = rtc.now();

    // change file name & index if on a different date
    if((rtcDay != now.day()))
    {
      rtcYear = now.year()%100;
      rtcMonth = now.month();
      rtcDay = now.day();
      file_idx = 0;
    }
  sprintf(file_name, "%02u%02u%02u_F%03u.txt", rtcYear, rtcMonth, rtcDay, file_idx);
  dataFile  = SD.open(file_name, FILE_WRITE);
  dataFile.println(rx_arr);
  dataFile.close();
  write_flag=false;
}
 
Something is off - seems that should be working.

90 chars 100 times/sec when UART can transfer 11,520 chars/sec - that shouldn't be difficult. The T_4.1 hardware FIFO isn't big (4 bytes?) - as long as interrupts aren't disabled though the service routines keep it flushed to the storage buffer.

Beta testing involved transferring across all 8 T_4.1 (or 7 T_4.0) UART ports at higher rates with verified data integrity. Both port to port on same Teensy or back and forth with a second Teensy.

With a reliable GND between the T1 and T2 the length and quality of the Tx to Rx wire would be suspect to check.
> Perhaps: replace T2 code with nothing but a loop reading available characters and Serial.print() to confirm the characters are arriving as expected { add a test for '\r' and then add a Serial.println() }
> that should show the expected repeating groups on regualr length lines.

The code for the Tx Teensy T1 isn't shown where the output chars are delivered with the '\r' inserted.
> perhaps it isn't properly constructing or transmitting the expected character data?
 
With a reliable GND between the T1 and T2 the length and quality of the Tx to Rx wire would be suspect to check.
> Perhaps: replace T2 code with nothing but a loop reading available characters and Serial.print() to confirm the characters are arriving as expected { add a test for '\r' and then add a Serial.println() }
> that should show the expected repeating groups on regualr length lines.

It only happens when the SD card portion of the code is included. Commenting out the following lines in the loop, prints out the serial data perfectly.
Code:
   if (write_flag == true)
      record_data();

Also tried it on a separate sketch printing out the serial data just to make sure, and I didn't see any scrambling happen.

Here is the code for my T1 which I am using to send the data. The string being sent is in the log_data() function :
Code:
#include <Wire.h>
#include <SPI.h>
#include <math.h>

// GPS
char SerChar;
static float latitude, longitude, speed_kph;
char gpsbuffer[85],gpstime[11];
int ndex = 0;
static boolean readGPS = true, GPSLock = false, firstFix = false;

#define gps Serial2
#define PMTK_SET_NMEA_UPDATE_1HZ  "$PMTK220,1000*1F"
#define PMTK_SET_NMEA_UPDATE_5HZ  "$PMTK220,200*2C"
#define PMTK_SET_NMEA_UPDATE_10HZ "$PMTK220,100*2F"

// turn on only the second sentence (GPRMC)
#define PMTK_SET_NMEA_OUTPUT_RMCONLY "$PMTK314,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0*29"
// turn on GPRMC and GGA
#define PMTK_SET_NMEA_OUTPUT_RMCGGA "$PMTK314,0,1,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0*28"
// turn on ALL THE DATA
#define PMTK_SET_NMEA_OUTPUT_ALLDATA "$PMTK314,1,1,1,1,1,1,0,0,0,0,0,0,0,0,0,0,0,0,0*28"
// turn off output
#define PMTK_SET_NMEA_OUTPUT_OFF "$PMTK314,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0*28"

#define PMTK_Q_RELEASE "$PMTK605*31"


// Accelerometer
const int MPU = 0x68; // MPU6050 I2C address when AD0 - GND
static double AccX, AccY, AccZ;
static double X_err, Y_err, Z_err;
int c = 0;

// Time Keeping
static uint16_t tick = 0;
static const uint8_t Time_Int = 9; // reading at 10/1000 second
static uint32_t CurrTime, PrevTime;

// Logging
static boolean log_en = false;      // Default at FALSE state

#define chip2 Serial6
char serial_buffer[1024];

void setup()
{
  chip2.addMemoryForWrite(serial_buffer, sizeof(serial_buffer));
  chip2.begin(115200);
  Serial.begin(115200);
  // Print Sketch Version & Name
  String sktch = __FILE__;
  sktch = (sktch.substring((sktch.indexOf("\r")), (sktch.lastIndexOf("\\")) + 1));
  Serial.print("Sketch Name: ");
  Serial.println(sktch);
  Serial.print("Flashed On : ");
  Serial.println(F(__DATE__ " " __TIME__));

  gps.begin(9600);
  gps.println(PMTK_SET_NMEA_OUTPUT_RMCONLY);
  gps.println(PMTK_SET_NMEA_UPDATE_5HZ);      //options - 1 Hz, 5 Hz, 10 Hz

  delay(500);

 // Setting up Accelerometer
  Wire.begin();                      // Initialize comunication
  Wire.beginTransmission(MPU);       // Start communication with MPU6050 // MPU=0x68
  Wire.write(0x6B);                  // Talk to the register 6B
  Wire.write(0x00);                  // Make reset - place a 0 into the 6B register
  Wire.endTransmission(true);        //end the transmission

  // Configure Accelerometer Sensitivity - Full Scale Range (default +/- 2g)
  Wire.beginTransmission(MPU);
  Wire.write(0x1C);                  //Talk to the ACCEL_CONFIG register (1C hex)
  Wire.write(0x10);                  //Set the register bits as 00010000 (+/- 8g full scale range)
  Wire.endTransmission(true);

  calculate_IMU_error();
  Serial.println("NOW RECORDING");
  delay(5000);

}

void loop()
{
  getGPS();
  if (GPSLock && log_en == true && firstFix == true)
  {
    log_data();
  }
}

void getGPS()
{
  while (gps.available() && readGPS == true)
  {
    SerChar = gps.read();
    gpsbuffer[ndex] = SerChar;
    ndex = ndex + 1;

    if (SerChar == '\n')
    {
      ndex = ndex - 1;
      gpsbuffer[ndex] = '\0';
      Serial.println(gpsbuffer);

      parseGPRMC(&latitude, &longitude, &speed_kph);
      Serial.println(F("---------------------"));
      Serial.print(F("Latitude: ")); Serial.println(latitude, 6);
      Serial.print(F("Longitude: ")); Serial.println(longitude, 6);


      if (abs(latitude) > 1.00 && abs(longitude) > 1.00)
      {
        GPSLock = true;
        log_en = true;
        firstFix = true;
        ndex = 0;
        break;
      }

      else
      {
        GPSLock = false;
        firstFix = false;
        log_en = false;
      }

      ndex = 0;
    }

  }
}


void log_data()
{
  CurrTime = millis();
  char dataChar[90];

  if (CurrTime - PrevTime > Time_Int)
  {
    Wire.beginTransmission(MPU);
    Wire.write(0x3B); // Start with register 0x3B (ACCEL_XOUT_H)
    Wire.endTransmission(false);
    Wire.requestFrom(MPU, 6, true); // Read 6 registers total, each axis value is stored in 2 registers

    AccX = ((int16_t)((Wire.read() << 8 | Wire.read())) / 4096.00) - X_err ; // X-axis value
    AccY = ((int16_t)((Wire.read() << 8 | Wire.read())) / 4096.00) - Y_err; // Y-axis value
    AccZ = ((int16_t)((Wire.read() << 8 | Wire.read())) / 4096.00) - Z_err ; // Z-axis value

    sprintf(dataChar, "%s , %d , %u, %.6f , %.6f , %.2f , %.2f , %.2f , %.2f" , gpstime, uint8_t(CurrTime - PrevTime), tick, latitude , longitude, speed_kph, AccX, AccY, AccZ);

    if (tick == 65535)
      tick = 0;
    else
      tick = tick + 1;
    PrevTime = CurrTime;

    //Serial.println(dataChar);   //prints string to send

    chip2.print(dataChar);  //only send content in string
    chip2.print('\r');  //Send return character to mark end of serial data 
  }

}

void calculate_IMU_error() 
{
  while (c < 500)
  {
    Serial.println(c);
    Serial.println("Calibrating...");
    
    Wire.beginTransmission(MPU);
    Wire.write(0x3B);
    Wire.endTransmission(false);
    Wire.requestFrom(MPU, 6, true);
    AccX = (int16_t)(Wire.read() << 8 | Wire.read()) / 4096.00 ;
    AccY = (int16_t)(Wire.read() << 8 | Wire.read()) / 4096.00 ;
    AccZ = (int16_t)(Wire.read() << 8 | Wire.read()) / 4096.00 ;
    
    X_err += AccX;
    Y_err += AccY;
    Z_err += AccZ;

    c++;
  }


  X_err = X_err / 500.00;
  Y_err = Y_err / 500.00;
  Z_err = Z_err / 500.00;

  Serial.println("X_Err: ");
  Serial.print(X_err);
  Serial.print("  AccErrorY: ");
  Serial.print(Y_err);
  Serial.print("  AccErrorZ: ");
  Serial.println(Z_err);
}


boolean parseGPRMC(float *lat, float *lon, float *speed_kph)
{
  char *tok;                         // Dummy Variable
  tok = strtok(gpsbuffer, ",");    // GPS Header
  //    if(strcmp(tok,"$GNRMC"))
  //      {
  strncpy(gpstime,strtok(NULL, ","),6);         // UTC Time
  gpstime[6] = '\0';
  tok = strtok(NULL, ",");         // Pos Status

  char *latp = strtok(NULL, ",");  // Latitude
  if (! latp) return false;
  char *latdir = strtok(NULL, ","); //Latitude Direction
  if (! latdir) return false;

  char *longp = strtok(NULL, ",");  // Longitude
  if (! longp) return false;
  char *longdir = strtok(NULL, ","); // Longitude Direction
  if (! longdir) return false;


  if (speed_kph != NULL)
  {
    char *speedp = strtok(NULL, ","); 
    if (! speedp) return false;

    *speed_kph = atof(speedp) * 1.852 ; //convert to km/h
  }


  double latitude = atof(latp);
  double longitude = atof(longp);

  // convert latitude from minutes to decimal
  float degrees = floor(latitude / 100);
  double minutes = latitude - (100 * degrees);
  minutes /= 60;
  degrees += minutes;

  // turn direction into + or -
  if (latdir[0] == 'S') degrees *= -1;

  *lat = degrees;

  // convert longitude from minutes to decimal
  degrees = floor(longitude / 100);
  minutes = longitude - (100 * degrees);
  minutes /= 60;
  degrees += minutes;

  // turn direction into + or -
  if (longdir[0] == 'W') degrees *= -1;

  *lon = degrees;

  
  (void) tok;
  
  char *ptr = strchr(gpstime,'.');
  if(ptr !=NULL)
    *ptr = '\0';
  (void) ptr;
  
  return true;
}
 
It only happens when the SD card portion of the code is included.

I'll be able to do some testing with 2 x T4.1 tomorrow, but in the meantime:

in log_data(), there is no protection against overwriting data_array[]. If that happens, because dataChar[] is on the stack, you'll likely get a crash/restart or some impossible-to-predict behavior. The easiest way to be sure you don't overwrite a C string buffer is to use snprintf() instead of sprintf(). snprintf() takes the size of the buffer as an argument and ensures that only that many bytes (including a terminating NULL character), get written to the string. instead of sprintf(dataChar,...) you could use snprintf(dataChar, sizeof(dataChar), ...)

Do the same thing for writing to gpstime[] and anywhere else you are using sprintf().

On the logging side, it's a little worrying to open/close a file every 10 ms. If one file is for a whole day, you have 86400 sec by 100 records/sec * 80 bytes/record = ~700MB. I'm really not sure what you might run into doing open/write/close over 8M times to write 700MB. Perhaps the worst case delays are much longer than the 50 ms I suggested previously, and the 1024 buffer you've allocated is not enough. That's only about 16 records, or 160 ms, so you could try increasing that to 4096 or even larger, so you know it's not an issue.
 
I did some testing, and I pretty quickly saw on the receive side an instance where a partial record was displayed, followed by a good record where the "tick" value increased by 40. I'm guessing this means that SD delay can be this long (400 ms?). I increased the serial buffer on the RX side to 16*1024, and I haven't seen any problems since.
 
I did some testing, and I pretty quickly saw on the receive side an instance where a partial record was displayed, followed by a good record where the "tick" value increased by 40. I'm guessing this means that SD delay can be this long (400 ms?). I increased the serial buffer on the RX side to 16*1024, and I haven't seen any problems since.

Indeed, the SD 'bookkeeping' delays can exceed 100ms. Thought to mention that reading the '50ms' value as posted - but the max value can depend on SD card in use and possibly the software ordering (open/write/close).

There are some posted observed values and there is a test example sketch in the library that will run and show Max/Min times IIRC.

Also buffering the data into 512 byte blocks is much more efficient at reducing overhead - though if a media block isn't currently on a 512B boundary it seems it would still result in two block updates rather than one clean block update. Buffering at 4KB seems the sweet spot if possible - some data might be lost ( some 46 records? ) but that would make the writes and open/close happen closer to twice a second ( 500ms instead of 10ms ) and probably result in more uniform timing and perhaps longer media life.
 
I did some testing, and I pretty quickly saw on the receive side an instance where a partial record was displayed, followed by a good record where the "tick" value increased by 40. I'm guessing this means that SD delay can be this long (400 ms?). I increased the serial buffer on the RX side to 16*1024, and I haven't seen any problems since.

Thank you so much for testing this. I increased the serial buffer to 16 * 1024 and tested for about an hour. Just like you had mentioned, I haven't noticed any issues so far. I will also be looking to add more sensors in the future and so my data string will also grow longer. Hoping the serial buffer size is large enough to handle that.

On the logging side, it's a little worrying to open/close a file every 10 ms. If one file is for a whole day, you have 86400 sec by 100 records/sec * 80 bytes/record = ~700MB. I'm really not sure what you might run into doing open/write/close over 8M times to write 700MB

Also I have now slightly modified the code for SD write to reduce the number of file open/close operations. Now I open the file once at the beginning, collect all the data, and close it just once when the "tick" variable rolls over. So a new file will be created every ~12 minutes with just one open and one close per file. Will look into buffering the SD writes when I have all my sensors connected.
 
Thank you so much for testing this. I increased the serial buffer to 16 * 1024 and tested for about an hour. Just like you had mentioned, I haven't noticed any issues so far. I will also be looking to add more sensors in the future and so my data string will also grow longer. Hoping the serial buffer size is large enough to handle that.

I think you can add quite a large buffer, assuming you have RAM space available.

Also I have now slightly modified the code for SD write to reduce the number of file open/close operations. Now I open the file once at the beginning, collect all the data, and close it just once when the "tick" variable rolls over. So a new file will be created every ~12 minutes with just one open and one close per file. Will look into buffering the SD writes when I have all my sensors connected.

I think this is a good improvement and you will see some benefit. How did you choose 12 minutes per file, which is 120 per day? It can get a bit unwieldy to have very large numbers of files, and there might be some middle ground between one file per day and 120.
 
How did you choose 12 minutes per file, which is 120 per day? It can get a bit unwieldy to have very large numbers of files, and there might be some middle ground between one file per day and 120.

I was initially planning to connect a FONA 3G module to transfer data, and from my testing with FONA I was able to only transfer about 14 MB of data into the memory and then upload it via FTP. I didn't want to introduce a new variable to keep track as I was able to use the "tick" variable (counts to 65535 which is equivalent to 655.35s of data because of 100 Hz sampling rate) to partition my data into appropriate sizes. Also I am looking at collecting the data if the gps speed is above 5 kph, so I will be skipping a lot of unnecessary data.

I am not entirely satisfied with using the 3G FONA module, but it's the only module I have. I am looking to find something that's easy to work with for transferring the file as a bulk. Wifi modules are a no go, as I need will need a pocket wifi thing to be carried around and a bigger power supply for it too
 
Back
Top