Corrupted Data?

Status
Not open for further replies.

likwid_courage

New member
I'm having an issue with saving data to a uSD on a Teensy 3.6, and more correctly reading back the data.

My main problem is looking back at the text file data, instead of seeing a collimated list, I see a huge string of what appears to be special and Chinese/Japanese/Korean characters.

I have the Teensy payload reading and recording a plethora of sensors (PM, pressure, temperature, etc.), as well as sending a signal to a transistor that will in turn open a separately powered solenoid to allow for the collection of an air sample. While I could read and trigger the opening of a solenoid valve with a second Teensy 3.2 in real time (communication via XBee radio), all of the data was being stored on a uSD card on the 3.6. No filenames or other files seem to be effected, so I don't think it's a corrupted SD card. All of the data/variables were stored as strings, which is what I'm leaning towards the issue being. Overall, the Teensy payload ran for about 10 hours straight off of a 9-volt battery (changing the battery once, but which I think was a crash, not the 9-volt dying).

So my big question is:
1. Is there any way to recover and/or make sense of the data that was being saved to the text file?

and

2. What might cause this to happen?

I'm attaching the sketch that was being used for reference, and will be around to answer questions as I'm sure I left out a lot as I'm a little frantic in trying to fix the issue (hopefully possible).

The only things I can think it might be due to are a crash of the Teensy, relying on the string variable, or possibly the large uSD card at 32gb. Any help or insight is greatly appreciated!
 

Attachments

  • Payload_02_Test.ino
    47.7 KB · Views: 53
Can you attach an example text file with the garbled data?

I see that you have output to the Serial monitor. It would help to see that too.

Pete
 
Here's a portion of the text file in question.

Unfortunately, I won't be able to start up the actual Teensy for a few days. It's at work, and out of an excess of caution we decided to quarantine after getting home (field work was across state lines).

At this point, I'd even be happy with a more detailed explanation of what causes this for the write up besides "microcontroller crashed". I ran into a weird storage problem a couple years ago as well with a raspberry pi finding that it could "only" store 2^14 files in a directory. I was hoping for a nice even number like that again that would point to some weird architecture thing or something, the full size file is 2,806kb.
 

Attachments

  • TEST.txt
    111.9 KB · Views: 66
The way to make sense of the data is to read the code and figure out why it isn't writing the ASCII text that you are expecting.

Pete
 
I've chopped your code down to just some of the stuff which writes to the SD files. I ran it on a T4.1 but that shouldn't make any difference to the result.
It produces normal text without any problems.
However, if you run it more than once, it APPENDS the new data to the end of the existing file. This means that if for some reason the file had junk at the beginning when it was first written, any subsequent good data will be added to the end of the file. Have you checked the end of the file - maybe there's good data there.

Code:
// From: https://forum.pjrc.com/threads/64593
#include <SD.h>                      /* For saving data to the built-in SD card on teensy3.6 */
#include <TimeLib.h>

const int chipSelect = BUILTIN_SDCARD;

int gpsyear = 20;              /* e.g. 2016 */
int gpsmon = 11;               /* 1 thru 12 UTC */
int gpsday = 20;               /* 1 thru 31 UTC */
int gpshour = 14;              /* 0 thru 23 UTC */
int gpsmin = 19;               /* 0 thru 59 UTC */
int gpssec = 0;               /* 0 thru 59 UTC */

uint16_t pm10_standard = 15569, pm25_standard = 6789, pm100_standard = 1234;
/********* Saving to SD *********/

// Saving to SD
char filename[] = "TESTING.txt"; // Data
char headername[] = "HEADER.txt"; //Header file
File myFile;

void saveData()
{
  //File myFile;
  myFile = SD.open(filename, FILE_WRITE);
  int m=month();
  int d=day();
  int y=year();
  int h=hour();
  int mi=minute();
  int sec=second();
  if (gpsmon < 10) myFile.print('0');
  myFile.print(gpsmon);
  myFile.print("/");
  if (gpsday < 10) myFile.print('0');
  myFile.print(gpsday);
  myFile.print("/");
  myFile.print(gpsyear);
  myFile.print(" ");
  if (gpshour < 10) myFile.print('0');
  myFile.print(gpshour, DEC);
  myFile.print(":");
  if (gpsmin < 10) myFile.print('0');
  myFile.print(gpsmin, DEC);
  myFile.print(":");
  if (gpssec < 10) myFile.print('0');
  myFile.print(gpssec, DEC);
  myFile.print(",");

  if (m < 10) myFile.print('0');
  myFile.print(m);
  myFile.print("/");
  if (d < 10) myFile.print('0');
  myFile.print(d);
  myFile.print("/");
  myFile.print(y);
  myFile.print(" ");
  if (h < 10) myFile.print('0');
  myFile.print(h);
  myFile.print(":");
  if (mi < 10) myFile.print('0');
  myFile.print(mi);
  myFile.print(":");
  if (sec < 10) myFile.print('0');
  myFile.print(sec);
  myFile.print(",");
  myFile.print(pm10_standard);
  myFile.print(",");
  myFile.print(pm25_standard);
  myFile.print(",");
  myFile.print(pm100_standard);

  myFile.println(",END");

  if (!SD.exists(filename)) {
    Serial.println("Error saving data to SD card!");
  }
  myFile.close();
}

void setup(void)
{
  Serial.begin(9600);
  while(!Serial);
  delay(500);
  Serial.println("About to try to set up SD card...");

  // Start-up the SD card
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    return;
  }

  Serial.println("SD card present!");

  Serial.println("Attempting to write header file...");

  // Write a Header File
  File hFile; //create header file object
  hFile = SD.open(headername, FILE_WRITE);
  hFile.println(" ");
  hFile.println("GPS_Date_Time_(GMT),Latitude_(DM),Longitude_(DM),Latitude_(DMS),Longitude_(DMS),Altitude_(),Course_Degrees,RTC_Date_Time_(PST),PM01_STD(ug/m3),PM2.5_STD(ug/m3),PM10_STD(ug/m3),PM01_ENV(ug/m3),PM2.5_ENV(ug/m3),PM10_ENV(ug/m3),Particles0.3_(um/0.1L_air),Particles0.5_(um/0.1L_air),Particles1.0_(um/0.1L_air),Particles2.5_(um/0.1L_air),Particles5.0_(um/0.1L_air),Particles10.0_(um/0.1L_air),CO2_(ppm),CO_raw(ppm),CO_temp_(C),CO_humid_(%),SHT_Temp_(C),SHT_Humid_(%),SHT_DewP_(C),BME_temp(C),BME_press(mb),BME_humid(%),BME_alt(m),MPL_Press_(hPa),Can_1_status,Can_2_status,airborne(0-Y 1-N),Calib_status");
  hFile.close();
  //hFile = SD.open("HEADER.txt", FILE_WRITE);
  /*
  //if(hFile){
  hFile.print("GPS_Date_Time_(GMT),Latitude_(DM),Longitude_(DM),Latitude_(DMS),Longitude_(DMS),Altitude_(),Course_Degrees,RTC_Date_Time_(PST),PM01_STD(ug/m3),PM2.5_STD(ug/m3),PM10_STD(ug/m3),PM01_ENV(ug/m3),PM2.5_ENV(ug/m3),PM10_ENV(ug/m3),Particles0.3_(um/0.1L_air),Particles0.5_(um/0.1L_air),Particles1.0_(um/0.1L_air),Particles2.5_(um/0.1L_air),Particles5.0_(um/0.1L_air),Particles10.0_(um/0.1L_air),CO2_(ppm),CO_raw(ppm),CO_temp_(C),CO_humid_(%),SHT_Temp_(C),SHT_Humid_(%),SHT_DewP_(C),BME_temp(C),BME_press(mb),BME_humid(%),BME_alt(m),MPL_Press_(hPa),Can_1_status,Can_2_status,airborne(0-Y 1-N),Calib_status"); //header data
  hFile.close();
  //}
  */
  Serial.println("Header file written!");
  if (!SD.exists(headername)) {
    Serial.println("CANNOT OPEN HEADER.txt");
  }

  for(int i = 0;i < 10;i++) {
    Serial.println(i);
    saveData();
    delay(10000);
  }
  Serial.println("DONE");
}

void loop(void)
{
}

After the first run, the TESTING.txt file contains this:
Code:
11/20/20 14:19:00,01/01/1970 00:00:01,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:11,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:21,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:31,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:41,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:51,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:01:01,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:01:11,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:01:21,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:01:31,15569,6789,1234,END

After the second run it contains this:
Code:
11/20/20 14:19:00,01/01/1970 00:00:01,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:11,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:21,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:31,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:41,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:51,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:01:01,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:01:11,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:01:21,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:01:31,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:01,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:11,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:21,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:31,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:41,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:00:51,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:01:01,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:01:11,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:01:21,15569,6789,1234,END
11/20/20 14:19:00,01/01/1970 00:01:31,15569,6789,1234,END

Pete
 
That's what I was hoping, that there would be a little bit of actual data in there to lend a clue. But looks like it's all special characters, beginning to end. My guess is either the PM, CO, or CO2 sensor (maybe teensy if it ran out of RAM or something?) ended up gagging and tossed out some ridiculous value that got written as essentially 1.4 million special characters and froze/crashed the teensy.
After I got the thing back up and running, we pretty much wanted to get right back to collecting samples, but I should have said hold on for 10 minutes to check the SD. Hindsight is 20/20.
 
In your text file, every byte has bit 8 set. That could have happen when you loaded the file in a text editor and the editor loaded a different code page due to some reason. And then the saved file would be corrupt. If you strip bit 8, the result does not look very useful.

For example
Code:
E3  A0  B0  E2  B1  8E  E3  84     hex data in your file
63  20  30  62  31  0E  63  04     bit 8 set to zero
 c  sp   0   b    1      c           ascii where sp is space
 
There is a possibility of array-overrun.
Code:
    // Send PM data over radio
    float PM_10=(float)data.pm10_standard;
    sprintf(SendPM_10, "zc%04.02f&", PM_10);
The problem is that specifying %04.02f to sprintf does not guarantee that it will print no more than 4 characters in this field.
For example,
Code:
  float PM_10 = 76423.45;
  sprintf(SendPM_10, "zc%04.02f&", PM_10);
  myFile.print(SendPM_10);
will print "zc76423.45&" which is longer than the declared length of SendPM_10. It doesn't matter that you don't print these values, just executing the sprintf can clobber the array.
You should take steps to ensure that these arrays can't be overrun.

Pete
 
Status
Not open for further replies.
Back
Top