Writing Vector to File on Teensy

[qwazzerman]

I'm making a datalogging program using Teensy 3.1 and an I2C sensor, with the SD card reader on the ILI9341.
I'm using vectors to store the data, since the logging duration is variable. The sampling rate is ~100Hz (every 10 ms), so rather than writing the data to a file continuously, I'm waiting until the end and writing everything all at once.
What's the best way to write these vectors to a file on an SD card? They'll contain up to 3000 values (in the most extreme case).

I've found two options that seem reasonable, but I'm a novice at C++, and I'm unsure if one is better suited to the Teensy environment.
Option 1:
(It doesn't use the SD library yet, since I'm waiting for my Teensy to arrive, but I'm hoping that making it work with the SD library is easy)

#include <iostream>
#include <fstream>
#include <vector>
#include <algorithm>
#include <iterator>
using namespace std;

int main()
{
	vector<double> foo;
	for (int i = 1; i < 100; i++)	//Build arbitrary vector
	{
	   foo.push_back(i);
	}

	ofstream outputFile("output.txt");
	outputFile << "Trial 1\t";
	copy(foo.begin(), foo.end(), ostream_iterator<double>(outputFile, "\t"));
	outputFile << endl;
}

Which gives me

Trial 1	1	2	3	4	5	6	7	8	9	10	11	12...

Option 2:
Adapted from the Datalogger example in the SD library

#include <SD.h>
#include <SPI.h>
#include <vector>

int chipSelect = 21;

int main()
{
   std::vector<double> foo;
	for (int i = 1; i < 100; i++)	//Build arbitrary vector
	{
		foo.push_back(i);
	}

   if (!SD.begin(chipSelect)) {
     Serial.println("Card failed, or not present");
      // don't do anything more:
     return;
    }
    Serial.println("card initialized.");

   File dataFile = SD.open("datalog.txt", FILE_WRITE);

  // if the file is available, write to it:
  if (dataFile) {
       dataFile.print("Trial 1");
       for (int i = 1; foo.size()-1; i++)
        {
          dataFile.print("\t");
          dataFile.print(foo[i]);
        }
          dataFile.close();

  }  
}

Or, would it be better to figure out how to write the data continuously as it's being collected?

 

[luni]

I would not recommend to use the STL for programming small microcontrollers unless you have some experience with the limitations of such systems. If you really want to use it I recommend reading "Real-Time C++: Efficient Object-Oriented and Template Microcontroller Programming" by Christopher Kormanyos or a similar book.

Regarding your datalogging I would start with something simple like

SdFat sdCard;
File logFile;

void setup()
{
	// setup SD_FAT
	// setup measuring hardware
	//....

	logFile = sdCard.open("myFile", O_WRITE | O_CREAT);
	if (!logFile.isOpen())
	{
		// Error handling		
	}
}


float readValue()
{	
	// you would read out your sensor here....

	// generate some interesting dummy numbers
	static float k = 1.0, 
	static float sign = -1.0, 
	static float pi_4 = 1.0;
	
	pi_4 += sign * 1.0 / (2.0 * k + 1); 

	sign *= -1; 
	k += 1; 
	
	return pi_4; 
}

void storeValue(float f, File& logFile)
{
	// Assuming that you want the floats stored as ASCII
	logFile.println(f);

	// If you want the floats stored asBinary: 
	//logFile.write(&f, sizeof(float))
}

void loop()
{
	float value = readValue();
	storeValue(value, logFile);
	delay(10); // simulate 100Hz data rate
}

The SDFat library should be fast enough to write the data on the fly. Of course you need to add some logic to close the file after you are done reading values. You can also try to open the file prior to writing the data and close it afterwards. I'm quite sure that this would still be fast enough.

 

[qwazzerman]

Apologies in advance if this is convoluted or confusing.

So the vector solution has a two-fold reason (which I now realize I should have mentioned earlier). In addition to writing the data all at once (which you've addressed), I'd be doing some rough data analysis on the whole dataset (taking an approximate derivative and finding the zero-crossings), returning a vector (or array, I suppose) of indices of waveform peaks. Since my derivative method is basically dx/dt, I really only need the current data point and the previous point. But I still need to write all of the peaks to a separate file. To minimize write latency, I'm guessing it's best to keep the waveform file open (rather than continually opening and closing). And, if I understand correctly, you can only have one file open at a time.
While the program is running, I need to access the value of the most recent peak so I can do some math to it, but the others can be discarded, as long as once the trial is complete, all of the peaks get written to a separate file. Would it be better to a) just discard the previous peak as a new one becomes available, then re-analyze the entire waveform after the trial finishes and get all the peaks at once, or b) store the peaks in some sort of array, maybe larger than I need, initialized to 'invalid' values (like 0 or -1), then replace each consecutive value in the array with the next peak? Or c) some other option?

...if that makes sense.

A couple additional questions:
- I had heard that the write latency for SD cards wasn't consistent/reliable - most writes are quick, but every so often a write command will take much longer. Is there a way I can avoid losing data points during this time?
- The sensor I'm using (Silicon Labs SI1143) sends an interrupt signal whenever it has a new sample, so rather than polling the sensor every 10ms and risking sampling the same data point twice / missing a point, I was planning on letting the interrupt trigger a read subroutine. Is there any reason this wouldn't work?

 

[luni]

.... I'd be doing some rough data analysis on the whole dataset (taking an approximate derivative and finding the zero-crossings), returning a vector (or array, I suppose) of indices of waveform peaks...

As mentioned, try to avoid std::vector, if you want or need to use STL containers better use std::array instead. This thing lives on the stack which will avoid possible memory fragmentation issues. Personally I'd use a simple c-array for that purpose. You said that you have 3000 peaks at most, that would make an array of about 12kB. Depending on the memory requirements of the rest of your program this should not be an issue with a Teensy 3.1.

...
Would it be better to a) just discard the previous peak as a new one becomes available, then re-analyze the entire waveform after the trial finishes and get all the peaks at once, or b) store the peaks in some sort of array, maybe larger than I need, initialized to 'invalid' values (like 0 or -1), then replace each consecutive value in the array with the next peak? Or c) some other option?

If you have enough free memory (~12kB) I'd go for b). You can then write the complete array to the file with one write operation which is much quicker than 3000 single write operations.

...A couple additional questions:
- I had heard that the write latency for SD cards wasn't consistent/reliable - most writes are quick, but every so often a write command will take much longer. Is there a way I can avoid losing data points during this time?
- The sensor I'm using (Silicon Labs SI1143) sends an interrupt signal whenever it has a new sample, so rather than polling the sensor every 10ms and risking sampling the same data point twice / missing a point, I was planning on letting the interrupt trigger a read subroutine. Is there any reason this wouldn't work?

You can certainly use an interupt to read out your sensor, avoid any lenthy processing in the service routine. But again, try to keep things simple at the begining. I would try to use something like the following simple flow

1. Check if new data arrived
2. Repeat 1) until new data arrived
3. Read out sensor
4. Process data and store in array
5. Continue at 1) until all data read
6. Write data to file

Polling time (time needed for 1) and 2)) will be in the µs region, so probably no problem with missing a point as long as 3) to 4) is much shorter than 10ms.

 

[WMXZ]

I'd be doing some rough data analysis on the whole dataset (taking an approximate derivative and finding the zero-crossings), returning a vector (or array, I suppose) of indices of waveform peaks.

While I understand the maths behind your approach, you could simply consider to keep three values in memory and

consider a peak at sample n-1 if x(n-1)>x(n) & x(n-1)>x(n-2)

.
as you know dx/dt is also zero for minimum.

this way you only have to copy 2 values

 

[mlu]

Lets hope there is not noise in the signal or there might be a huge number of of peaks.
Probably there is need for some signal conditioning/filtering before detecting peaks.