Generic data logger object

Status
Not open for further replies.
@mborgerson - @defragster
Just as a test I used the SDFat-beta BufferedPrint example and used it to log the data to the SD Card on SPI and it worked without a problem.

So question is what really needs to be changed, besides the SDConfig to use an external card, or is it a different issue.

@mborgerson - parafoils are not the easiest thing in the world to deal with so it sounds like you had tons of fun on that project.
 
Your comment that some applications really don't fit the timer-based buffering and storage model is spot on. The GPS logger is really a bit of a stretch. It might be be better done by increasing the size of the serial port buffer and a much simpler logging scheme. The timer model might be more applicable to a U-Blox GPS that can return binary packets on request at higher than 1HZ rates.
...

GPS was @mjs513 particular use case - indeed buffer size control was needed in the uNav work we worked on together That was on a T_3.6 that was getting 500Hz IMU updates - but only when data output was SPI_to_SPI to a second Teensy to offload the logging USB Print to Telemetry Viewer using a library tonton81 kindly crafted - that mjs513 is back to apparently so he can play with robots or something :) When it did the intensive calcs they had to be delayed if GPS data was inbound - or GPS data had to wait for the calcs to complete to be read.

But there or otherwise logging that can be selectively delayed can be critical to the process.

If the code summary in p#72 is generally complete - that change would allow elimination of the timer _isr that may have no value or even adverse affects.

Not sure about multiple files on some FS Media - like SD - others may handle it well - @mjs513 may know if T_4.1_SPIFFS to FLASH or PSRAM can handle multiple files? Those would provide access to 8 or 16 MB of storage with uniform and quick time response - where SD cards can be finicky with extreme write times even with a single open file as it does.

Having the code on github will be cool. If you posted a current copy @mjs513 or myself might make a repository for it and you could join an then migrate it to your user account and own it as you get familiar with github ... it can be a major mystery at times if not used properly ... at least that is what I find in trying to use it at times beyond simple updates.
 
defragster said:
Not sure about multiple files on some FS Media - like SD - others may handle it well - @mjs513 may know if T_4.1_SPIFFS to FLASH or PSRAM can handle multiple files? Those would provide access to 8 or 16 MB of storage with uniform and quick time response - where SD cards can be finicky with extreme write times even with a single open file as it does.

SPIFFS are not set up for PSRAM, only for FLASH, never tested with 2 FLASH Chips though. I would be reluctant to use PSRAM for logging data though, if you were to loose power you would loose whatever data you saved. FLASH on the other hand would be well suited.

As to your question the answer is yes, you can have multiple files using SPIFFS. There is also a MTP responder for SPIFFS that I put together so you can pull your data files from the FLASH like you would do with the SD Card.
 
With some simple mods to the generic logger, I am now able to use two logger objects to save data to two different files at different collection rates. Apparently, SDFat 2.0B has no problems having two files open for writing to the same SD Card. I have one logger saving event data at 10Hz and the other saving simulated temperature data at 1KHz. When I've verified that the loggers are saving what I intend, I'll post the example code and a zip of the modified generic logger library.

I took a quick look at the no-timer "Log Now" stuff and I can see a potential problem. Since the generic logger doesn't write to disk until it gets a full chunk of data (normally about 100mSec of data), it might never write it to SD if you don't get enough "Log Now" data to fill a chunk. That is usually not a problem with standard timed logging, as you would lose, at most, the last 100mSec of data.

You could set up a 10Hz timed logger, then implement a binary writer function that only writes a record when the foreground program has new data. It is equivalent to having the logger poll your program for new data at a fixed rate. However this approach has the same problem---you may not fill a chunk of data to write to the SD card for quite some time if the data is available intermittently.

Since it appears that SDFat can write to two open files, it might be simpler to just write to a separate file for the intermittent data while the generic logger handles regularly timed data.

The generic logger accumulates chunks of data to write to the SD card so that it can use the more efficient multiple-sector writes to minimize power consumption and the time spent writing to the SD card. It works well for medium to high-speed timed logging where you need to make sure that data is acquired with minimal timing jitter (for later spectral analysis, etc.). It is probably not the best solution for low-speed or intermittent logging.
 
I've attached a zip file with the latest incarnation of the generic data logger. It has been updated to allow the use of more than one logger object in a program. I have added an example that uses a slow and a faster logger in a simple demo program. The other sketches in the examples folder have been updated with the two changes in function calls that are needed (new parameters in InitStorage and StartLogger). Note that some of the programs use my MTP Responder library. If you don't want to use that, you can eliminate the StartMTP() function.
View attachment DataLogger.zipI
 
Last edited:
I've attached a zip file with the latest incarnation of the generic data logger. It has been updated to allow the use of more than one logger object in a program. I have added an example that uses a slow and a faster logger in a simple demo program. The other sketches in the examples folder have been updated with the two changes in function calls that are needed (new parameters in InitStorage and StartLogger). Note that some of the programs use my MTP Responder library. If you don't want to use that, you can eliminate the StartMTP() function.
View attachment 20754I

Thank you for the updated zip.

PS. I noticed that you had a custom MTP Responder Library for you logger when I was going through it. Like how you used the pointer for the file system instead of the default. Didn't see it posted anywhere though. Care to share :)
 
Got the new zip, Thanks.

I see this code that gave bad results in fast log testing on 8MB PSRAM buffer:
Code:
// need to make sure this works as intended
  if(buffneeded64 > 0x0FFFFFF00){ // calculation will overflow in uint32_t

The working edit I had was in post #67

Not sure about 'unflushed data loss' issue? The only change is not starting a timer to call user _isr to call TimerChore(void)
> TimerChore(void) is still called - just on a user defined schedule?
> Would same buffer full still trigger a write/flush? Loss could only be part of a buffer.
> This syncInterval :: bool DataLogger::StartLogger(const char *filename, uint16_t syncInterval, void(*afptr)()){
Could still be triggered to flush with call to CheckLogger()
 
Thank you for the updated zip.

PS. I noticed that you had a custom MTP Responder Library for you logger when I was going through it. Like how you used the pointer for the file system instead of the default. Didn't see it posted anywhere though. Care to share :)

Here is a link to the MTP library I am using. I haven't posted it before because you have to edit some of the core USB libraries to get it to work. If you've gone through that before, you should be able to get a USB link for MTP + Serial working.

View attachment MTP.zip
 
Here is a link to the MTP library I am using. I haven't posted it before because you have to edit some of the core USB libraries to get it to work. If you've gone through that before, you should be able to get a USB link for MTP + Serial working.

View attachment 20757

Thanks @mborgerson. Wouldn't be the first time I edited the some of the core files :)

Oh if I can ask you one more question - can this be used with a T4.0 using an external card reader? Besides the SDConfig anything you would recommend to change?
 
Thanks @mborgerson. Wouldn't be the first time I edited the some of the core files :)

Oh if I can ask you one more question - can this be used with a T4.0 using an external card reader? Besides the SDConfig anything you would recommend to change?

I haven't worked with external card readers on the T4.0. I got out my binocular microscope, finest tweezers and soldering tip and coerced a micro-sd card socket to bond with the misfit pads on the bottom of the T4.0. Although there's a bit of stress in the relationship, they haven't yet parted ways :)

I've attached CoreStuff.zip, which has my modified boards.txt, and usb_desc.h. You should save copies of your originals before you drop the files in the proper places and remove the TX_ from the beginning of the file names.

Here is a more recent piece of example code--a data logger I am using to record solar data from a small solar cube that I built this week.
Code:
#include <MTP.h>
#include <Storage.h>
#include <usb1_mtp.h>


#include <DataLogger.h>
#include <CMDHandler.h>
#include <ADC.h>

const int admarkpin  = 12;

// Add stuff for MTP
#define DO_DEBUG 0


MTPStorage_SD storage;
MTPD       mtpd(&storage);

SdFs *fsptr;

// instantiate a datalogger object
DataLogger mydl;
CMDHandler mycmds;



bool MTPLoopOK = true;   // start up with MTP Active---no loop only when logging
bool autostarted = false;

#define LEDON digitalWriteFast(ledpin, HIGH);
#define LEDOFF digitalWriteFast(ledpin, LOW);
const char compileTime [] = " Compiled on " __DATE__ " " __TIME__;


const char *autoname = "Solar";
const char *extype = "005";


void LoggerISR(void) {  // call the data logger collection ISR handler
  mydl.TimerChore();
}


#define SAMPLERATE 1000
#define SAVEINTERVAL 2 // seconds between records


// use our own buffer on the T3.6
//  rawdat is 16 bytes and we need at least 1200
#define MAXBUFFER 24000
uint8_t mybuffer[MAXBUFFER];

char logfilename[64];

bool logging = false;
uint64_t fsreserved;
elapsedMillis el_autostart;

// raw data for buffer  16 bytes long

struct rawdat {
  uint32_t icount;
  float ival, vs, vt;
};

// processed data storage structure
struct datrec {
  uint32_t unixtime;
  float mAvg, mPeak, vsAvg, vtAvg;
};

// instantiate a new ADC object
ADC *adc = new ADC(); // adc object;


#define AUTODELAY 300000 // 5 minutes
void setup() {
  uint32_t bufflen;
  TLoggerStat *tsp;

  pinMode(ledpin, OUTPUT);
  pinMode(A3, INPUT);  // A3 is voltage at teensy
  pinMode(A2, INPUT);  // A2 is  solar/battery voltage
  pinMode(A0, INPUT);  // a0 is current sensor output

  Serial.begin(9600);
  delay(500);

  Serial.print("\n\nSolar Data Logger  ");
  Serial.println(compileTime);

  mydl.SetDBPrint(false);  // turn on debug output
  fsptr = mydl.InitStorage();// try starting SD Card and file system
  if (fsptr == NULL) {
    // initializing SD Card failed
    fastBlink();
  }

  // now try to initialize buffer.  Check that request falls within
  // size of local buffer or if it will fit on heap
  bufflen = mydl.InitializeBuffer(sizeof(rawdat), SAMPLERATE, 1200, mybuffer);
  if ((bufflen == 0) || (bufflen > MAXBUFFER)) {
    Serial.println("Not enough buffer space!  Reduce buffer time or sample rate.");
    fastBlink();
  }

  adc->adc0->setAveraging(1 ); // set number of averages
  adc->adc0->setResolution(12); // set bits of resolution
  adc->adc0->setReference(ADC_REFERENCE::REF_1V2);  // use the 1.2V reference
  adc->adc0->setConversionSpeed(ADC_CONVERSION_SPEED::MED_SPEED); // change the conversion speed
  adc->adc0->setSamplingSpeed(ADC_SAMPLING_SPEED::MED_SPEED); // change the sampling speed
  LEDON
  delay(1000);  // Need a delay while SDFat checks free clusers for first time
  tsp =  mydl.GetStatus();
  LEDOFF

  Serial.printf("Free Space: %6.2f\n", (float)(tsp->spaceavailable) / (1024.0 * 1024.0));
  Serial.println("Calculating reserved space.");
  fsreserved = tsp->spaceavailable / 20; //reserve the last 5% of space available;

  StartMTP();
  // Now attach our customized callback functions
  mydl.AttachCollector(&myCollector); // specify our collector callback function
  mydl.AttachWriter(&myBinaryWriter); // logger saves processed binary data you generate
  mydl.AttachDisplay(&myBinaryDisplay, 2000); // display written data once per 2 seconds

  mydl.AutoFile(autoname, extype, 6);

  delay(200);
  el_autostart = 0;
  InitCommands();
}

void InitCommands(void) {
  mycmds.AddCommand(&StartLogging, "SL", 0);
  mycmds.AddCommand(&QuitLogging, "QL", 0);
  mycmds.AddCommand(&ShowStatus, "SS", 0);
  mycmds.AddCommand(&Directory, "DI", 0);

}

void Directory(void *cmdline) {
  mydl.ShowDirectory();

}

void loop() {
  // put your main code here, to run repeatedly:

  TLoggerStat *tsp;
  do {
    tsp =  mydl.GetStatus();
    delay(1);
  } while (tsp->spaceavailable == 0);

  if (logging)mydl.CheckLogger(); // check for data to write to SD at regular intervals
  if ((tsp->spaceavailable < fsreserved) && logging) { //we are near end of card
    Serial.println("Halting logging.  Getting near end of SD Card.");
    Serial.printf("Space Available = %6.1f Space reserved = %6.1f\n",
                  (float)tsp->spaceavailable / (1024.0 * 1024.0), (float)fsreserved / (1024.0 * 1024.0));
    QuitLogging(NULL);
    ShowStatus(NULL);
    logging = false;
  }
  if (mycmds.CheckCommandInput()) {
    el_autostart = 0;   // reset autostart timer
  }

  if (MTPLoopOK) {
    mtpd.loop();
  }
  if (!autostarted) {
    if (el_autostart > AUTODELAY) {
      StartLogging(NULL);
    }
  }
  //asm("WFI\n");
}

void StartMTP(void) {
  Serial.println("Starting MTP Responder");
  usb_mtp_configure();
  if (!Storage_init(fsptr)) {
    Serial.println("Could not initialize MTP Storage!");
    fastBlink();
  } else MTPLoopOK = true;
}



void StartLogging(void *cmdline) {
  TLoggerStat *tsp;
  tsp =  mydl.GetStatus();
  Serial.println("Starting Logger.");
  logging = true;
  autostarted = true;
  MTPLoopOK = false;
  mydl.MakeFileName(autoname, extype);
  strncpy(logfilename, tsp->filename, strlen(tsp->filename));
  mydl.StartLogger(logfilename, 1000);  // sync once per second

  Serial.print("\n");
}

void QuitLogging(void *cmdline) {
  Serial.println("Stopping Logger.");
  mydl.StopLogger();
  logging = false;
  MTPLoopOK = true;
}


// blink at 500Hz forever to signal unrecoverable error
void fastBlink(void) {
  while (1) {
    LEDON
    delay(100);
    LEDOFF
    delay(100);
  }
}

// can be called before, during, and after logging.
void ShowStatus(void *cmdline) {
  TLoggerStat *tsp;
  tsp =  mydl.GetStatus();



  Serial.println("\nLogger Status:");
  Serial.printf("Bytes Written: %lu\n", tsp->byteswritten);
  Serial.printf("Collection time: %lu seconds\n", tsp->collectionend - tsp->collectionstart);
  Serial.printf("Max Collection delay: %lu microseconds\n", tsp->maxcdelay);
  Serial.printf("Average Write Time: %6.3f milliseconds\n", tsp->avgwritetime / 1000.0);
  Serial.printf("Maximum Write Time: %6.3f milliseconds\n\n", tsp->maxwritetime / 1000.0);


}

/***************************************************
     Callback function to handle user-defined collection
     and buffering
  struct rawdat{
  uint32_t spare
  float ival, vs, vt;
  };
 ******************************************************/

// called from the datalogger timer handler
// The collector simply collects three adc inputs
// and saves them in a rawdat record
void myCollector(  void* vdp) {
  struct rawdat *rp;
  rp =  (struct rawdat *)vdp;

  uint16_t adval;
  float vadc, ival;
  adval = (uint16_t)adc->adc0->analogRead(A0); // MAX 471current sensor
  vadc = 5.0 * 1.2 * (float)adval / 4096.0; // parallel resistance is 4K for sensor
  ival = 500.0 * vadc;  // 500mA/volt
  rp->ival = ival;

  adval = (uint16_t)adc->adc0->analogRead(A2); // Solar Volts
  vadc = 5.0 * 1.2 * (float)adval / 4096.0; // correct for 4:1 attenuator
  rp->vs = vadc;

  adval = (uint16_t)adc->adc0->analogRead(A3); // T3 Power  Volts
  vadc = 5.0 * 1.2 * (float)adval / 4096.0;// correct for 4:1 attenuator
  rp->vt = vadc;

}

char *TString(time_t tm) {
  static char dstr[12];
  // sprintf(dstr,"%02u/%02u/%04u       ",month(tm),day(tm),year(tm));
  sprintf(dstr, "%02u:%02u:%02u", hour(tm), minute(tm), second(tm));
  return dstr;
}


// called from the datalogger CheckLoggger function
void myBinaryDisplay( void* vdp) {
  struct datrec *dp;
  dp  = (struct datrec *)vdp;
  if (!logging) return;
  Serial.printf("%s\t", TString(dp->unixtime));
  Serial.printf("%4.1f\t%4.1f\t%4.3f\t%4.3f\n", dp->mAvg, dp->mPeak, dp->vsAvg, dp->vtAvg);

}


/************************************************
   struct datrec {
  uint32_t unixtime,
  float mAvg, mPeak,vsAvg, vtAvg;
  };

 ***************************************************/

// Used to write processed processed binary data.  Input to function is
// a pointer to a rawdat record.
// This gets called once for each rawdat buffered at the 1000Hz rate.
// A binary record is saved every saveinterval seconds
uint16_t myBinaryWriter(void *bdp, void* rdp) {
  static uint16_t rcounter = 0;
  uint16_t rval;
  static struct datrec dr;
  struct rawdat *rp;
  rp = (struct rawdat *)bdp;
  static float isum, vssum, vtsum, ipeak;

  char** sptr;
  sptr  = (char**) rdp;
  // add data to sums and check peaks
  isum += rp->ival;
  if (rp->ival > ipeak) ipeak = rp->ival;
  vssum += rp->vs;
  vtsum += rp->vt;

  rcounter++;
  if (rcounter >= (SAMPLERATE * SAVEINTERVAL)) {
    dr.mPeak = ipeak;
    dr.mAvg = isum / rcounter;
    dr.vsAvg = vssum / rcounter;
    dr.vtAvg = vtsum / rcounter;
    dr.unixtime = now();

    ipeak = 0.0;
    isum = 0.0;
    vssum = 0.0;
    vtsum = 0.0;

    *sptr = (char *)&dr;
    rval = sizeof(dr);
    rcounter = 0;
  } else {
    *sptr = NULL;
    rval = 0;// return nothing until end of interval
  }
  return rval;
}

This code shows how I use the MTP responder---and prevent calls to the MTP loop function while logging. It also shows how to queue binary data at 1KHz, then process that data and save peaks and averages every two seconds.
 

Attachments

  • CoreStuff.zip
    19.8 KB · Views: 87
Hello,

I'm interested in using this code to record data from four analog inputs and 3 axes of accelerometer data (LIS3DH). The code works very well with analog inputs on the T4.1, but after inserting code for the accelerometer in place of the analogRead statements the code hangs when I start the logging. I'm looking at recording data at a minimum of 500 Hz, but I'm not sure if that's possible because the program freezes. Would it be a better idea to access the accelerometer using just the Wire I2C library? I'm wondering if the Sparkfun library is too slow.

Thanks
 
Hello,

I'm interested in using this code to record data from four analog inputs and 3 axes of accelerometer data (LIS3DH). The code works very well with analog inputs on the T4.1, but after inserting code for the accelerometer in place of the analogRead statements the code hangs when I start the logging. I'm looking at recording data at a minimum of 500 Hz, but I'm not sure if that's possible because the program freezes. Would it be a better idea to access the accelerometer using just the Wire I2C library? I'm wondering if the Sparkfun library is too slow.

Thanks

I suspect that the Sparkfun library isn't really the problem. I think the wire library also runs at a 400KHz clock---which means an absolute maximum of 50Kbytes/second, or 20 microseconds per byte. A quick glance at the LIS3DH data sheet seems to indicate that it will have to transfer a minimum of about 20 bytes checking status, then sending register numbers and reading two bytes per channel. That's a minimum of 400 microseconds per reading. That has to be added to the analog read times and happen at 2 millisecond intervals in the logger collection interrupt service routine. If the LIS3DH isn't ready when you ask for data, you could be taking too long in the data collection function.

Here are some steps you can take:

1. Do a simple sketch with some bracketed micros() calls to find out how long it takes to collect LIS3DH data. You really need the maximum delay to be under a millisecond---and even at that you are spending half the CPU cycles in the collection interrupt routine.

2. If you're not too picky about the absolute timing of the LIS3DH data, you could set up a separate loop to collect LIS3DH data from an interval timer handler at whatever maximum speed you can get with a state machine that triggers the first transfer at the interval timer interrupt, then collects subsequent bytes with the interrupt at the end of each transfer. The data would go into a global array which the logger collection handler would read for storage.

3. Figure out if there's a setting for the LIS3DH that will allow you to use multi-byte transfers so you can request all six bytes of data in a single transfer.

If I was reluctant to become a Wire and LIS3DH guru (and I am), i'd spend some time looking at analog-output accelerometers like the Sparkfun ADXL337 breakout board ($9.95 Qty 1). I've successfully used similar accelerometers to measure the motion of an oceanographic logger on a deep-sea mooring. The specs say it is good for the 500Hz bandwidth you want, and collecting 3 more analog channels will take a LOT less time than collecting 3 channels via WIRE---especially if you make good use of the two different ADCs in the T4.1. Of course you'd have to compare the noise levels and resolution of the two sensors to guide your decision.

For some users the big draw of the LIS3DH is that it can trigger an interrupt in response to events like free-fall or an impact. It doesn't sound like you need that capability.
 
Thanks for the reply. Honestly, I think that using the ADXL337 would be much more simple - I looked into it briefly but I didn't know it was good for the 500 Hz bandwidth, so thank you for bringing that up!

I'm going to look into getting a 337 or another analog output accelerometer. The 337 would be a good starting point, and if I need higher bandwidth I can look at the ADXL1002 etc. I'm trying to do vibration analysis, so although I said 500 Hz, I want to get the highest bandwidth possible.

The deep-sea monitoring project sounds interesting - would you be able to share more about that?

Thanks again!
 
Thanks for the reply. Honestly, I think that using the ADXL337 would be much more simple - I looked into it briefly but I didn't know it was good for the 500 Hz bandwidth, so thank you for bringing that up!

I'm going to look into getting a 337 or another analog output accelerometer. The 337 would be a good starting point, and if I need higher bandwidth I can look at the ADXL1002 etc. I'm trying to do vibration analysis, so although I said 500 Hz, I want to get the highest bandwidth possible.

The deep-sea monitoring project sounds interesting - would you be able to share more about that?

Thanks again!

The instrument involved is called a Chi-Pod. You can find a description here: http://mixing.coas.oregonstate.edu/research/moored_mixing/

The first of these instruments were deployed in about 2005---in pre-Teensy times. PJRC was selling 8051 and MP3 player boards. The ChiPods used MSP430 MPUs from Texas Instruments.
 
Status
Not open for further replies.
Back
Top