Forum Rule: Always post complete source code & details to reproduce any issue!
Page 3 of 4 FirstFirst 1 2 3 4 LastLast
Results 51 to 75 of 87

Thread: Generic data logger object

  1. #51
    Quote Originally Posted by WMXZ View Post
    For ExFat you need SdFat-beta
    Thanks - will try.

  2. #52
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,339
    Ok going through the examples seems everything is based on a timer call back. Question is if I just want to log data every 200ms based on the data available test in a loop:
    Code:
    if ((newIMUData == 1 && rtk.upDated == true)) {
    how would I modify the say the ASCII example?

    EDIT: @defragster, recognize the if test

  3. #53
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    11,780
    Quote Originally Posted by mjs513 View Post
    Ok going through the examples seems everything is based on a timer call back. Question is if I just want to log data every 200ms based on the data available test in a loop:
    Code:
    if ((newIMUData == 1 && rtk.upDated == true)) {
    how would I modify the say the ASCII example?

    EDIT: @defragster, recognize the if test
    Sublime finds :: 22 matches across 22 files >> ... _GPSimuDK\1Apr18\uNavINS_MPU9250_I2C_MAGV4\uNavINS _MPU9250_I2C_MAGV4.ino :: if (newIMUData == 1) {

    Is that asking for save as ASCII versus binary? Or selective rejection of a callback when no data to add? Or log of data based on Sketch behavior not a timer? ... or some combination?

    Have been away from this a bit ... once the callback is issued it seems expected the data records will be created. If the Timer is setup for 200ms update the 'prior' string could be provided?

    Not sure I have current code - if it ever got to github ... Not sure if there is a public way to say 'LogThisNow( _record_ )' without having it as a timer based logger?

  4. #54
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,339
    @defragster
    Guess thats exactly what I am looking for: 'LogThisNow( _record_ )' - too bad

  5. #55
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    11,780
    Quick glance in :: ...\libraries\DataLogger\Examples\Default__LoggerT est\Default__LoggerTest.ino

    StartLogger could take a third param that defaults to '1' - when passed '0' it might skip the :: dlTimer.begin(&LoggerISR, 1000000 / collectionrate); // start at collectionrate

    USER HACK would be make this local function void:
    Code:
    //  The ISR handler for an interval timer must be at global scope,
    //  and not in the DataLogger object scope.  THUS,  This function to
    //  transfer back to the object so that the buffering can use all
    //  the datalogger private variables.
    void LoggerISR(void) {  // call the data logger collection ISR handler
      // mydl.TimerChore();
    }

    Then manually as desired call that :: DataLogger::TimerChore(void):: mydl.TimerChore();
    Code:
    		// this function is called from the global level interval timer ISR.
    		//  It has access to alll the class variables and will call the collector function.
    		void TimerChore(void);
    Collectionrate could be passed zero to say no timer - but it appears to be used in buffer calc so it needs to know how the general nature of the logging rate in InitializeBuffer().
    Code:
      collectionrate = crate;
      mystatus.collectionrate = crate;

  6. #56
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,339
    Morning @defragster
    Thanks for looking I will give it a try. BTW heres the latest piece of code I am using with Brian's updated uNavIns. Oh, think I figured out what the primary cause of yaw drift is - the GPS.
    Attached Files Attached Files

  7. #57
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,339
    @defragster -- seems to hang as soon as it trys to write the first pack. May have to do something else

  8. #58
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    11,780
    Quote Originally Posted by mjs513 View Post
    @defragster -- seems to hang as soon as it trys to write the first pack. May have to do something else
    Odd - is that with no timer init and manual call - or NOP ISR on timer and manual call?

    Did you set up for GDB to track the fault? Post the LIB and Sketch as you have it and I can do that under GDB and see if it gives an easy answer for fun.

    Wow Throwback code - I saved it - the board is as left here beside me - though may be canabalized a bit - probably not get back to it - stupid GPS input - I was hoping it would evolve to have GPS be removable ... and not so computer intensive as it grew to be.

  9. #59
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,339
    @defragster
    Seems to hanging on this line:
    Code:
      cptr((void*)collectPtr); // call data collection function
    in datalogger::TimeChore

  10. #60
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    11,780
    Odd - that is set in StartLogger() {collectPtr = chunkPtrs[collectchunk];} - and should not care about how the mydl.TimerChore(); gets called as that is expected to be in user code so it is hands off on the _isr the timer triggers.

    Did that code make it to github? Not sure if I have the same/latest version as you - and can't see your edits from here

  11. #61
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,339
    Quote Originally Posted by defragster View Post
    Odd - that is set in StartLogger() {collectPtr = chunkPtrs[collectchunk];} - and should not care about how the mydl.TimerChore(); gets called as that is expected to be in user code so it is hands off on the _isr the timer triggers.

    Did that code make it to github? Not sure if I have the same/latest version as you - and can't see your edits from here
    I attached it in post #56 but here is the latest i am working with.

    You know you can just hard code the GPS lat, long etc instead of attaching a GPS.
    Attached Files Attached Files

  12. #62
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    11,780
    Opps - I thought that was just posted for throwback fun - didn't realize that is where the datalogger usage was

  13. #63
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,339
    Quote Originally Posted by defragster View Post
    Opps - I thought that was just posted for throwback fun - didn't realize that is where the datalogger usage was
    Oops, my bad that version didn;t have the datalogger in it = my bad . sorry

  14. #64
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    11,780
    Opened the 2nd zip.

    Looks like the StartLogging(); may not have been triggered when that is called.

    As noted in p#60 - the logger has to be running to have that memory valid!

    See one of the examples and put the StartLogger() function in setup? Or check the logger status before calling the mydl.TimerChore();

  15. #65
    Senior Member
    Join Date
    Feb 2018
    Location
    Corvallis, OR
    Posts
    111
    Sorry about the delay in response to questions---but I was on a camping trip where I tested a version of the data logger software to collect GPS data. GPS collection is a bit of a PITA in that you get a swarm of NMEA sentences as text that you have to parse to return compact binary values for logging. I used TinyGPS to do the parsing, then saved the parsed results as a binary record for later conversion to a KML file that I could display in Google Earth. (The conversion to KML is off topic here as it involves Matlab and some library function I have on a 30-day trial.)

    The logger also uses a simple command handler that I have been using for about 15 years. It allows you to use 2-letter commands with up to 4 parameters. I'll add the command handler at the end of this post.

    Here is the GPS logger code:

    Code:
    #include <MTP.h>
    #include <Storage.h>
    #include <usb1_mtp.h>
    
    #include <TinyGPS.h>
    #include <DataLogger.h>
    #include <CMDHandler.h>
    #include "MTP.h"
    
    const int admarkpin  = 12;
    
    const int gpspwrpin = 2;
    // Add stuff for MTP
    #define DO_DEBUG 1
    
    
    MTPStorage_SD storage;
    MTPD       mtpd(&storage);
    
    TinyGPS mygps;
    SdFs *fsptr;
    
    // instantiate a datalogger object
    DataLogger mydl;
    CMDHandler mycmds;
    
    bool MTPLoopOK = true;   // start up with MTP Active---no loop only when logging
    bool autostarted = false;
    #define GPSON digitalWriteFast(gpspwrpin, HIGH);
    #define GPSOFF digitalWriteFast(gpspwrpin, LOW);
    
    #define LEDON digitalWriteFast(ledpin, HIGH);
    #define LEDOFF digitalWriteFast(ledpin, LOW);
    const char compileTime [] = "GPS Logger Compiled on " __DATE__ " " __TIME__;
    
    
    const char *autoname = "GPS";
    const char *extype = "002";
    
    
    void LoggerISR(void) {  // call the data logger collection ISR handler
      mydl.TimerChore();
    }
    
    
    #define SAMPLERATE 100
    #define SAVEINTERVAL 1 // seconds between records
    
    
    // use our own buffer on the T3.6
    #define MAXBUFFER 20000
    uint8_t mybuffer[MAXBUFFER];
    
    char logfilename[64];
    uint32_t gpsChars = 0;
    bool echoflag = false;
    bool logging = false;
    uint64_t fsreserved;
    elapsedMillis el_autostart;
    
    // string input for buffer  16 bytes long
    // not all bytes are filled, thus the length is used
    struct gpsstring{
      uint8_t len;
      char sdata[15];
    };
    
    // processed data storage structure
    struct datrec {
      uint32_t unixtime, ltime;
      int32_t lat, lon;
      float felev, fcourse, fspeed;
      float fhdop;
    };
    
    
    
    #define AUTODELAY 300000 // 5 minutes
    void setup() {
      uint32_t bufflen;
      TLoggerStat *tsp;
    
      pinMode(ledpin, OUTPUT);
      pinMode(gpspwrpin, OUTPUT);
    
      Serial.begin(9600);
      Serial1.begin(9600);
    
      delay(500);
    
      Serial.print("\n\nGPS Data Logger  ");
      Serial.println(compileTime);
    
      mydl.SetDBPrint(false);  // turn on debug output
      fsptr = mydl.InitStorage();// try starting SD Card and file system
      if (fsptr == NULL) { 
        // initializing SD Card failed
        fastBlink();
      }
    
      // now try to initialize buffer.  Check that request falls within
      // size of local buffer or if it will fit on heap
      bufflen = mydl.InitializeBuffer(sizeof(gpsstring), SAMPLERATE, 1200, mybuffer);
      if ((bufflen == 0) || (bufflen > MAXBUFFER)) {
        Serial.println("Not enough buffer space!  Reduce buffer time or sample rate.");
        fastBlink();
      }
    
      
      LEDON
      delay(1000);  // Need a delay while SDFat checks free clusers for first time
      tsp =  mydl.GetStatus();
      LEDOFF
    
      Serial.printf("Free Space: %6.2f\n", (float)(tsp->spaceavailable) / (1024.0 * 1024.0));
      Serial.println("Calculating reserved space.");
      fsreserved = tsp->spaceavailable / 20; //reserve the last 5% of space available;
    
      StartMTP();
      // Now attach our customized callback functions
      mydl.AttachCollector(&myCollector); // specify our collector callback function
      mydl.AttachWriter(&myBinaryWriter); // logger saves processed binary data you generat
      mydl.AttachDisplay(&myBinaryDisplay, 20000); // display written data once per 5 seconds
      GPSON
      mydl.AutoFile(autoname, extype, 6);
    
      delay(200);
      el_autostart = 0;
      InitCommands();
    }
    
    void InitCommands(void){
      mycmds.AddCommand(&StartLogging,"SL",0);
      mycmds.AddCommand(&QuitLogging, "QL",0);
      mycmds.AddCommand(&ShowStatus, "SS",0);
      mycmds.AddCommand(&EchoFlag, "EF",0);
      mycmds.AddCommand(&Directory, "DI",0);
    }
    
    void EchoFlag(void *cmdline){
      echoflag = !echoflag; 
      Serial.println();
    }
    
    void Directory(void *cmdline){
      mydl.ShowDirectory();
      
    }
    
    void loop() {
      // put your main code here, to run repeatedly:
    
      TLoggerStat *tsp;
      do{
        tsp =  mydl.GetStatus();
        delay(1);
      } while(tsp->spaceavailable == 0);
    
      if(logging)mydl.CheckLogger();  // check for data to write to SD at regular intervals
      if ((tsp->spaceavailable < fsreserved) && logging) { //we are near end of card
        Serial.println("Halting logging.  Getting near end of SD Card.");
        Serial.printf("Space Available = %6.1f Space reserved = %6.1f\n", 
                                 (float)tsp->spaceavailable / (1024.0 * 1024.0), (float)fsreserved / (1024.0 * 1024.0));
        QuitLogging(NULL);
        ShowStatus(NULL);
        logging = false;
      }
      if(mycmds.CheckCommandInput()){
        el_autostart = 0;   // reset autostart timer
      }
    
      if(MTPLoopOK){
        mtpd.loop();
      }
      if(!autostarted){
        if(el_autostart > AUTODELAY){
          StartLogging(NULL);
        }
      }
      asm("WFI\n");
    }
    
    void StartMTP(void){
      Serial.println("Starting MTP Responder");
      usb_mtp_configure();
      if(!Storage_init(fsptr)) {
        Serial.println("Could not initialize MTP Storage!"); 
        fastBlink();
       } else MTPLoopOK = true;
    }
    
    
    void StartLogging(void *cmdline) {
      TLoggerStat *tsp;
      tsp =  mydl.GetStatus();
      Serial.println("Starting Logger.");
      logging = true;
      autostarted = true;
      MTPLoopOK = false;
      mydl.MakeFileName(autoname, extype);
      strncpy(logfilename, tsp->filename, strlen(tsp->filename));
      mydl.StartLogger(logfilename, 1000);  // sync once per second
    
      Serial.print("\n");
    }
    
    void QuitLogging(void *cmdline) {
      Serial.println("Stopping Logger.");
      mydl.StopLogger();
      logging = false;
      MTPLoopOK = true;
    }
    
    
    // blink at 500Hz forever to signal unrecoverable error
    void fastBlink(void) {
      while (1) {
        LEDON
        delay(100);
        LEDOFF
        delay(100);
      }
    }
    
    // can be called before, during, and after logging.
    void ShowStatus(void *cmdline) {
      TLoggerStat *tsp;
      tsp =  mydl.GetStatus();
      Serial.println("\nLogger Status:");
      Serial.printf("Bytes Written: %lu\n", tsp->byteswritten);
      Serial.printf("Collection time: %lu seconds\n", tsp->collectionend - tsp->collectionstart);
      Serial.printf("Max Collection delay: %lu microseconds\n", tsp->maxcdelay);
      Serial.printf("Average Write Time: %6.3f milliseconds\n", tsp->avgwritetime / 1000.0);
      Serial.printf("Maximum Write Time: %6.3f milliseconds\n\n", tsp->maxwritetime / 1000.0);
      Serial.printf("GPS Characters processed: %lu\n", gpsChars);
    
    }
    
    /***************************************************
         Callback function to handle user-defined collection
         and logging.
     ******************************************************/
    
    // called from the datalogger timer handler
    // The collector simply collects GPS input characters
    // and processes them through mygps;
    // the binary data strucure isn't used at this point
    void myCollector(  void* vdp) {
    
      uint8_t len;
      struct gpsstring *gstp;
      gstp = (struct gpsstring *)(vdp);
      len = Serial1.available();
      if(len > 15) len = 15;
      gstp->len = len;
      len = Serial1.readBytes(&gstp->sdata[0], len);
    }
    
    
    
    // called from the datalogger CheckLoggger function
    void myBinaryDisplay( void* vdp) {
      struct datrec *dp;
      dp = (struct datrec *)vdp;
      if (!logging) return;
      Serial.printf("Lat: %10.6f  Lon: %12.6f elev: %6.1f  HDOP: %4.2f Sats: %u\n", 
            (double)dp->lat/1e6, (double)dp->lon/1e6, dp->felev, dp->fhdop,mygps.satellites());
    }
    
    /************************************************
     * struct datrec {
      uint32_t unixtime, ltime;
      uint32_t flat, flon;
      float felev, fcourse, fspeed;
      float fhdop;
    };
    
     ***************************************************/
    
    // Used to write processed GPS binary data.  Input to function is
    // a pointer to a gpsstr record.  We process the string input
    // through mygps.  This gets called once for each string buffered
    // at the 100Hz rate.
    // A binary record is saved every saveinterval seconds
    uint16_t myBinaryWriter(void *bdp, void* rdp) {
      static uint16_t rcounter = 0;
      static struct datrec dr;
      char** sptr;
      sptr  = (char**) rdp;
      uint32_t age, ldate;
      uint16_t i, rval;
      struct gpsstring *gsptr;
    
    
       gsptr = (struct gpsstring *)bdp;
      // first process the input string at bdp
      for(i= 0; i< gsptr->len; i++) mygps.encode(gsptr->sdata[i]);
      if(echoflag)Serial.write(gsptr->sdata, gsptr->len);
      gpsChars += gsptr->len;
      rcounter++;
      if(rcounter >= (SAMPLERATE * SAVEINTERVAL)){
        rcounter = 0;
        mygps.get_datetime(&ldate, &dr.ltime, &age);
        mygps.get_position(&dr.lat, &dr.lon, &age);
        dr.unixtime = now();
        dr.felev = mygps.f_altitude()/0.3048;
        dr.fcourse = mygps.f_course();
        dr.fspeed = mygps.f_speed_mph();
        dr.fhdop = (float)mygps.hdop()/100.0;
        *sptr = (char *)&dr;
         rval = sizeof(dr);
      } else {
        *sptr = NULL;
        rval = 0;// return nothing until end of interval
      }
    
    
      return rval;
    }

    Another example that may be useful is one that I've been working on today. In this example, I collect current and voltage data from a small solar battery charger at 1KHz using 3 ADC channels. Rather than save all the data, I use a custom binary writer function to average 2 seconds of data and record the peak current during the interval. I think it is a good illustration of how you can sample at 1KHz, examine and process the data, and save at a much lower data rate. It's still a work in progress, so you are getting what you paid for!

    Code:
    #include <MTP.h>
    #include <Storage.h>
    #include <usb1_mtp.h>
    
    
    #include <DataLogger.h>
    #include <CMDHandler.h>
    #include <ADC.h>
    
    const int admarkpin  = 12;
    
    // Add stuff for MTP
    #define DO_DEBUG 0
    
    
    MTPStorage_SD storage;
    MTPD       mtpd(&storage);
    
    SdFs *fsptr;
    
    // instantiate a datalogger object
    DataLogger mydl;
    CMDHandler mycmds;
    
    
    
    bool MTPLoopOK = true;   // start up with MTP Active---no loop only when logging
    bool autostarted = false;
    
    #define LEDON digitalWriteFast(ledpin, HIGH);
    #define LEDOFF digitalWriteFast(ledpin, LOW);
    const char compileTime [] = " Compiled on " __DATE__ " " __TIME__;
    
    
    const char *autoname = "Solar";
    const char *extype = "005";
    
    
    void LoggerISR(void) {  // call the data logger collection ISR handler
      mydl.TimerChore();
    }
    
    
    #define SAMPLERATE 1000
    #define SAVEINTERVAL 2 // seconds between records
    
    
    // use our own buffer on the T3.6
    //  rawdat is 16 bytes and we need at least 1200
    #define MAXBUFFER 24000
    uint8_t mybuffer[MAXBUFFER];
    
    char logfilename[64];
    
    bool logging = false;
    uint64_t fsreserved;
    elapsedMillis el_autostart;
    
    // raw data for buffer  16 bytes long
    
    struct rawdat {
      uint32_t icount;
      float ival, vs, vt;
    };
    
    // processed data storage structure
    struct datrec {
      uint32_t unixtime;
      float mAvg, mPeak, vsAvg, vtAvg;
    };
    
    // instantiate a new ADC object
    ADC *adc = new ADC(); // adc object;
    
    
    #define AUTODELAY 300000 // 5 minutes
    void setup() {
      uint32_t bufflen;
      TLoggerStat *tsp;
    
      pinMode(ledpin, OUTPUT);
      pinMode(A3, INPUT);  // A3 is voltage at teensy
      pinMode(A2, INPUT);  // A2 is  solar/battery voltage
      pinMode(A0, INPUT);  // a0 is current sensor output
    
      Serial.begin(9600);
      delay(500);
    
      Serial.print("\n\nSolar Data Logger  ");
      Serial.println(compileTime);
    
      mydl.SetDBPrint(false);  // turn on debug output
      fsptr = mydl.InitStorage();// try starting SD Card and file system
      if (fsptr == NULL) {
        // initializing SD Card failed
        fastBlink();
      }
    
      // now try to initialize buffer.  Check that request falls within
      // size of local buffer or if it will fit on heap
      bufflen = mydl.InitializeBuffer(sizeof(rawdat), SAMPLERATE, 1200, mybuffer);
      if ((bufflen == 0) || (bufflen > MAXBUFFER)) {
        Serial.println("Not enough buffer space!  Reduce buffer time or sample rate.");
        fastBlink();
      }
    
      adc->adc0->setAveraging(1 ); // set number of averages
      adc->adc0->setResolution(12); // set bits of resolution
      adc->adc0->setReference(ADC_REFERENCE::REF_1V2);  // use the 1.2V reference
      adc->adc0->setConversionSpeed(ADC_CONVERSION_SPEED::MED_SPEED); // change the conversion speed
      adc->adc0->setSamplingSpeed(ADC_SAMPLING_SPEED::MED_SPEED); // change the sampling speed
      LEDON
      delay(1000);  // Need a delay while SDFat checks free clusers for first time
      tsp =  mydl.GetStatus();
      LEDOFF
    
      Serial.printf("Free Space: %6.2f\n", (float)(tsp->spaceavailable) / (1024.0 * 1024.0));
      Serial.println("Calculating reserved space.");
      fsreserved = tsp->spaceavailable / 20; //reserve the last 5% of space available;
    
      StartMTP();
      // Now attach our customized callback functions
      mydl.AttachCollector(&myCollector); // specify our collector callback function
      mydl.AttachWriter(&myBinaryWriter); // logger saves processed binary data you generate
      mydl.AttachDisplay(&myBinaryDisplay, 2000); // display written data once per 2 seconds
    
      mydl.AutoFile(autoname, extype, 6);
    
      delay(200);
      el_autostart = 0;
      InitCommands();
    }
    
    void InitCommands(void) {
      mycmds.AddCommand(&StartLogging, "SL", 0);
      mycmds.AddCommand(&QuitLogging, "QL", 0);
      mycmds.AddCommand(&ShowStatus, "SS", 0);
      mycmds.AddCommand(&Directory, "DI", 0);
    
    }
    
    void Directory(void *cmdline) {
      mydl.ShowDirectory();
    
    }
    
    void loop() {
      // put your main code here, to run repeatedly:
    
      TLoggerStat *tsp;
      do {
        tsp =  mydl.GetStatus();
        delay(1);
      } while (tsp->spaceavailable == 0);
    
      if (logging)mydl.CheckLogger(); // check for data to write to SD at regular intervals
      if ((tsp->spaceavailable < fsreserved) && logging) { //we are near end of card
        Serial.println("Halting logging.  Getting near end of SD Card.");
        Serial.printf("Space Available = %6.1f Space reserved = %6.1f\n",
                      (float)tsp->spaceavailable / (1024.0 * 1024.0), (float)fsreserved / (1024.0 * 1024.0));
        QuitLogging(NULL);
        ShowStatus(NULL);
        logging = false;
      }
      if (mycmds.CheckCommandInput()) {
        el_autostart = 0;   // reset autostart timer
      }
    
      if (MTPLoopOK) {
        mtpd.loop();
      }
      if (!autostarted) {
        if (el_autostart > AUTODELAY) {
          StartLogging(NULL);
        }
      }
      //asm("WFI\n");
    }
    
    void StartMTP(void) {
      Serial.println("Starting MTP Responder");
      usb_mtp_configure();
      if (!Storage_init(fsptr)) {
        Serial.println("Could not initialize MTP Storage!");
        fastBlink();
      } else MTPLoopOK = true;
    }
    
    
    
    void StartLogging(void *cmdline) {
      TLoggerStat *tsp;
      tsp =  mydl.GetStatus();
      Serial.println("Starting Logger.");
      logging = true;
      autostarted = true;
      MTPLoopOK = false;
      mydl.MakeFileName(autoname, extype);
      strncpy(logfilename, tsp->filename, strlen(tsp->filename));
      mydl.StartLogger(logfilename, 1000);  // sync once per second
    
      Serial.print("\n");
    }
    
    void QuitLogging(void *cmdline) {
      Serial.println("Stopping Logger.");
      mydl.StopLogger();
      logging = false;
      MTPLoopOK = true;
    }
    
    
    // blink at 500Hz forever to signal unrecoverable error
    void fastBlink(void) {
      while (1) {
        LEDON
        delay(100);
        LEDOFF
        delay(100);
      }
    }
    
    // can be called before, during, and after logging.
    void ShowStatus(void *cmdline) {
      TLoggerStat *tsp;
      tsp =  mydl.GetStatus();
    
    
    
      Serial.println("\nLogger Status:");
      Serial.printf("Bytes Written: %lu\n", tsp->byteswritten);
      Serial.printf("Collection time: %lu seconds\n", tsp->collectionend - tsp->collectionstart);
      Serial.printf("Max Collection delay: %lu microseconds\n", tsp->maxcdelay);
      Serial.printf("Average Write Time: %6.3f milliseconds\n", tsp->avgwritetime / 1000.0);
      Serial.printf("Maximum Write Time: %6.3f milliseconds\n\n", tsp->maxwritetime / 1000.0);
    
    
    }
    
    /***************************************************
         Callback function to handle user-defined collection
         and buffering
      struct rawdat{
      uint32_t spare
      float ival, vs, vt;
      };
     ******************************************************/
    
    // called from the datalogger timer handler
    // The collector simply collects three adc inputs
    // and saves them in a rawdat record
    void myCollector(  void* vdp) {
      struct rawdat *rp;
      rp =  (struct rawdat *)vdp;
    
      uint16_t adval;
      float vadc, ival;
      adval = (uint16_t)adc->adc0->analogRead(A0); // current sensor
      vadc = 5.0 * 1.2 * (float)adval / 4096.0; // parallel resistance is 4K
      ival = 500.0 * vadc;  // 500mA/volt
      rp->ival = ival;
    
      adval = (uint16_t)adc->adc0->analogRead(A2); // SOlar Volts
      vadc = 5.0 * 1.2 * (float)adval / 4096.0;
      rp->vs = vadc;
    
      adval = (uint16_t)adc->adc0->analogRead(A3); // T3 Power  Volts
      vadc = 5.0 * 1.2 * (float)adval / 4096.0;
      rp->vt = vadc;
    
    }
    
    char *TString(time_t tm) {
      static char dstr[12];
      // sprintf(dstr,"%02u/%02u/%04u       ",month(tm),day(tm),year(tm));
      sprintf(dstr, "%02u:%02u:%02u", hour(tm), minute(tm), second(tm));
      return dstr;
    }
    
    
    // called from the datalogger CheckLoggger function
    void myBinaryDisplay( void* vdp) {
      struct datrec *dp;
      dp  = (struct datrec *)vdp;
      if (!logging) return;
      Serial.printf("%s\t", TString(dp->unixtime));
      Serial.printf("%4.1f\t%4.1f\t%4.3f\t%4.3f\n", dp->mAvg, dp->mPeak, dp->vsAvg, dp->vtAvg);
    
    }
    
    
    /************************************************
       struct datrec {
      uint32_t unixtime,
      float mAvg, mPeak,vsAvg, vtAvg;
      };
    
     ***************************************************/
    
    // Used to write processed processed binary data.  Input to function is
    // a pointer to a rawdat record.
    // This gets called once for each rawdat buffered at the 1000Hz rate.
    // A binary record is saved every saveinterval seconds
    uint16_t myBinaryWriter(void *bdp, void* rdp) {
      static uint16_t rcounter = 0;
      uint16_t rval;
      static struct datrec dr;
      struct rawdat *rp;
      rp = (struct rawdat *)bdp;
      static float isum, vssum, vtsum, ipeak;
    
      char** sptr;
      sptr  = (char**) rdp;
      // add data to sums and check peaks
      isum += rp->ival;
      if (rp->ival > ipeak) ipeak = rp->ival;
      vssum += rp->vs;
      vtsum += rp->vt;
    
      rcounter++;
      if (rcounter >= (SAMPLERATE * SAVEINTERVAL)) {
        dr.mPeak = ipeak;
        dr.mAvg = isum / rcounter;
        dr.vsAvg = vssum / rcounter;
        dr.vtAvg = vtsum / rcounter;
        dr.unixtime = now();
    
        ipeak = 0.0;
        isum = 0.0;
        vssum = 0.0;
        vtsum = 0.0;
    
        *sptr = (char *)&dr;
        rval = sizeof(dr);
        rcounter = 0;
      } else {
        *sptr = NULL;
        rval = 0;// return nothing until end of interval
      }
    
    
      return rval;
    }
    
    /************************************************
       Time Display Functions for 16x2 display
     * *************************************************/
    
    char *DTGString(time_t tm) {
      static char dstr[32];
      sprintf(dstr, "%02u/%02u/%04u       ", month(tm), day(tm), year(tm));
      sprintf(&dstr[17], "%02u:%02u:%02u        ", hour(tm), minute(tm), second(tm));
      return dstr;
    }

    Finally, here is a zip file with the command handler. You can unzip it into your libraries folder in a folder named "CmdHandler"
    Attached Files Attached Files
    Last edited by mborgerson; 06-24-2020 at 03:09 AM. Reason: Spellling and zip file inclusion problems

  16. #66
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    11,780
    @mborgerson - nice you were just camping and not gone. 15 years of logging - no wonder it popped up being done so fast and seems so well developed Did this ever make it to github to keep track of current code?

    @mjs513 hasn't replied - it seems his fault was caused by trying to log data in loop before startLogging set the memory pointer for storage?

    In prior posts it seems there is no 'expected' way to do 'on demand' logging without initiating a timer for auotmatic logging?

    Ideas above were to let the timer start but not have active call to TimerChore() in the _isr but just call it when logging is desired ... via the callBack.

    Also the _isr could be done away with if the timer were not started - may require a 'weak' LoggerISR() in the library to allow the user to not create it?

    Adding an optional thrid param {default=1} as noted perhaps to StartLogger() - could be passed as Zero to skip starting the timer for the _isr call?

  17. #67
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    11,780
    I find this DIFF in my version of DataLogger\DataLogger.cpp to last posted ZIP on prior page to get the overflow detect to work as I was using it:
    Code:
      
      // nee to make sure this works as intended
      bufferneeded = buffneeded64/ 1000;
      if(buffneeded64 != ((uint64_t)bufferneeded*1000)){ // calculation will overflow in uint32_t
    		if (dbprint)iosptr->printf("Possible overflow in buffer size calculation using %lu?\n", bufferneeded);
    		return 0;
      }
    Versus this in that last zip:
    Code:
      // nee to make sure this works as intended
      if(buffneeded64 > 0x0FFFFFF00){ // calculation will overflow in uint32_t
    		if (dbprint)iosptr->println("Possible overflow in buffer size calculation");
    		return 0;
      }

  18. #68
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,339
    @mborgerson - Hope you enjoyed your camping trip and got some good data out of it Thanks for posting the additional examples. I have been using a modified Bolderflight GPS by Chirs O. Actually lets you set some parameters on the UBLOX direct from the sketch.

    @defragster - @mborgerson: As for my testing testing yesterday I a couple of things I took the code for the datalogger out of the GPS/IMU sketch and made it stand alone and just used some dummy data to see if it would work. Bottom line is at least i verified the method we used to remove the reliance on the timer would work. It worked whether i started the logger in setup or in the loop.

    Now for full disclosure which i forgot to mention I was using a External SD Card, FAT32, and am using ascii writer, maybe binary would be better.

    As for the full GPS/IMU sketch. I was trying to use one of Brian's SD Card backpack boards with the T4.0 and every time i tried to access the SD card it would hang the sketch. Note the example sketches for the datalogger worked no problem. I put a T4.1 and tried it and it worked but was acting funny. The first time I would start the logger nothing would be logged. If i quite and started it again it would log data to the card. So something funny.

    Going to play some more with it today and see what happens.

  19. #69
    Senior Member
    Join Date
    Feb 2018
    Location
    Corvallis, OR
    Posts
    111
    I've considered the possibility of event-triggered logging, and I think it could be implemented with a modification of the solar logger example. For event-triggered logging, the attached binary writer would examine the buffered raw data packets as they are passed to the binary writer function. If the logging condition is met, the writer passes back a pointer to the desired data (which may be the raw data or some manipulation, as occurs in the solar logger). It the data doesn't meet the logging criteria, the writer passes back a NULL pointer and zero length and nothing is written to the SD card.

    The writer function could also implement a minimum number of records to write after a trigger event to make sure the falling edge of the data is fully captured. Capturing the pre-trigger leading edge is a bit more complex and probably involves maintaining a separate precursor queue at the main program level.

    I considered this option for my GPS logger. It would log data only when speed was greater than 1.5MPH, so that I wouldn't have long, useless files collected overnight while I was parked at a campsite. I rejected it for a number of reasons:

    1. A fundamental rule for oceanographic loggers is: Collect Everything and let the scientists sort it out later. You never know when a paper or thesis subject is hiding in the data from a sensor that wasn't the main subject of the deployment.
    2. Storage is cheap. Retracing a route or re-deploying a buoy is expensive. We still make choices about sampling rates based on data characteristics, but it usually isn't because we're short of SD card space. Rather, it is the battery capacity that is needed to power the sensors, MPU and SD card that is the key.
    3. I generally process and display my data sets using MatLab. It's easy to add filtering to the functions that read and display the data. That said, my Matlab Work folder is growing quite large with the data and processing functions. To paraphrase the old saying about federal spending: A gigabyte here and a gigabyte there, and pretty soon you're talking about real storage issues! (the Ocean Mixing Group at OSU buys their storage as multiple 6TB RAID NAS boxes--a couple to take on cruises and a few more for lab and office backup and data analysis.)

  20. #70
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,339
    @mborgerson
    Believe me I say I understand your fundamental rules. Been there and done that.

    In some cases though you need to process the data real-time so say an autonomous vehicle can make decisions on obstacles or position in 3D space. In this particular application, Kalman filtering to determine Yaw, Pitch and Roll we are tying GPS to IMU data, i.e, when both are available we update the filter, hence, the criteria:
    Code:
    if ((newIMUData == 1 && rtk.upDated == true))
    So when I new IMU data AND rtK (GPS) data has been updated I process the filter update:
    Code:
      if ((newIMUData == 1 && rtk.upDated == true)) {
        newIMUData = 0;
        getIMU();    //Filter updated to get Yaw Pitch and Roll
    then once this occurs I log the data.

    Right now i am dumping the data to Serial and using something called Telemetry Viewer to view plots in real time. But once I go mobile I want to be able to log the data for later analysis. Right now i am dumping a ton of GPS data and only the YPR of the filter
    Code:
      dp->iTOW      = rtk.iTOW*2.7778E-07;
      dp->tempIMU   = IMU.getTemperature_C();
      dp->fixType   = rtk.fixType;
      dp->lat1      = rtk.lat1;
      dp->lon1      = rtk.lon1;
      dp->hMSL      = rtk.hMSL;
      dp->velN      = rtk.velN;
      dp->velE      = rtk.velE;
      dp->velD      = rtk.velD;
      dp->pitch     = ypr[1];
      dp->roll      = ypr[2];
      dp->yaw       = ypr[0];
    probably should dump more but that is later because like you said there may be some other things buried in the data.

    Anyway that's what i am trying to do.

  21. #71
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,339
    I did make a couple of changes but right now its hanging on File open using an external Card reader on a T4.0

    Code:
    bool DataLogger::OpenDataFile(const char *filename) {
    Serial.print("Opening FIle  "); Serial.println(filename);
      if (! dataFile.open(filename, O_WRITE | O_CREAT | O_TRUNC)) {
    	  Serial.println("Failed to open File"); Serial.flush();
      if (dbprint)iosptr->println("Open File failed");
        return false;
      }
    So I am about to give up on this.

  22. #72
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    11,780
    Adding the OnDemand [ useTimer=false below } method of logging seems it would be a useful and important decision - it is basically there - but the _isr() timer driven logging may not always be proper.

    Suppose the GPS goes offline when in power saving mode or no reception - or if the GPS message of 5 or 10 Hz happens to get out of sync with the Teensy Timer - it would be much better to just have the Teensy OnDemand log the GPS message as received - when it is valid and changing.

    Again - not sure the code I have is the most current - but in reading it the obvious changes should generally be covered - in general:
    Code:
    void LoggerISR(void)  __attribute__ ((weak));
    void LoggerISR(void) { // when not using Timer this satisfies the linker, but if user fails to define with timer will hide the issue 
    }
    
    bool  StartLogger(const char*filename, uint16_t syncInterval, bool useTimer=true);
    
    bool DataLogger::StartLogger(const char *filename, uint16_t syncInterval, bool useTimer ) 
    // ...
    if ( useTimer) dlTimer.begin(&LoggerISR, 1000000 / collectionrate); // start at collectionrate

  23. #73
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,339
    Quote Originally Posted by defragster View Post
    Adding the OnDemand [ useTimer=false below } method of logging seems it would be a useful and important decision - it is basically there - but the _isr() timer driven logging may not always be proper.

    Suppose the GPS goes offline when in power saving mode or no reception - or if the GPS message of 5 or 10 Hz happens to get out of sync with the Teensy Timer - it would be much better to just have the Teensy OnDemand log the GPS message as received - when it is valid and changing.

    Again - not sure the code I have is the most current - but in reading it the obvious changes should generally be covered - in general:
    Code:
    void LoggerISR(void)  __attribute__ ((weak));
    void LoggerISR(void) { // when not using Timer this satisfies the linker, but if user fails to define with timer will hide the issue 
    }
    
    bool  StartLogger(const char*filename, uint16_t syncInterval, bool useTimer=true);
    
    bool DataLogger::StartLogger(const char *filename, uint16_t syncInterval, bool useTimer ) 
    // ...
    if ( useTimer) dlTimer.begin(&LoggerISR, 1000000 / collectionrate); // start at collectionrate
    Makes sense for an on demand application to me. The issue I am having may be associated that I am trying to use an external card reader on SPI and SDIO.

  24. #74
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    11,780
    Quote Originally Posted by mjs513 View Post
    Makes sense for an on demand application to me. The issue I am having may be associated that I am trying to use an external card reader on SPI and SDIO.
    Another good point there - having a timer _isr() fire without regard to the state of the system could be bad. Having a 1-5+ms SD card write interrupt some ongoing process could present system problems. Allowwing the sketch to selectively queue one or more data records to write 'when time allows' would be another use case for on demand logging.

    Though as noted as long as the logger has started - which appeared to be the problem in post #61 code - prepping the file and memory as it does - then the _isr() could just set a flag and it isn't apparent there is any reason the sketch couldn't then call the mydl.TimerChore(); from within the sketch outside the _isr() and manage what is logged when.

  25. #75
    Senior Member
    Join Date
    Feb 2018
    Location
    Corvallis, OR
    Posts
    111
    Your comment that some applications really don't fit the timer-based buffering and storage model is spot on. The GPS logger is really a bit of a stretch. It might be be better done by increasing the size of the serial port buffer and a much simpler logging scheme. The timer model might be more applicable to a U-Blox GPS that can return binary packets on request at higher than 1HZ rates.

    MJS513's project brings back memories. About 15 years ago I worked on the guidance, navigation, and control software for autonomous cargo delivery parafoil systems. It had all the complexity one could wish for:

    Environmental extremes: out the door at 20,000 feet at 30 degrees F, landing in the Arizona desert at 115F.
    Complex guidance decisions: If a parafoil system falls below the glide path, you can't bump the throttle to gain altitude.
    Multi-rate sensors: IMU reporting at 100Hz, GPS at 5Hz.
    Modified EKF with lots of floating point math.
    Data storage on SD Cards for post-flight analysis.
    Real-time telemetry via 900MHz radios---with some command overrides required by range safety at Yuma Proving Ground.
    Adverse winds and parafoil problems sometimes had us chasing systems through the cattle range and cotton fields of southern Arizona.

    I'm now considering whether some of the multi-rate logging issues might be best addressed by having two or more datalogger objects, each logging a particular sensor or group that reports at the same rate. The separate logger files would have time stamps for later synchronization and analysis. A lot of this will depend on whether SDFat beta can handle having two files open for writing at the same time. At a minimum, I will have to edit the logger startup so that both instances would be using the same file system.

    Speaking of changes, I have signed up for a GitHub account and am working through introductory tutorials. I hope to have the datalogger code up on GitHub sometime in the next few weeks. I also want to add the stuff to make it a more useful library, like key word coloring and a readme file.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •