Forum Rule: Always post complete source code & details to reproduce any issue!
Page 4 of 4 FirstFirst ... 2 3 4
Results 76 to 87 of 87

Thread: Generic data logger object

  1. #76
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,325
    @mborgerson - @defragster
    Just as a test I used the SDFat-beta BufferedPrint example and used it to log the data to the SD Card on SPI and it worked without a problem.

    So question is what really needs to be changed, besides the SDConfig to use an external card, or is it a different issue.

    @mborgerson - parafoils are not the easiest thing in the world to deal with so it sounds like you had tons of fun on that project.

  2. #77
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    11,764
    Quote Originally Posted by mborgerson View Post
    Your comment that some applications really don't fit the timer-based buffering and storage model is spot on. The GPS logger is really a bit of a stretch. It might be be better done by increasing the size of the serial port buffer and a much simpler logging scheme. The timer model might be more applicable to a U-Blox GPS that can return binary packets on request at higher than 1HZ rates.
    ...
    GPS was @mjs513 particular use case - indeed buffer size control was needed in the uNav work we worked on together That was on a T_3.6 that was getting 500Hz IMU updates - but only when data output was SPI_to_SPI to a second Teensy to offload the logging USB Print to Telemetry Viewer using a library tonton81 kindly crafted - that mjs513 is back to apparently so he can play with robots or something When it did the intensive calcs they had to be delayed if GPS data was inbound - or GPS data had to wait for the calcs to complete to be read.

    But there or otherwise logging that can be selectively delayed can be critical to the process.

    If the code summary in p#72 is generally complete - that change would allow elimination of the timer _isr that may have no value or even adverse affects.

    Not sure about multiple files on some FS Media - like SD - others may handle it well - @mjs513 may know if T_4.1_SPIFFS to FLASH or PSRAM can handle multiple files? Those would provide access to 8 or 16 MB of storage with uniform and quick time response - where SD cards can be finicky with extreme write times even with a single open file as it does.

    Having the code on github will be cool. If you posted a current copy @mjs513 or myself might make a repository for it and you could join an then migrate it to your user account and own it as you get familiar with github ... it can be a major mystery at times if not used properly ... at least that is what I find in trying to use it at times beyond simple updates.

  3. #78
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,325
    Quote Originally Posted by defragster
    Not sure about multiple files on some FS Media - like SD - others may handle it well - @mjs513 may know if T_4.1_SPIFFS to FLASH or PSRAM can handle multiple files? Those would provide access to 8 or 16 MB of storage with uniform and quick time response - where SD cards can be finicky with extreme write times even with a single open file as it does.
    SPIFFS are not set up for PSRAM, only for FLASH, never tested with 2 FLASH Chips though. I would be reluctant to use PSRAM for logging data though, if you were to loose power you would loose whatever data you saved. FLASH on the other hand would be well suited.

    As to your question the answer is yes, you can have multiple files using SPIFFS. There is also a MTP responder for SPIFFS that I put together so you can pull your data files from the FLASH like you would do with the SD Card.

  4. #79
    Senior Member
    Join Date
    Feb 2018
    Location
    Corvallis, OR
    Posts
    111
    With some simple mods to the generic logger, I am now able to use two logger objects to save data to two different files at different collection rates. Apparently, SDFat 2.0B has no problems having two files open for writing to the same SD Card. I have one logger saving event data at 10Hz and the other saving simulated temperature data at 1KHz. When I've verified that the loggers are saving what I intend, I'll post the example code and a zip of the modified generic logger library.

    I took a quick look at the no-timer "Log Now" stuff and I can see a potential problem. Since the generic logger doesn't write to disk until it gets a full chunk of data (normally about 100mSec of data), it might never write it to SD if you don't get enough "Log Now" data to fill a chunk. That is usually not a problem with standard timed logging, as you would lose, at most, the last 100mSec of data.

    You could set up a 10Hz timed logger, then implement a binary writer function that only writes a record when the foreground program has new data. It is equivalent to having the logger poll your program for new data at a fixed rate. However this approach has the same problem---you may not fill a chunk of data to write to the SD card for quite some time if the data is available intermittently.

    Since it appears that SDFat can write to two open files, it might be simpler to just write to a separate file for the intermittent data while the generic logger handles regularly timed data.

    The generic logger accumulates chunks of data to write to the SD card so that it can use the more efficient multiple-sector writes to minimize power consumption and the time spent writing to the SD card. It works well for medium to high-speed timed logging where you need to make sure that data is acquired with minimal timing jitter (for later spectral analysis, etc.). It is probably not the best solution for low-speed or intermittent logging.

  5. #80
    Senior Member
    Join Date
    Feb 2018
    Location
    Corvallis, OR
    Posts
    111
    I've attached a zip file with the latest incarnation of the generic data logger. It has been updated to allow the use of more than one logger object in a program. I have added an example that uses a slow and a faster logger in a simple demo program. The other sketches in the examples folder have been updated with the two changes in function calls that are needed (new parameters in InitStorage and StartLogger). Note that some of the programs use my MTP Responder library. If you don't want to use that, you can eliminate the StartMTP() function.
    DataLogger.zipI
    Last edited by mborgerson; 06-26-2020 at 09:59 PM. Reason: spelling errors

  6. #81
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,325
    Quote Originally Posted by mborgerson View Post
    I've attached a zip file with the latest incarnation of the generic data logger. It has been updated to allow the use of more than one logger object in a program. I have added an example that uses a slow and a faster logger in a simple demo program. The other sketches in the examples folder have been updated with the two changes in function calls that are needed (new parameters in InitStorage and StartLogger). Note that some of the programs use my MTP Responder library. If you don't want to use that, you can eliminate the StartMTP() function.
    DataLogger.zipI
    Thank you for the updated zip.

    PS. I noticed that you had a custom MTP Responder Library for you logger when I was going through it. Like how you used the pointer for the file system instead of the default. Didn't see it posted anywhere though. Care to share

  7. #82
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    11,764
    Got the new zip, Thanks.

    I see this code that gave bad results in fast log testing on 8MB PSRAM buffer:
    Code:
    // need to make sure this works as intended
      if(buffneeded64 > 0x0FFFFFF00){ // calculation will overflow in uint32_t
    The working edit I had was in post #67

    Not sure about 'unflushed data loss' issue? The only change is not starting a timer to call user _isr to call TimerChore(void)
    > TimerChore(void) is still called - just on a user defined schedule?
    > Would same buffer full still trigger a write/flush? Loss could only be part of a buffer.
    > This syncInterval :: bool DataLogger::StartLogger(const char *filename, uint16_t syncInterval, void(*afptr)()){
    Could still be triggered to flush with call to CheckLogger()

  8. #83
    Senior Member
    Join Date
    Feb 2018
    Location
    Corvallis, OR
    Posts
    111
    Quote Originally Posted by mjs513 View Post
    Thank you for the updated zip.

    PS. I noticed that you had a custom MTP Responder Library for you logger when I was going through it. Like how you used the pointer for the file system instead of the default. Didn't see it posted anywhere though. Care to share
    Here is a link to the MTP library I am using. I haven't posted it before because you have to edit some of the core USB libraries to get it to work. If you've gone through that before, you should be able to get a USB link for MTP + Serial working.

    MTP.zip

  9. #84
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,325
    Quote Originally Posted by mborgerson View Post
    Here is a link to the MTP library I am using. I haven't posted it before because you have to edit some of the core USB libraries to get it to work. If you've gone through that before, you should be able to get a USB link for MTP + Serial working.

    MTP.zip
    Thanks @mborgerson. Wouldn't be the first time I edited the some of the core files

    Oh if I can ask you one more question - can this be used with a T4.0 using an external card reader? Besides the SDConfig anything you would recommend to change?

  10. #85
    Senior Member
    Join Date
    Jul 2014
    Posts
    2,665
    Quote Originally Posted by mborgerson View Post
    Here is a link to the MTP library I am using.
    BTW, there is an example mtp-logger in MTP_t4, which could also be used to test/develp mtp aware data logger.

  11. #86
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    5,325
    Quote Originally Posted by WMXZ View Post
    BTW, there is an example mtp-logger in MTP_t4, which could also be used to test/develp mtp aware data logger.
    Thanks - missed the updates to MTP_T4 See the logger example now.

  12. #87
    Senior Member
    Join Date
    Feb 2018
    Location
    Corvallis, OR
    Posts
    111
    Quote Originally Posted by mjs513 View Post
    Thanks @mborgerson. Wouldn't be the first time I edited the some of the core files

    Oh if I can ask you one more question - can this be used with a T4.0 using an external card reader? Besides the SDConfig anything you would recommend to change?
    I haven't worked with external card readers on the T4.0. I got out my binocular microscope, finest tweezers and soldering tip and coerced a micro-sd card socket to bond with the misfit pads on the bottom of the T4.0. Although there's a bit of stress in the relationship, they haven't yet parted ways

    I've attached CoreStuff.zip, which has my modified boards.txt, and usb_desc.h. You should save copies of your originals before you drop the files in the proper places and remove the TX_ from the beginning of the file names.

    Here is a more recent piece of example code--a data logger I am using to record solar data from a small solar cube that I built this week.
    Code:
    #include <MTP.h>
    #include <Storage.h>
    #include <usb1_mtp.h>
    
    
    #include <DataLogger.h>
    #include <CMDHandler.h>
    #include <ADC.h>
    
    const int admarkpin  = 12;
    
    // Add stuff for MTP
    #define DO_DEBUG 0
    
    
    MTPStorage_SD storage;
    MTPD       mtpd(&storage);
    
    SdFs *fsptr;
    
    // instantiate a datalogger object
    DataLogger mydl;
    CMDHandler mycmds;
    
    
    
    bool MTPLoopOK = true;   // start up with MTP Active---no loop only when logging
    bool autostarted = false;
    
    #define LEDON digitalWriteFast(ledpin, HIGH);
    #define LEDOFF digitalWriteFast(ledpin, LOW);
    const char compileTime [] = " Compiled on " __DATE__ " " __TIME__;
    
    
    const char *autoname = "Solar";
    const char *extype = "005";
    
    
    void LoggerISR(void) {  // call the data logger collection ISR handler
      mydl.TimerChore();
    }
    
    
    #define SAMPLERATE 1000
    #define SAVEINTERVAL 2 // seconds between records
    
    
    // use our own buffer on the T3.6
    //  rawdat is 16 bytes and we need at least 1200
    #define MAXBUFFER 24000
    uint8_t mybuffer[MAXBUFFER];
    
    char logfilename[64];
    
    bool logging = false;
    uint64_t fsreserved;
    elapsedMillis el_autostart;
    
    // raw data for buffer  16 bytes long
    
    struct rawdat {
      uint32_t icount;
      float ival, vs, vt;
    };
    
    // processed data storage structure
    struct datrec {
      uint32_t unixtime;
      float mAvg, mPeak, vsAvg, vtAvg;
    };
    
    // instantiate a new ADC object
    ADC *adc = new ADC(); // adc object;
    
    
    #define AUTODELAY 300000 // 5 minutes
    void setup() {
      uint32_t bufflen;
      TLoggerStat *tsp;
    
      pinMode(ledpin, OUTPUT);
      pinMode(A3, INPUT);  // A3 is voltage at teensy
      pinMode(A2, INPUT);  // A2 is  solar/battery voltage
      pinMode(A0, INPUT);  // a0 is current sensor output
    
      Serial.begin(9600);
      delay(500);
    
      Serial.print("\n\nSolar Data Logger  ");
      Serial.println(compileTime);
    
      mydl.SetDBPrint(false);  // turn on debug output
      fsptr = mydl.InitStorage();// try starting SD Card and file system
      if (fsptr == NULL) {
        // initializing SD Card failed
        fastBlink();
      }
    
      // now try to initialize buffer.  Check that request falls within
      // size of local buffer or if it will fit on heap
      bufflen = mydl.InitializeBuffer(sizeof(rawdat), SAMPLERATE, 1200, mybuffer);
      if ((bufflen == 0) || (bufflen > MAXBUFFER)) {
        Serial.println("Not enough buffer space!  Reduce buffer time or sample rate.");
        fastBlink();
      }
    
      adc->adc0->setAveraging(1 ); // set number of averages
      adc->adc0->setResolution(12); // set bits of resolution
      adc->adc0->setReference(ADC_REFERENCE::REF_1V2);  // use the 1.2V reference
      adc->adc0->setConversionSpeed(ADC_CONVERSION_SPEED::MED_SPEED); // change the conversion speed
      adc->adc0->setSamplingSpeed(ADC_SAMPLING_SPEED::MED_SPEED); // change the sampling speed
      LEDON
      delay(1000);  // Need a delay while SDFat checks free clusers for first time
      tsp =  mydl.GetStatus();
      LEDOFF
    
      Serial.printf("Free Space: %6.2f\n", (float)(tsp->spaceavailable) / (1024.0 * 1024.0));
      Serial.println("Calculating reserved space.");
      fsreserved = tsp->spaceavailable / 20; //reserve the last 5% of space available;
    
      StartMTP();
      // Now attach our customized callback functions
      mydl.AttachCollector(&myCollector); // specify our collector callback function
      mydl.AttachWriter(&myBinaryWriter); // logger saves processed binary data you generate
      mydl.AttachDisplay(&myBinaryDisplay, 2000); // display written data once per 2 seconds
    
      mydl.AutoFile(autoname, extype, 6);
    
      delay(200);
      el_autostart = 0;
      InitCommands();
    }
    
    void InitCommands(void) {
      mycmds.AddCommand(&StartLogging, "SL", 0);
      mycmds.AddCommand(&QuitLogging, "QL", 0);
      mycmds.AddCommand(&ShowStatus, "SS", 0);
      mycmds.AddCommand(&Directory, "DI", 0);
    
    }
    
    void Directory(void *cmdline) {
      mydl.ShowDirectory();
    
    }
    
    void loop() {
      // put your main code here, to run repeatedly:
    
      TLoggerStat *tsp;
      do {
        tsp =  mydl.GetStatus();
        delay(1);
      } while (tsp->spaceavailable == 0);
    
      if (logging)mydl.CheckLogger(); // check for data to write to SD at regular intervals
      if ((tsp->spaceavailable < fsreserved) && logging) { //we are near end of card
        Serial.println("Halting logging.  Getting near end of SD Card.");
        Serial.printf("Space Available = %6.1f Space reserved = %6.1f\n",
                      (float)tsp->spaceavailable / (1024.0 * 1024.0), (float)fsreserved / (1024.0 * 1024.0));
        QuitLogging(NULL);
        ShowStatus(NULL);
        logging = false;
      }
      if (mycmds.CheckCommandInput()) {
        el_autostart = 0;   // reset autostart timer
      }
    
      if (MTPLoopOK) {
        mtpd.loop();
      }
      if (!autostarted) {
        if (el_autostart > AUTODELAY) {
          StartLogging(NULL);
        }
      }
      //asm("WFI\n");
    }
    
    void StartMTP(void) {
      Serial.println("Starting MTP Responder");
      usb_mtp_configure();
      if (!Storage_init(fsptr)) {
        Serial.println("Could not initialize MTP Storage!");
        fastBlink();
      } else MTPLoopOK = true;
    }
    
    
    
    void StartLogging(void *cmdline) {
      TLoggerStat *tsp;
      tsp =  mydl.GetStatus();
      Serial.println("Starting Logger.");
      logging = true;
      autostarted = true;
      MTPLoopOK = false;
      mydl.MakeFileName(autoname, extype);
      strncpy(logfilename, tsp->filename, strlen(tsp->filename));
      mydl.StartLogger(logfilename, 1000);  // sync once per second
    
      Serial.print("\n");
    }
    
    void QuitLogging(void *cmdline) {
      Serial.println("Stopping Logger.");
      mydl.StopLogger();
      logging = false;
      MTPLoopOK = true;
    }
    
    
    // blink at 500Hz forever to signal unrecoverable error
    void fastBlink(void) {
      while (1) {
        LEDON
        delay(100);
        LEDOFF
        delay(100);
      }
    }
    
    // can be called before, during, and after logging.
    void ShowStatus(void *cmdline) {
      TLoggerStat *tsp;
      tsp =  mydl.GetStatus();
    
    
    
      Serial.println("\nLogger Status:");
      Serial.printf("Bytes Written: %lu\n", tsp->byteswritten);
      Serial.printf("Collection time: %lu seconds\n", tsp->collectionend - tsp->collectionstart);
      Serial.printf("Max Collection delay: %lu microseconds\n", tsp->maxcdelay);
      Serial.printf("Average Write Time: %6.3f milliseconds\n", tsp->avgwritetime / 1000.0);
      Serial.printf("Maximum Write Time: %6.3f milliseconds\n\n", tsp->maxwritetime / 1000.0);
    
    
    }
    
    /***************************************************
         Callback function to handle user-defined collection
         and buffering
      struct rawdat{
      uint32_t spare
      float ival, vs, vt;
      };
     ******************************************************/
    
    // called from the datalogger timer handler
    // The collector simply collects three adc inputs
    // and saves them in a rawdat record
    void myCollector(  void* vdp) {
      struct rawdat *rp;
      rp =  (struct rawdat *)vdp;
    
      uint16_t adval;
      float vadc, ival;
      adval = (uint16_t)adc->adc0->analogRead(A0); // MAX 471current sensor
      vadc = 5.0 * 1.2 * (float)adval / 4096.0; // parallel resistance is 4K for sensor
      ival = 500.0 * vadc;  // 500mA/volt
      rp->ival = ival;
    
      adval = (uint16_t)adc->adc0->analogRead(A2); // Solar Volts
      vadc = 5.0 * 1.2 * (float)adval / 4096.0; // correct for 4:1 attenuator
      rp->vs = vadc;
    
      adval = (uint16_t)adc->adc0->analogRead(A3); // T3 Power  Volts
      vadc = 5.0 * 1.2 * (float)adval / 4096.0;// correct for 4:1 attenuator
      rp->vt = vadc;
    
    }
    
    char *TString(time_t tm) {
      static char dstr[12];
      // sprintf(dstr,"%02u/%02u/%04u       ",month(tm),day(tm),year(tm));
      sprintf(dstr, "%02u:%02u:%02u", hour(tm), minute(tm), second(tm));
      return dstr;
    }
    
    
    // called from the datalogger CheckLoggger function
    void myBinaryDisplay( void* vdp) {
      struct datrec *dp;
      dp  = (struct datrec *)vdp;
      if (!logging) return;
      Serial.printf("%s\t", TString(dp->unixtime));
      Serial.printf("%4.1f\t%4.1f\t%4.3f\t%4.3f\n", dp->mAvg, dp->mPeak, dp->vsAvg, dp->vtAvg);
    
    }
    
    
    /************************************************
       struct datrec {
      uint32_t unixtime,
      float mAvg, mPeak,vsAvg, vtAvg;
      };
    
     ***************************************************/
    
    // Used to write processed processed binary data.  Input to function is
    // a pointer to a rawdat record.
    // This gets called once for each rawdat buffered at the 1000Hz rate.
    // A binary record is saved every saveinterval seconds
    uint16_t myBinaryWriter(void *bdp, void* rdp) {
      static uint16_t rcounter = 0;
      uint16_t rval;
      static struct datrec dr;
      struct rawdat *rp;
      rp = (struct rawdat *)bdp;
      static float isum, vssum, vtsum, ipeak;
    
      char** sptr;
      sptr  = (char**) rdp;
      // add data to sums and check peaks
      isum += rp->ival;
      if (rp->ival > ipeak) ipeak = rp->ival;
      vssum += rp->vs;
      vtsum += rp->vt;
    
      rcounter++;
      if (rcounter >= (SAMPLERATE * SAVEINTERVAL)) {
        dr.mPeak = ipeak;
        dr.mAvg = isum / rcounter;
        dr.vsAvg = vssum / rcounter;
        dr.vtAvg = vtsum / rcounter;
        dr.unixtime = now();
    
        ipeak = 0.0;
        isum = 0.0;
        vssum = 0.0;
        vtsum = 0.0;
    
        *sptr = (char *)&dr;
        rval = sizeof(dr);
        rcounter = 0;
      } else {
        *sptr = NULL;
        rval = 0;// return nothing until end of interval
      }
      return rval;
    }
    This code shows how I use the MTP responder---and prevent calls to the MTP loop function while logging. It also shows how to queue binary data at 1KHz, then process that data and save peaks and averages every two seconds.
    Attached Files Attached Files

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •