Generic data logger object

[mborgerson]

After posting some example data logger code in other threads, I have decided to try to write a generic logger object which can become the basis of a data logger library.

One of the first things I need to figure out is how to have the logger object allocate it's own storage for the internal buffers and/or queues. The logical way is to have the logger object examine the user-defined storage structure size, then initialize queues and buffers and use 'new' inside the object to get the required memory from the heap. (Of course, the object will free that memory in the destructor.) I haven't explicitly used new/delete or malloc()/free() much at all in my embedded programming.

Some questions occur to me:

Are there limits on how much memory I should allocate in this fashion?
Is there a way to determine how much memory is available in the heap? (So I can set up a buffer that leaves some space for other parts of the program)

Can I add some code to make the SD Card look like a USB-connected device that can be used to upload the logged data from the SD card?


Here are some of the more complex public methods that I envision:

datalogger::InitStorage(uint16_t structsize, uint32_t collectionrate, uint16_t bufferMSec);
The user passes the size of their data structure, the collection rate and the number of milliseconds of buffer capacity. From those values the object calculates the amount of buffer memory that it requires. There will probably need to be some indicator of whether the memory allocation succeeded and how much memory was allocated.

datalogger::AttachCollector( void *aptr);
The user passes in a pointer to their data collection function. The function will be called from the internal IntervalTimer with a pointer parameter that the user typecasts to their structure type, then fills in the data to be saved. I envision some further questions about what devices can be used to do the collection, as the data is collected inside an interrupt handler.

datalogger:AttachDisplay(void *aptr);
The user passes in a pointer to a data display function. That function will be called when the user requests a display of the logged data. The user will write a function that does an appropriate display of the data. The user function will receive a pointer to the data to be displayed.

When I get a bit further along, I'll post some code showing the header file for the class.

 

[defragster]

That is promising - the one example posted this week was fun to play with - I took it to the T_4.1 beta page with two almost 1 MB buffer for the SDFat write using external 8M QSPI RAM chip - I added a SPARSE Print to make the Print spew easier to parse - and used it to show data log anomalies I have to get back to.

As far as ways to see free heap - look at this for T_4 :: github.com/FrankBoesing/T4_PowerButton/blob/master/T4_PowerButton.cpp
Other ways are needed and around for other MCU's - this looks promising for T_3.2's - maybe others? :: github.com/FrankBoesing/RamMonitor

As far as exposing data over USB - that is a separate project - there is a thread or so on that … one may be this :: MTP-Responder-Contribution

 

[mborgerson]

First release of DataLogger class

The DataLogger class provides the collection, buffering and storage elements for a generic data logging application. It has the following characteristics:

* A user-defined storage structure allows many different types of data to be collected.

* The internal buffer of the data logger is organized as an array of bytes grouped into chunks of data, each of which is an integer multiple of the storage record size for efficient writing to SD card.

* The amount of buffer to use is calculated based on the record size, collection rate, and the expected maximum delay during SD Card writes. A 1000 milliSecond delay is probably appropriate for slower SD cards using FAT32.

* The buffer can reside either in a user-defined and allocated array of bytes, or it can be allocated from the heap. When you initialize the buffer it returns the number of bytes it expects to use. If your specified collection rate, structure size and expected delay result in a buffer requirement larger than your allocated array or the available heap, you should halt the program and change your code.

* The data buffer is divided into 10 equal-sized chunks, which are collected an written using a FIFO queue.

* Data collection is timed by an IntervalTimer and collection occurs in the timer's interrupt routine, so it collects at fixed intervals independent of SD card write delays or other lengthy operations.

* Callback functions from the datalogger to the user code allow customized data collection, display and SDC writing. It is very simple to switch from the efficient storage of binary data records to the storage of CSV strings.

* The data writer callback function can be set up to handle event-triggered logging or filtering and decimation of the input data stream.

* The data logger keeps a lot of internal statistics on things like collection timing, SD card write delays, and the amount of data written. The status record is defined at the global level in the header file so that you can write your own status display functions.

* A display callback function can be added to show the data collected at a user-specified interval to allow monitoring of sensor performance. You can access that data through a function that returns a pointer to the most recently written record.

* Data displays and debug information are sent to the USB Serial port by default, but can be switched to a hardware serial port if desired.

* The DataLogger class uses SDFat 2.0 so that it has access to ExFat for very efficient storage of data to SD cards with pre-allocated contiguous files.

This is a beta version of the code and the DataLogger object may change significantly over the next month or two.

The DataLogger class header file and CPP source are attached in a .zip file. To use the class, put the unzipped DataLogger folder in your sketch/library folder and restart the Arduino IDE. You will then be able to add the library to your sketch with the "Sketch/Include Library" menu.

The DataLogger folder includes an Examples folder with sketches illustrating the ways in which you can customize your data logger. The example code has been tested on a Teensy 3.6. I expect to run some tests on my newly-arrived T4.1 over the next few days.


I expect to put the class files and examples on GitHub sometime in the future, but getting that done is a learning curve to experience at another time.

View attachment DataLogger.zip

 

[defragster]

@mborgerson:
Looks like a decent library from the prior example. Having multiple buffer chunks looks good - was going to do that to the sample. Have seen other libs and posts with 'high speed loggers' that I never looked into to compare but your code was nicely readable and looks clean enough.

Not sure if you watched the T_4.1 beta thread - I used the prior code to log to 8MB PSRAM with mods {removed actual SD write to just test PSRAM working with _isr and loop access} to that - and some bad pointers from manual buffer alloc() gave false fail indications.

It did fail at 1MB with the sample collect of the 16 byte test struct. Was logging under, 1MB but the reason it failed at 1MB in that case was 16 bit uint collectchunk { prior code changed to uint32_t saveidx }

volatile uint16_t collectchunk, writechunk;

Those should be made uint32_t if they are indexing into large memory pools. I got a new T_4.1 planning to solder on two 8MB PSRAMs that will give 16MB contiguous.
Generally on 32 bit ARM's just using 32 bit will be faster.

 

[defragster]

Hi - running a local copy of : Timing_LoggerTest.ino
Nice work! It worked in the default case. And when modified to use PSRAM on T-4.1.

Nice and easy to setup and use and point to the PSRAM buffer. Great to see valid Time/date stamp on the files! THough I see this build note:

T:\arduino-1.8.12\hardware\teensy\avr\libraries\Time/time.h:1:2: warning: #warning "Please include TimeLib.h, not Time.h. Future versions will remove Time.h" [-Wcpp]
#warning "Please include TimeLib.h, not Time.h. Future versions will remove Time.h"

One thing is this sketch would do well with a way to Verify - open the file to read and in this case check for sequential 'numrecords' values, perhaps that the other times are consistent and always increasing by 1 when they change? Have logger, when stopped, offer a way to open the file for read and return the sample records, knowing the filename and having the struct info on hand would make that a nice feature to return records? I did that in the prior sketch to have the sketch auto check the data because it was too much to see on the SerMon display and make sense of and as binary data hard to see without writing another program offline.

One anomaly - to get full use of memory ( 16 MB ) this change was needed to prevent overflow in :: ...\DataLogger\DataLogger.cpp

  bufferneeded = (([U]structsize * collectionrate[/U] ) / 1000) [B]* mSecLen[/B];

It worked in this case - not sure if a different order change makes sense - or casting into 64 bit uint until the divide.

Took a bit - and a look at the code - to understand how the buffer size is calculated. Then I added a print of that and saw it was truncated after adding this to the ino:

  if ((bufflen == 0) || (bufflen > MAXBUFFER)) {
    Serial.println("Not enough buffer space!  Reduce buffer time or sample rate.");
    fastBlink();
  }
  else {
    Serial.printf("Buffer size needed %lu\n", bufflen);
  }

It was truncating due to the order of operations when setting:

#define SAMPLERATE 250000
#define BUFFERMSEC 1000
char logfilename[64];

// use our own buffer on the T3.6
#define MAXBUFFER 7000000
// uint8_t mybuffer[MAXBUFFER];
uint8_t *mybuffer = (uint8_t *)0x70000000;

Those settings gave this:

buffer at: 0x70000000 with length 4000000 
chunk size is 400000   or 25000 data records
Buffer size needed 4000000

I added a Loop Count per second to see what was going on - the delay(2) on checking for input had to go. It is added to the 5 second printing here at the line end:

1524.140,   Records:381025000   Overflows:   16   loopCntCurr: 3
1529.340,   Records:382325000   Overflows:   16   loopCntCurr: 2
1534.440,   Records:383600000   Overflows:   16   loopCntCurr: 1766110
1539.440,   Records:384850000   Overflows:   16   loopCntCurr: 647864
1544.440,   Records:386100000   Overflows:   16   loopCntCurr: 1858585
1549.240,   Records:387300000   Overflows:   16   loopCntCurr: 4
1554.440,   Records:388600000   Overflows:   16   loopCntCurr: 2404175
…
1589.440,   Records:397350000   Overflows:   16   loopCntCurr: 2739043
1594.440,   Records:398600000   Overflows:   16   loopCntCurr: 2222278

Logger Status:
MBytes Written:  1990.35
Collection time: 1596 seconds
Max Collection delay: 1 microseconds
Average Write Time: 70.747 milliseconds
Maximum Write Time: 514.071 milliseconds

That shows normally about 2 million loop()'s per second - except when the SD write stalls. And it looks like 16 Overflows?

So I bumped to these settings to avoid the overflow at that high sample rate:

#define SAMPLERATE 250000
#define BUFFERMSEC 3000
char logfilename[64];

// use our own buffer on the T3.6
#define MAXBUFFER 15000000
// uint8_t mybuffer[MAXBUFFER];
uint8_t *mybuffer = (uint8_t *)0x70000000;

It it looks to have started okay with 12MB of buffer used. I changed the code to print each second's loop count rather than each five. Also those latest settings seem to fit in memory and have prevented Overflow.

buffer at: 0x70000000 with length 12000000 
chunk size is 1200000   or 75000 data records
Buffer size needed 12000000
Starting Logger.
File name is:  <LOG_0046.bin>
Pre-allocating 1GB file space 
Allocation succeeded.
Opened  File LOG_0046.bin
 
 lCnt: 5798771, lCnt: 233267, lCnt: 2, lCnt: 1, lCnt: 1,
   4.540,   Records: 1125000   Overflows:    0 	 lCnt: 1, lCnt: 2875979, lCnt: 1888361, lCnt: 3605322, lCnt: 3151245,
   9.640,   Records: 2400000   Overflows:    0 	 lCnt: 2847929, lCnt: 193767, lCnt: 2, lCnt: 2, lCnt: 2,
  14.140,   Records: 3525000   Overflows:    0 	 lCnt: 2067750, lCnt: 3, lCnt: 772324, lCnt: 2018135, lCnt: 2898249,
  19.540,   Records: 4875000   Overflows:    0 	 lCnt: 1404425, lCnt: 2975773, lCnt: 3680696, lCnt: 1975690, lCnt: 2374501,
  24.640,   Records: 6150000   Overflows:    0 	 lCnt: 2, lCnt: 3122582, lCnt: 3939747, lCnt: 2427998, lCnt: 1,
  28.840,   Records: 7200000   Overflows:    0 	 lCnt: 3, lCnt: 2739317, lCnt: 3315775, lCnt: 52805, lCnt: 1579606,
  34.540,   Records: 8625000   Overflows:    0 	 lCnt: 2072821, lCnt: 2285362, lCnt: 1147757, lCnt: 129729,
  39.640,   Records: 9900000   Overflows:    0 	 lCnt: 3223082, lCnt: 1976284, lCnt: 592270, lCnt: 2502392, lCnt: 2457422, lCnt: 3,
  44.440,   Records:11100000   Overflows:    0 	 lCnt: 2952963, lCnt: 3517981, lCnt: 496039, lCnt: 1284406, lCnt: 3,
  49.540,   Records:12375000   Overflows:    0 	 lCnt: 2, lCnt: 1650703, lCnt: 3653001, lCnt: 2447, lCnt: 2500369,
  54.640,   Records:13650000   Overflows:    0 	 lCnt: 2868919, lCnt: 1987013, lCnt: 3073759, lCnt: 2326527, lCnt: 229316,
  59.440,   Records:14850000   Overflows:    0 	 lCnt: 3, lCnt: 3, lCnt: 2333092, lCnt: 1322661, lCnt: 1,
…
 239.440,   Records:59850000   Overflows:    0 	 lCnt: 1, lCnt: 3, lCnt: 2115988, lCnt: 2, lCnt: 2,
 244.240,   Records:61050000   Overflows:    0 	 lCnt: 1591842, lCnt: 2, lCnt: 3, lCnt: 338483, lCnt: 2,
 249.340,   Records:62325000   Overflows:    0 	 lCnt: 3, lCnt: 1, lCnt: 1, lCnt: 2, lCnt: 1,
 254.140,   Records:63525000   Overflows:    0 	 lCnt: 1, lCnt: 1, lCnt: 1, lCnt: 1, lCnt: 1,
 259.240,   Records:64800000   Overflows:    0 	 lCnt: 1, lCnt: 1, lCnt: 1, lCnt: 1,
 264.040,   Records:66000000   Overflows:    0 	 lCnt: 1, lCnt: 2, lCnt: 2, lCnt: 562541, lCnt: 4357189, lCnt: 2596467,
…
1794.640,   Records:448650000   Overflows:    0 	 lCnt: 3, lCnt: 2, lCnt: 2744547, lCnt: 1377938, lCnt: 2,
1799.440,   Records:449850000   Overflows:    0 	 lCnt: 1, lCnt: 1, lCnt: 2, lCnt: 206504,
1804.540,   Records:451125000   Overflows:    0 	 lCnt: 4304783, lCnt: 2587558, lCnt: 3, lCnt: 1, lCnt: 1446262, lCnt: 1325149,
1809.640,   Records:452400000   Overflows:    0 	 lCnt: 391003, lCnt: 3, lCnt: 3,
Logger Status:
MBytes Written:  2821.95
Collection time: 1814 seconds
Max Collection delay: 1 microseconds
Average Write Time: 227.386 milliseconds
Maximum Write Time: 849.430 milliseconds

 

[mborgerson]

I'll be trying out some of the T4.1 tests later today. I got the headers and PSRAM chip soldered on yesterday, so the hardware is ready to go.

I agree that there should be some way to verify files besides plugging the SD card into a PC and analyzing the file with MatLab. Adding a better user interface so that a user can select files and easily issue commands with parameters is a project in progress. One way I may approach this is to add a callback for a "playback file" option that has a file name parameter, but will use the last file recorded if the name parameter is empty.. That avoids having to add a UI to get and verify a file name. The callback would open the file and send each record to a user function that could check for things like missing records. For something like an analog logger, the callback could extract one or two channels and send them out in a format compatible with the graphing option in the Arduino IDE.

One pitfall of a generic playback option is that, if you try to play back a file recorded with a different data structure, you're going to get garbage out. One possibility is to have some sort of hash key based on the data structure encoded in the file name. Then, a program could refuse to play back files that did not match the current program's hash key.

 

[mborgerson]

The calculation of bufferneeded does need some work. Your example of:

bufferneeded = ((structsize * collectionrate ) / 1000) * mSecLen;

Is OK for very fast collection, but will return zero for a 4-byte structure sampled 100 times per second.

 

[XFer]

Change to

bufferneeded = (mSecLen * structsize * collectionrate ) / 1000;

with a further check to avoid 0

 

[defragster]

Change to

bufferneeded = (mSecLen * structsize * collectionrate ) / 1000;

with a further check to avoid 0

The calculation of bufferneeded does need some work. Your example of:

bufferneeded = ((structsize * collectionrate ) / 1000) * mSecLen;

Is OK for very fast collection, but will return zero for a 4-byte structure sampled 100 times per second.

@mborgerson: Indeed that change worked for my FAST samples large number - but expected/noted it was not likely ideal for all cases large/small.

@XFer: that was the original code that failed for 32 bit overflow.

Doing it as 64 bit (uint64_t) would allow the multiple to work in any case and then not lose anything when it came to the '/1000'

 

[defragster]

I'll be trying out some of the T4.1 tests later today. I got the headers and PSRAM chip soldered on yesterday, so the hardware is ready to go.

I agree that there should be some way to verify files besides plugging the SD card into a PC and analyzing the file with MatLab. Adding a better user interface so that a user can select files and easily issue commands with parameters is a project in progress. One way I may approach this is to add a callback for a "playback file" option that has a file name parameter, but will use the last file recorded if the name parameter is empty.. That avoids having to add a UI to get and verify a file name. The callback would open the file and send each record to a user function that could check for things like missing records. For something like an analog logger, the callback could extract one or two channels and send them out in a format compatible with the graphing option in the Arduino IDE.

One pitfall of a generic playback option is that, if you try to play back a file recorded with a different data structure, you're going to get garbage out. One possibility is to have some sort of hash key based on the data structure encoded in the file name. Then, a program could refuse to play back files that did not match the current program's hash key.

Indeed - playback noted was for 'current log record'. Anything else would possibly fail or need lots of UI to pick the file. And maybe require a secondary file with structure details. That might be handy anyhow in the end - showing log start/stop time&date and log struct details in some fashion.
For current log active or stopped - just having a way to retrieve and test records would be a way to validate or debug a system. In the prior example version I took out SD WRITE and instead parsed the buffer data to be written.

 

[XFer]

Yes I took for granted it was an uint64_t

 

[mborgerson]

Xfer and defragster: I took your comments to heart and made the following changes:

1. The internal calculations for the buffer size now use a uint64_t and check for values out of the uint32_t range.
2. I added a number of sanity checks on the input values which limit structure size and the number of bytes/second
the logger would have to write to the SD card.
3. I changed byteswritten to a uint64_t, since it only takes about 40 minutes for a really fast setup, like that used by
defragster, to have the bytes written exceed the 4GB limit of a uint32_t.
4. I added a playback routine and a fast logger example that makes use of PSRAM on a T4.1 to log lots of timing data
very quickly. The playback output goes to a verifier function that checks for missed storage records.

I'll add the new stuff to the library folder and upload it in the morning, after I do some more testing.

A few notes:

A. spell check wants to change "defragster" to "defroster"! "defroster" seems like a cool screen name, so perhaps I'll claim it.
B. You need to download and install Teensyduino 1.52 beta (I have beta6) to get the T4.1 menus and be able to use the
PSRAM.

Thanks for your input.

 

[mborgerson]

Here is an update library source and some new examples that illustrate file playback. One program is designed to log timing data very quickly on a T4.1, then verify that no records were skipped during logging. The ASCII example now lets you play back the data to the USB serial port. This could be used to retrieve data using a serial terminal that can log received data to a file.

My long-term plan is to be able to have the logger become an MTP responder, so that you can upload the files from the Teensy as if it were an attached as an MTP USB disk. I've gotten the basic MTP test programs to work on the T3.6, but not the T4.That means that full MTP capability awaits my education and, probably a lot of work by other programmers working on the MTP functionality.

 

[defragster]

@mborgerson- defroster ... funny. Do you have a github account to post with? Not sure if I saw and adjusted anything else as I went. Most of the sample sketch cnages were noted for feedback.

Looking forward to it to beat on the PSRAM and verify. That will save me from hacking on it.

Had a thought about having a callback after the data went to SD with a pointer to the buffer before it got marked as free for next filling. That is basically what I did in the prior sketch.

Looking at the .CheckLogger() the noInterrupts();/interrupts(); isn't pretty (millions of times per second) - maybe you could make this pattern work :: TeensyTimerTool

That works to READ volatile variables that can change in an _isr as that "do{" can repeat and readychunks might get a '++' - which looks like most of that except maybe:

  numready += readychunks; // numready would be set to zero before entering do { __LDREXW( ...
  readychunks = 0;  // clear number ready

That allows interrupts - ANY interrupt will force a repeat of that do{}while();

 

[JBeale]

Sorry to be slightly off-topic, but do you data-acquisition folks have a type or brand of SD card you're happy with? In my experiences with other hardware (Raspberry Pi) it seems these memory cards tend to be the weakest link in a system if you do a whole lot of writing to them.

 

[defragster]

Sorry to be slightly off-topic, but do you data-acquisition folks have a type or brand of SD card you're happy with? In my experiences with other hardware (Raspberry Pi) it seems these memory cards tend to be the weakest link in a system if you do a whole lot of writing to them.

Card selection is a mystery AFAIK - SanDisk should be good - but all may not be real first quality. PJRC notes SanDisk … but that is older ? may just be based on speed at the time not long term usability.

 

[defragster]

@mborgerson - using the new code the 64 bit math still allows compiler to cheat doing 32 bit then upgrading the lost data into 64 bits - adding this cast in both places works:

  buffneeded64 = ([B][U](uint64_t)[/U][/B]structsize * collectionrate * mSecLen);
  if(buffneeded64 > 0xFFFFFF00){ // calculation will overflow in uint32_t
		if (dbprint)iosptr->println("Possible overflow in buffer size calculation");
		//return 0;
  }

But then in my case trying to use 12 MB of 16MB of RAM the overflow test fails. Removed the return 0; there and I get the flag print, as above - but I get usable 12 MB of RAM:

result is this running like before:

Data Logger Timing Example   Compiled on May 16 2020 00:01:35
Card Type is exFAT
File system initialization done.
RTC has set the system time
Possible overflow in buffer size calculation
Buffer initialization complete.
buffer at: 0x70000000 with length 12000000 
chunk size is 1200000   or 75000 data records
Buffer size needed 12000000

Now to go see about the 'Verification done' on buffer data.

And maybe try removing the no_int code. Running this sample with 250,000 _isr()'s per second. This is with data use now at 14,400,000 buffer bytes.

#define SAMPLERATE 250000
#define BUFFERMSEC 3600
char logfilename[64];

// use our own buffer on the T3.6
#define MAXBUFFER 15000000
// uint8_t mybuffer[MAXBUFFER];
uint8_t *mybuffer = (uint8_t *)0x70000000;

 

[defragster]

>> ERROR: 32GB Disk filled and logging halted without warning.

> And restarting the Sketch just hangs on start with no feedback

Put SD into PC and deleted files - the sketch still could not run? Then it got as far as this and I added sketch format - which did not correct the problem:

  if (!mydl.InitStorage()) { // try starting SD Card and file system
mydl.FormatSD(true); // format exFat for all sizes
    // initialize SD Card failed
//    fastBlink();
  }

It would then START - but could not create file. did a manual 'f' format from Menu items and that also left it unusable to start logging.

Had to return SD to PC to Format. { expect this is SDFat format issue - or leftover from filling disk and some disk struct not updated the SDFat_format doesn't repair}

Logger running with this code change to REMOVE noInterrupts()::

  if ( 0 ) { // this code replaced below
  // block interrupts and update mystatus with variables
  // that are changed in timer interrupt handler
    noInterrupts();
    numready = readychunks;
    readychunks = 0;  // clear number ready
    mystatus.bufferoverflows = bufferoverflows;
    mystatus.maxcdelay = maxcdelay;
    interrupts();
  }
  numready = 0;
  uint16_t readyHold;
  do { // account for interrupts that may alter vars during read.
    __LDREXW(&logger_safe_read);
    readyHold = numready;
    numready += readychunks;
    readychunks -= (numready - readyHold);  // account for number found
    mystatus.bufferoverflows = bufferoverflows;
    mystatus.maxcdelay = maxcdelay;
  } while ( __STREXW(1, &logger_safe_read));

 

[mborgerson]

I'll work over these issues today:

1. type casts to make sure the buffer needed calculation is done with uint64_t internally.
2. Investigate ways to handle "disk Full" situations. The logger really shouldn't render an SD card unusable if
it runs out of storage space.
3. Look over options for the noInterrupt/interrupt situation. One option would be to mask only the timer interrupt.
In other loggers on other platforms, I've been careful to have the IRQ handlers either write or read volatile variables
while the foreground routines do the opposite for each variable. I've also looked at the assembly code generated to
make sure the read or write was an atomic instruction. In those cases shutting off interrupts was not so much an
issue. And, for the most part, there were a lot fewer interrupts happening---none in USB, ADCs, SDCs, etc.etc.

Most of my loggers over the past two decades have been low to medium speed (1-3KB/sec) and very low power (20mA for
storage, peripherals and sensors). When I do those things on the Teensy, the interval timer is running at about 100 to 400 Hz
and there is a WFI() in the main loop. I don't have the main loop running as fast as possible burning up CPU cycles just to
increment a loop counter. I quite often test my Teensy stuff at 48MHz and I usually find that things work fine as the sensors
and SD card hardly notice the drop in CPU speed. It's only when I try to do things like correlation detection, multiple digital
filters and FFTs that I have to crank the speed back up to Warp 1 (>150MHz ).

 

[defragster]

Cool.
Full use of PSRAM for a logger while doing SDIO writes seemed like a natural(real world) offering of this code as a way to test this on the new T_4.1 - so this is a bit extreme - but it works.
In looking the Verify code wasn't apparent yet?
WFI on T_4.x's is different as the sysTick won't wake it, but a real timer will.
Check that 'LDREXW' code - it will run straight through if no interrupt triggers in the do section. It will repeat as often as SOME/ANY interrupt fires - across that loop. It is used as linked to get micros() extension to millis/sysTick using ARM cycle counter, and that runs in about 38 cycles on average.

I didn't try to read the binary data on the PC - I assume it would have read the files - it deleted them - but something still prevented the sketch from access, even after format.

 

[mborgerson]

On the subject of SD Cards:

Five to 10 years ago, I used a lot of SanDisk cards with good results. Then about five years ago SanDisk cards started getting counterfeited on a large scale. A co-worker at Oregon State got a batch of SanDisk 32GB cards which had about a 50% failure rate in testing. It turns out that they were counterfeit. Their printed labels were very good copies of the original SanDisk labels. However, there were some small differences in the laser-etched notations on the cards themselves. Quite often, the cards had only half the capacity they should have. I suppose there might have been speed issues also, but they were masked by the fact that the loggers used them in SPI mode at very low clock rates.

I'm getting good results now on Samsung micro sd cards. You can get a 5-pack of Samsung EVO 32GB cards for about $33 on Amazon. A 2-pack of SanDisk 128GB cards is about $33. Whether you want to thoroughly test each incoming card depends on your project's tolerance for errors. If you're going to deploy a logger 100m deep in the middle of the Indian Ocean for six months, you should THOROUGHLY test for errors before deployment.

Perhaps I will write a test program to at least verify the capacity of a micro-Sd card. It's pretty easy to detect off-size cards. You write a test pattern about once each 128KB and read it back. If the test pattern for block 400,000 appears at some other location, you've got a problem!

 

[mborgerson]

This update to the data logger has the following changes:

1. Used method proposed by @defragster to avoid blocking interrupts on every call to CheckLogger.

2. Added code which checks free space on the SD card and stops writing data when too close to the
End of the SD free space. It prevents damage to the card file system, but can still allow some
overruns near the end of the card. It seems that a file write in the last few MB of free space can
overwork the card's internal processor as it shifts, erases, and remaps blocks. Some writes at
the end of the card took over 3.6 seconds to complete! The best solution seems to be to stop
logging at least 100MB before the end of the free space.
3. Tried again to make sure that the calculation of the buffer space needed does not overflow
any 32-bit calculations. This still needs more exhaustive testing--which is probably best done
in a simpler sketch.


The file verification code is present only in the FastLoggerT4_1 sketch. It seems do work and did
catch errors at the end of the SD cards when chunk write times went past the buffer size. I fund no
errors before that point.

I did test the card overflow problem both on a 2GB card and on a 16GB Samsung EVO card. The
latter card took about 1hour and 10 minutes to fill on a T4.1 at 600MHz.

 

[defragster]

Wow this post got buried in the NEW list before I saw it … Good to see the updates.

My T_4.1 ran the old code to FILL the SD again with a 28+ GB file where it dies and would not restart. Moved the SD to Windows and found PowerShell :: PS C:\tmp> format-hex .\LOG_1305.bin
Could dump the file that took 19 minutes to move to the hard drive. The data dumping looked like expected data representation as far as I looked.

#1: hope it looked right _ I thought it might be done a time or two until I re- processed the holes. Any time there is an external write it is dangerous - luckily the micros() usage was all reading.

#2: good - that will prevent the disk full fail. Indeed the juggling the last free space could lead to odd effects so best to avoid that.

#3: Will see how the (casted) math works with 16 MB of space

FastLoggerT4_1 - got it will look there instead of the 'Timing...'

 

[defragster]

Looking at the FastLogger:

At first this would not work to even start - just gave 'fail ???':

#define SAMPLERATE 250000
#define BUFFERMSEC 3950
char logfilename[64];

// use our own buffer on the T4.1
// With one PSRam chip, we get 8MB of buffer--so leave about a MByte
// for other uses
#define MAXBUFFER 15900000

Now it does with change below ( there was a missing cast on the recalc that is gone now ):

Buffer initialization complete.

buffer at: 0x70000000 with length 15800000 
chunk size is 1580000   or 98750 data records
End of buffer at 0x70f116c0
Starting Logger.

File name is:  <LOG_0001.bin>
Pre-allocating 1GB file space 
Allocation succeeded.

Opened  File LOG_0001.bin

Made the code look like this - the added print is not right but it helped - on failure I had no idea what the math result was for needed buffer:

  buffneeded64 = ((uint64_t)structsize * collectionrate * mSecLen);
  // need to make sure this works as intended
  bufferneeded = buffneeded64/ 1000;
  if(buffneeded64 != ((uint64_t)bufferneeded*1000)){ // calculation will overflow in uint32_t
		if (dbprint)iosptr->printf("Possible overflow in buffer size calculation using %lu?\n", bufferneeded);
		return 0;
  }

Now need to check Verify function. I stopped logging and hit 'v' and it just hung?

Verify after logging works. Problem was hitting 'v' during logging breaks the system after this:

Allocation succeeded.

Opened  File LOG_0032.bin
 
   3.950,   Records:    987500   Overflows:    0
Verifying last file

Can't play back while logging

I put this in sketch - problem is in the library response?

void VerifyFile(void) {
  Serial.println("Verifying last file");
[B]  if ( logging == true ){
    Serial.println("Can't verying while logging!");
    return;
  }
[/B]

I added millitime verify to the sketch to check another 4 bytes to be the same or +1 as okay.

 

[defragster]

BTW - doing a format between runs works fine. It was only that odd case of the filled to full that caused trouble with the built in format.

So far with some edits for the prior post all seems to be working well using 15,800,000 Bytes for logging across two PSRAMs on a T_4.1

Now to walk away and let the 32GB SD card fill.

Logger Status:

MBytes Written:   999.01   SdCard free space: 29421.11 MBytes
Collection time: 263 seconds
Max Collection delay: 1 microseconds
Average Write Time: 279.631 milliseconds
Maximum Write Time: 621.274 milliseconds

<edit> switching to packed ... good long no error - NO SD - run:

 V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@ 2794.867,   Records: 349358400   Overflows:    0
 V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@ 2799.705,   Records: 349963200   Overflows:    0
 V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@ 2804.889,   Records: 350611200   Overflows:    0
 V@  V@ 
Logger Status:

MBytes Written: 11893.41   SdCard free space: 18526.71 MBytes
Collection time: 2806 seconds
Max Collection delay: 7 microseconds
Average Write Time: 79.007 milliseconds
Maximum Write Time: 79.137 milliseconds

Packed struct run started:

buffer at: 0x70000000 with length 14256000 
chunk size is 1425600   or 43200 data records
End of buffer at 0x70d98780

Logger Status:
MBytes Written:     0.00   SdCard free space: 29396.12 MBytes
...
Data Logger Directory
2020-05-18 18:57            0 LOG_1857.bin

Starting Logger.

File name is:  <LOG_1945.bin>
Pre-allocating 1GB file space 
Allocation succeeded.

Opened  File LOG_1945.bin
 
 V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@  V@    4.493,   Records:    561600   Overflows:    0
...