Low latency data logger

Status
Not open for further replies.

michaelstetner

New member
I wanted to share a sketch I wrote to acquire data and store it on an SD card. In my tests using a Teensy 3.6 and a SanDisk Ultra 16GB SDHC micro SD card, I was able to record an analog pin at 25 kHz. That's pretty fast :cool:

There has been some discussion of how to do this on this forum, but I couldn't find a complete working example, so I made by own.

It uses the beta version of the SdFat library written by Bill Greiman, which is available at https://github.com/greiman/SdFat-beta. I have tested this sketch with revision ffbccb0 of SdFat-beta. This code was inpired by SdFat's LowLatencyLogger and TeensySdioDemo examples.

I hope my code will help you make your own loggers.

Download the code here:
https://github.com/MichaelStetner/TeensyDataLogger
 
The code explicitly limits sample rate with : const uint32_t sampleIntervalMicros = 40; // 40 us interval = 25 kHz

and to acquire it is not using the ADC MDA code but :: data->adc[0] = analogRead(SENSOR_PIN);

It does log at least 12 bytes for each data sample - looks like 300,000 bytes per second
 
Your code is missing what makes the SdFat LowLatencyLogger low latency - the pre-allocated, pre-erased file. When using standard file writes, I have seen rare latency spikes of 850ms (vs. 40ms max for the pre-allocated, pre-erased file). Your buffers would overflow.

SdFat with the SDIO interface is pretty good with calling yield, the maximum latency I have seen there is 75us.
 
Just out of interest (as the above is vaguely related to my dual-Can full-speed SD logging), I have a very simple question :

If an analogue pin is 16bit precision (2 bytes) and you are happy to add a timestamp (uint32, 4bytes) once every 10 samples, then
your 300,000 bytes/sec will get you 125kHz logging to SD and your buffer would need to be 300Kb out of the available 1Mb Ram to cover for a potential worst case 1 second latency?
 
then your 300,000 bytes/sec will get you 125kHz logging to SD and your buffer would need to be 300Kb out of the available 1Mb Ram to cover for a potential worst case 1 second latency?
Teensy 3.6 has 256kB RAM (3.5 has 192kB, 3.2 has 64kB).
 
Ooops! Thank you.

That's due to MY memory, - probably getting close to it's maximum number of writes by now...
 
It does log at least 12 bytes for each data sample - looks like 300,000 bytes per second

So for every two bytes of data you record another 10 bytes of...

What?

Looking at the code I see 4 bytes of timing data added. But that is just an admission that you don't have good control on the actual conversion process. Worse is that the sampling interval is not well controlled which would be a problem if you wanted to run an FFT on the data.

Get better control on the conversion process. You could configure the ADC to be triggered by a PDB or configure it for continuous conversions. Have a look at the audio library for an example of using the PDB.

Then you only need to record two bytes of actual data for each sample.

Hmmm. I looked at that SDIO code and I cannot find where it sends ACMD23 before CMD25. This tells the card how many blocks to expect which improves write speed.
 
Your code is missing what makes the SdFat LowLatencyLogger low latency - the pre-allocated, pre-erased file. When using standard file writes, I have seen rare latency spikes of 850ms (vs. 40ms max for the pre-allocated, pre-erased file). Your buffers would overflow.

Version 5 of the SD Simplified Specification finally added some detail where there had been none on recording speeds. A lot of detail. One of those details is that for a FAT update you can expect up to a 750ms delay. (Table 4-61) Average is 100ms. That and varying quality in FAT libraries is why when I need speed I take steps to avoid file system updates. For the Logomatic I found a large contiguous free area to write to and delayed all FAT updates until file close time. For now I am skipping a file system completely. There are still periodic stalls but they are manageable.

I don't have enough buffer space to be able to write data in the size required to get rated speed on a speed class 10 SDHC card (512K!) but 16KB does pretty well at >6MB/sec. (Recording Unit size is another of those details previously missing.)
 
For a fast datalogger having far compacter data stored is really very useful, aka probably needed.

Using blocks with 1 timestamp, measurement frequency and several ten thousand measurements is far better. You could even consider that the measurement has only 12-bit accuracy an packing 12-bit values. Or probably easier use a compression on the blocks, for example mini-LZO of LZ4.
 
Hmmm. I looked at that SDIO code and I cannot find where it sends ACMD23 before CMD25. This tells the card how many blocks to expect which improves write speed.

I had looked multiple times into this: AFAIK, there is no ACMD23 in SDIO mode, this seems to be SPI mode only.
 
The specification is quite clear on ACMD23 being a mandatory command. If it weren't supported then sending the command to the card would result in an error.

It doesn't. At least not for me.
 

Memory cards are not SDIO cards and the SDIO specification means nothing to them. You should be looking at version 6 (I just noticed this was out when grabbing the link) of the Physical Layer Simplified Specification, Part 1.. See table 4-21 in section 4.7.3.

Memory cards have been required to support ACMD23 since version 1 of the specification.
 
Memory cards are not SDIO cards and the SDIO specification means nothing to them. You should be looking at version 6 (I just noticed this was out when grabbing the link) of the Physical Layer Simplified Specification, Part 1.. See table 4-21 in section 4.7.3.

Memory cards have been required to support ACMD23 since version 1 of the specification.

Is this thread not talking about T3.6 with SDIO interface to SD cards?
But it you wanted to be right...
 
That is your reply that talks about MMC but not OP, which talks about T3.6 with SDIO (build in mode for SD card).

Go back and read it again. It said: "I wanted to share a sketch I wrote to acquire data and store it on an SD card. In my tests using a Teensy 3.6 and a SanDisk Ultra 16GB SDHC micro SD card, "

And then read the SD specification:
In addition to the SD Memory Card, there is the SD I/O (SDIO) Card. The SDIO Card specification is
defined in a separate specification named: "SDIO Card Specification" that can be obtained from the SD
Association. The SDIO Specification defines an SD card that may contain interfaces between various I/O
units and an SD Host. The SDIO card may contain memory storage capability as well as its I/O
functionality. The Memory portion of SDIO card shall be fully compatible to the given Physical Layer
Specification. The SDIO card is based on and compatible with the SD Memory card.
The Teensy 3.6 includes a Secure Digital Host Controller which implements SD mode communication with SD memory or I/O cards. The topic here is using it with SD memory cards. SD memory cards are required to implement ACMD23.
 
Status
Not open for further replies.
Back
Top