Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 18 of 18

Thread: Low latency data logger

  1. #1

    Low latency data logger

    I wanted to share a sketch I wrote to acquire data and store it on an SD card. In my tests using a Teensy 3.6 and a SanDisk Ultra 16GB SDHC micro SD card, I was able to record an analog pin at 25 kHz. That's pretty fast

    There has been some discussion of how to do this on this forum, but I couldn't find a complete working example, so I made by own.

    It uses the beta version of the SdFat library written by Bill Greiman, which is available at https://github.com/greiman/SdFat-beta. I have tested this sketch with revision ffbccb0 of SdFat-beta. This code was inpired by SdFat's LowLatencyLogger and TeensySdioDemo examples.

    I hope my code will help you make your own loggers.

    Download the code here:
    https://github.com/MichaelStetner/TeensyDataLogger

  2. #2
    You ought to be able to do much better than that. I was able to record at over 160KSPS on the Logomatic V1 and it used SPI for the SD card.

  3. #3
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,904
    The code explicitly limits sample rate with : const uint32_t sampleIntervalMicros = 40; // 40 us interval = 25 kHz

    and to acquire it is not using the ADC MDA code but :: data->adc[0] = analogRead(SENSOR_PIN);

    It does log at least 12 bytes for each data sample - looks like 300,000 bytes per second

  4. #4
    Senior Member
    Join Date
    Jan 2013
    Posts
    843
    Your code is missing what makes the SdFat LowLatencyLogger low latency - the pre-allocated, pre-erased file. When using standard file writes, I have seen rare latency spikes of 850ms (vs. 40ms max for the pre-allocated, pre-erased file). Your buffers would overflow.

    SdFat with the SDIO interface is pretty good with calling yield, the maximum latency I have seen there is 75us.

  5. #5
    Just out of interest (as the above is vaguely related to my dual-Can full-speed SD logging), I have a very simple question :

    If an analogue pin is 16bit precision (2 bytes) and you are happy to add a timestamp (uint32, 4bytes) once every 10 samples, then
    your 300,000 bytes/sec will get you 125kHz logging to SD and your buffer would need to be 300Kb out of the available 1Mb Ram to cover for a potential worst case 1 second latency?

  6. #6
    Senior Member
    Join Date
    Jan 2013
    Posts
    843
    Quote Originally Posted by Darcy View Post
    then your 300,000 bytes/sec will get you 125kHz logging to SD and your buffer would need to be 300Kb out of the available 1Mb Ram to cover for a potential worst case 1 second latency?
    Teensy 3.6 has 256kB RAM (3.5 has 192kB, 3.2 has 64kB).

  7. #7
    Ooops! Thank you.

    That's due to MY memory, - probably getting close to it's maximum number of writes by now...

  8. #8
    Quote Originally Posted by defragster View Post
    It does log at least 12 bytes for each data sample - looks like 300,000 bytes per second
    So for every two bytes of data you record another 10 bytes of...

    What?

    Looking at the code I see 4 bytes of timing data added. But that is just an admission that you don't have good control on the actual conversion process. Worse is that the sampling interval is not well controlled which would be a problem if you wanted to run an FFT on the data.

    Get better control on the conversion process. You could configure the ADC to be triggered by a PDB or configure it for continuous conversions. Have a look at the audio library for an example of using the PDB.

    Then you only need to record two bytes of actual data for each sample.

    Hmmm. I looked at that SDIO code and I cannot find where it sends ACMD23 before CMD25. This tells the card how many blocks to expect which improves write speed.

  9. #9
    Quote Originally Posted by tni View Post
    Your code is missing what makes the SdFat LowLatencyLogger low latency - the pre-allocated, pre-erased file. When using standard file writes, I have seen rare latency spikes of 850ms (vs. 40ms max for the pre-allocated, pre-erased file). Your buffers would overflow.
    Version 5 of the SD Simplified Specification finally added some detail where there had been none on recording speeds. A lot of detail. One of those details is that for a FAT update you can expect up to a 750ms delay. (Table 4-61) Average is 100ms. That and varying quality in FAT libraries is why when I need speed I take steps to avoid file system updates. For the Logomatic I found a large contiguous free area to write to and delayed all FAT updates until file close time. For now I am skipping a file system completely. There are still periodic stalls but they are manageable.

    I don't have enough buffer space to be able to write data in the size required to get rated speed on a speed class 10 SDHC card (512K!) but 16KB does pretty well at >6MB/sec. (Recording Unit size is another of those details previously missing.)

  10. #10
    Senior Member
    Join Date
    Dec 2016
    Posts
    120
    For a fast datalogger having far compacter data stored is really very useful, aka probably needed.

    Using blocks with 1 timestamp, measurement frequency and several ten thousand measurements is far better. You could even consider that the measurement has only 12-bit accuracy an packing 12-bit values. Or probably easier use a compression on the blocks, for example mini-LZO of LZ4.

  11. #11
    Senior Member
    Join Date
    Jul 2014
    Posts
    3,150
    Quote Originally Posted by UhClem View Post
    Hmmm. I looked at that SDIO code and I cannot find where it sends ACMD23 before CMD25. This tells the card how many blocks to expect which improves write speed.
    I had looked multiple times into this: AFAIK, there is no ACMD23 in SDIO mode, this seems to be SPI mode only.

  12. #12
    The specification is quite clear on ACMD23 being a mandatory command. If it weren't supported then sending the command to the card would result in an error.

    It doesn't. At least not for me.

  13. #13
    Senior Member
    Join Date
    Jul 2014
    Posts
    3,150
    Quote Originally Posted by UhClem View Post
    The specification is quite clear on ACMD23 being a mandatory command. If it weren't supported then sending the command to the card would result in an error.

    It doesn't. At least not for me.
    it is NOT mandatory for SDIO; see page 60 of https://www.sdcard.org/developers/ov..._Card_Spec.pdf

  14. #14
    Quote Originally Posted by WMXZ View Post
    it is NOT mandatory for SDIO; see page 60 of https://www.sdcard.org/developers/ov..._Card_Spec.pdf
    Memory cards are not SDIO cards and the SDIO specification means nothing to them. You should be looking at version 6 (I just noticed this was out when grabbing the link) of the Physical Layer Simplified Specification, Part 1.. See table 4-21 in section 4.7.3.

    Memory cards have been required to support ACMD23 since version 1 of the specification.

  15. #15
    Senior Member
    Join Date
    Jul 2014
    Posts
    3,150
    Quote Originally Posted by UhClem View Post
    Memory cards are not SDIO cards and the SDIO specification means nothing to them. You should be looking at version 6 (I just noticed this was out when grabbing the link) of the Physical Layer Simplified Specification, Part 1.. See table 4-21 in section 4.7.3.

    Memory cards have been required to support ACMD23 since version 1 of the specification.
    Is this thread not talking about T3.6 with SDIO interface to SD cards?
    But it you wanted to be right...

  16. #16
    Quote Originally Posted by WMXZ View Post
    Is this thread not talking about T3.6 with SDIO interface to SD cards?
    No. SD interface to SD(HC) memory cards.

  17. #17
    Senior Member
    Join Date
    Jul 2014
    Posts
    3,150
    Quote Originally Posted by UhClem View Post
    No. SD interface to SD(HC) memory cards.
    That is your reply that talks about MMC but not OP, which talks about T3.6 with SDIO (build in mode for SD card).

  18. #18
    Quote Originally Posted by WMXZ View Post
    That is your reply that talks about MMC but not OP, which talks about T3.6 with SDIO (build in mode for SD card).
    Go back and read it again. It said: "I wanted to share a sketch I wrote to acquire data and store it on an SD card. In my tests using a Teensy 3.6 and a SanDisk Ultra 16GB SDHC micro SD card, "

    And then read the SD specification:
    In addition to the SD Memory Card, there is the SD I/O (SDIO) Card. The SDIO Card specification is
    defined in a separate specification named: "SDIO Card Specification" that can be obtained from the SD
    Association. The SDIO Specification defines an SD card that may contain interfaces between various I/O
    units and an SD Host. The SDIO card may contain memory storage capability as well as its I/O
    functionality. The Memory portion of SDIO card shall be fully compatible to the given Physical Layer
    Specification. The SDIO card is based on and compatible with the SD Memory card.
    The Teensy 3.6 includes a Secure Digital Host Controller which implements SD mode communication with SD memory or I/O cards. The topic here is using it with SD memory cards. SD memory cards are required to implement ACMD23.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •