Best Teensy for a datalogger?

powerfly

Active member
Hi, I've been looking at Teensy's and am currently working on a 4.1 to use as a data-logger. I have however seen on this forum that some people recommend using a 3.6 instead. which Teensy is the best to use for a small, battery powered data-logger? The Teensy cannot use a laptop for this, currently I am recording data to a microSD card. my main concern is a high fidelity signal.

I'm hoping to be able to have at least 12 channels of 12 bit A2D with a 10 kHz sampling rate continuously. The more channels the better (so if It could do 25 channels that would be great). I may be able to significantly decrease the data-logging rate if that is an issue.

I'm guessing there may be a trade-off between the 3.6 or 3.5 having better quality data, but not being able to sample it as fast, but I'm not sure. Is there a sound theoretical calculation to figure out the maximum capabilities of these devices?

Alternatively, would you instead say a Teensy isn't the ideal tool, and that I should opt for a different microcontroller instead?

It would also be good if the setup can withstand mechanical shock, but I guess most boards would fare pretty similarly.

Any help would be much appreciated, so thanks in advance if you do :)
 
Use Teensy 4.1. The ability to add PSRAM will give you the needed buffer to temporarily hold collected data for those rare occasions when the SD card is busy for a "long" time. Teensy 3.6 doesn't have this, and it's also discontinued.

Aside from buffering data while the SD card is busy, the other important thing to do is write to the card in chunks of 4K or larger. If you do small writes, or if writes aren't a multiple of 512 bytes, SD card performance is much slower.
 
@Paul so you have a sample sketch with an implementation of such a buffer? Managing when to write to the SD and calculating data input size etc..
 
@Paul so you have a sample sketch with an implementation of such a buffer? Managing when to write to the SD and calculating data input size etc..

I recommend looking at the TeensySdioLogger example in the SdFat library. It uses a RingBuf template class that is part of the library. Unless you change the source in Ringbuf.h, the buffer is statically allocated, so it's in RAM1.

In my testing with Sandisk Ultra 32GB card, the time to write a 512-byte sector is never more than 6 us, and the longest "busy" time for the SD card is about 40 ms. The key to always having non-blocking writes to SD is to check file.isBusy() before writing. If you do that, and never write more than 512 bytes, you can avoid the situation of file.write() not returning immediately. I size my RingBuf to hold 50 ms of data (some cushion over 40), so if my data logging rate is 1 MB/s, the buffer size is 50KB (1MB/s * 0.050s = 50KB).

To minimize delays in the SD, it is important to use SdFat's preAllocate() function as shown in the TeensySdioLogger example. To the best of my knowledge, there is no speed advantage to writing in chunks larger than 512 bytes because the buffer in the SD card is only 512 bytes, so SdFat will break larger writes into 512-byte writes internally. With this method, I have tested logging at 1 MB/s for 10 minutes to create a 600 MB file, with only 1% of CPU time spent on SD logging. I have also tested shorter periods at up to 10 MB/s. I tried buffering in PSRAM and found that it works, but it has problems of its because it's so much slower than on-chip RAM.

The program further below is based on the TeensySdioLogger example, but with an IntervalTimer to simulate a logging interrupt. There is a lot of benchmarking logic, but it boils down to this simple diagram. The RingBuf class has functions for reading/writing the buffer from both ISR and task level.

Code:
   [ISR]---->[RingBuf]----->[loop]----->[SD]

Here is the program output for logging at 2 MB/s for 10 seconds.

Code:
Log for 10 seconds at 2.00 MB/s in 256 byte chunks
Pre-allocated file 20971520 bytes
RingBuf 104857 bytes
Start dataTimer (period = 122 us)
.........
Stop dataTimer
40960 writes in 9.995 s (0.192 s writing to file = 1.924 %)
File is full
fileSize     =   20971520 before sync()
rb.bytesUsed =          0 before sync()
fileSize     =   20971520 after sync()
rb.bytesUsed =          0 after sync()
rbMaxUsed    =      87552
min write us =       4.60
max write us =       6.26
min busy  us =         17
max busy  us =      41832
file.close()

Code:
// Test Teensy SDIO with write busy in a data logger demo.
//
// The driver writes to the uSDHC controller's FIFO then returns while the
// controller writes the data to the SD.  The first sector puts the controller
// in write mode and takes about 11 usec on a Teensy 4.1. About 5 usec is
// required to write a sector when the controller is in write mode.

#include <SdFat.h>
#include <RingBuf.h>

//******************************************************************************
// global variables
//******************************************************************************
#define SD_CONFIG	(SdioConfig(FIFO_SDIO))		// use Teensy SDIO
#define LOGGING_TIME_S	(10)				// s
#define DATA_RATE_BPS	((2)*(1024*1024))		// bytes/s
#define LOG_FILE_SIZE	(DATA_RATE_BPS * LOGGING_TIME_S)// total bytes
#define RING_BUF_SIZE	(DATA_RATE_BPS / 20)		// 50-ms buffer at BPS
#define BUFLEN		(256)				// bytes per write
#define C2US(c)		((c)*(1E6/F_CPU))		// CPU cycles to us

//******************************************************************************
// global variables
//******************************************************************************
IntervalTimer dataTimer;		// IntervalTimer for ISR-level writes
RingBuf<FsFile,RING_BUF_SIZE> rb;	// ISR --> RingBuf --> loop --> SD file
SdFs     sd;				// SdFat type
FsFile   file;				// SdFat file type
size_t   rbMaxUsed = 0;			// RingBuf max bytes (useful diagnostic)
char     buf[BUFLEN];			// test buffer
uint32_t error;				// RingBuf/file error code

//******************************************************************************
// IntervalTimer callback -- write BUFLEN bytes to RingBuf
//******************************************************************************
void dataTimerCallback( void )
{
#if (SD_FAT_VERSION == 20102)		// #if SdFat 2.1.1
  rb.memcpyIn( buf, BUFLEN );		//   write to RingBuf via rb.memcpyIn()
#elif (SD_FAT_VERSION >= 20202)		// #elif SdFat >= 2.2.0
  rb.beginISR();			//   begin interrupt access
  rb.write( buf, BUFLEN );		//   write to RingBuf via rb.write()
  rb.endISR();				//   end interrupt access
#endif					// #endif
  if (rb.getWriteError())		// if write error occurred
    error = 1;				//   set global error code
}  

//******************************************************************************
// setup()
//******************************************************************************
void setup()
{
  Serial.begin(9600);
  while (!Serial && millis() < 3000) {}

  Serial.printf( "%s  %s\n", __DATE__, __TIME__ ); 
  Serial.printf( "Teensy %s\n",
#if defined(ARDUINO_TEENSY35)
  "3.5" );
#elif defined(ARDUINO_TEENSY41)
  "4.1" );
#endif
  Serial.printf( "Teensyduino version %1lu\n", TEENSYDUINO );
  Serial.printf( "SdFat version %s\n", SD_FAT_VERSION_STR );

  // Initialize the SD.
  if (!sd.begin(SD_CONFIG)) {
    sd.initErrorHalt(&Serial);
  }
  
  // these 2 lines are necessary to enable cycle counting on T3.x
  #if (defined(KINETISL) || defined(KINETISK))		// if Teensy LC or 3.x
  ARM_DEMCR    |= ARM_DEMCR_TRCENA;			//   enable debug/trace
  ARM_DWT_CTRL |= ARM_DWT_CTRL_CYCCNTENA;		//   enable cycle counter
  #endif
}

//******************************************************************************
// loop()  open file, preAllocate, init RingBuf, log data, print results/stats
//******************************************************************************
void loop()
{ 
  while (Serial.available()) { Serial.read(); }
  Serial.println( "Type any character to begin" );
  while (!Serial.available()) {}
  
  Serial.printf( "Log for %1lu seconds at %1.2f MB/s in %1lu byte chunks\n",
		LOGGING_TIME_S, (float)(DATA_RATE_BPS/(1024*1024.0)), BUFLEN );
  Serial.printf( "Pre-allocated file %1lu bytes\n", LOG_FILE_SIZE );
  Serial.printf( "RingBuf %1lu bytes\n", RING_BUF_SIZE );

  // Open or create file - truncate existing file.
  if (!file.open( "logfile.txt", O_RDWR | O_CREAT | O_TRUNC )) {
    Serial.println( "open failed\n" );
  }
  // File must be pre-allocated to avoid huge delays searching for free clusters.
  else if (!file.preAllocate( LOG_FILE_SIZE )) {
     Serial.println( "preAllocate failed\n" );
     file.close();
  }
  // init the file and RingBuf
  else {
    rb.begin(&file);
  }
  
  // Init data buffer with random data
  randomSeed( micros() ); 
  for (int i=0; i<BUFLEN; i++)
    buf[i] = 0x30+random( 10 );
  buf[BUFLEN-1] = '\n';
  
  uint32_t timer_period_us = 1E6 * BUFLEN / DATA_RATE_BPS;
  Serial.printf( "Start dataTimer (period = %1lu us)\n", timer_period_us );
  dataTimer.begin( dataTimerCallback, timer_period_us );

  uint32_t count = 0, start_ms = millis(), start_busy = 0;
  bool busy = false;
  error = 0;
  elapsedMillis ms = 0;
  uint32_t sum_busy=0, min_busy=0xFFFFFFFF, max_busy=0;
  uint32_t sum_write=0, min_write=0xFFFFFFFF, max_write=0;
  while (error == 0 && millis() - start_ms < LOGGING_TIME_S*1000) {
	  
    if (ms >= 1000) { Serial.print( "." ); ms -= 1000; }
    
    // number of bytes in RingBuf
    size_t n = rb.bytesUsed(); 
    if (n > rbMaxUsed) {
      rbMaxUsed = n;
    }

    // bytes in RingBuf now will fit, but any more will exceed file size
    if ((n + file.curPosition()) > (LOG_FILE_SIZE - BUFLEN/*_MAX*/)) {
      error = 2; // file full
    }

    // write one sector (512 bytes) from RingBuf to file
    // Not busy only allows one sector before possible busy wait
    if (file.isBusy()) {
      if (!busy) {
        busy = true;
        start_busy = ARM_DWT_CYCCNT;
      }
    }
    else if (busy) {
      busy = false;
      uint32_t busy_cyc = ARM_DWT_CYCCNT - start_busy;
      sum_busy += busy_cyc;
      if (busy_cyc < min_busy) min_busy = busy_cyc;
      if (busy_cyc > max_busy) max_busy = busy_cyc;
    }
    if (n >= 512 && !busy) {
      uint32_t start_write = ARM_DWT_CYCCNT;
      if (512 != rb.writeOut(512)) {
        error = 1; // write error
      }
      uint32_t write_cyc = ARM_DWT_CYCCNT - start_write;
      sum_write += write_cyc;
      if (count > 0) {
        if (write_cyc < min_write) min_write = write_cyc;
        if (write_cyc > max_write) max_write = write_cyc;
      }
      count++;
    }
  }
  
  uint32_t duration_ms = millis() - start_ms;
  Serial.printf( "\nStop dataTimer\n" );
  dataTimer.end();
  
  double duration_s = duration_ms/1000.0;
  double write_s = C2US(sum_write)/1E6;
  double write_percent = 100*(write_s/duration_s);
  Serial.printf( "%1lu writes in %1.3lf s (%1.3lf s writing to file = %1.3lf %c)\n",
		count, duration_s, write_s, write_percent, '%' );
  switch (error) {
    case 0:   Serial.printf( "No error\n" );			break;
    case 1:   Serial.printf( "Not enough space in RingBuf\n" );	break;
    case 2:   Serial.printf( "File is full\n" );		break;
    case 3:   Serial.printf( "Write from RingBuf failed" );	break;
    default:  Serial.printf( "Undefined error %1lu\n", error );	break;
  }

  // write any remaining RingBuf data to file
  Serial.printf( "fileSize     = %10lu before sync()\n", (uint32_t)file.fileSize() );
  Serial.printf( "rb.bytesUsed = %10lu before sync()\n", (uint32_t)rb.bytesUsed() );
  rb.sync();
  
  // file and buffer stats
  Serial.printf( "fileSize     = %10lu after sync()\n", (uint32_t)file.fileSize() );
  Serial.printf( "rb.bytesUsed = %10lu after sync()\n", (uint32_t)rb.bytesUsed() );
  Serial.printf( "rbMaxUsed    = %10lu\n", (uint32_t)rbMaxUsed );
  Serial.printf( "min write us = %10.2lf\n", C2US(min_write) );
  Serial.printf( "max write us = %10.2lf\n", C2US(max_write) );
  Serial.printf( "min busy  us = %10.0lf\n", C2US(min_busy) );
  Serial.printf( "max busy  us = %10.0lf\n", C2US(max_busy) );
  
  // print first N line(s) of file.
  int lines=0;
  if (lines > 0) {
    Serial.printf( "First %1d line(s) of file\n", lines );
    file.truncate();
    file.rewind();
  }
  for (int n=0; n < lines && file.available();) {
    int c = file.read();
    if (c < 0) break;
    Serial.write(c);
    if (c == '\n') n++;
  }
 
  // close file
  file.close();
  Serial.printf( "file.close()\n\n" );
}
 
I recommend looking at the TeensySdioLogger example in the SdFat library. ...

That code worked here:
Code:
Teensy 4.1
Teensyduino version 159
SdFat version 2.1.2
Type any character to begin
Log for 10 seconds at 2.00 MB/s in 256 byte chunks
Pre-allocated file 20971520 bytes
RingBuf 104857 bytes
Start dataTimer (period = 122 us)
.........
Stop dataTimer
40843 writes in 10.000 s (0.193 s writing to file = 1.932 %)
No error
fileSize     =   20911616 before sync()
rb.bytesUsed =        153 before sync()
fileSize     =   20911769 after sync()
rb.bytesUsed =          0 after sync()
rbMaxUsed    =     104857
min write us =       4.62
max write us =       6.39
min busy  us =         17
max busy  us =      53708
file.close()
 
Code:
Pre-allocated file 20971520 bytes
RingBuf 104857 bytes
fileSize     =   20911769 after sync()
rbMaxUsed    =     104857
max busy  us =      53708

Interesting that it says your RingBuf was full at some point, but there was no error, and your final fileSize is slightly less than what was preAllocated. Your max busy time is more than 50 ms, so could you try increasing the RingBuf size a little bit via the RING_BUF_SIZE macro? You could just double it to be sure it won't become full.
 
I tried some different ways of allocating memory for SdFat's RingBuf.

1) Modify code to define large buffer outside and class and pass its address via begin(). First used malloc() to create buffer on static, with results virtually the same as for statically allocated buffer in RAM1.

2) Define large buffer in EXTMEM and pass to RingBuf via begin(). This results in about 2x of the CPU time for writing a 20MB file in 10 seconds.

3) Define RingBuf as EXTMEM, so large buffer and all other class members are in PSRAM. This results in CPU time about 5x compared to RAM.

So, if you need a buffer larger than will fit in RAM, you can buffer in PSRAM, but with a speed penalty.
 
I tried some different ways of allocating memory for SdFat's RingBuf.

1) Modify code to define large buffer outside and class and pass its address via begin(). First used malloc() to create buffer on static, with results virtually the same as for statically allocated buffer in RAM1.

2) Define large buffer in EXTMEM and pass to RingBuf via begin(). This results in about 2x of the CPU time for writing a 20MB file in 10 seconds.

3) Define RingBuf as EXTMEM, so large buffer and all other class members are in PSRAM. This results in CPU time about 5x compared to RAM.

So, if you need a buffer larger than will fit in RAM, you can buffer in PSRAM, but with a speed penalty.

how much noise did you find that you got? were working with a Teensy 4.1 atm but finding that we get quite a lot of noise even with a clean input signal (about 100 divisions at it worst on the 12 bit data). I've tried putting the data logger in an alu box (to act as a faraday cage), and connecting a 10k resistor between signal and GND to help but nothing has worked so far.

Use Teensy 4.1. The ability to add PSRAM will give you the needed buffer to temporarily hold collected data for those rare occasions when the SD card is busy for a "long" time. Teensy 3.6 doesn't have this, and it's also discontinued.

Aside from buffering data while the SD card is busy, the other important thing to do is write to the card in chunks of 4K or larger. If you do small writes, or if writes aren't a multiple of 512 bytes, SD card performance is much slower.

I've found a 4.1 gives rise to noise like this even when it's supposed to be flat (with a clean input signal and basically no ambient noise):

Skärmbild 2023-10-19 162504.png

connecting a 10k resistor between GND and the signal didn't help either. at it's worst, this is about 100 divisions (for 12 bit data). is there a way to improve this?
 
how much noise did you find that you got? were working with a Teensy 4.1 atm but finding that we get quite a lot of noise even with a clean input signal (about 100 divisions at it worst on the 12 bit data). I've tried putting the data logger in an alu box (to act as a faraday cage), and connecting a 10k resistor between signal and GND to help but nothing has worked so far.

The tests I wrote about were just of the SD logging with arbitrary "data". If you have a question about ADC, please post a short sketch that shows the issue.
 
Back
Top