Long time log on SD card

pd0lew · Jun 19, 2024

Dear all,

I use a Teensy 4.1 and a SD card 32Gb for data logging I log about 29 bytes/min (15mB year) this must be run for two years for a long time test project.

My question how reliable is a SD card because I have many write operation during the two years is that still a good idea?

Please let me know your thoughts....

thanks,
Johan

defragster · Jun 19, 2024

Not seen any long term 2 year runs to an SD card noted?

Given it is T_4.1: One idea would be a Bottom soldered QSPI FLASH chip - 64+MB would hold 2 years of logging and could be sent to SD card daily or weekly for backup and retrieval that would cut down on the SD card abuse and also make sure that at least one copy survived.

This shows 64 MB and 128 MB from ...\GitHub\LittleFS\src\LittleFS.cpp

Code:

{{0xEF, 0x40, 0x20}, 32, 256, 65536, 0xDC, 67108864, 3500, 2000000, "W25Q512JV*Q"}, // Winbond W25Q512JV*Q
{{0xEF, 0x40, 0x21}, 32, 256, 65536, 0xDC, 134217728, 3500, 2000000, "W25Q01JV*Q"},// Winbond W25Q01JV*Q
...
{{0xEF, 0x70, 0x20}, 32, 256, 65536, 0xDC, 67108864, 3500, 2000000, "W25Q512JV*M (DTR)"}, // Winbond W25Q512JV*M (DTR)

And 32MB, but that might cut it close at 2*15MB, and cause more wear from LittleFS storage - especially at the end of year 2.

And these larger NAND's (128 and 256 MB) are also supported from ...\GitHub\LittleFS\src\LittleFS_NAND.cpp:

Code:

    {{0xEF, 0xAA, 0x21}, 24, 2048, 131072, 0, 131596288, 2000, 15000, "W25N01GVZEIG"},  //Winbond W25N01G
...
    {{0xEF, 0xAA, 0x22}, 24, 2048, 131072, 0, 265289728, 2000, 15000, "W25N02KVZEIR"},  //Winbond W25N02G
    {{0xEF, 0xBB, 0x21}, 24, 2048, 131072, 0, 265289728, 2000, 15000, "W25M02"},  //Winbond W25M02

Angelo · Jun 20, 2024

One source of problem would be SDcard socket. The contacts could/would oxydize.

The scenario proposed by defragster is very good.
You could also use flash connected on normal SPI or I2C. Using 2 parts, each on its own SPI or I2C, it would give you some sort of RAID storage

And if you implement regular/hourly/daily backups on the SD, it would be very good.

You could use the flash whithout file system, to avoid wear.

Angelo

pd0lew · Jun 20, 2024

Thank you all, pretty good idea.

A lot of EEproms are not available this moment I like to use this one W25N01GVZEIG (Mouser nr. 454-W25Q512JVEIQ) will this go to work with a Teensy 4.1 ?

Best,
Johan

pd0lew · Jun 21, 2024

Perhaps this is a better approach to use a USB stich on the Teensy 4.1 USB host.... are there any example programs for testing. I looked a bit arround but not find a lot of info. I like to log on the USB stick as example temperature and pressure.

Please help,
Best,
Johan

wwatson · Jun 21, 2024

pd0lew said:
Perhaps this is a better approach to use a USB stich on the Teensy 4.1 USB host.... are there any example programs for testing. I looked a bit arround but not find a lot of info. I like to log on the USB stick as example temperature and pressure.

Please help,
Best,
Johan

Here is a simple example of logging data to a USB thumb (stick) drive:

Code:

/*
  MSC USB Drive datalogger
 
 This example shows how to log data from three analog sensors
 to an MSC USB drive using the mscFS library.
 
 created  24 Nov 2010
 modified 9 Apr 2012
 by Tom Igoe
 modified 17 Nov 2020
 by Warren Watson
 
 This example code is in the public domain.
 */

#include <USBHost_t36.h>

// Setup USBHost_t36 and as many HUB ports as needed.
USBHost myusb;
USBHub hub1(myusb);
USBHub hub2(myusb);
USBHub hub3(myusb);
USBHub hub4(myusb);

// Setup MSC for the number of USB Drives you are using. (Two for this example)
// Mutiple  USB drives can be used. Hot plugging is supported. There is a slight
// delay after a USB MSC device is plugged in. This is waiting for initialization
// but after it is initialized ther should be no delay.
USBDrive myDrive(myusb);
USBFilesystem firstPartition(myusb);

int record_count = 0;
bool write_data = false;
File dataFile;

void setup()
{

  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect.
  }

  // Start USBHost_t36, HUB(s) and USB devices.
  myusb.begin();

  Serial.print("\nInitializing USB MSC drive...");

  while (!firstPartition) {
    myusb.Task();
  }
  Serial.println("USB drive initialized.");

  menu();
}

void loop()
{
  if ( Serial.available() ) {
    char rr;
    rr = Serial.read();
    switch (rr) {
      case 'l': listFiles(); break;
      case 'e': eraseFile(); break;
      case 's':
        {
          Serial.println("\nLogging Data!!!");
          write_data = true;   // sets flag to continue to write data until new command is received
          // opens a file or creates a file if not present,  FILE_WRITE will append data to
          // to the file created.
          dataFile = firstPartition.open("datalog.txt", FILE_WRITE);
          logData();
        }
        break;
      case 'x': stopLogging(); break;
      case 'd': dumpLog(); break;
      case '\r':
      case '\n':
      case 'h': menu(); break;
    }
    while (Serial.read() != -1) ; // remove rest of characters.
  }

  if(write_data) logData();
}

void logData()
{
    // make a string for assembling the data to log:
    String dataString = "";
 
    // read three sensors and append to the string:
    for (int analogPin = 0; analogPin < 3; analogPin++) {
      int sensor = analogRead(analogPin);
      dataString += String(sensor);
      if (analogPin < 2) {
        dataString += ",";
      }
    }
 
    // if the file is available, write to it:
    if (dataFile) {
      dataFile.println(dataString);
      // print to the serial port too:
      Serial.println(dataString);
      record_count += 1;
    } else {
      // if the file isn't open, pop up an error:
      Serial.println("error opening datalog.txt");
    }
    delay(100); // run at a reasonable not-too-fast speed for testing
}

void stopLogging()
{
  Serial.println("\nStopped Logging Data!!!");
  write_data = false;
  // Closes the data file.
  dataFile.close();
  Serial.printf("Records written = %d\n", record_count);
}


void dumpLog()
{
  Serial.println("\nDumping Log!!!");
  // open the file.
  dataFile = firstPartition.open("datalog.txt");

  // if the file is available, write to it:
  if (dataFile) {
    while (dataFile.available()) {
      Serial.write(dataFile.read());
    }
    dataFile.close();
  } 
  // if the file isn't open, pop up an error:
  else {
    Serial.println("error opening datalog.txt");
  }
}

void menu()
{
  Serial.println();
  Serial.println("Menu Options:");
  Serial.println("\tl - List files on disk");
  Serial.println("\te - Erase log file on disk");
  Serial.println("\ts - Start Logging data (Restarting logger will append records to existing log)");
  Serial.println("\tx - Stop Logging data");
  Serial.println("\td - Dump Log");
  Serial.println("\th - Menu");
  Serial.println();
}

void listFiles()
{
  Serial.print("\n Space Used = ");
  Serial.println(firstPartition.usedSize());
  Serial.print("Filesystem Size = ");
  Serial.println(firstPartition.totalSize());

  printDirectory(firstPartition);
}

void eraseFile() {
  Serial.println("\nErasing datalog.txt");
  firstPartition.remove("datalog.txt");  // Erase datalog.txt
  Serial.println("\ndatalog.txt erased !");
}

void printDirectory(FS &fs) {
  Serial.println("Directory\n---------");
  printDirectory(fs.open("/"), 0);
  Serial.println();
}

void printDirectory(File dir, int numSpaces) {
   while(true) {
     File entry = dir.openNextFile();
     if (! entry) {
       //Serial.println("** no more files **");
       break;
     }
     printSpaces(numSpaces);
     Serial.print(entry.name());
     if (entry.isDirectory()) {
       Serial.println("/");
       printDirectory(entry, numSpaces+2);
     } else {
       // files have sizes, directories do not
       printSpaces(36 - numSpaces - strlen(entry.name()));
       Serial.print("  ");
       Serial.print(entry.size(), DEC);
       DateTimeFields datetime;
       if (entry.getModifyTime(datetime)) {
         printSpaces(4);
         printTime(datetime);
       }
       Serial.println();
     }
     entry.close();
   }
}

void printSpaces(int num) {
  for (int i=0; i < num; i++) {
    Serial.print(" ");
  }
}

void printTime(const DateTimeFields tm) {
  const char *months[12] = {
    "January","February","March","April","May","June",
    "July","August","September","October","November","December"
  };
  if (tm.hour < 10) Serial.print('0');
  Serial.print(tm.hour);
  Serial.print(':');
  if (tm.min < 10) Serial.print('0');
  Serial.print(tm.min);
  Serial.print("  ");
  Serial.print(months[tm.mon]);
  Serial.print(" ");
  Serial.print(tm.mday);
  Serial.print(", ");
  Serial.print(tm.year + 1900);
}

It presents a little menu:

Code:

Menu Options:
        l - List files on disk
        e - Erase log file on disk
        s - Start Logging data (Restarting logger will append records to existing log)
        x - Stop Logging data
        d - Dump Log
        h - Menu

that shows a few options that are available. Search the forum for "datalogger" to find more examples.

mborgerson · Jun 21, 2024

Angelo said:
One source of problem would be SDcard socket. The contacts could/would oxydize.

Angelo

I don't recall any corrosion problems with SD cards in oceanographic loggers in about 100 logger years of deployment over 50 or more loggers. We did take precautions:

1. The loggers are in watertight cases (Well DOH! they're deployed 10 to 400 meters below the sea surface).
2. The SD cards have gold-plated contacts. I suspect the contact points on the card socket are also gold plated.
3. We started with new cards and only used them once in the logger. This eliminated contact wear that might affect the gold plating.
4. Before sealing the logger we added several small dessicant packets in empty spaces in the logger. We used the type of packets that are used when shipping and storing semiconductors.
5. The logger case was flushed with dry nitrogen before it was closed up.
6. We used circuit boards that were well cleaned and dried before assembling the loggers.

We took those precautions not only because the loggers and their sensors cost $5-10K to build and calibrate, but because an oceanographic research cruise to deploy the loggers is expensive. A regional-class research vessel has an operating cost of about $31,000/per day. A month-long cruise to the equator and back is going to cost about a million bucks! Add in the salaries of the scientists and engineers not part of the ship's crew, and you can understand why oceanographers try to have multiple research groups working on the same cruise.

pd0lew · Jun 22, 2024

wwatson said:

Here is a simple example of logging data to a USB thumb (stick) drive:

Code:

/*
  MSC USB Drive datalogger
 
 This example shows how to log data from three analog sensors
 to an MSC USB drive using the mscFS library.
 
 created  24 Nov 2010
 modified 9 Apr 2012
 by Tom Igoe
 modified 17 Nov 2020
 by Warren Watson
 
 This example code is in the public domain.
 */

#include <USBHost_t36.h>

// Setup USBHost_t36 and as many HUB ports as needed.
USBHost myusb;
USBHub hub1(myusb);
USBHub hub2(myusb);
USBHub hub3(myusb);
USBHub hub4(myusb);

// Setup MSC for the number of USB Drives you are using. (Two for this example)
// Mutiple  USB drives can be used. Hot plugging is supported. There is a slight
// delay after a USB MSC device is plugged in. This is waiting for initialization
// but after it is initialized ther should be no delay.
USBDrive myDrive(myusb);
USBFilesystem firstPartition(myusb);

int record_count = 0;
bool write_data = false;
File dataFile;

void setup()
{

  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect.
  }

  // Start USBHost_t36, HUB(s) and USB devices.
  myusb.begin();

  Serial.print("\nInitializing USB MSC drive...");

  while (!firstPartition) {
    myusb.Task();
  }
  Serial.println("USB drive initialized.");

  menu();
}

void loop()
{
  if ( Serial.available() ) {
    char rr;
    rr = Serial.read();
    switch (rr) {
      case 'l': listFiles(); break;
      case 'e': eraseFile(); break;
      case 's':
        {
          Serial.println("\nLogging Data!!!");
          write_data = true;   // sets flag to continue to write data until new command is received
          // opens a file or creates a file if not present,  FILE_WRITE will append data to
          // to the file created.
          dataFile = firstPartition.open("datalog.txt", FILE_WRITE);
          logData();
        }
        break;
      case 'x': stopLogging(); break;
      case 'd': dumpLog(); break;
      case '\r':
      case '\n':
      case 'h': menu(); break;
    }
    while (Serial.read() != -1) ; // remove rest of characters.
  }

  if(write_data) logData();
}

void logData()
{
    // make a string for assembling the data to log:
    String dataString = "";
 
    // read three sensors and append to the string:
    for (int analogPin = 0; analogPin < 3; analogPin++) {
      int sensor = analogRead(analogPin);
      dataString += String(sensor);
      if (analogPin < 2) {
        dataString += ",";
      }
    }
 
    // if the file is available, write to it:
    if (dataFile) {
      dataFile.println(dataString);
      // print to the serial port too:
      Serial.println(dataString);
      record_count += 1;
    } else {
      // if the file isn't open, pop up an error:
      Serial.println("error opening datalog.txt");
    }
    delay(100); // run at a reasonable not-too-fast speed for testing
}

void stopLogging()
{
  Serial.println("\nStopped Logging Data!!!");
  write_data = false;
  // Closes the data file.
  dataFile.close();
  Serial.printf("Records written = %d\n", record_count);
}


void dumpLog()
{
  Serial.println("\nDumping Log!!!");
  // open the file.
  dataFile = firstPartition.open("datalog.txt");

  // if the file is available, write to it:
  if (dataFile) {
    while (dataFile.available()) {
      Serial.write(dataFile.read());
    }
    dataFile.close();
  }
  // if the file isn't open, pop up an error:
  else {
    Serial.println("error opening datalog.txt");
  }
}

void menu()
{
  Serial.println();
  Serial.println("Menu Options:");
  Serial.println("\tl - List files on disk");
  Serial.println("\te - Erase log file on disk");
  Serial.println("\ts - Start Logging data (Restarting logger will append records to existing log)");
  Serial.println("\tx - Stop Logging data");
  Serial.println("\td - Dump Log");
  Serial.println("\th - Menu");
  Serial.println();
}

void listFiles()
{
  Serial.print("\n Space Used = ");
  Serial.println(firstPartition.usedSize());
  Serial.print("Filesystem Size = ");
  Serial.println(firstPartition.totalSize());

  printDirectory(firstPartition);
}

void eraseFile() {
  Serial.println("\nErasing datalog.txt");
  firstPartition.remove("datalog.txt");  // Erase datalog.txt
  Serial.println("\ndatalog.txt erased !");
}

void printDirectory(FS &fs) {
  Serial.println("Directory\n---------");
  printDirectory(fs.open("/"), 0);
  Serial.println();
}

void printDirectory(File dir, int numSpaces) {
   while(true) {
     File entry = dir.openNextFile();
     if (! entry) {
       //Serial.println("** no more files **");
       break;
     }
     printSpaces(numSpaces);
     Serial.print(entry.name());
     if (entry.isDirectory()) {
       Serial.println("/");
       printDirectory(entry, numSpaces+2);
     } else {
       // files have sizes, directories do not
       printSpaces(36 - numSpaces - strlen(entry.name()));
       Serial.print("  ");
       Serial.print(entry.size(), DEC);
       DateTimeFields datetime;
       if (entry.getModifyTime(datetime)) {
         printSpaces(4);
         printTime(datetime);
       }
       Serial.println();
     }
     entry.close();
   }
}

void printSpaces(int num) {
  for (int i=0; i < num; i++) {
    Serial.print(" ");
  }
}

void printTime(const DateTimeFields tm) {
  const char *months[12] = {
    "January","February","March","April","May","June",
    "July","August","September","October","November","December"
  };
  if (tm.hour < 10) Serial.print('0');
  Serial.print(tm.hour);
  Serial.print(':');
  if (tm.min < 10) Serial.print('0');
  Serial.print(tm.min);
  Serial.print("  ");
  Serial.print(months[tm.mon]);
  Serial.print(" ");
  Serial.print(tm.mday);
  Serial.print(", ");
  Serial.print(tm.year + 1900);
}

It presents a little menu:

Code:

Menu Options:
        l - List files on disk
        e - Erase log file on disk
        s - Start Logging data (Restarting logger will append records to existing log)
        x - Stop Logging data
        d - Dump Log
        h - Menu

that shows a few options that are available. Search the forum for "datalogger" to find more examples.

Works out of the box good example. I can use this now with some modifications in my project! Tested with a 16Gb and Teensy 4.1 logged about
131271 records without any issues. Thanks for helping! best regards Johan

wwatson · Jun 22, 2024

pd0lew said:
Works out of the box good example. I can use this now with some modifications in my project! Tested with a 16Gb and Teensy 4.1 logged about
131271 records without any issues. Thanks for helping! best regards Johan

Your welcome, Good Luck...

pd0lew · Jun 23, 2024

Dear all,

I modified the code as example for those who need a decent USB stick logger. The time, date and a pressure value is with headers in the .csv file.
Time is taken when your project is loaded and when a battery is installed your RTC keeps the time for the logging index.

Some message added and a start / stop button.

Tested the logger with more than 500000 records written,

Have fun playing with huge data collection.

Best,
Johan

Code:

/*
  MSC USB Drive datalogger
 
 This example shows how to log data from three analog sensors
 to an MSC USB drive using the mscFS library.
 
 created  24 Nov 2010
 modified 9 Apr 2012
 by Tom Igoe
 modified 17 Nov 2020
 by Warren Watson
 modified 23 June 2024
 by J.G. Holstein
 This example code is in the public domain.
 */



#include <USBHost_t36.h>
#include <TimeLib.h> 
#include <SSD1322_for_Adafruit_GFX.h>
#include <U8g2lib.h>
#include <SPI.h>
#include "Profont29.h"
#include "Profont22.h"
#include "Profont17.h"
#include "Profont13.h"
#include "Profont15.h"
#include "Profont11.h"

#define OLED_CLK 13
#define OLED_MOSI 11
#define OLED_CS 10
#define OLED_DC 9
#define OLED_RESET 255
Adafruit_SSD1322 lcd(256, 64, &SPI, OLED_DC, OLED_RESET, OLED_CS);

USBHost myusb;
USBHub hub1(myusb);
USBHub hub2(myusb);
USBHub hub3(myusb);
USBHub hub4(myusb);

USBDrive myDrive(myusb);
USBFilesystem firstPartition(myusb);

const int toggleSwitchPin = 1; // start stop datalog
bool loggingActive = false;
int switchState = LOW;
int lastSwitchState = LOW;
unsigned long lastDebounceTime = 0;
unsigned long debounceDelay = 200;
long record_count = 0;
bool write_data = false;
File dataFile;
bool usbInitialized = false;

time_t getTeensy3Time() {
  return Teensy3Clock.get();
}

void setup() {
  lcd.begin();
  lcd.clearDisplay();
  lcd.display();
  lcd.setRotation(0);
  lcd.setContrast(155);
  Serial.begin(115200);
 
  setSyncProvider(getTeensy3Time);    // synchronise time and date after uploading firmware
 
  lcd.setFont(&Profont17);
  lcd.setCursor(2, 15);
  lcd.print("USB drive data logging");
  lcd.setCursor(2, 40);
  lcd.print("Push start for logging");
  lcd.display();
  //setTime(16, 42, 00, 12, 8, 2022); // use this for GUI Pascal to set time and date if needed
  pinMode(toggleSwitchPin, INPUT_PULLUP);
 
  myusb.begin();

  Serial.print("\nInitializing USB MSC drive...");

  while (!firstPartition) {
    myusb.Task();
  }
  Serial.println("USB drive initialized.");
  usbInitialized = true;
 
  menu();
}

void loop() {
  int reading = digitalRead(toggleSwitchPin);
 
  if (!firstPartition) {
    usbInitialized = false;
    Serial.println("USB drive removed.");
    lcd.setCursor(40, 40);
    lcd.print("USB drive removed.");
    lcd.display();
    lcd.clearDisplay();
  }

  if (!usbInitialized) {
    Serial.print("\nReinitializing USB MSC drive...");
    while (!firstPartition) {
      myusb.Task();
    }
    Serial.println("USB drive reinitialized.");
    usbInitialized = true;
     lcd.setCursor(2, 15);
    lcd.print("Push start for logging");
    lcd.setCursor(35, 40);
    lcd.print("USB drive inserted");
    lcd.setCursor(55, 63);
    lcd.print("USB drive ready");
    lcd.display();
    lcd.clearDisplay();
  }

   if (reading != lastSwitchState) {
    lastDebounceTime = millis();
    
  }

  if ((millis() - lastDebounceTime) > debounceDelay) {
    if (reading != switchState) {
      switchState = reading;
      if (switchState == LOW) {
        Serial.println(switchState);
        loggingActive = !loggingActive;
        if (loggingActive) {
          startLogging();
        } else {
          stopLogging();
        }
      }
    }
  }

  lastSwitchState = reading;

  if (write_data) logData();

  if (Serial.available()) {
    char rr;
    rr = Serial.read();
    
    switch (rr) {
      case 'l': listFiles(); break;
      case 'e': eraseFile(); break;
      case 'd': dumpLog(); break;
      case '\r':
      case '\n':
      case 'h': menu(); break;
    }
    while (Serial.read() != -1);
  }
}

void startLogging() {
  Serial.println("\nLogging Data!!!");
  write_data = true;   
  dataFile = firstPartition.open("datalog.csv", FILE_WRITE);
  if (record_count == 0) {
    dataFile.println("Date,Time,Pressure");
  }
}

void stopLogging() {
  Serial.println("\nStopped Logging Data!!!");
  write_data = false;
  dataFile.close();
  lcd.setCursor(40, 15);
  lcd.print("Logging stopped. ");
  lcd.setCursor(2, 40);
  lcd.print("USB drive, ");
  lcd.setCursor(2, 62);
  lcd.print("can be removed safely.");
  lcd.display();
  lcd.clearDisplay();
  Serial.printf("Records written = %d\n", record_count);
}

void logData() {
  lcd.setCursor(2, 16);
  lcd.print("USB drive data logging,");
  lcd.setCursor(0, 40);
  lcd.print("in progress! ");
  lcd.setCursor(0, 63);
  lcd.print("Records written= ");
  lcd.print(record_count);
  lcd.display();
  lcd.clearDisplay();
  String dataString = "";
 
  char dateBuffer[11];  // YYYY-MM-DD
  snprintf(dateBuffer, sizeof(dateBuffer), "%04d-%02d-%02d", year(), month(), day());

  char timeBuffer[9];  // HH:MM:SS
  snprintf(timeBuffer, sizeof(timeBuffer), "%02d:%02d:%02d", hour(), minute(), second());
  // test with A0 as sensor
  int sensor = analogRead(A0); 
  dataString += dateBuffer;
  dataString += ",";
  dataString += timeBuffer;
  dataString += ",";
  dataString += String(sensor);

  if (dataFile) {
    dataFile.println(dataString);
    Serial.println(dataString);
    record_count += 1;
  } else {
      Serial.println("error opening datalog.csv");
  }
  delay(100); // log interval test 100ms
}

void dumpLog() {
  Serial.println("\nDumping Log!!!");
  // open the file.
  dataFile = firstPartition.open("datalog.csv");

  if (dataFile) {
    while (dataFile.available()) {
      Serial.write(dataFile.read());
    }
    dataFile.close();
  }
 
  else {
    Serial.println("error opening datalog.csv");
  }
}

void menu() {
  Serial.println();
  Serial.println("Menu Options:");
  Serial.println("\tl - List files on disk");
  Serial.println("\te - Erase log file on disk");
  Serial.println("\td - Dump Log");
  Serial.println("\th - Menu");
  Serial.println();
}

void listFiles() {
  Serial.print("\n Space Used = ");
  Serial.println(firstPartition.usedSize());
  Serial.print("Filesystem Size = ");
  Serial.println(firstPartition.totalSize());
  printDirectory(firstPartition);
}

void eraseFile() {
  Serial.println("\nErasing datalog.csv");
  firstPartition.remove("datalog.csv"); 
  Serial.println("\ndatalog.csv erased !");
}

void printDirectory(FS &fs) {
  Serial.println("Directory\n---------");
  printDirectory(fs.open("/"), 0);
  Serial.println();
}

void printDirectory(File dir, int numSpaces) {
  while (true) {
    File entry = dir.openNextFile();
    if (!entry) {
      break;
    }
    printSpaces(numSpaces);
    Serial.print(entry.name());
    if (entry.isDirectory()) {
      Serial.println("/");
      printDirectory(entry, numSpaces + 2);
    } else {
      printSpaces(36 - numSpaces - strlen(entry.name()));
      Serial.print("  ");
      Serial.print(entry.size(), DEC);
      DateTimeFields datetime;
      if (entry.getModifyTime(datetime)) {
        printSpaces(4);
        printTime(datetime);
      }
      Serial.println();
    }
    entry.close();
  }
}

void printSpaces(int num) {
  for (int i = 0; i < num; i++) {
    Serial.print(" ");
  }
}

void printTime(const DateTimeFields tm) {
  const char *months[12] = {
    "January", "February", "March", "April", "May", "June",
    "July", "August", "September", "October", "November", "December"
  };
  if (tm.hour < 10) Serial.print('0');
  Serial.print(tm.hour);
  Serial.print(':');
  if (tm.min < 10) Serial.print('0');
  Serial.print(tm.min);
  Serial.print("  ");
  Serial.print(months[tm.mon]);
  Serial.print(" ");
  Serial.print(tm.mday);
  Serial.print(", ");
  Serial.print(tm.year + 1900);
}

pd0lew · Jun 23, 2024

There is a important NOTE: please set in Tab Tools, MTP Disk (Experimental).

pd0lew · Jun 23, 2024

jmarsh · Jun 23, 2024

pd0lew said:
There is a important NOTE: please set in Tab Tools, MTP Disk (Experimental).

Why? I can't see any reference to MTP in the code.

mborgerson · Jun 23, 2024

pd0lew said:

Dear all,

I modified the code as example for those who need a decent USB stick logger. The time, date and a pressure value is with headers in the .csv file.
Time is taken when your project is loaded and when a battery is installed your RTC keeps the time for the logging index.

Some message added and a start / stop button.

Tested the logger with more than 500000 records written,

Have fun playing with huge data collection.

Best,
Johan

Code:

/*
  MSC USB Drive datalogger
 
 This example shows how to log data from three analog sensors
 to an MSC USB drive using the mscFS library.
 
 created  24 Nov 2010
 modified 9 Apr 2012
 by Tom Igoe
 modified 17 Nov 2020
 by Warren Watson
 modified 23 June 2024
 by J.G. Holstein
 This example code is in the public domain.
 */



#include <USBHost_t36.h>
#include <TimeLib.h>
#include <SSD1322_for_Adafruit_GFX.h>
#include <U8g2lib.h>
#include <SPI.h>
#include "Profont29.h"
#include "Profont22.h"
#include "Profont17.h"
#include "Profont13.h"
#include "Profont15.h"
#include "Profont11.h"

#define OLED_CLK 13
#define OLED_MOSI 11
#define OLED_CS 10
#define OLED_DC 9
#define OLED_RESET 255
Adafruit_SSD1322 lcd(256, 64, &SPI, OLED_DC, OLED_RESET, OLED_CS);

USBHost myusb;
USBHub hub1(myusb);
USBHub hub2(myusb);
USBHub hub3(myusb);
USBHub hub4(myusb);

USBDrive myDrive(myusb);
USBFilesystem firstPartition(myusb);

const int toggleSwitchPin = 1; // start stop datalog
bool loggingActive = false;
int switchState = LOW;
int lastSwitchState = LOW;
unsigned long lastDebounceTime = 0;
unsigned long debounceDelay = 200;
long record_count = 0;
bool write_data = false;
File dataFile;
bool usbInitialized = false;

time_t getTeensy3Time() {
  return Teensy3Clock.get();
}

void setup() {
  lcd.begin();
  lcd.clearDisplay();
  lcd.display();
  lcd.setRotation(0);
  lcd.setContrast(155);
  Serial.begin(115200);
 
  setSyncProvider(getTeensy3Time);    // synchronise time and date after uploading firmware
 
  lcd.setFont(&Profont17);
  lcd.setCursor(2, 15);
  lcd.print("USB drive data logging");
  lcd.setCursor(2, 40);
  lcd.print("Push start for logging");
  lcd.display();
  //setTime(16, 42, 00, 12, 8, 2022); // use this for GUI Pascal to set time and date if needed
  pinMode(toggleSwitchPin, INPUT_PULLUP);
 
  myusb.begin();

  Serial.print("\nInitializing USB MSC drive...");

  while (!firstPartition) {
    myusb.Task();
  }
  Serial.println("USB drive initialized.");
  usbInitialized = true;
 
  menu();
}

void loop() {
  int reading = digitalRead(toggleSwitchPin);
 
  if (!firstPartition) {
    usbInitialized = false;
    Serial.println("USB drive removed.");
    lcd.setCursor(40, 40);
    lcd.print("USB drive removed.");
    lcd.display();
    lcd.clearDisplay();
  }

  if (!usbInitialized) {
    Serial.print("\nReinitializing USB MSC drive...");
    while (!firstPartition) {
      myusb.Task();
    }
    Serial.println("USB drive reinitialized.");
    usbInitialized = true;
     lcd.setCursor(2, 15);
    lcd.print("Push start for logging");
    lcd.setCursor(35, 40);
    lcd.print("USB drive inserted");
    lcd.setCursor(55, 63);
    lcd.print("USB drive ready");
    lcd.display();
    lcd.clearDisplay();
  }

   if (reading != lastSwitchState) {
    lastDebounceTime = millis();
  
  }

  if ((millis() - lastDebounceTime) > debounceDelay) {
    if (reading != switchState) {
      switchState = reading;
      if (switchState == LOW) {
        Serial.println(switchState);
        loggingActive = !loggingActive;
        if (loggingActive) {
          startLogging();
        } else {
          stopLogging();
        }
      }
    }
  }

  lastSwitchState = reading;

  if (write_data) logData();

  if (Serial.available()) {
    char rr;
    rr = Serial.read();
  
    switch (rr) {
      case 'l': listFiles(); break;
      case 'e': eraseFile(); break;
      case 'd': dumpLog(); break;
      case '\r':
      case '\n':
      case 'h': menu(); break;
    }
    while (Serial.read() != -1);
  }
}

void startLogging() {
  Serial.println("\nLogging Data!!!");
  write_data = true; 
  dataFile = firstPartition.open("datalog.csv", FILE_WRITE);
  if (record_count == 0) {
    dataFile.println("Date,Time,Pressure");
  }
}

void stopLogging() {
  Serial.println("\nStopped Logging Data!!!");
  write_data = false;
  dataFile.close();
  lcd.setCursor(40, 15);
  lcd.print("Logging stopped. ");
  lcd.setCursor(2, 40);
  lcd.print("USB drive, ");
  lcd.setCursor(2, 62);
  lcd.print("can be removed safely.");
  lcd.display();
  lcd.clearDisplay();
  Serial.printf("Records written = %d\n", record_count);
}

void logData() {
  lcd.setCursor(2, 16);
  lcd.print("USB drive data logging,");
  lcd.setCursor(0, 40);
  lcd.print("in progress! ");
  lcd.setCursor(0, 63);
  lcd.print("Records written= ");
  lcd.print(record_count);
  lcd.display();
  lcd.clearDisplay();
  String dataString = "";
 
  char dateBuffer[11];  // YYYY-MM-DD
  snprintf(dateBuffer, sizeof(dateBuffer), "%04d-%02d-%02d", year(), month(), day());

  char timeBuffer[9];  // HH:MM:SS
  snprintf(timeBuffer, sizeof(timeBuffer), "%02d:%02d:%02d", hour(), minute(), second());
  // test with A0 as sensor
  int sensor = analogRead(A0);
  dataString += dateBuffer;
  dataString += ",";
  dataString += timeBuffer;
  dataString += ",";
  dataString += String(sensor);

  if (dataFile) {
    dataFile.println(dataString);
    Serial.println(dataString);
    record_count += 1;
  } else {
      Serial.println("error opening datalog.csv");
  }
  delay(100); // log interval test 100ms
}

void dumpLog() {
  Serial.println("\nDumping Log!!!");
  // open the file.
  dataFile = firstPartition.open("datalog.csv");

  if (dataFile) {
    while (dataFile.available()) {
      Serial.write(dataFile.read());
    }
    dataFile.close();
  }
 
  else {
    Serial.println("error opening datalog.csv");
  }
}

void menu() {
  Serial.println();
  Serial.println("Menu Options:");
  Serial.println("\tl - List files on disk");
  Serial.println("\te - Erase log file on disk");
  Serial.println("\td - Dump Log");
  Serial.println("\th - Menu");
  Serial.println();
}

void listFiles() {
  Serial.print("\n Space Used = ");
  Serial.println(firstPartition.usedSize());
  Serial.print("Filesystem Size = ");
  Serial.println(firstPartition.totalSize());
  printDirectory(firstPartition);
}

void eraseFile() {
  Serial.println("\nErasing datalog.csv");
  firstPartition.remove("datalog.csv");
  Serial.println("\ndatalog.csv erased !");
}

void printDirectory(FS &fs) {
  Serial.println("Directory\n---------");
  printDirectory(fs.open("/"), 0);
  Serial.println();
}

void printDirectory(File dir, int numSpaces) {
  while (true) {
    File entry = dir.openNextFile();
    if (!entry) {
      break;
    }
    printSpaces(numSpaces);
    Serial.print(entry.name());
    if (entry.isDirectory()) {
      Serial.println("/");
      printDirectory(entry, numSpaces + 2);
    } else {
      printSpaces(36 - numSpaces - strlen(entry.name()));
      Serial.print("  ");
      Serial.print(entry.size(), DEC);
      DateTimeFields datetime;
      if (entry.getModifyTime(datetime)) {
        printSpaces(4);
        printTime(datetime);
      }
      Serial.println();
    }
    entry.close();
  }
}

void printSpaces(int num) {
  for (int i = 0; i < num; i++) {
    Serial.print(" ");
  }
}

void printTime(const DateTimeFields tm) {
  const char *months[12] = {
    "January", "February", "March", "April", "May", "June",
    "July", "August", "September", "October", "November", "December"
  };
  if (tm.hour < 10) Serial.print('0');
  Serial.print(tm.hour);
  Serial.print(':');
  if (tm.min < 10) Serial.print('0');
  Serial.print(tm.min);
  Serial.print("  ");
  Serial.print(months[tm.mon]);
  Serial.print(" ");
  Serial.print(tm.mday);
  Serial.print(", ");
  Serial.print(tm.year + 1900);
}

This example is very useful for its code to set up and use a USB thumb drive for storage. However, it has some limitations that will need to be addressed to convert it to a long-term logger useful for scientific or engineering use:

1. The CSV format, while useful for saving data that can be read by Excel or other spreadsheets, will limit simple import of your data. Excel can only handle 1,048,576 rows of data. At 10 rows per second, you will reach that limit in about 1.2 days.

2. The data is saved ~10 times per second, but the time stamp has only 1-second resolution. Adding a field with the millis() value would allow a way to calculate the actual time at which a record is saved.

3. The Serial.println(data string) is a waste of CPU time and system power. People can't read and understand 10 lines of data per second. It would be better to display only one or two lines per second, or display the data only on request from the user.

4. For real long-term logging (months or more), storing a binary structure, multi-level buffering, interrupt-based collection, and power management would be needed. (Power Management: WFI() between samples, Using ping-pong buffers of a megabyte or two in PSRAM and turning off USB Drive power between buffer writes).

Overcoming these limitations will always be a challenge. However, the learning curve is less challenging when you have example code that shows how to initialize the storage medium then open, write to, and close files. The example code here does that and should prove useful to a lot of Teensy users over the next few years.

pd0lew · Jun 23, 2024

This an example my real project log 1 time a minute, no reason to make worries at all.

Best,

Johan

pd0lew · Jun 23, 2024

MTP Disk (Experimental) you are right not needed at all,...

Thanks for the remark.

Best,

Johan

pd0lew · Jun 25, 2024

The USB datalogger is ready, I log 3 variables Time, Date and measuring Pressure. I measure once a minute and that during 30 days, after 30 days I make a new file with name as datalog_2024_06_25_201802.csv the unit has also a realtime clock.

Best,
Johan

pd0lew · Jun 25, 2024

Great great product the Teensy 4.1, very happy with this powerful beast.

wwatson · Jun 25, 2024

pd0lew said:
Great great product the Teensy 4.1, very happy with this powerful beast.

Glad to see it's working for you

pd0lew · Jul 30, 2024

Now my meter is nearly ready it is time to show it on YouTube ....
Vacuum meter use USB disk for Datalogging

Thanks all for the help!
Thanks PJRC for this great microcontroller!

Johan

wwatson · Jul 30, 2024

Very Interesting datalogger...

pd0lew · Aug 5, 2024

The meter acts now as a proportional valve controller, here i used a teensy 4.0 with ADS1115 AD converter.

Long time log on SD card

Well-known member

Senior Member+

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member

Well-known member