Setting create time on files written to the SD card

Rezo · Jan 10, 2023

Using a Teensy Micromod with SDIO interface to log data.
Each time the data log option is enabled, a new file is created.

I am able to write the timestamp into the file itself (a CSV) but I am having a hard time setting the timestamp on the create field of the file.

Time is captured over CAN and fed into the Time library.
I've tried the following approaches but with no success:

#1
Before initialising mtp and the SD card, I set the time in the file system.
If I do this after I set up mtp/SD I get a weird timestamp when testing on the bench with USB plugged into the computer and no CAN network (so time will default to Jan 1st 1970, but it prints the current day/month with the year at 2030)

Code:

FLASHMEM void FSdateTime(uint16_t* date, uint16_t* time) { 
  // return date using FS_DATE macro to format fields
   *date = FS_DATE(year(), (uint8_t)month(), (uint8_t)day());
  
  // return time using FS_TIME macro to format fields
   *time = FS_TIME((uint8_t)hour(), (uint8_t)minute(), (uint8_t)second());
}


FLASHMEM void sdBegin(){
    FsDateTime::setCallback(FSdateTime);
    MTP.begin();
  // Add SD Card
    if (SD.begin(BUILTIN_SDCARD)) {
        MTP.addFilesystem(SD, "SD Card");
    } 
    else {
    } 
    
}

#2
When the file is created, I call the following code:

Code:

DateTimeFields ts;
    ts.hour = hour();
    ts.min = minute();
    ts.sec = second();
    ts.year = year()-1900;
    ts.mon = month();
    ts.mday = day();
    csvFile.setCreateTime(ts);

But this doesn't work at all and I get a timestamp when the card was produced (sometime in 2019)

Can anyone suggest another method that is known to work with SD.h and MTP?

KurtE · Jan 10, 2023

Is the normal SD setup for setting the Create and modify date/times not working?

You created your own callback, but so does SD.

That is:

Code:

bool SDClass::begin(uint8_t csPin) {
#ifdef __arm__
	FsDateTime::setCallback(dateTime);
#endif

When a file is created, the underlying SDFat code should be setting the create and modify date/times. Typically, only needed to do it manually, like this when for example, you are doing a copy of a file
and you wish to set the create data of the new file to that you copied.

Also your create date time fields may be off, as month is 0 biased.

Code:

	virtual bool getCreateTime(DateTimeFields &tm) {
		uint16_t fat_date, fat_time;
		if (!sdfatfile.getCreateDateTime(&fat_date, &fat_time)) return false;
		if ((fat_date == 0) && (fat_time == 0)) return false;
		tm.sec = FS_SECOND(fat_time);
		tm.min = FS_MINUTE(fat_time);
		tm.hour = FS_HOUR(fat_time);
		tm.mday = FS_DAY(fat_date);
		tm.mon = FS_MONTH(fat_date) - 1;
		tm.year = FS_YEAR(fat_date) - 1900;
		return true;
	}

If you need your own callback try resetting it after the call to SD.begin()

Rezo · Jan 10, 2023

Im not sure about your first question? I didn’t see a set date/Time function in the SD wrapper class.
If you mean it should automatically set the date/time itself using its own callback - it’s not working.

Do I need my own callback if I am using the Time library?

KurtE · Jan 10, 2023

What I was trying to say is did you try out the currently built in stuff and it failed?

For example, here is a sketch that I hacked up:

Code:

//=============================================================================
// Simple Sketch that initializes MTP and adds one SD drive as the only
// storage.
// Notes:
//    If SD.h is included before MTP_Teensy.h, the code will automatically
//    inert a handler callback that will try to detect when the SD is inserted
//    or removed.
//
//    The host often times will not automatically refresh to show the updated
//    status of the drive after events such as this.  However doing a refresh
//    example F5 on Windows, will update and show the new status.
//
//    If MTP_Teensy.h is included before SD.h, by default, this checking
//    for drive insertions is not installed.  You can tell the addFilesystem that
//    the storage is an SD drive like:
//        MTP.addFilesystem(SD, "SD Card", MTP_FSTYPE_SD);
//
//=============================================================================
#include <SD.h>
#include <MTP_Teensy.h>

#define CS_SD BUILTIN_SDCARD  // Works on T_3.6 and T_4.1
//#define CS_SD 10  // Works on SPI with this CS pin
void setup() {
  // mandatory to begin the MTP session.
  MTP.begin();

  while(!Serial && millis() < 5000);
  Serial.begin(115200);
  MTP.loop();
  // Add SD Card
  if (!SD.begin(CS_SD)) Serial.println("Failed to open SD");
  MTP.addFilesystem(SD, "SD Card");

  // lets create a new file and see what it's date/times are
  SD.remove("/test_file.txt");
  Serial.println("After remove file");

  File sdfile = SD.open("/test_file.txt", FILE_WRITE);
  if (sdfile) {
    sdfile.println("This is a test file");
    sdfile.close();

    sdfile = SD.open("/test_file.txt", FILE_READ);
    Serial.printf("Test file opened size: %llu ", sdfile.size());
    DateTimeFields dtf;
    sdfile.getCreateTime(dtf);
    Serial.printf(" Created: %U/%u/%u %u:%u:%u", dtf.mon + 1, dtf.mday, dtf.year + 1900, dtf.hour, dtf.min, dtf.sec);
    sdfile.getModifyTime(dtf);
    Serial.printf(" modify: %U/%u/%u %u:%u:%u\n", dtf.mon + 1, dtf.mday, dtf.year + 1900, dtf.hour, dtf.min, dtf.sec);
  } else {
    Serial.println("Failed to create file");
  }
}

void loop() {
  MTP.loop();  //This is mandatory to be placed in the loop code.
}

And the output:

Code:

Add **SDClass** file system
After remove file
Test file opened size: 21  Created: 1/10/2023 9:15:10 modify: 1/10/2023 9:15:10
&&&&& Dump MTPStorage Loop Data &&&&&
	Callback Data:		0	0x0	0
		1	0x8209	1
		2	0x0	0
		3	0x0	0
		4	0x0	0
	File Systems:
		0	0x200029e8	1

Note: I have some debug stuff turned on so there were the extra messages.

Also here is image showing the MTP directory with today's date and time

Rezo · Jan 10, 2023

I'll clarify and provide some more context.
The device is a data logger that connect to a car.
When the unit is powered up, it requests the date/time over CAN from the car, and feeds it to the time lib. This works flawlessly. I can also print the Time library provided timestamp within the created file.
But, don't get the right timestamp on the file metadata.

Now, If I plug the unit into my computer over USB, without any CAN comms, and generate a data log, it write the correct timestamp to the file metadata as I believe either MTP or FS is getting that from the USB handshake (?)

Therefore, the sketch you posted will work as expected, but only when the device is connected to a computer over USB.
My question is, does the SD/FS class know how to read the time/date from the Time library, or do I need to feed it manually (as I did with a callback or some other method)?

KurtE · Jan 10, 2023

I am pretty sure that the Teensy does not retrieve the current date/time over usb or the like. Could be wrong.

I believe that the teensy has the date/time when the teensy was programmed, and maybe uses this to initialize the teensy Real Time clock. Now if the Teensy has the appropriate power
on the VBat pin, it will use that the keep the RTC up to date, even when the teensy is not powered up: https://www.pjrc.com/store/teensy41.html#timing

MichaelMeissner · Jan 10, 2023

KurtE said:
I am pretty sure that the Teensy does not retrieve the current date/time over usb or the like. Could be wrong.

I believe that the teensy has the date/time when the teensy was programmed, and maybe uses this to initialize the teensy Real Time clock. Now if the Teensy has the appropriate power
on the VBat pin, it will use that the keep the RTC up to date, even when the teensy is not powered up: https://www.pjrc.com/store/teensy41.html#timing

IIRC, in the special case when the RTC is powered via the VBAT pin, it does upload the time when you are flash in code. From the timing document:

The RTC keeps track of date / time. The Time library is typically used together with the RTC. Teensy Loader automatically initializes the RTC to your PC's time while uploading. If a coin cell is connected to VBAT, the RTC will continue keeping time while power is turned off.

Rezo · Jan 10, 2023

KurtE said:
I am pretty sure that the Teensy does not retrieve the current date/time over usb or the like. Could be wrong.

I believe that the teensy has the date/time when the teensy was programmed, and maybe uses this to initialize the teensy Real Time clock. Now if the Teensy has the appropriate power
on the VBat pin, it will use that the keep the RTC up to date, even when the teensy is not powered up: https://www.pjrc.com/store/teensy41.html#timing

Well, it works somehow

when powered by USB via my Mac - I have no idea how or why.

All I am trying to figure out it how I can set the timestamp on the files from the Time lib - still no lead there.

I will note that I am not using the RTC. I fetch the time only once, when the Teensy is powered up. It does not perform any syncing while on (it's only on for an hour or two at a time)

defragster · Jan 10, 2023

MichaelMeissner said:
IIRC, in the special case when the RTC is powered via the VBAT pin, it does upload the time when you are flash in code. From the timing document:

yes, IIRC:
T_3.x puts a known value in one of the 8 RTC DWORDS - if that isn't there it assumes the RTC Batt isn't active and pushes the compile time, and that magic value. If it does exist at reset, then it assumes the RTC is powered and running.

<edit> Warm restarting while powered the RTC is held even without battery

KurtE · Jan 11, 2023

As I tried to mention earlier...

Rezo said:
Using a Teensy Micromod with SDIO interface to log data.
#1
Before initialising mtp and the SD card, I set the time in the file system.
If I do this after I set up mtp/SD I get a weird timestamp when testing on the bench with USB plugged into the computer and no CAN network (so time will default to Jan 1st 1970, but it prints the current day/month with the year at 2030)

Code:

FLASHMEM void sdBegin(){ FsDateTime::setCallback(FSdateTime); MTP.begin(); // Add SD Card if (SD.begin(BUILTIN_SDCARD)) { MTP.addFilesystem(SD, "SD Card"); } else { } }

You set the callback to your function:
Then you call SD.begin() which does the FsDateTime::setCallback to it's own function.
If you really want yours, maybe make your call AFTER the SD.begin() call.

Rezo said:
#2
When the file is created, I call the following code:

Code:

DateTimeFields ts; ts.hour = hour(); ts.min = minute(); ts.sec = second(); ts.year = year()-1900; ts.mon = month(); ts.mday = day(); csvFile.setCreateTime(ts);

I believe this one is wrong for month...
Should 0 bias the month like:

Code:

DateTimeFields ts;
    ts.hour = hour();
    ts.min = minute();
    ts.sec = second();
    ts.year = year()-1900;
    ts.mon = month() - 1;
    ts.mday = day();
    csvFile.setCreateTime(ts);

EDIT: Note the Callback function FsDateTime::setCallback(FSdateTime);
is called by SDFat when files are created.

Code:

   /** Set the date/time callback function.
   *
   * \param[in] dateTime The user's call back function.  The callback.
   * function is of the form:
   *
   * \code
   * void dateTime(uint16_t* date, uint16_t* time) {
   *   uint16_t year;
   *   uint8_t month, day, hour, minute, second;
   *
   *   // User gets date and time from GPS or real-time clock here.
   *
   *   // Return date using FS_DATE macro to format fields.
   *   *date = FS_DATE(year, month, day);
   *
   *   // Return time using FS_TIME macro to format fields.
   *   *time = FS_TIME(hour, minute, second);
   * }
   * \endcode
   *
   * Sets the function that is called when a file is created or when
   * a file's directory entry is modified by sync(). All timestamps,
   * access, creation, and modify, are set when a file is created.
   * sync() maintains the last access date and last modify date/time.
   *
   */
  void setCallback(void (*dateTime)(uint16_t* date, uint16_t* time));

You should not need to call it your self, unless you want it to deviate from the current time. I mentioned as an example MTP needs to call it when you copy a file to your storage, as it wants the new file to not have the current date/time but instead the date/time of the file that was copied (or moved)

defragster · Jan 11, 2023

@KurtE - Good Note on setCallback. I was working with those examples back then and didn't notice that - Seemed it 'just worked'.

Rezo · Jan 12, 2023

Okay, so I was doing some reading and now stuff you guys are saying is making sense; timestamp in the RTC when flashing, which explains why my time stamps looked spot on when connected to the computer over USB.
Now, both methods in my OP didn't work.
But I did the following and it seems to work:

Code:

FLASHMEM void FSdateTime(uint16_t* date, uint16_t* time) { 
  // return date using FS_DATE macro to format fields
   *date = FS_DATE(year(), (uint8_t)month(), (uint8_t)day());
  // return time using FS_TIME macro to format fields
   *time = FS_TIME((uint8_t)hour(), (uint8_t)minute(), (uint8_t)second());
}


FLASHMEM void sdBegin(){
    MTP.begin();
  // Add SD Card
    if (SD.begin(BUILTIN_SDCARD)) {
        MTP.addFilesystem(SD, "SD Card");
        SdFile::dateTimeCallback(FSdateTime);
    } 
    else {
}

If there is another method you feel is better - I'm all open to try it out

Now, I have another question, will calling csvFile.flush() in set intervals affect the create/modified fields on the file?
Is there even a need to call flush every so often when data logging for long periods of times? The file is opened when the log starts and is closed only when the data log is stopped.

Setting create time on files written to the SD card

Rezo

Well-known member

KurtE

Senior Member+

Rezo

Well-known member

KurtE

Senior Member+

Rezo

Well-known member

KurtE

Senior Member+

MichaelMeissner

Senior Member+

Rezo

Well-known member

defragster

Senior Member+

KurtE

Senior Member+

defragster

Senior Member+

Rezo

Well-known member