uSD card write time and coding around it

Status
Not open for further replies.
T

TurboStreetCar

Active member
Hello, ive created a datalogger from a Teensy 3.6 and using the onboard uSD card slot to save the data in CSV format.

I have pretty much all of it working properly but i have a few questions about the behavior ive noticed. Ill preface this by saying i have ZERO formal education or training in writing code or the technical aspects of how these micro-controlers work. Its just been pure hobby, but i pick up on it pretty quick. The current logger code is two pages and a total of 540 lines of code.

The logger is a CAN logger that pulls messages from the bus, identifies the address, pulls the data contained in the message, and formats it into a char array for each parameter. THEN every 50mS it formats all the parameter char arrays into a string and stores it in a 8KB buffer. When the buffer reaches 7730bytes (full minus a complete max length line of data) it prints to the uSD Card.

My question stems from how the data is moved to the card.

With my 50mS interval i see an exact 50mS interval as measured and logged in the data from every time it enters the loop.

If i change the interval to 20mS, i see single peaks that push the recorded interval to ~30mS.

Does the code pause on myFile.println()? Or does the actual writing to SD take place in the background with no effect on the running code? Could the latency be that while the buffer is being "printed" to SD, the code is modifying the buffer?
 
SD cards contain their own micro controller that assigns memory to keep wear consistent and can put the transfer on hold while it thinks about things or even shuffles blocks around. If 100% repeatable timing is the aim you may need to look at getting memory chips and interacting with them directly.
 
The standard SD file writes are synchronous and wait for the SD card to do its thing. That can be a rather long time, close to a whole second.

All my MicroSD cards have a >800ms worst case latency with standard file writes (might only happen once per several hours). Using pre-allocated, pre-erased files, that worst-case latency is less than 45ms for all of them.

Getting asynchronous behavior is not trivial. One way to do it (using interrupts for data acquisition):
https://forum.pjrc.com/threads/43834-Real-low-latency-logging-for-Teensy-3-5-3-6-SDIO-SD

The other other way is to rely on the SdFat-beta library to call 'yield()' while the SD card is busy internally reshuffling data (move your data acquisition into 'yield()').

\\

The easy way out might be to simply use a huge buffer for the incoming CAN messages. At 1MBit/s, you have a worst case maximum of around 12000 messages per second (in practice, with realistic message sizes more like 8000 - 10000). You can adjust the RX buffer size:

https://github.com/collin80/FlexCAN_Library/blob/master/FlexCAN.h#L15
 
Thanks for the input!

The write time isn't so much the issue as the 8000 byte buffer allows around 6 seconds of buffer time before information needs to be moved to the SD. Also i nave no issue with loosing CAN messages.

It's more the pausing/lag in the teensy code that concerns me. As if there is a delay function in the main loop. No data is lost, but the timestamp get shifted 10-30mS at each pause.

What exactly does yield() do?

Thanks for the help, my inexperience with code makes understanding some of these complex functions a slow process. It's like my brain has a really small SD car buffer. Takes time to process haha.
 
Last edited:
TurboStreetCar said:
What exactly does yield() do?
Click to expand...
'yield()' is invoked once per 'loop()' and by various libraries when there is a long delay (library calls block).

Normally, it's just calling the serialEvent functions:
https://forum.pjrc.com/threads/43241-what-is-outside-loop()

'yield()' is declared as a weak symbol, so if you define your own 'yield()' function, it replaces the default one.

\\

It's more the pausing/lag in the teensy code that concerns me. As if there is a delay function in the main loop. No data is lost, but the timestamp get shifted 10-30mS at each pause.
Click to expand...
That's just the SD card taking its time. When you call the SD file write functions, that kind of delay will happen very often.
 
Can i use yield() to allow the main loop to continue to run while the SD card figures out what it wants to do?

I only need to write to the SD when the buffer is filled. I can create large and/or multiple buffers that EASILY take longer to fill then the write function.

But it doesnt seem to make a difference how large or numerous the buffers are when the whole code stops. At that point the code is dead in the water, so the empty buffers cant be updated while the full buffer is written.
 
So. Figured it out!

Im such a newbie i didnt realize i could do this.

My code was layed out like this:

***disclaimer, this code is prolly horrendously inefficient and breaks a bunch of code etiquette, but im new and dont know any better****

Code: 
***
variable declaratons
**
void setup() {
  pin modes set
  Serial and SD initialization.
  date and time functions.
  
}

void loop() {

//Check if can message is available
  if (Can0.available()) {
    Can0.read(rxmsg);
    switch (rxmsg.id) {

      //can message filter by address and variable data updating.

    }
  }

//When logging is active
  if (logging == 1) {

//When defined interval of time has passed.
    if ((millis() - beginTime) >= timerInterval) {
      
      if (runonce == 0) {
        digitalWrite(loggingActiveLED, HIGH);
        runonce++;

//print header to SD
        setFileName();
        myFile.open(fileName, O_CREAT | O_WRITE);
        myFile.println(header);
        myFile.close();

//first write of parameters to first SD buffer in order.
        switch (currentBuffer) {
          case 1:
            bufferLength = setParamatersFirst(paramaterBuffer1);
            break;
          case 2:
            bufferLength = setParamatersFirst(paramaterBuffer2);
            break;
        }
      }

//After buffer is filled, first write to new buffer in order.
      else {
        if (bufferFull == true) {
          switch (currentBuffer) {
            case 1:
              bufferLength = setParamatersFirst(paramaterBuffer1);
              break;
            case 2:
              bufferLength = setParamatersFirst(paramaterBuffer2);
              break;            
          }
          bufferFull = false;
        }
        else {

//add parameters to existing buffer that isnt full.
          switch (currentBuffer) {
            case 1:
              bufferLength = addParamaters(paramaterBuffer1);
              break;
            case 2:
              bufferLength = addParamaters(paramaterBuffer2);
              break;
          }

//when buffer is full, send full buffer to SD and set new buffer to be used.
          if (bufferLength >= 9770) {
            myFile.open(fileName, O_APPEND | O_WRITE);
            switch (currentBuffer) {
              case 1:
                myFile.println(paramaterBuffer1);
                currentBuffer = 2;
                break;
              case 2:
                myFile.println(paramaterBuffer2);
                currentBuffer = 1;
                break;
            }
            myFile.close();
            bufferFull = true;
          }
        }
      }
    }
  }

//when logging is turned off, sends existing selected buffer to SD to save unsaved data
  else {
    if (runonce == 1) {
      //myFile.open(fileName,  O_APPEND | O_WRITE);
      switch (currentBuffer) {
        case 1:
          myFile.println(paramaterBuffer1);
          currentBuffer = 2;
          break;
        case 2:
          myFile.println(paramaterBuffer2);
          currentBuffer = 1;
          break;
      }
      myFile.close();
      digitalWrite(loggingActiveLED, LOW);
      runonce = 0;
    }
  }

//reset button state for debounce
  pastButtonState = currentButtonState;
}

I had the idea that since the close() function is what is causing the delay, and since i dont really need to close the file until im done logging, i just commented out all the open() and close() lines between the first open() and the last close().

SO now it only opens the file once when it enters the logging loop, and only closes the file once when it leaves the logging loop.

Now it only takes about 800ms to run threw the loop. The delay at close is no longer an issue because its after logging has stopped!!

Such a fun hobby.
 
Other then being susceptible to data loss if power is lost, Is there any downside to leaving the file open for the duration of the logging period?
 
this 'CAN' your logging, I presume it's from your car? You could always close the file when you see the can stream start closing, or when the RPMs drop a certain threshold, then open the file and start logging when it detects the level go above a specific threshold

this way you could pull the card out knowing its already safe
 
Yes that's kind of what the plan is. Except it's on a push button. I just wasn't sure if there's a rule of thumb for how much data to write before closing the file. I'm not well versed in the process of writing to uSD, so I wasn't sure if there were benefits to closing the file more frequently then at the finish of the desired logging period.
 
TurboStreetCar said:
I just wasn't sure if there's a rule of thumb for how much data to write before closing the file.
Click to expand...

There is a maximal file size in any FAT system (Wikipedia may help). This may not be an issue for your data rate.
Also, think about what you wanted to do with data. (e.g. read whole file into memory of your computer).
 
TurboStreetCar said:
The file is just getting saved to be reviewed at a later time. Won't be read back by the teensy or manipulated in any way.
Click to expand...

That what I mean, analysis on a PC.
e.g. Excel had some limits of 64*1014 lines per sheet (not sure if it is still valid). So any number of lines > the sheet limit makes your analysis a little bit more difficult.
Nothing serious, but something to consider when designing a system.
 
I believe that sheet limit was raised, but it would be opened in a log viewing program so that's not much concern.

Was more worried about buffers in the background on the teensy or uSD side of things.

If println() stored the information in a buffer and then close() writes the buffer to the uSD. I was concerned about overflowing a buffer and having data loss or some other problem.

But if I can println() up to the max file size for FAT without consequence then there's no real reason to close() before the logging period is completed.

With the size of the data per line, it would take a couple days straight to reach 4GB.
 
Status
Not open for further replies.
Back
Top