CAN logger and SD card speed.

[rkam]

Hi,

A few days ago I got my Teensy 3.6, and I'm now trying to make a dual channel CAN logger out of it.
Currently the aim is to be able to log two 500kbps channels, but 1Mbps would be even better.

Logging to serial port seems to be working well, but logging to an SD card might be problematic.

Below are output from Greimans SdFat-beta library with the card I am using.

1:

size,write,read
bytes,KB/sec,KB/sec
512,16039.19,18710.54
1024,17056.88,18889.52
2048,17123.70,19061.90
4096,17295.31,19191.60
8192,17312.16,19126.09
16384,17325.54,19266.40
32768,17304.02,19272.50

totalMicros 6522071
yieldMicros 186429
yieldCalls 176
yieldMaxUsec 1349
kHzSdClk 45000
Done
Type '1' for SdFatSdioEX or '2' for SdFatSdio


2:

size,write,read
bytes,KB/sec,KB/sec
512,415.15,2068.71
1024,561.05,2899.53
2048,1593.08,5062.67
4096,4137.94,9203.65
8192,5899.78,11952.82
16384,9984.05,16415.39
32768,10952.01,17668.97

totalMicros 56682983
yieldMicros 56134934
yieldCalls 81362
yieldMaxUsec 134753
kHzSdClk 45000
Done

The question is how much or little I can/have to write to avoid loosing any CAN frames.
I believe the CAN buffer is 6 frames long, and 6 full frames would take about 1.5ms to receive.
If the CAN frames contain less than 8 bytes of payload, the time to fill the buffer will be less.

Using DMA method, I was able to write a block of 512 bytes in about 2.5ms.
A 32 byte block was written in 0.19ms
I chose DMA method to have the most free CPU time, but that might not be required.

Is it OK to write in 32 byte blocks, or in general anything less than 512?
Will there be a longer delay when I reach 512 bytes in total?

I will try to perform some tests to check this, but if anyone can tell me to just forget it, that would save some time.

Thanks


View attachment canloggerserial.inoView attachment sdwrite32byte.ino

 

[tni]

Worst-case SD latency is several hundred milliseconds. The low latency logger code works around that by pre-allocating and pre-erasing a file.

Teensy 3.6 has quite a bit of RAM, so if you use interrupts for CAN and copy messages to a large buffer, that could also work.

 

[rkam]

Thanks,

Low latency logger seems to be using a logging interval of 2ms which is longer than the time to fill the CAN input buffer, so it will be too slow.
But if there is a risk of intermittant long delays that can be avoided using techniques from that example, they could be useful.

The interrupt driven CAN library might come in handy if the SD card writing could be split into lower priority operations with short enough latency.
I hadn't noticed that one, but I have been using SavvyCAN and GVRET on a DUE. SD card writing was never sucessful on those, but I didn't put much work into it.

RAM will fill up in a couple of seconds. I need to be able to store all CAN frames continously.

If I could write somewhere arund 128 bytes in 0.8 ms and be sure the time was never longer, then 500kbps logging should be possible. (12 full frames every 1.5ms)
I haven't calculated worst case timing of short CAN frames though.

 

[tni]

Low latency logger seems to be using a logging interval of 2ms which is longer than the time to fill the CAN input buffer, so it will be too slow.

You can use whatever log interval you want. The only restriction is that you need to write a whole sector / 512 bytes at a time. IIRC, some SD cards have consistent performance of better than 0.7ms for sector writes to pre-erased files. Some discussion on low latency logging is here.

The interrupt driven CAN library might come in handy if the SD card writing could be split into lower priority operations with short enough latency.

They are interrupts and interrupt things. Seriously, interrupts work while the SD transfer is happening. So you can fill up the RAM while transfer occurs.

 

[rkam]

If the SD writing library doesn't mind being interrupted, then using CAN interrupts is probably the safest way to go.

It's a bit confusing to read all the stuff about SD cards.
Like; you cannot write less than 512 bytes of data, but you can write some of it, do something else (interrupt), then write the rest.
I suppose that depends on the definition of "you". User or library.

 

[tni]

Like; you cannot write less than 512 bytes of data, but you can write some of it, do something else (interrupt), then write the rest.

AFAIK, the minimum clock frequency allowed for data transfers is 0 - so you can have arbitrary delays. However, the write commands only support full sectors. So if you were to write your own SD driver, you can have an interface with smaller blocks and have an unfinished transfer (but eventually you have to write the full sector). You wouldn't be able to do anything else with the card while the transfer is not complete.

 

[rkam]

I tried my example above changed to write 16 128-byte blocks and recorded the microseconds:

2766
2769 3
2771 2
4071 1300
4073 2
4075 2
4077 2
5679 1602
5682 3
5684 2
5686 2
7296 1610
7298 2
7300 2
7302 2

By the results it looks like Greimans SDIO DMA library will fill a buffer and then spend more time actually writing when 512 bytes are ready.

 

[Bill Greiman]

Every SD card can have an occasional long write latency. The card you are using had a latency of over 134 ms in the DMA test.

yieldMaxUsec 134753

The SD spec allows up to 250 ms for SDHC cards and 500 ms for SDXC cards.

I find high end cards like a Samsung pro select have much lower write latency. Max of around 10-20 ms.

You must buffer data to avoid loss. I use an RTOS for fast logging, not the Arduino IDE. I can log ADC data at a million samples per second without data loss with a high end STM32 using the ChibiOS/RT RTOS. I allocate up to 100 KB of buffer.

 

[CollinK]

I'm actually porting the GVRET code over to Teensy so that I can use the new FlexCAN interrupt driven code along with the on-board sdCard slot to do logging to sdCard with dual CAN. My goal is to use a 32768 byte buffer and send out large batches to the card all at once. That's kind of what GVRET does on the Due but I hadn't tested sdcard writing in a long, long time so maybe it got broken along the way? Or, perhaps the trouble is that I never bothered to check for whether the sdcard was still busy with a long write latency and then I tried to write again and clobbered things. I didn't know anything about sdcard write latency when I originally wrote the sdcard logging a couple of years ago.

Anyway, since he's on this thread I'd like to say "Thank you Bill Greiman for writing the awesome sdcard library!" Really, that's a tremendous help and a very nice library.

 

[rkam]

As long as it is possible to interrupt the SD writing once or twice every millisecond, a buffer should work.
But I'm not sure if any or both of the two methods in the SdFat-beta will allow that.
I selected method 2 hoping that it would leave me with more CPU time, but I probably don't need much CPU-time to just receive CAN messages.
Maybe method 1 would be better?
Still a yieldMaxUsec of 1349 is pretty tight if it cannot be interrupted.
And short CAN frames or 1Mbps would be impossible since the CAN receive buffer would overflow before I could move the data to a RAM buffer.

 

[rkam]

There is a pretty high chance that failing SD card writing with GVRET and DUE is my fault.
I was however able to identify the card and get its size, so the connections should be OK.

 

[CollinK]

The processor in the Teensy supports nested interrupts. You can literally interrupt an interrupt. So, I don't see why it shouldn't be possible for the CAN code to interrupt the sdcard code so long as the sdcard code doesn't turn off interrupts for a long time (which I highly doubt) and so long as the CAN interrupt is of a higher priority than the sdcard interrupt. That I don't know about but it's easy to change in the library so no big deal there. The CAN code doesn't sit around in the interrupt handler long so long as you don't have callbacks enabled and doing long tasks. So, between the two they should be able to happily coexist and interrupt. Life should be good. Of course, testing is always a good idea. I will be doing so pretty soon.

 

[Bill Greiman]

The processor in the Teensy supports nested interrupts. You can literally interrupt an interrupt. So, I don't see why it shouldn't be possible for the CAN code to interrupt the sdcard code so long as the sdcard code doesn't turn off interrupts for a long time (which I highly doubt) and so long as the CAN interrupt is of a higher priority than the sdcard interrupt. That I don't know about but it's easy to change in the library so no big deal there. The CAN code doesn't sit around in the interrupt handler long so long as you don't have callbacks enabled and doing long tasks. So, between the two they should be able to happily coexist and interrupt. Life should be good. Of course, testing is always a good idea. I will be doing so pretty soon.

Yes you can avoid using an RTOS. I just don't like to reinvent multi-threading, semaphores, buffer queues, and other schedulers stuff for each project.

SdFat with DMA on Teensy 3.6 uses a DMA done interrupt with a very short ISR.

// ISR
void sdhc_isr() {
  SDHC_IRQSIGEN = 0;
  m_irqstat = SDHC_IRQSTAT;
  SDHC_IRQSTAT = m_irqstat;
  m_dmaBusy = false;
}

In an RTOS, the last line is replaced with a semaphore give to wake the SD thread. In a data logger I use a sensor done interrupt to wake the data queuing thread. ChibiOS does a context switch in about a half micros-second and is tickless so there is no unnecessary scheduler overhead.

SdFat on Teensy with the Arduino IDE just spins in a loop that calls yield() until dmaBusy is set false.

 

[rkam]

Sidenote:
Testing my traffic generator Kvaser USBcan Light 2xHS with CanKing software.
The software only allows me to send continous burst data at one channel.
500kbps with a message length of 1 is pumped out at 8 frames per ms. 0.125ms each.
1000Mbps gives the same result, so this is probably a limitation in the software or hardware.