Fast Data Logger

I'm using Paul's Teensy-specific fork of SdFat that is included with TeensyDuino. Are you actually downloading from Bill Greiman's github?
 
Yes- but now I see Paul’s fork on github. I’m trying that now. I had been using the one in the Teensy hardware libraries which I assume is the one included in Teensyduino, but maybe it was out of date.
 
slash2 said:
Yes- but now I see Paul’s fork on github. I’m trying that now. I had been using the one in the Teensy hardware libraries which I assume is the one included in Teensyduino, but maybe it was out of date.
Click to expand...

You should use the one that is in whatever version of the Teensy package you are using. You don't need to do a special download/install from Paul's github, and in general you should not do that.
 
I ran what I downloaded with my program and got:

max Buf Usage (bytes): 533532

using a Dual buffer of 899072 bytes

so theoretically it worked! I'll need to check the data, but this is encouraging.

I'm using Teensy 1.59.0, which looks like the most recent, but that's the SDFat I had trouble with. Maybe I should reinstall it and try it.
 
jmarsh said:
So all the formatter is doing is filling sectors with 0xFF bytes (or zeroes) rather than trimming them.
Click to expand...
It is kind of important that the file system be aligned to the SD card things like Recording Units. It will work if it isn't but speed will be compromised.

If the OS just sees a series of blocks I wouldn't have had an SD card reader be unable to deal with a SD 2.0 card many years ago. (Not a high capacity problem.) Clearly the driver was issuing more than read and write commands to the card.
 
slash2 said:
I ran what I downloaded with my program and got:

max Buf Usage (bytes): 533532

using a Dual buffer of 899072 bytes

so theoretically it worked! I'll need to check the data, but this is encouraging.

I'm using Teensy 1.59.0, which looks like the most recent, but that's the SDFat I had trouble with. Maybe I should reinstall it and try it.
Click to expand...

That's good. I don't understand "I ran what I downloaded with my program". If you have installed Teensy 1.59, that included the Teensy-specific version of SdFat, and that is the one you should use. If you update to Teensy 1.60 beta 5 (latest), it has the same version of SdFat as 1.59.
 
UhClem said:
If the OS just sees a series of blocks I wouldn't have had an SD card reader be unable to deal with a SD 2.0 card many years ago. (Not a high capacity problem.) Clearly the driver was issuing more than read and write commands to the card.
Click to expand...
You would if it was an older reader that didn't handle SDHC cards. The addressing mode was changed from bytes to sectors and an extra initialization command is required. This is all handled by the reader, not the PC.
 
jmarsh said:
You would if it was an older reader that didn't handle SDHC cards. The addressing mode was changed from bytes to sectors and an extra initialization command is required. This is all handled by the reader, not the PC.
Click to expand...
Looks like I wasted my time stating it wasn't a high capacity problem.
 
Because the capacity doesn't play a part? There are both regular SD and SDHC cards that are 4GB...
 
jmarsh said:
Because the capacity doesn't play a part? There are both regular SD and SDHC cards that are 4GB...
Click to expand...
It has been almost two decades but I found some notes on that project for the SparkFun Logomatic. That USB SD card reader could read and write the card but the the results were corrupted. Small files were fine but larger ones weren't.
 
I updated my Arduino and Teesnyduino to the latest. Ran a half hour at 4.5Msmpl/sec (18Mbytes/sec) with the RAM1 and RAM2 buffer, though it looks like a single buffer would be enough. Buffer usage in bytes on Y and sample number /8100 on X. It's interesting that the buffer usage is much different than what I saw before. This is encouraging - just need to test with real data just to be sure. Thanks Joe (and everyone) for all the advice!

1770685176867.png
 
slash2 said:
I updated my Arduino and Teesnyduino to the latest. Ran a half hour at 4.5Msmpl/sec (18Mbytes/sec) with the RAM1 and RAM2 buffer, though it looks like a single buffer would be enough. Buffer usage in bytes on Y and sample number /8100 on X. It's interesting that the buffer usage is much different than what I saw before. This is encouraging - just need to test with real data just to be sure. Thanks Joe (and everyone) for all the advice!
Click to expand...

You're welcome, and that chart looks good. Seems like your max busy time is only about 21 or 22 ms, compared to the 40 I'm getting. I wonder why that difference exists? In any case, let us know what happens when you test with real data.

Can you please say again what you changed/fixed to get these better results? Was it just that you weren't using the Teensy-specific version of SdFat?
 
I believe so. I did a lot of testing at home and it had a somewhat older Arduino on it. That’s where the earlier charts came from. Then I updated to the latest arduino and teensy and removed the arduino/libraries/sdfat libraries just to be sure. That produced the working chart. The odd thing is I also did some testing at work and there I had the latest preV2 Arduino and teensy and I got bad results. However it may have been using the Arduino SDfat library.

Something just occurred to me - I remember your code listed the teensy and sdfat versions. I need to add that - then I’ll know what it is using.

Edit…
I reported a bad result using your code earlier here and it showed;

Teensyduino version 159
SdFat version 2.1.2

You had showed results that you had run:

Teensyduino version 160
SdFat version 2.1.2

so same sdfat version! So very odd.

Now I did make another change and that is I stated doing the full format with the Teensy formatter on the T4 rather than doing a full format on the PC with the SDcard org formatter. Maybe that has something to do with it. I could test that.
 
Last edited:
slash2 said:
Good info, thanks. So if the card is fully formatted, will the block erase be skipped? I should test that.

What oceanographic company? I used to work at SeaLink which got bought up by EG&G. We mostly did releases.
Click to expand...
I worked at WETLabs in Philomath, OR for about 5 years in the late '90s. The was before they got acquired by SeaBird. I got bummed out by some startup shenanigans and lots of overtime near the holidays to finish government research projects, so I started my own one-person consulting company. I developed a number of different sensors and data loggers for moored turbulence measurements for the Oregon State Ocean Mixing Group. I also worked there part time as a faculty research assistant several times between 2010 and 2020. Apropos this newsgroup, the last project was adding a turbulence sensor to an autonomous, drifting, vertical profiler developed by Scripps. We used a T3.5 to acquire and log the data. We needed the processing power to do some complex initial data reduction--which needed lots of math, including FFTs--to reduce the raw data to something that could be reported back via satellite while the profiler was on the surface for a few minutes between trips down to 150 meters. I was just in at the start--mostly contributing Teensy expertise. I was about 72 by then and almost completely retired. I was fully retired by the time of the first sea tests. You can see the Flipping Chi Solo here: OSU Mixing Instrumentation

I also did initial designs on the PPOD, Chi-Pod, and GuST loggers. The first ChiPods in the early 2000's used TattleTale processors from Onset Computer and stored data on small Toshiba hard drives that were at the end of a long series of buffers (RAM, then CF Cards) and were only turned on about once every 15 minutes. Later versions used a TI MSP430 processor with a multiplexer connecting to 4 x 4GB SD cards run in SPI mode.

Needless to say, I've seen a lot of challenges and changes in embedded systems for data logging since I got out of the Navy in 1974.
 
Nice designs! I remember the TattleTale. I was in the oceanographic domain about 40 years ago. I did the firmware for the 8011 Deck Unit in Forth. Out of curiosity I just looked and they still make it!

8011
 
slash2 said:
I reported a bad result using your code earlier here and it showed;

Teensyduino version 159
SdFat version 2.1.2

You had showed results that you had run:

Teensyduino version 160
SdFat version 2.1.2

so same sdfat version! So very odd.
Click to expand...
The SdFat version alone does not indicate whether you were using Bill Greiman's SdFat or Paul's Teensy-specific fork. When Paul forks a version, he makes changes to it, but doesn't change the SdFat version number. At one point you posted results that had a later version, so I knew you weren't using the SdFat contained in TeensyDuino.

It would be helpful if you would report results using my sketch. That's the only way we can compare apples to apples. I've probably already said this somewhere in this thread, but the starting point for my sketch was SdFat example TeensySdioLogger. The main difference relative to bench.ino is the use of isBusy() to avoid calling write() when the SD is busy. That is important in my applications, which are control systems that must execute the control loop and communicate with other devices on a schedule. The RingBuf allows data to be buffered when the SD is busy for those long periods (~40 ms), while still allowing the CPU to execute the control loop on schedule at 100 or 1000 or 10000 Hz, whatever it happens to be.
 
30 minute run:

36863999 writes in 1787.904 s (171.923 s writing to file = 9.616 %)
File is full
fileSize = 18874367488 before sync()
rb.bytesUsed = 512 before sync()
fileSize = 18874368000 after sync()
rb.bytesUsed = 0 after sync()
rbMaxUsed = 239104
avg write us = 4.66
min write us = 4.58
max write us = 7.59
avg busy us = 18.03
min busy us = 15.37
max busy us = 21087.28
file.close()

I ran my firmware on my board and got max buffer of: 572,680 (32msec)

I also ran my firmware on a standard T4.1 board. To make it work on a standard board I created a frequency output of 4.5Mhz and feed into my data clock input. That one gave me 2,398,152 bytes of buffer - not good.

I ran these a couple of more times and switched the SD Cards between them each time, both Sandisk; one is an Industrial V30 and the other is an Extreme Plus V30. The Extreme plus works on both, the Industrial fails on both.
 
Last edited:
slash2 said:
30 minute run:

36863999 writes in 1787.904 s (171.923 s writing to file = 9.616 %)
File is full
fileSize = 18874367488 before sync()
rb.bytesUsed = 512 before sync()
fileSize = 18874368000 after sync()
rb.bytesUsed = 0 after sync()
rbMaxUsed = 239104
avg write us = 4.66
min write us = 4.58
max write us = 7.59
avg busy us = 18.03
min busy us = 15.37
max busy us = 21087.28
file.close()

I ran my firmware on my board and got max buffer of: 572,680 (32msec)

I also ran it on a standard T4.1 board. To make it work on a standard board I created a frequency output of 4.5Mhz and feed into my data clock input. That one gave me 2,398,152 bytes of buffer - not good. I need to figure that one out.
Click to expand...

Thanks. For my sketch, that was a 30-minute run at 10 MB/s, right? Since your max busy is only 21 ms, you could change the line below to size the buffer for ~24 ms and try a short (1 minute) run at 18 MB/s. If it completes without overflowing the buffer, you would know you can run with code and data in TCM and have a good estimate of %CPU required for SD logging at 18 MB/s. The question after that would be what %CPU does your data processing require, and is there time for both.

Code: 
#define RING_BUF_SIZE    (DATA_RATE_BPS / 42)        // ~24-ms buffer at BPS
 
I ran both cards with your code. I had to make the buffer on the 18mb/s = (DATA_RATE_BPS / 84)
The card matters greatly, even though both are V30.


CardSD IndustrialSD Express PlusSD IndustrialSD Express Plus
Speed (mbytes/sec)10101818
Latency (usecs)115,07832,733139,08835,201
 
slash2 said:
I ran both cards with your code. I had to make the buffer on the 18mb/s = (DATA_RATE_BPS / 84)
The card matters greatly, even though both are V30.
Click to expand...

Thank you for these results. Seems like you can support your target 18 MB/s with the Extreme Plus card and your RAM1/RAM2 buffer.
 
You must log in or register to reply here.
Back
Top