Low-level direct IO for SD cards

Status
Not open for further replies.
R

RolandXX

Member
SD card read/writes are rather slow.
Since I just want to have a fast mass storage block device and I don't realy need a file system (e.g. FAT32), I could consider doing low level block based read/write accesses to the SD card.
Can someone help with the following questions:
- what is the expected speed-up with low level IO ?
- has someone a solution ?
- how to search for a solution ? Does is make sense to take the SD library as a starting point?
 
I don't think you can write code that is faster than the inbuilt sd controller with 4 bit parallel access which does it in hardware.

The bottleneck with SD is random adressing.
You can't make it faster with code. Its a SD inherent problem.
But you can preallocate files. The lib supports that.
 
Danke für die leider schlechte Nachricht.
Ich bin davon ausgegangen, dass der Filesystem-Overhead für die magere Transferrate verantwortlich ist.

Die "gefuehlte" Transferrate bei anderen Rechnern ist eigentlich aber sehr viel höher als "schlappe" 70KB/sec.
 
RolandXX said:
Danke für die leider schlechte Nachricht.
Ich bin davon ausgegangen, dass der Filesystem-Overhead für die magere Transferrate verantwortlich ist.

Die "gefuehlte" Transferrate bei anderen Rechnern ist eigentlich aber sehr viel höher als "schlappe" 70KB/sec.
Click to expand...

What are you saying?
 
Speed should be much faster than 70 kbytes/sec. Maybe you are doing something wrong?

You need to show us the problem. So far, you have not even said which Teensy you are using, which version of Teensyduino & Arduino, which SD card, or whether the SD card is connected by 1 bit SPI or 4 bit SDIO on the built in socket on Teensy 3.5, 3.6, 4.1.

On this forum, when a problem is reported, we generally expect you to show the complete program. The concept is a program anyone else on this forum can copy into Arduino and run on a Teensy to reproduce the same problem. Do NOT show only a fragment of your code. Give us the complete program so we can recreate the problem without guesswork!
 
I am using a Teensy 4.1 and a 32GB FAT32 formated micro SD card in the buildin socket.
I used the following test function to measure the transfer rate. The buffer size "nBytes" is 1kByte and I am writing howOften=10*1024 the buffer.
Something wrong ?

Code: 
[QUOTE]
int saveFile(char *buffer, int nBytes, int howOften) {
 
  File  myFile = SD.open("testfile.dat", FILE_WRITE);
  if (myFile) {
    myFile.seek(0);
    for (int i = 0; i < howOften; i++) {
      myFile.write(buffer, nBytes);
    }
    myFile.close();
  }
  return 1;
}[/QUOTE]
 
RolandXX said:
SD card read/writes are rather slow.
Click to expand...
It depends a lot on your application. I went the low level route with no file system because I needed higher write speeds.

When writing data to an SD card you get best performance when writing in large, and I do mean large, blocks. (See SD specification.) There is a library around that sneaks up on this by starting the write of a large block and then leaving the SD card hanging between 512 byte blocks until it is done. Great if you are only writing large chunks of data and doing nothing else.

In the past when I have measured write speed I store the current value from a hardware timer into a 512 byte buffer and write that buffer. Then I can read the data and process the timer data to measure the time between 512 byte blocks. For example:
single.gif
 
RolandXX said:
Something wrong ?
Click to expand...

You gave us a code fragment, not a complete program to easily reproduce the problem.

Please, post a complete program, so anyone can copy it into Arduino and run it on their Teensy to reproduce the slow performance. DO NOT make us guess the rest of the required code!
 
WMXZ said:
windows formatting works for me without issue. only if you want to format >32GB as Fat32 of <32GB as exFAT, one needs a more flexible tool.
Click to expand...

In the past there were issues with Windows-Format & SD. Maybe they fixed them?
 
I am using a Teensy 4.1 and use the buildin SD socket with a 32GB micro SD card.
Code: 
#include <SPI.h>
#include <SD.h>
int SDOkay = 0;
void setup() {
  Serial.begin(9600);
  SDOkay = SD.begin(BUILTIN_SDCARD);
}
#define BSIZE (1 * 1024)
#define N_TRANSFERS (10 * 1024)
#define M_BYTES (float)(BSIZE * N_TRANSFERS / (1024*1024))
char buf[BSIZE];

void loop() {
  int t;
  float bw;
  if (Serial.available()) {
          Serial.read();
          Serial.printf("starting file write with %f MBytes\n", M_BYTES);
          t = millis();
          saveFile(buf, BSIZE, N_TRANSFERS);
          t = millis() - t;
          bw = M_BYTES / (1e-3 * (float)t);
          Serial.printf("Time for transfer %d msec, BW = %f MByte/sec\n", t, bw);
  }
}
int saveFile(char *buffer, int nBytes, int howOften) {
  if (!SDOkay) {
    return 0;
  }
  File  myFile = SD.open("testfile.dat", FILE_WRITE);
  if (myFile) {
    myFile.seek(0);
    for (int i = 0; i < howOften; i++) {
      myFile.write(buffer, nBytes);
    }
    myFile.close();
  }
  return 1;
}
 
Thank you Paul for insisting on code that can be shared.
After extracting the file IO from other activities in a much more complex program, I got a factor of 10 increase in BW.
I need to check for some "back ground" IRQ's.
Here the code (measuring 780kByte/sec)

Code: 
#include <SPI.h>
#include <SD.h>
int SDOkay = 0;
void setup() {
  Serial.begin(9600);
  SDOkay = SD.begin(BUILTIN_SDCARD);
}
#define BSIZE (10 * 1024)
#define N_TRANSFERS (1 * 1024)
#define M_BYTES (float)(BSIZE * N_TRANSFERS / (1024*1024))
char buf[BSIZE];

void loop() {
  int t;
  float bw;
  if (Serial.available()) {
          Serial.read();
          Serial.printf("starting file write with %f MBytes\n", M_BYTES);
          t = millis();
          saveFile(buf, BSIZE, N_TRANSFERS);
          t = millis() - t;
          bw = M_BYTES / (1e-3 * (float)t);
          Serial.printf("Time for transfer %d msec, BW = %f MByte/sec\n", t, bw);
  }
}
int saveFile(char *buffer, int nBytes, int howOften) {
  if (!SDOkay) {
    return 0;
  }
  File  myFile = SD.open("testfile.dat", FILE_WRITE);
  if (myFile) {
    myFile.seek(0);
    for (int i = 0; i < howOften; i++) {
      myFile.write(buffer, nBytes);
    }
    myFile.close();
  }
  return 1;
}
 
One issue is that you don't measure the transfer speed but the sum of the time for open, seek, write and close. That's not valid.
If you do that in your real program, change it. Open once, write as much you need and close when you're ready.
And don't overwrite existing files.
 
RolandXX said:
Wow, this is great. What's wrong with my setup ?
Click to expand...

there is nothing wrong with your setup.
all is wrong with your sd card, it is most likely highly fragmented.
if not done, reformat card, preferable as exFAT, (FAT32 is slower than exFAT)
 
I experimented with your suggestions:
1. Formating with exFAT. Teensy didn't accepted the SD Card anymore. ?
2. Fresh FAT32 fomated card to remove fragmentation. Almost no change 810kB/sec instead of 790
3 Creating a new file instead of writing into an existing. 806kB/sec instead of 816kB/sec.

Another surprising effect is that if I run the test multple times, I get number from 580 up to 806 kB/sec
 
RolandXX said:
1. Formating with exFAT. Teensy didn't accepted the SD Card anymore. ?
Click to expand...
Are you sure you are using TD1.54? In other words are all earlier versions deleted?
Prior to TD1.54 SD library did not accept exFAT.
suggest, to remove all Arduino from Computer and re-install Arduino and Teensyduino
 
Status
Not open for further replies.
Back
Top