Slow to list 1000+ files

[Jeroen]

I'm building an audio project that has like one or two thousand RAW mono audio samples, each a second, max 5 seconds.

I just want to store all the filenames in an array on startup but with 1300 files that takes about a minute and half! That can't be right, right??

#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SD.h>
#include <SerialFlash.h>

AudioPlaySdRaw           playRaw1;
// Use one of these 3 output types: Digital I2S, Digital S/PDIF, or Analog DAC
AudioOutputI2S           audioOutput;
AudioConnection          patchCord1(playRaw1, 0, audioOutput, 0);
AudioConnection          patchCord2(playRaw1, 0, audioOutput, 1);
AudioControlSGTL5000     sgtl5000_1;

// Use these with the Teensy Audio Shield
#define SDCARD_CS_PIN    10
#define SDCARD_MOSI_PIN  7
#define SDCARD_SCK_PIN   14


char *samples[2000];
File fileRoot;
int fileCount;

void setup() {
  Serial.begin(9600);

  AudioMemory(8);

  sgtl5000_1.enable();
  sgtl5000_1.volume(0.5);

  SPI.setMOSI(SDCARD_MOSI_PIN);
  SPI.setSCK(SDCARD_SCK_PIN);
  if (!(SD.begin(SDCARD_CS_PIN))) {
    while (1) {
      Serial.println("Unable to access the SD card");
      delay(500);
    }
  }

  fileRoot = SD.open("/SAMPLES");
  int startS =  millis() / 1000;
  Serial.print(millis());
  readDB(fileRoot);
  int endS = millis() / 1000;

  Serial.print("files stored in array: ");
  Serial.println(fileCount, DEC);
  Serial.print("Took: ");
  Serial.println(endS-startS, DEC);
}


void readDB(File dir) {
  // int i = 0;
  while(true) {
    File entry = dir.openNextFile();

    if (!entry) {
      // no more files
      break;
    }
    fileCount ++;
    // i++;
    Serial.println(fileCount, DEC);

    entry.close();
  }
}

void loop() {
}

Also, it starts of kind of fast (still superslow if you ask me) but gets slower as the list grows longer.

Surely this cannot be a daunting task for the Teensy 4.0?

 

[Jeroen]

OK I went browsing through this: https://github.com/greiman/SdFat/tree/master/examples/TeensySdioDemo

And distilled an example down to:

/*
 * Example use of chdir(), ls(), mkdir(), and  rmdir().
 */
#include "SdFat.h"
#include "sdios.h"

// SD_FAT_TYPE = 0 for SdFat/File as defined in SdFatConfig.h,
// 1 for FAT16/FAT32, 2 for exFAT, 3 for FAT16/FAT32 and exFAT.
#define SD_FAT_TYPE 0

#ifndef SDCARD_SS_PIN
const uint8_t SD_CS_PIN = SS;
#else  // SDCARD_SS_PIN
// Assume built-in SD is used.
const uint8_t SD_CS_PIN = SDCARD_SS_PIN;
#endif  // SDCARD_SS_PIN

#if SD_FAT_TYPE == 0
SdFat sd;
File file;
File root;
#elif SD_FAT_TYPE == 1
SdFat32 sd;
File32 file;
File32 root;
#elif SD_FAT_TYPE == 2
SdExFat sd;
ExFile file;
ExFile root;
#elif SD_FAT_TYPE == 3
SdFs sd;
FsFile file;
FsFile root;
#endif  // SD_FAT_TYPE

// Create a Serial output stream.
ArduinoOutStream cout(Serial);
//------------------------------------------------------------------------------
// Store error strings in flash to save RAM.
#define error(s) sd.errorHalt(&Serial, F(s))
//------------------------------------------------------------------------------
void setup() {
  Serial.begin(9600);

  // Wait for USB Serial
  while (!Serial) {
    SysCall::yield();
  }
  delay(1000);

  cout << F("Type any character to start\n");
  while (!Serial.available()) {
    SysCall::yield();
  }

  // Initialize the SD card.
  if (!sd.begin(SdSpiConfig(SD_CS_PIN, DEDICATED_SPI, SD_SCK_MHZ(50)))) {
    Serial.println("AssBummer");
  }

  int rootFileCount = 0;
  if (!root.open("/")) {
    error("open root");
  }
  while (file.openNext(&root, O_RDONLY)) {
    if (!file.isHidden()) {
      rootFileCount++;
    }
    file.close();
  }

  cout << F("\nList of files on the SD.\n");
  sd.ls("/", LS_R);
}
//------------------------------------------------------------------------------
// Nothing happens in loop.
void loop() {}

Not sure what the best mode is for the SD card slot on the Teensy Audio board but this works

sd.begin(SdSpiConfig(SD_CS_PIN, DEDICATED_SPI, SD_SCK_MHZ(50)));

The same file listing now completes in like I don't even measured it but a blink of an eye

I assume that storing it in an array won't slow things down, but I'll report back here if it does

 

[Jeroen]

Let me first stress that I am a n00b at Arduino programming.

So now I have trouble getting SDFAT library to work alongside Teensy libs.

#include <Audio.h>

This already throws errors:

n file included from /Applications/Teensyduino.app/Contents/Java/hardware/teensy/avr/libraries/Audio/play_sd_raw.h:32:0,
                 from /Applications/Teensyduino.app/Contents/Java/hardware/teensy/avr/libraries/Audio/Audio.h:124,
                 from /Users/jeroen/Code/arduino/sdfatio/sdfatio.ino:2:
/Applications/Teensyduino.app/Contents/Java/hardware/teensy/avr/libraries/SD/SD.h:26:0: warning: "FILE_READ" redefined
 #define FILE_READ O_READ
 ^
In file included from /Users/jeroen/Code/arduino/libraries/SdFat/src/ExFatLib/ExFatFile.h:797:0,
                 from /Users/jeroen/Code/arduino/libraries/SdFat/src/ExFatLib/ExFatVolume.h:28,
                 from /Users/jeroen/Code/arduino/libraries/SdFat/src/ExFatLib/ExFatLib.h:27,
                 from /Users/jeroen/Code/arduino/libraries/SdFat/src/SdFat.h:33,
                 from /Users/jeroen/Code/arduino/sdfatio/sdfatio.ino:1:
...

Multiple libraries were found for "SD.h"
 Used: /Applications/Teensyduino.app/Contents/Java/hardware/teensy/avr/libraries/SD
 Not used: /Applications/Teensyduino.app/Contents/Java/libraries/SD
Error compiling for board Teensy 4.0.

 

[Jeroen]

Nice 1 man show I got going on here

OK well now I moved to teensyduino beta 7, used the exact same code of my original post.

Finishes instantly, MASSIVE SPEEDUP

p.s. I'm, sure I'll be back to talk to myself some more, stay tuned everybody!!

 

[Frank B]

happens... there a lot of threads/ posts where I answer my questions... and lot without any answer..

Worse is when you talk to someone, he answeres, then you ask a question, and no answer comes. I hate that deeply.
Everthing is better than that. Yes, a "i don't know" or "i dont want to answer that" or just "No" is way better.

 

[Jeroen]

Thanks Frank! we are not alone

 

[PaulStoffregen]

So now I have trouble getting SDFAT library to work alongside Teensy libs.

Install 1.54-beta7.

https://forum.pjrc.com/threads/66357-Teensyduino-1-54-Beta-7

We're now using SdFat for everything. The old slow SD lib was replaced by a thin wrapper which just uses SdFat.


Also, you might consider lowering the SPI clock speed. I believe 25 MHz is the official limit for the SD card spec (in SPI mode) and 30 MHz is the official limit for Teensy. Faster speeds usually work, but technically they're considered overclocking.

 

[Jeroen]

Thanks Paul

I did that (see above) and now I'm just using standard Teensy example code. I don't see any speeed settings in that so I'm guessing you use the default (25? 30? Mhz) setting in the beta code.