Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 8 of 8

Thread: Slow to list 1000+ files

  1. #1
    Member
    Join Date
    Feb 2021
    Location
    netherlands
    Posts
    28

    Slow to list 1000+ files

    I'm building an audio project that has like one or two thousand RAW mono audio samples, each a second, max 5 seconds.

    I just want to store all the filenames in an array on startup but with 1300 files that takes about a minute and half! That can't be right, right??

    Code:
    #include <Audio.h>
    #include <Wire.h>
    #include <SPI.h>
    #include <SD.h>
    #include <SerialFlash.h>
    
    AudioPlaySdRaw           playRaw1;
    // Use one of these 3 output types: Digital I2S, Digital S/PDIF, or Analog DAC
    AudioOutputI2S           audioOutput;
    AudioConnection          patchCord1(playRaw1, 0, audioOutput, 0);
    AudioConnection          patchCord2(playRaw1, 0, audioOutput, 1);
    AudioControlSGTL5000     sgtl5000_1;
    
    // Use these with the Teensy Audio Shield
    #define SDCARD_CS_PIN    10
    #define SDCARD_MOSI_PIN  7
    #define SDCARD_SCK_PIN   14
    
    
    char *samples[2000];
    File fileRoot;
    int fileCount;
    
    void setup() {
      Serial.begin(9600);
    
      AudioMemory(8);
    
      sgtl5000_1.enable();
      sgtl5000_1.volume(0.5);
    
      SPI.setMOSI(SDCARD_MOSI_PIN);
      SPI.setSCK(SDCARD_SCK_PIN);
      if (!(SD.begin(SDCARD_CS_PIN))) {
        while (1) {
          Serial.println("Unable to access the SD card");
          delay(500);
        }
      }
    
      fileRoot = SD.open("/SAMPLES");
      int startS =  millis() / 1000;
      Serial.print(millis());
      readDB(fileRoot);
      int endS = millis() / 1000;
    
      Serial.print("files stored in array: ");
      Serial.println(fileCount, DEC);
      Serial.print("Took: ");
      Serial.println(endS-startS, DEC);
    }
    
    
    void readDB(File dir) {
      // int i = 0;
      while(true) {
        File entry = dir.openNextFile();
    
        if (!entry) {
          // no more files
          break;
        }
        fileCount ++;
        // i++;
        Serial.println(fileCount, DEC);
    
        entry.close();
      }
    }
    
    void loop() {
    }
    Also, it starts of kind of fast (still superslow if you ask me) but gets slower as the list grows longer.

    Surely this cannot be a daunting task for the Teensy 4.0?

  2. #2
    Member
    Join Date
    Feb 2021
    Location
    netherlands
    Posts
    28
    OK I went browsing through this: https://github.com/greiman/SdFat/tre...TeensySdioDemo

    And distilled an example down to:

    Code:
    /*
     * Example use of chdir(), ls(), mkdir(), and  rmdir().
     */
    #include "SdFat.h"
    #include "sdios.h"
    
    // SD_FAT_TYPE = 0 for SdFat/File as defined in SdFatConfig.h,
    // 1 for FAT16/FAT32, 2 for exFAT, 3 for FAT16/FAT32 and exFAT.
    #define SD_FAT_TYPE 0
    
    #ifndef SDCARD_SS_PIN
    const uint8_t SD_CS_PIN = SS;
    #else  // SDCARD_SS_PIN
    // Assume built-in SD is used.
    const uint8_t SD_CS_PIN = SDCARD_SS_PIN;
    #endif  // SDCARD_SS_PIN
    
    #if SD_FAT_TYPE == 0
    SdFat sd;
    File file;
    File root;
    #elif SD_FAT_TYPE == 1
    SdFat32 sd;
    File32 file;
    File32 root;
    #elif SD_FAT_TYPE == 2
    SdExFat sd;
    ExFile file;
    ExFile root;
    #elif SD_FAT_TYPE == 3
    SdFs sd;
    FsFile file;
    FsFile root;
    #endif  // SD_FAT_TYPE
    
    // Create a Serial output stream.
    ArduinoOutStream cout(Serial);
    //------------------------------------------------------------------------------
    // Store error strings in flash to save RAM.
    #define error(s) sd.errorHalt(&Serial, F(s))
    //------------------------------------------------------------------------------
    void setup() {
      Serial.begin(9600);
    
      // Wait for USB Serial
      while (!Serial) {
        SysCall::yield();
      }
      delay(1000);
    
      cout << F("Type any character to start\n");
      while (!Serial.available()) {
        SysCall::yield();
      }
    
      // Initialize the SD card.
      if (!sd.begin(SdSpiConfig(SD_CS_PIN, DEDICATED_SPI, SD_SCK_MHZ(50)))) {
        Serial.println("AssBummer");
      }
    
      int rootFileCount = 0;
      if (!root.open("/")) {
        error("open root");
      }
      while (file.openNext(&root, O_RDONLY)) {
        if (!file.isHidden()) {
          rootFileCount++;
        }
        file.close();
      }
    
      cout << F("\nList of files on the SD.\n");
      sd.ls("/", LS_R);
    }
    //------------------------------------------------------------------------------
    // Nothing happens in loop.
    void loop() {}
    Not sure what the best mode is for the SD card slot on the Teensy Audio board but this works

    Code:
    sd.begin(SdSpiConfig(SD_CS_PIN, DEDICATED_SPI, SD_SCK_MHZ(50)));
    The same file listing now completes in like I don't even measured it but a blink of an eye

    I assume that storing it in an array won't slow things down, but I'll report back here if it does :P

  3. #3
    Member
    Join Date
    Feb 2021
    Location
    netherlands
    Posts
    28
    Let me first stress that I am a n00b at Arduino programming.

    So now I have trouble getting SDFAT library to work alongside Teensy libs.

    #include <Audio.h>

    This already throws errors:



    Code:
    n file included from /Applications/Teensyduino.app/Contents/Java/hardware/teensy/avr/libraries/Audio/play_sd_raw.h:32:0,
                     from /Applications/Teensyduino.app/Contents/Java/hardware/teensy/avr/libraries/Audio/Audio.h:124,
                     from /Users/jeroen/Code/arduino/sdfatio/sdfatio.ino:2:
    /Applications/Teensyduino.app/Contents/Java/hardware/teensy/avr/libraries/SD/SD.h:26:0: warning: "FILE_READ" redefined
     #define FILE_READ O_READ
     ^
    In file included from /Users/jeroen/Code/arduino/libraries/SdFat/src/ExFatLib/ExFatFile.h:797:0,
                     from /Users/jeroen/Code/arduino/libraries/SdFat/src/ExFatLib/ExFatVolume.h:28,
                     from /Users/jeroen/Code/arduino/libraries/SdFat/src/ExFatLib/ExFatLib.h:27,
                     from /Users/jeroen/Code/arduino/libraries/SdFat/src/SdFat.h:33,
                     from /Users/jeroen/Code/arduino/sdfatio/sdfatio.ino:1:
    ...
    
    Multiple libraries were found for "SD.h"
     Used: /Applications/Teensyduino.app/Contents/Java/hardware/teensy/avr/libraries/SD
     Not used: /Applications/Teensyduino.app/Contents/Java/libraries/SD
    Error compiling for board Teensy 4.0.

  4. #4
    Member
    Join Date
    Feb 2021
    Location
    netherlands
    Posts
    28
    Nice 1 man show I got going on here

    OK well now I moved to teensyduino beta 7, used the exact same code of my original post.

    Finishes instantly, MASSIVE SPEEDUP

    p.s. I'm, sure I'll be back to talk to myself some more, stay tuned everybody!!

  5. #5
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    8,282
    happens... there a lot of threads/ posts where I answer my questions... and lot without any answer..

    Worse is when you talk to someone, he answeres, then you ask a question, and no answer comes. I hate that deeply.
    Everthing is better than that. Yes, a "i don't know" or "i dont want to answer that" or just "No" is way better.

  6. #6
    Member
    Join Date
    Feb 2021
    Location
    netherlands
    Posts
    28
    Thanks Frank! we are not alone

  7. #7
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    24,096
    Quote Originally Posted by Jeroen View Post
    So now I have trouble getting SDFAT library to work alongside Teensy libs.
    Install 1.54-beta7.

    https://forum.pjrc.com/threads/66357...no-1-54-Beta-7

    We're now using SdFat for everything. The old slow SD lib was replaced by a thin wrapper which just uses SdFat.


    Also, you might consider lowering the SPI clock speed. I believe 25 MHz is the official limit for the SD card spec (in SPI mode) and 30 MHz is the official limit for Teensy. Faster speeds usually work, but technically they're considered overclocking.

  8. #8
    Member
    Join Date
    Feb 2021
    Location
    netherlands
    Posts
    28
    Thanks Paul

    I did that (see above) and now I'm just using standard Teensy example code. I don't see any speeed settings in that so I'm guessing you use the default (25? 30? Mhz) setting in the beta code.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •