Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 14 of 14

Thread: Serial1.addMemoryForRead Buffer usage expands as I increase buffer size

  1. #1
    Member
    Join Date
    Jul 2019
    Location
    United Kingdom
    Posts
    52

    Serial1.addMemoryForRead Buffer usage expands as I increase buffer size

    I'm trying to receive a 300,000 bytes SysEx message over midi with a Teensy 3.6 and to save it to the onboard SD card.
    Thanks for Paul's suggestion to use Serial1.addMemoryForRead(serial1buffer, sizeOfSerial1buffer); and to write in small chunks.
    This is a lot better but I'm getting strange results for Serial1.available().
    If I set an unsigned char serial1buffer[2000]; then my reported Serial1.available() is invariably either small (eg 50 or large eg 1900, just below the available buffer. Sometimes Serial1.available() is greater than 2063 and I miss data. So I increase the size... If I change to serial1buffer[10000]; then the available is either small eg 50 or large eg 9999, again just below the available buffer! Does the SD card writing know how much buffer is available and put its feet up for a rest until the buffer is nearly full? No matter how big I make the buffer, it just uses more of it and still misses data occasionally! What am I doing wrong?
    Thanks

    1.8.13, Teensyduino 1.53. I'm using the core library from github from 12th Jan.



    Code:
    #include <SD.h>
    #include <Adafruit_CharacterOLED.h>//include the OLED library 
    #define SD_CS BUILTIN_SDCARD
    
    const int errorReporting = 0;
    Sd2Card card;
    int dumpByteCount;
    int chunk[6100];
    unsigned char serial1buffer[2000];
    int sizeOfSerial1buffer;
    int chunkIndex = 0; int chunkNumber = 0;
    int chunkSize = 1200;
    //******************************************************************************  setup *************************************************
    void setup() {
      Serial.begin(9600);//diagnostic serial port
      Serial1.begin(31250);
      delay(100);
      sizeOfSerial1buffer = sizeof(serial1buffer);
      Serial1.addMemoryForRead(serial1buffer, sizeOfSerial1buffer);
    
      if (!SD.begin(SD_CS)) {
        Serial.println("Initialization of SD card failed");
      }
    
      if (!card.init(SPI_HALF_SPEED, SD_CS)) {
        Serial.println("Card initialization failed.");  while (true) {}; // freeze
      } //infinite loop}
      else {
        Serial.println("SD card is present  ");
      }
    
      removeStreamFiles();
    }
    
    void loop() {
      if (Serial1.available()) {
        chunk[chunkIndex] = Serial1.read();
    
        if (chunkIndex == chunkSize - 1) {
          chunkIndex = -1;
          Serial.println(" ");
          Serial.print("Before call "); Serial.println(Serial1.available());
          writeChunkToSD(chunkNumber);
          chunkNumber ++;
        }
    
        chunkIndex++;
      }
    }
    
    void writeChunkToSD(int index) {
      char filename[30];
      String fname = "Temp" + String(index) + ".dat";
      char fullPathName[30];
      fname.toCharArray(filename, 16);
      snprintf(fullPathName, 16, "%s",  filename);
    
      File dataFile = SD.open(fullPathName, FILE_WRITE);
      Serial.print("Before write "); Serial.println(Serial1.available());
      for (int k = 0; k < chunkSize; k++) {
        dataFile.print(String(chunk[k]));
        dataFile.print(",");
      }
      dataFile.close();
      Serial.print("After write "); Serial.println(Serial1.available());
    }
    
    void removeStreamFiles() {
      char filename[30];
      for (int i = 0; i < 1000; i++) {
        String fname = "Temp" + String(i) + ".dat";
        fname.toCharArray(filename, 16);
        Serial.print("Checking "); Serial.println(filename);
        if (SD.exists(filename)) {
          Serial.print("Removing ");
          Serial.println(filename);
          SD.remove(filename);
        }
      }
    }
    Example output with 2000 size:

    Before call 0

    Before write 42

    After write 2023



    Before call 0

    Before write 40

    After write 74



    Before call 1

    Before write 43

    After write 75



    Before call 0

    Before write 42

    After write 1883



    Before call 1

    Before write 43

    After write 76



    Before call 0

    Before write 41

    After write 75



    Before call 0

    Before write 42

    After write 1890



    Before call 1

    Before write 47

    After write 81



    Before call 0

    Before write 42

    After write 75



    Before call 1

    Before write 1851

    After write 1884



    Before call 1

    Before write 41

    After write 75



    Before call 0

    Before write 42

    After write 76



    Before call 1

    Before write 1851

    After write 1884



    HTML Code:
    C:\Program Files (x86)\Arduino\arduino-builder -dump-prefs -logger=machine -hardware C:\Program Files (x86)\Arduino\hardware -hardware C:\Users\Steve\AppData\Local\Arduino15\packages -hardware F:\Documents\Arduino\hardware -tools C:\Program Files (x86)\Arduino\tools-builder -tools C:\Program Files (x86)\Arduino\hardware\tools\avr -tools C:\Users\Steve\AppData\Local\Arduino15\packages -built-in-libraries C:\Program Files (x86)\Arduino\libraries -libraries F:\Documents\Arduino\libraries -fqbn=teensy:avr:teensy36:usb=serial,speed=180,opt=o2std,keys=en-gb -ide-version=10813 -build-path C:\Users\Steve\AppData\Local\Temp\arduino_build_871834 -warnings=none -build-cache C:\Users\Steve\AppData\Local\Temp\arduino_cache_46956 -verbose F:\Documents\Arduino\LFE\Utilities\SysEx_Dump_2demo\SysEx_Dump_2demo.ino
    C:\Program Files (x86)\Arduino\arduino-builder -compile -logger=machine -hardware C:\Program Files (x86)\Arduino\hardware -hardware C:\Users\Steve\AppData\Local\Arduino15\packages -hardware F:\Documents\Arduino\hardware -tools C:\Program Files (x86)\Arduino\tools-builder -tools C:\Program Files (x86)\Arduino\hardware\tools\avr -tools C:\Users\Steve\AppData\Local\Arduino15\packages -built-in-libraries C:\Program Files (x86)\Arduino\libraries -libraries F:\Documents\Arduino\libraries -fqbn=teensy:avr:teensy36:usb=serial,speed=180,opt=o2std,keys=en-gb -ide-version=10813 -build-path C:\Users\Steve\AppData\Local\Temp\arduino_build_871834 -warnings=none -build-cache C:\Users\Steve\AppData\Local\Temp\arduino_cache_46956 -verbose F:\Documents\Arduino\LFE\Utilities\SysEx_Dump_2demo\SysEx_Dump_2demo.ino
    Using board 'teensy36' from platform in folder: C:\Program Files (x86)\Arduino\hardware\teensy\avr
    Using core 'teensy3' from platform in folder: C:\Program Files (x86)\Arduino\hardware\teensy\avr
    Detecting libraries used...
    "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-g++" -E -CC -x c++ -w -g -Wall -ffunction-sections -fdata-sections -nostdlib -fno-exceptions -fpermissive -felide-constructors -std=gnu++14 -Wno-error=narrowing -fno-rtti -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -D__MK66FX1M0__ -DTEENSYDUINO=153 -DARDUINO=10813 -DARDUINO_TEENSY36 -DF_CPU=180000000 -DUSB_SERIAL -DLAYOUT_UNITED_KINGDOM "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\cores\\teensy3" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\sketch\\SysEx_Dump_2demo.ino.cpp" -o nul -DARDUINO_LIB_DISCOVERY_PHASE
    Alternatives for SD.h: [SD@1.2.4 SD@1.2.2]
    ResolveLibrary(SD.h)
      -> candidates: [SD@1.2.4 SD@1.2.2]
    "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-g++" -E -CC -x c++ -w -g -Wall -ffunction-sections -fdata-sections -nostdlib -fno-exceptions -fpermissive -felide-constructors -std=gnu++14 -Wno-error=narrowing -fno-rtti -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -D__MK66FX1M0__ -DTEENSYDUINO=153 -DARDUINO=10813 -DARDUINO_TEENSY36 -DF_CPU=180000000 -DUSB_SERIAL -DLAYOUT_UNITED_KINGDOM "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\cores\\teensy3" "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\libraries\\SD" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\sketch\\SysEx_Dump_2demo.ino.cpp" -o nul -DARDUINO_LIB_DISCOVERY_PHASE
    Alternatives for Adafruit_CharacterOLED.h: [Adafruit_CharacterOLED-master]
    ResolveLibrary(Adafruit_CharacterOLED.h)
      -> candidates: [Adafruit_CharacterOLED-master]
    "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-g++" -E -CC -x c++ -w -g -Wall -ffunction-sections -fdata-sections -nostdlib -fno-exceptions -fpermissive -felide-constructors -std=gnu++14 -Wno-error=narrowing -fno-rtti -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -D__MK66FX1M0__ -DTEENSYDUINO=153 -DARDUINO=10813 -DARDUINO_TEENSY36 -DF_CPU=180000000 -DUSB_SERIAL -DLAYOUT_UNITED_KINGDOM "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\cores\\teensy3" "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\libraries\\SD" "-IF:\\Documents\\Arduino\\libraries\\Adafruit_CharacterOLED-master" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\sketch\\SysEx_Dump_2demo.ino.cpp" -o nul -DARDUINO_LIB_DISCOVERY_PHASE
    Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\File.cpp
    Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\SD.cpp
    Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\cache_t3.cpp
    Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\card_t3.cpp
    Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\dir_t3.cpp
    Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\fat_t3.cpp
    Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\file_t3.cpp
    Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\init_t3.cpp
    Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\utility\NXP_SDHC.cpp
    Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\utility\Sd2Card.cpp
    Alternatives for SPI.h: [SPI@1.0]
    ResolveLibrary(SPI.h)
      -> candidates: [SPI@1.0]
    Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\utility\SdFile.cpp
    Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\utility\SdVolume.cpp
    Using cached library dependencies for file: F:\Documents\Arduino\libraries\Adafruit_CharacterOLED-master\Adafruit_CharacterOLED.cpp
    Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SPI\SPI.cpp
    Generating function prototypes...
    "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-g++" -E -CC -x c++ -w -g -Wall -ffunction-sections -fdata-sections -nostdlib -fno-exceptions -fpermissive -felide-constructors -std=gnu++14 -Wno-error=narrowing -fno-rtti -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -D__MK66FX1M0__ -DTEENSYDUINO=153 -DARDUINO=10813 -DARDUINO_TEENSY36 -DF_CPU=180000000 -DUSB_SERIAL -DLAYOUT_UNITED_KINGDOM "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\cores\\teensy3" "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\libraries\\SD" "-IF:\\Documents\\Arduino\\libraries\\Adafruit_CharacterOLED-master" "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\libraries\\SPI" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\sketch\\SysEx_Dump_2demo.ino.cpp" -o "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\preproc\\ctags_target_for_gcc_minus_e.cpp" -DARDUINO_LIB_DISCOVERY_PHASE
    "C:\\Program Files (x86)\\Arduino\\tools-builder\\ctags\\5.8-arduino11/ctags" -u --language-force=c++ -f - --c++-kinds=svpf --fields=KSTtzns --line-directives "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\preproc\\ctags_target_for_gcc_minus_e.cpp"
    Compiling sketch...
    "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/precompile_helper" "C:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr/cores/teensy3" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834" "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-g++" -x c++-header -O2 -g -Wall -ffunction-sections -fdata-sections -nostdlib -MMD -fno-exceptions -fpermissive -felide-constructors -std=gnu++14 -Wno-error=narrowing -fno-rtti -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -D__MK66FX1M0__ -DTEENSYDUINO=153 -DARDUINO=10813 -DARDUINO_TEENSY36 -DF_CPU=180000000 -DUSB_SERIAL -DLAYOUT_UNITED_KINGDOM "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr/cores/teensy3" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/pch/Arduino.h" -o "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/pch/Arduino.h.gch"
    Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\pch\Arduino.h.gch
    "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-g++" -c -O2 -g -Wall -ffunction-sections -fdata-sections -nostdlib -MMD -fno-exceptions -fpermissive -felide-constructors -std=gnu++14 -Wno-error=narrowing -fno-rtti -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -D__MK66FX1M0__ -DTEENSYDUINO=153 -DARDUINO=10813 -DARDUINO_TEENSY36 -DF_CPU=180000000 -DUSB_SERIAL -DLAYOUT_UNITED_KINGDOM "-IC:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/pch" "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\cores\\teensy3" "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\libraries\\SD" "-IF:\\Documents\\Arduino\\libraries\\Adafruit_CharacterOLED-master" "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\libraries\\SPI" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\sketch\\SysEx_Dump_2demo.ino.cpp" -o "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\sketch\\SysEx_Dump_2demo.ino.cpp.o"
    Compiling libraries...
    Compiling library "SD"
    Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\cache_t3.cpp.o
    Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\dir_t3.cpp.o
    Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\file_t3.cpp.o
    Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\fat_t3.cpp.o
    Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\card_t3.cpp.o
    Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\init_t3.cpp.o
    Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\SD.cpp.o
    Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\File.cpp.o
    Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\utility\Sd2Card.cpp.o
    Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\utility\NXP_SDHC.cpp.o
    Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\utility\SdFile.cpp.o
    Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\utility\SdVolume.cpp.o
    Compiling library "Adafruit_CharacterOLED-master"
    Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\Adafruit_CharacterOLED-master\Adafruit_CharacterOLED.cpp.o
    Compiling library "SPI"
    Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SPI\SPI.cpp.o
    Compiling core...
    Using precompiled core: C:\Users\Steve\AppData\Local\Temp\arduino_cache_46956\core\core_2d97c2c6649b57b15124d16c7952f707.a
    Linking everything together...
    "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-gcc" -O2 -Wl,--gc-sections,--relax,--defsym=__rtc_localtime=1614852616 "-TC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\cores\\teensy3/mk66fx1m0.ld" -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -o "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.elf" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\sketch\\SysEx_Dump_2demo.ino.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\File.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\SD.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\cache_t3.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\card_t3.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\dir_t3.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\fat_t3.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\file_t3.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\init_t3.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\utility\\NXP_SDHC.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\utility\\Sd2Card.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\utility\\SdFile.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\utility\\SdVolume.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\Adafruit_CharacterOLED-master\\Adafruit_CharacterOLED.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SPI\\SPI.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/..\\arduino_cache_46956\\core\\core_2d97c2c6649b57b15124d16c7952f707.a" "-LC:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834" -larm_cortexM4lf_math -lm -lstdc++
    "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-objcopy" -O ihex -j .eeprom --set-section-flags=.eeprom=alloc,load --no-change-warnings --change-section-lma .eeprom=0 "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.elf" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.eep"
    "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-objcopy" -O ihex -R .eeprom "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.elf" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.hex"
    "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/stdout_redirect" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.lst" "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-objdump" -d -S -C "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.elf"
    "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/stdout_redirect" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.sym" "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-objdump" -t -C "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.elf"
    "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/teensy_post_compile" -file=SysEx_Dump_2demo.ino "-path=C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834" "-tools=C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/" -board=TEENSY36
    Multiple libraries were found for "SD.h"
     Used: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD
     Not used: C:\Program Files (x86)\Arduino\libraries\SD
    Using library SD at version 1.2.2 in folder: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD 
    Using library Adafruit_CharacterOLED-master in folder: F:\Documents\Arduino\libraries\Adafruit_CharacterOLED-master (legacy)
    Using library SPI at version 1.0 in folder: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SPI 
    "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-size" -A "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.elf"
    Sketch uses 46036 bytes (4%) of program storage space. Maximum is 1048576 bytes.
    Global variables use 31892 bytes (12%) of dynamic memory, leaving 230252 bytes for local variables. Maximum is 262144 bytes.

  2. #2
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    24,000
    Just to confirm, which version of Teensyduino do you have installed? In Arduino, click Help > About to check.

    Serial1.addMemoryForRead() is relatively new for Teensy 3.x.

  3. #3
    Member
    Join Date
    Jul 2019
    Location
    United Kingdom
    Posts
    52
    About reports: 1.8.13, Teensyduino 1.53. I just installed the latest core from github and no change.

    Sometimes it seems to be the writing that's filling the buffer (big jump between Before write and After write) and sometimes it's the SD.open that's filling the buffer (big jump between Before call and Before write)

    With unsigned char serial1buffer[20000];
    Before call 0
    Before write 19850
    After write 19979


    Before call 0
    Before write 43
    After write 167



    Before call 1
    Before write 41
    After write 19979

    Before call 1
    Before write 43
    After write 169



    Before call 1
    Before write 43
    After write 19981


    Before call 1
    Before write 43
    After write 171

    Before call 1
    Before write 43
    After write 19976

    Before call 1
    Before write 19851
    After write 19980

    Before call 1
    Before write 43
    After write 19978
    Last edited by sw_hunt; 03-04-2021 at 11:44 AM.

  4. #4
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    8,230
    Please notice, that your chunk size of 1200 is not optimal - a block is 512 bytes - so (2*512) would be better. Not sure how Bills library handles that. Perhaps it has its own buffer, so your chunk size may be ok.
    Then, writing a new file is very time intensive.The library needs to scan the File allocation table, add a new entry, etc etc before the file can be written.
    Same for opening an existing file. For speed, the best is to open the file once, and then simply append data, and to close when everything is done.
    Adressing a block on SD takes time.

  5. #5
    Member
    Join Date
    Jul 2019
    Location
    United Kingdom
    Posts
    52
    Thanks Frank B. Noted re block size. I believe I'm removing all files before the main loop starts so all file opens are new files, so I'm not opening an existing file. Once a file is opened, I do this and then close it, never to open it again in this sketch:

    File dataFile = SD.open(fullPathName, FILE_WRITE);
    Serial.print("Before write "); Serial.println(Serial1.available());
    for (int k = 0; k < chunkSize; k++) {
    dataFile.print(String(chunk[k]));
    dataFile.print(",");
    }
    dataFile.close();

    Sorry if I missed your point?

  6. #6
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    8,230
    I said writing a new file is slow. Or, more exactly, opening a new file for writing is slow.

    Edit: Opening to append data to a file is slow, too.
    Best and fastest is to open and close a file once, outside a loop, if possible.
    May be not be possible in every case, but you get the idea..
    Last edited by Frank B; 03-04-2021 at 12:32 PM.

  7. #7
    Member
    Join Date
    Jul 2019
    Location
    United Kingdom
    Posts
    52
    Right, so the answer is to not keep opening files! So I'm now opening the file once and dumping the whole 300k bytes into it as it comes in, then closing it. Then I can read in chunks from the file at my leisure using file.position() Works a treat Thanks Frank B.

  8. #8
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    8,891
    I know playing around with MTP and the like, I believe if you do all writes in units of sectors or clusters, it can run a whole lot faster.
    For example I know it was a lot faster with 4K blocks than when I dumped USB blocks in with their data as it came in (about 500 bytes first one and 512 after)... So you might try that.

    Again the addBuffer for serial stuff on T3.x has only been added for the most recent beta releases.

    I am not sure if there are other viable options for some of this.

    Things like: I don't remember if the SD code will call yield when waiting for a write to complete? If so you can play around with some different ways to try to read data from the Serial port.
    Things like: SerialEvent1
    Which gets called when anything calls yield and there is unread data in the Serial1 read buffer. You could then simply read the data and move to some big buffer, maybe circular/ring buffer...

    Or maybe Interval Timer which you enable when you are going to do a write and it checks and moves data...

    But first thing I would do is to optimize the write sizes and see if that takes care of it for you...

    Also I have not played enough with SD stuff here to know if it helps or not.
    But on some systems, you can sometimes do things like: You know that your file is going to be 300KB in size.
    So when you create the file, you try to get the file to be fully allocated. Again I have not played enough here, to know if I tell the file to seek to that position or the like, and then seek back to zero, if that
    will pre-allocate all of the clusters for you...

  9. #9
    Member
    Join Date
    Jul 2019
    Location
    United Kingdom
    Posts
    52
    Thanks KurtE. Interesting idea about pre-allocating the file.

  10. #10
    Member
    Join Date
    Jul 2019
    Location
    United Kingdom
    Posts
    52
    Is there a better way to pre-allocate disk space than this? Some writes (prints) still take over 50ms. Something quicker (this takes 10 seconds) or something that guarantees quick prints of 512 bytes by reducing the amount of housekeeping that SD has to do?

    char filename[10] = "Temp.dat";
    File dataFile = SD.open(filename, FILE_WRITE);
    for (int i = 0; i < (4194304); i++){ // 4MB
    dataFile.print(" ");
    }
    dataFile.flush();
    dataFile.seek(0);

  11. #11
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    13,756
    @sw_hunt: Is the current Beta TD 1.54 Beta 7 in use?

    It has changed the SD code to defer to the SdFat code that seems to benchmark some 5 or more times faster on writes. Benchmark as seen showed 10MB /sec - depending on card and write method

  12. #12
    Member
    Join Date
    Jul 2019
    Location
    United Kingdom
    Posts
    52
    No beta, I'm using Teensyduino 1.53 with the latest core from github. I looked at the beta thread and there seemed to be lots of issues still so I've not risked it.

    I confess I really don't understand the different versions of SD and what I'm using. The compiler report in the first post mentions several alternatives but I don't know how to force it to use the right one. The IDE tells me there is a later version of SdFat available (2.0.5 vs current 2.0.2) but I know that if I let it install, another sketch of mine will stop working! That other sketch starts:

    #include <MTP.h>

    MTPStorage_SD storage;
    MTPD mtpd(&storage);

    Sd2Card card;

    Maybe I should start a thread to ask how to resolve that issue? Then I can install the latest SD for the sketch I'm having slow writes with.
    Last edited by sw_hunt; 03-07-2021 at 12:27 PM. Reason: corrected IDE update comment

  13. #13
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    8,891
    Again I am not as experienced with the FAT system in SDFat and the like, although I understand a lot more of it than I used to. And I am pretty good at using a global search function in sublime text.

    In the old Dos/Windows days, there were ways to ask the system to preallocate a size for a file. This was done when for example maybe your program or other programs used multiple files and for speed you wanted to pre-allocate the space and also maybe try to do it without fragmentation...

    Looks like there are functions for this: Like: If you look at some of the SDFat examples:
    Code:
      if (!binFile.open(binName, O_RDWR | O_CREAT)) {
        error("open binName failed");
      }
      Serial.print(F("Allocating: "));
      Serial.print(MAX_FILE_SIZE_MiB);
      Serial.println(F(" MiB"));
      if (!binFile.preAllocate(MAX_FILE_SIZE)) {
        error("preAllocate failed");
      }
    @PaulStoffregen - Wonder if the preAllocate should be added to FS.h?

  14. #14
    Member
    Join Date
    Jul 2019
    Location
    United Kingdom
    Posts
    52
    I tried updating SdFat to 2.0.5 and now my other MTP.h sketch works, so I guess something in the latest update or in the latest core has fixed it. Anyway with SdFat 2.0.5, my sketch still has a few slow writes. The sketch is using SD, but did I read somewhere that that in turn uses SdFat?

    I've also been experimenting with different values of Serial1.addMemoryForRead(serial1buffer, sizeof(serial1buffer));
    Having it commented out or having a value > 192 is a lot worse. 64 or 128 is best.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •