Serial1.addMemoryForRead Buffer usage expands as I increase buffer size

Status
Not open for further replies.

sw_hunt

Well-known member
I'm trying to receive a 300,000 bytes SysEx message over midi with a Teensy 3.6 and to save it to the onboard SD card.
Thanks for Paul's suggestion to use Serial1.addMemoryForRead(serial1buffer, sizeOfSerial1buffer); and to write in small chunks.
This is a lot better but I'm getting strange results for Serial1.available().
If I set an unsigned char serial1buffer[2000]; then my reported Serial1.available() is invariably either small (eg 50 or large eg 1900, just below the available buffer. Sometimes Serial1.available() is greater than 2063 and I miss data. So I increase the size... If I change to serial1buffer[10000]; then the available is either small eg 50 or large eg 9999, again just below the available buffer! Does the SD card writing know how much buffer is available and put its feet up for a rest until the buffer is nearly full? No matter how big I make the buffer, it just uses more of it and still misses data occasionally! What am I doing wrong?
Thanks

1.8.13, Teensyduino 1.53. I'm using the core library from github from 12th Jan.



Code:
#include <SD.h>
#include <Adafruit_CharacterOLED.h>//include the OLED library 
#define SD_CS BUILTIN_SDCARD

const int errorReporting = 0;
Sd2Card card;
int dumpByteCount;
int chunk[6100];
unsigned char serial1buffer[2000];
int sizeOfSerial1buffer;
int chunkIndex = 0; int chunkNumber = 0;
int chunkSize = 1200;
//******************************************************************************  setup *************************************************
void setup() {
  Serial.begin(9600);//diagnostic serial port
  Serial1.begin(31250);
  delay(100);
  sizeOfSerial1buffer = sizeof(serial1buffer);
  Serial1.addMemoryForRead(serial1buffer, sizeOfSerial1buffer);

  if (!SD.begin(SD_CS)) {
    Serial.println("Initialization of SD card failed");
  }

  if (!card.init(SPI_HALF_SPEED, SD_CS)) {
    Serial.println("Card initialization failed.");  while (true) {}; // freeze
  } //infinite loop}
  else {
    Serial.println("SD card is present  ");
  }

  removeStreamFiles();
}

void loop() {
  if (Serial1.available()) {
    chunk[chunkIndex] = Serial1.read();

    if (chunkIndex == chunkSize - 1) {
      chunkIndex = -1;
      Serial.println(" ");
      Serial.print("Before call "); Serial.println(Serial1.available());
      writeChunkToSD(chunkNumber);
      chunkNumber ++;
    }

    chunkIndex++;
  }
}

void writeChunkToSD(int index) {
  char filename[30];
  String fname = "Temp" + String(index) + ".dat";
  char fullPathName[30];
  fname.toCharArray(filename, 16);
  snprintf(fullPathName, 16, "%s",  filename);

  File dataFile = SD.open(fullPathName, FILE_WRITE);
  Serial.print("Before write "); Serial.println(Serial1.available());
  for (int k = 0; k < chunkSize; k++) {
    dataFile.print(String(chunk[k]));
    dataFile.print(",");
  }
  dataFile.close();
  Serial.print("After write "); Serial.println(Serial1.available());
}

void removeStreamFiles() {
  char filename[30];
  for (int i = 0; i < 1000; i++) {
    String fname = "Temp" + String(i) + ".dat";
    fname.toCharArray(filename, 16);
    Serial.print("Checking "); Serial.println(filename);
    if (SD.exists(filename)) {
      Serial.print("Removing ");
      Serial.println(filename);
      SD.remove(filename);
    }
  }
}

Example output with 2000 size:

Before call 0

Before write 42

After write 2023



Before call 0

Before write 40

After write 74



Before call 1

Before write 43

After write 75



Before call 0

Before write 42

After write 1883



Before call 1

Before write 43

After write 76



Before call 0

Before write 41

After write 75



Before call 0

Before write 42

After write 1890



Before call 1

Before write 47

After write 81



Before call 0

Before write 42

After write 75



Before call 1

Before write 1851

After write 1884



Before call 1

Before write 41

After write 75



Before call 0

Before write 42

After write 76



Before call 1

Before write 1851

After write 1884



HTML:
C:\Program Files (x86)\Arduino\arduino-builder -dump-prefs -logger=machine -hardware C:\Program Files (x86)\Arduino\hardware -hardware C:\Users\Steve\AppData\Local\Arduino15\packages -hardware F:\Documents\Arduino\hardware -tools C:\Program Files (x86)\Arduino\tools-builder -tools C:\Program Files (x86)\Arduino\hardware\tools\avr -tools C:\Users\Steve\AppData\Local\Arduino15\packages -built-in-libraries C:\Program Files (x86)\Arduino\libraries -libraries F:\Documents\Arduino\libraries -fqbn=teensy:avr:teensy36:usb=serial,speed=180,opt=o2std,keys=en-gb -ide-version=10813 -build-path C:\Users\Steve\AppData\Local\Temp\arduino_build_871834 -warnings=none -build-cache C:\Users\Steve\AppData\Local\Temp\arduino_cache_46956 -verbose F:\Documents\Arduino\LFE\Utilities\SysEx_Dump_2demo\SysEx_Dump_2demo.ino
C:\Program Files (x86)\Arduino\arduino-builder -compile -logger=machine -hardware C:\Program Files (x86)\Arduino\hardware -hardware C:\Users\Steve\AppData\Local\Arduino15\packages -hardware F:\Documents\Arduino\hardware -tools C:\Program Files (x86)\Arduino\tools-builder -tools C:\Program Files (x86)\Arduino\hardware\tools\avr -tools C:\Users\Steve\AppData\Local\Arduino15\packages -built-in-libraries C:\Program Files (x86)\Arduino\libraries -libraries F:\Documents\Arduino\libraries -fqbn=teensy:avr:teensy36:usb=serial,speed=180,opt=o2std,keys=en-gb -ide-version=10813 -build-path C:\Users\Steve\AppData\Local\Temp\arduino_build_871834 -warnings=none -build-cache C:\Users\Steve\AppData\Local\Temp\arduino_cache_46956 -verbose F:\Documents\Arduino\LFE\Utilities\SysEx_Dump_2demo\SysEx_Dump_2demo.ino
Using board 'teensy36' from platform in folder: C:\Program Files (x86)\Arduino\hardware\teensy\avr
Using core 'teensy3' from platform in folder: C:\Program Files (x86)\Arduino\hardware\teensy\avr
Detecting libraries used...
"C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-g++" -E -CC -x c++ -w -g -Wall -ffunction-sections -fdata-sections -nostdlib -fno-exceptions -fpermissive -felide-constructors -std=gnu++14 -Wno-error=narrowing -fno-rtti -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -D__MK66FX1M0__ -DTEENSYDUINO=153 -DARDUINO=10813 -DARDUINO_TEENSY36 -DF_CPU=180000000 -DUSB_SERIAL -DLAYOUT_UNITED_KINGDOM "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\cores\\teensy3" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\sketch\\SysEx_Dump_2demo.ino.cpp" -o nul -DARDUINO_LIB_DISCOVERY_PHASE
Alternatives for SD.h: [SD@1.2.4 SD@1.2.2]
ResolveLibrary(SD.h)
  -> candidates: [SD@1.2.4 SD@1.2.2]
"C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-g++" -E -CC -x c++ -w -g -Wall -ffunction-sections -fdata-sections -nostdlib -fno-exceptions -fpermissive -felide-constructors -std=gnu++14 -Wno-error=narrowing -fno-rtti -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -D__MK66FX1M0__ -DTEENSYDUINO=153 -DARDUINO=10813 -DARDUINO_TEENSY36 -DF_CPU=180000000 -DUSB_SERIAL -DLAYOUT_UNITED_KINGDOM "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\cores\\teensy3" "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\libraries\\SD" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\sketch\\SysEx_Dump_2demo.ino.cpp" -o nul -DARDUINO_LIB_DISCOVERY_PHASE
Alternatives for Adafruit_CharacterOLED.h: [Adafruit_CharacterOLED-master]
ResolveLibrary(Adafruit_CharacterOLED.h)
  -> candidates: [Adafruit_CharacterOLED-master]
"C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-g++" -E -CC -x c++ -w -g -Wall -ffunction-sections -fdata-sections -nostdlib -fno-exceptions -fpermissive -felide-constructors -std=gnu++14 -Wno-error=narrowing -fno-rtti -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -D__MK66FX1M0__ -DTEENSYDUINO=153 -DARDUINO=10813 -DARDUINO_TEENSY36 -DF_CPU=180000000 -DUSB_SERIAL -DLAYOUT_UNITED_KINGDOM "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\cores\\teensy3" "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\libraries\\SD" "-IF:\\Documents\\Arduino\\libraries\\Adafruit_CharacterOLED-master" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\sketch\\SysEx_Dump_2demo.ino.cpp" -o nul -DARDUINO_LIB_DISCOVERY_PHASE
Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\File.cpp
Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\SD.cpp
Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\cache_t3.cpp
Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\card_t3.cpp
Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\dir_t3.cpp
Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\fat_t3.cpp
Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\file_t3.cpp
Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\init_t3.cpp
Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\utility\NXP_SDHC.cpp
Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\utility\Sd2Card.cpp
Alternatives for SPI.h: [SPI@1.0]
ResolveLibrary(SPI.h)
  -> candidates: [SPI@1.0]
Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\utility\SdFile.cpp
Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD\utility\SdVolume.cpp
Using cached library dependencies for file: F:\Documents\Arduino\libraries\Adafruit_CharacterOLED-master\Adafruit_CharacterOLED.cpp
Using cached library dependencies for file: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SPI\SPI.cpp
Generating function prototypes...
"C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-g++" -E -CC -x c++ -w -g -Wall -ffunction-sections -fdata-sections -nostdlib -fno-exceptions -fpermissive -felide-constructors -std=gnu++14 -Wno-error=narrowing -fno-rtti -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -D__MK66FX1M0__ -DTEENSYDUINO=153 -DARDUINO=10813 -DARDUINO_TEENSY36 -DF_CPU=180000000 -DUSB_SERIAL -DLAYOUT_UNITED_KINGDOM "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\cores\\teensy3" "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\libraries\\SD" "-IF:\\Documents\\Arduino\\libraries\\Adafruit_CharacterOLED-master" "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\libraries\\SPI" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\sketch\\SysEx_Dump_2demo.ino.cpp" -o "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\preproc\\ctags_target_for_gcc_minus_e.cpp" -DARDUINO_LIB_DISCOVERY_PHASE
"C:\\Program Files (x86)\\Arduino\\tools-builder\\ctags\\5.8-arduino11/ctags" -u --language-force=c++ -f - --c++-kinds=svpf --fields=KSTtzns --line-directives "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\preproc\\ctags_target_for_gcc_minus_e.cpp"
Compiling sketch...
"C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/precompile_helper" "C:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr/cores/teensy3" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834" "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-g++" -x c++-header -O2 -g -Wall -ffunction-sections -fdata-sections -nostdlib -MMD -fno-exceptions -fpermissive -felide-constructors -std=gnu++14 -Wno-error=narrowing -fno-rtti -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -D__MK66FX1M0__ -DTEENSYDUINO=153 -DARDUINO=10813 -DARDUINO_TEENSY36 -DF_CPU=180000000 -DUSB_SERIAL -DLAYOUT_UNITED_KINGDOM "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr/cores/teensy3" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/pch/Arduino.h" -o "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/pch/Arduino.h.gch"
Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\pch\Arduino.h.gch
"C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-g++" -c -O2 -g -Wall -ffunction-sections -fdata-sections -nostdlib -MMD -fno-exceptions -fpermissive -felide-constructors -std=gnu++14 -Wno-error=narrowing -fno-rtti -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -D__MK66FX1M0__ -DTEENSYDUINO=153 -DARDUINO=10813 -DARDUINO_TEENSY36 -DF_CPU=180000000 -DUSB_SERIAL -DLAYOUT_UNITED_KINGDOM "-IC:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/pch" "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\cores\\teensy3" "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\libraries\\SD" "-IF:\\Documents\\Arduino\\libraries\\Adafruit_CharacterOLED-master" "-IC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\libraries\\SPI" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\sketch\\SysEx_Dump_2demo.ino.cpp" -o "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\sketch\\SysEx_Dump_2demo.ino.cpp.o"
Compiling libraries...
Compiling library "SD"
Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\cache_t3.cpp.o
Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\dir_t3.cpp.o
Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\file_t3.cpp.o
Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\fat_t3.cpp.o
Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\card_t3.cpp.o
Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\init_t3.cpp.o
Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\SD.cpp.o
Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\File.cpp.o
Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\utility\Sd2Card.cpp.o
Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\utility\NXP_SDHC.cpp.o
Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\utility\SdFile.cpp.o
Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SD\utility\SdVolume.cpp.o
Compiling library "Adafruit_CharacterOLED-master"
Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\Adafruit_CharacterOLED-master\Adafruit_CharacterOLED.cpp.o
Compiling library "SPI"
Using previously compiled file: C:\Users\Steve\AppData\Local\Temp\arduino_build_871834\libraries\SPI\SPI.cpp.o
Compiling core...
Using precompiled core: C:\Users\Steve\AppData\Local\Temp\arduino_cache_46956\core\core_2d97c2c6649b57b15124d16c7952f707.a
Linking everything together...
"C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-gcc" -O2 -Wl,--gc-sections,--relax,--defsym=__rtc_localtime=1614852616 "-TC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\cores\\teensy3/mk66fx1m0.ld" -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -o "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.elf" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\sketch\\SysEx_Dump_2demo.ino.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\File.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\SD.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\cache_t3.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\card_t3.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\dir_t3.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\fat_t3.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\file_t3.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\init_t3.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\utility\\NXP_SDHC.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\utility\\Sd2Card.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\utility\\SdFile.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SD\\utility\\SdVolume.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\Adafruit_CharacterOLED-master\\Adafruit_CharacterOLED.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834\\libraries\\SPI\\SPI.cpp.o" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/..\\arduino_cache_46956\\core\\core_2d97c2c6649b57b15124d16c7952f707.a" "-LC:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834" -larm_cortexM4lf_math -lm -lstdc++
"C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-objcopy" -O ihex -j .eeprom --set-section-flags=.eeprom=alloc,load --no-change-warnings --change-section-lma .eeprom=0 "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.elf" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.eep"
"C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-objcopy" -O ihex -R .eeprom "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.elf" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.hex"
"C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/stdout_redirect" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.lst" "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-objdump" -d -S -C "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.elf"
"C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/stdout_redirect" "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.sym" "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-objdump" -t -C "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.elf"
"C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/teensy_post_compile" -file=SysEx_Dump_2demo.ino "-path=C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834" "-tools=C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/" -board=TEENSY36
Multiple libraries were found for "SD.h"
 Used: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD
 Not used: C:\Program Files (x86)\Arduino\libraries\SD
Using library SD at version 1.2.2 in folder: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SD 
Using library Adafruit_CharacterOLED-master in folder: F:\Documents\Arduino\libraries\Adafruit_CharacterOLED-master (legacy)
Using library SPI at version 1.0 in folder: C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SPI 
"C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-size" -A "C:\\Users\\Steve\\AppData\\Local\\Temp\\arduino_build_871834/SysEx_Dump_2demo.ino.elf"
Sketch uses 46036 bytes (4%) of program storage space. Maximum is 1048576 bytes.
Global variables use 31892 bytes (12%) of dynamic memory, leaving 230252 bytes for local variables. Maximum is 262144 bytes.
 
Just to confirm, which version of Teensyduino do you have installed? In Arduino, click Help > About to check.

Serial1.addMemoryForRead() is relatively new for Teensy 3.x.
 
About reports: 1.8.13, Teensyduino 1.53. I just installed the latest core from github and no change.

Sometimes it seems to be the writing that's filling the buffer (big jump between Before write and After write) and sometimes it's the SD.open that's filling the buffer (big jump between Before call and Before write)

With unsigned char serial1buffer[20000];
Before call 0
Before write 19850
After write 19979


Before call 0
Before write 43
After write 167



Before call 1
Before write 41
After write 19979

Before call 1
Before write 43
After write 169



Before call 1
Before write 43
After write 19981


Before call 1
Before write 43
After write 171

Before call 1
Before write 43
After write 19976

Before call 1
Before write 19851
After write 19980

Before call 1
Before write 43
After write 19978
 
Last edited:
Please notice, that your chunk size of 1200 is not optimal - a block is 512 bytes - so (2*512) would be better. Not sure how Bills library handles that. Perhaps it has its own buffer, so your chunk size may be ok.
Then, writing a new file is very time intensive.The library needs to scan the File allocation table, add a new entry, etc etc before the file can be written.
Same for opening an existing file. For speed, the best is to open the file once, and then simply append data, and to close when everything is done.
Adressing a block on SD takes time.
 
Thanks Frank B. Noted re block size. I believe I'm removing all files before the main loop starts so all file opens are new files, so I'm not opening an existing file. Once a file is opened, I do this and then close it, never to open it again in this sketch:

File dataFile = SD.open(fullPathName, FILE_WRITE);
Serial.print("Before write "); Serial.println(Serial1.available());
for (int k = 0; k < chunkSize; k++) {
dataFile.print(String(chunk[k]));
dataFile.print(",");
}
dataFile.close();

Sorry if I missed your point?
 
I said writing a new file is slow. Or, more exactly, opening a new file for writing is slow.

Edit: Opening to append data to a file is slow, too.
Best and fastest is to open and close a file once, outside a loop, if possible.
May be not be possible in every case, but you get the idea..
 
Last edited:
Right, so the answer is to not keep opening files! So I'm now opening the file once and dumping the whole 300k bytes into it as it comes in, then closing it. Then I can read in chunks from the file at my leisure using file.position() Works a treat :) Thanks Frank B.
 
I know playing around with MTP and the like, I believe if you do all writes in units of sectors or clusters, it can run a whole lot faster.
For example I know it was a lot faster with 4K blocks than when I dumped USB blocks in with their data as it came in (about 500 bytes first one and 512 after)... So you might try that.

Again the addBuffer for serial stuff on T3.x has only been added for the most recent beta releases.

I am not sure if there are other viable options for some of this.

Things like: I don't remember if the SD code will call yield when waiting for a write to complete? If so you can play around with some different ways to try to read data from the Serial port.
Things like: SerialEvent1
Which gets called when anything calls yield and there is unread data in the Serial1 read buffer. You could then simply read the data and move to some big buffer, maybe circular/ring buffer...

Or maybe Interval Timer which you enable when you are going to do a write and it checks and moves data...

But first thing I would do is to optimize the write sizes and see if that takes care of it for you...

Also I have not played enough with SD stuff here to know if it helps or not.
But on some systems, you can sometimes do things like: You know that your file is going to be 300KB in size.
So when you create the file, you try to get the file to be fully allocated. Again I have not played enough here, to know if I tell the file to seek to that position or the like, and then seek back to zero, if that
will pre-allocate all of the clusters for you...
 
Is there a better way to pre-allocate disk space than this? Some writes (prints) still take over 50ms. Something quicker (this takes 10 seconds) or something that guarantees quick prints of 512 bytes by reducing the amount of housekeeping that SD has to do?

char filename[10] = "Temp.dat";
File dataFile = SD.open(filename, FILE_WRITE);
for (int i = 0; i < (4194304); i++){ // 4MB
dataFile.print(" ");
}
dataFile.flush();
dataFile.seek(0);
 
@sw_hunt: Is the current Beta TD 1.54 Beta 7 in use?

It has changed the SD code to defer to the SdFat code that seems to benchmark some 5 or more times faster on writes. Benchmark as seen showed 10MB /sec - depending on card and write method
 
No beta, I'm using Teensyduino 1.53 with the latest core from github. I looked at the beta thread and there seemed to be lots of issues still so I've not risked it.

I confess I really don't understand the different versions of SD and what I'm using. The compiler report in the first post mentions several alternatives but I don't know how to force it to use the right one. The IDE tells me there is a later version of SdFat available (2.0.5 vs current 2.0.2) but I know that if I let it install, another sketch of mine will stop working! That other sketch starts:

#include <MTP.h>

MTPStorage_SD storage;
MTPD mtpd(&storage);

Sd2Card card;

Maybe I should start a thread to ask how to resolve that issue? Then I can install the latest SD for the sketch I'm having slow writes with.
 
Last edited:
Again I am not as experienced with the FAT system in SDFat and the like, although I understand a lot more of it than I used to. And I am pretty good at using a global search function in sublime text.

In the old Dos/Windows days, there were ways to ask the system to preallocate a size for a file. This was done when for example maybe your program or other programs used multiple files and for speed you wanted to pre-allocate the space and also maybe try to do it without fragmentation...

Looks like there are functions for this: Like: If you look at some of the SDFat examples:
Code:
  if (!binFile.open(binName, O_RDWR | O_CREAT)) {
    error("open binName failed");
  }
  Serial.print(F("Allocating: "));
  Serial.print(MAX_FILE_SIZE_MiB);
  Serial.println(F(" MiB"));
  [COLOR="#FF0000"]if (!binFile.preAllocate(MAX_FILE_SIZE)[/COLOR]) {
    error("preAllocate failed");
  }
@PaulStoffregen - Wonder if the preAllocate should be added to FS.h?
 
I tried updating SdFat to 2.0.5 and now my other MTP.h sketch works, so I guess something in the latest update or in the latest core has fixed it. Anyway with SdFat 2.0.5, my sketch still has a few slow writes. The sketch is using SD, but did I read somewhere that that in turn uses SdFat?

I've also been experimenting with different values of Serial1.addMemoryForRead(serial1buffer, sizeof(serial1buffer));
Having it commented out or having a value > 192 is a lot worse. 64 or 128 is best.
 
Hi all,

Would LOVE to use the addMemoryforRead functionality, because only one of my projects requires a rather large receive buffer.
Since I am not using the Arduino IDE for projects with more than 100 lines of code,
would just copying the teensy3 core from github to the cores directory in .platformio do the trick?

Alternatively, is PJRC or platformio the party responsible for updating the teensy cores in platform ?

Thanks
 
Hi all,

Would LOVE to use the addMemoryforRead functionality, because only one of my projects requires a rather large receive buffer.
Since I am not using the Arduino IDE for projects with more than 100 lines of code,
would just copying the teensy3 core from github to the cores directory in .platformio do the trick?

Alternatively, is PJRC or platformio the party responsible for updating the teensy cores in platform ?

Thanks

Its Platformio, and it already has the newest Version.
 
First time I ever used the command "pio package update" ... who knew?
(I assumed the monthly automatic updates took care of all the extras too)

Had to manually remove the tools-teensy folder for some reason. But it works now.

Thanks a million Frank.
 
Status
Not open for further replies.
Back
Top