Serial characters mysteriously missing

Status
Not open for further replies.

CWashburn

Active member
Hello all,

I'm working on a portable GPS logger based on the T3.6, but I've been stuck on a weird issue I've been having regarding missing characters.

My code reads raw NMEA data from the GPS, stores and stores it in a 256-byte buffer. When the buffer is full, it dumps the contents into a .txt file an SD card. The problem is, it will randomly skip characters, sometimes only 2-3, other times whole sentences. What makes this more interesting is that if I comment out all of the data-logging code, I no longer have this issue.

My initial reaction was to add a 4700uF capacitor to the VDD line (overkill, I know :D) in case current spikes during the SD write-cycle somehow caused the GPS to glitch out, but that didn't seem to fix the problem. For what it's worth, I'm using a PNY High Performance 32GB card in the T3.6's builtin slot. Although this could very well be the problem, my gut tells me it isn't the case.

Now I'm wondering if the SD write routine has a higher interrupt priority than the Serial read buffer, which would cause it to completely miss chunks of data.

I have a few theories as to what it could be, but I would like a fresh set of eyes to check my code and make sure there isn't something I missed.


Here's my code.

Code:
#include <SD.h>
#include <SPI.h>


File file;
const int BUFSIZE = 256;
char charbuf[BUFSIZE];
int buflevel = 0;

const int led1 = 24;
const int led2 = 26;


void setup(){

  pinMode(led1, OUTPUT);
  digitalWrite(led1, HIGH);
  pinMode(led2, OUTPUT);
  digitalWrite(led2, HIGH);
  
  Serial.begin(115200);
  Serial4.begin(115200);  //GPS device
  while(!Serial);

  Serial.print("Initializing SD card...");

  while(!SD.begin(BUILTIN_SDCARD)) {
    Serial.println("initialization failed!");
    digitalWrite(led2, LOW);
    delay(750);
    digitalWrite(led2, HIGH);
    delay(750);
  }
  Serial.println("initialization done.");

  if(SD.exists("NMEA.txt"))
    SD.remove("NMEA.txt");  
  delay(100);
  file = SD.open("NMEA.txt", FILE_WRITE);
  
  if(file){
    Serial.print("Writing to NMEA.txt...");
    Serial.println("done.");
    file.close();
  }
  else{
    Serial.println("error opening NMEA.txt");
  }  
}



void loop(){
  
  while(Serial4.available()){  //if a new character is available

    charbuf[buflevel] = Serial4.read(); //copy it into the buffer

    //if(charbuf[buflevel] == '$'){ //dump buffer contents to SD at the beginning of each NMEA sentence
    if(buflevel == BUFSIZE-1){ //dump buffer contents to SD when the buffer is full   

           
      file = SD.open("NMEA.txt", FILE_WRITE);
    
      if(file){    
        digitalWrite(led1, LOW);      //turn on led
        file.write(charbuf, buflevel);   //write buffer data to SD
        file.close();
        Serial.write(charbuf, buflevel); //dump buffer contents to Serial monitor as well        
        digitalWrite(led1, HIGH);     //turn off led
        buflevel = 0;                 //reset buffer level to zero
      } 
      else{
        Serial.println("error opening NMEA.txt");
      }      
    }
    
    buflevel++;
  }
}



And here's an example of the data I'm getting.

Capture.PNG
 
You probably want to use Serial1 or Serial2 as these have 8 byte hardware fifos.

From: https://www.pjrc.com/teensy/td_uart.html:

On Teensy 3.2, 3.5, 3.6, Serial1 and Serial2 have 8 byte transmit and receive FIFOs, which allow for higher speed baud rates, even when other libraries create interrupt latency.

All serial ports on all Teensy boards use interrupt-based transmit and receive buffering, to allow your program to avoid waiting when writing short messages, and to allow data to be reliably received even if your program must spend time performing other tasks.

On Teensy 3.6 when running faster than 120 MHz, writing to EEPROM may momentarily interfere with Serial1 and Serial2 baud rates. Reading EEPROM does not interfere. Serial3 to Serial6 are not affected.

If your GPS supports hardware flow control, you may want to enable it as well. See the link above for more details.
 
You probably want to use Serial1 or Serial2 as these have 8 byte hardware fifos.

Ok, that makes sense.

If your GPS supports hardware flow control, you may want to enable it as well.

Can you clarify what you mean? I'm just using your run-of-the-mill NEO-6M GPS module.
 
Ok, that makes sense.

Can you clarify what you mean? I'm just using your run-of-the-mill NEO-6M GPS module.

The Neo 6M GPSes I saw via a quick search don't have RTS/CTS.

RTS/CTS (request to send, clear to send) is what is called hardware flow control, where the hardware indicates whether it is safe to send the bytes down the line. There are two different methods of using CTS/RTS (https://www.lammertbies.nl/comm/info/RS-232_flow_control.html).

Teensy uses the first method described where the RTS of the Teensy is connected to the CTS of the device, and vice versa. If the buffer is full, Teensy says slow down, I can't receive any more data. The device sending the data is supposed to look at this before sending data.
 
IIRC doing block writes of 512 bytes is more efficient - it will fill a block in one write without the need to have the card manage half write/read/rewrite.

That might speed the logging.

Moving to Serial 1 or 2 would be best to maximize hardware and minimize needed interrupts.

And possibly attempts could be made to read data from serial buffer between steps in the log process to make sure the Rcv buffer doesn't overflow when a long pause for SD write happens.

Data rate at 115200 is slow enough - last I used was 460800 with no loss on Serial2, with attention to detail - though did extend the hardware serial buffer to 128 Bytes IIRC because the system would hang long enough to cause loss doing calcs at the wrong time with a 100+ byte message. Not sure what string length of the messages here is - or the repeat rate 1, 5 or 10 Hz?

Suggest perhaps using charbuf to full twice then transferring to logbuf[512] - then complete buffer empty Serial read to charbuff before starting the logbuf[512] write. Or just make two [512] buffers and toggle between read and log with them.

If charbuf was all but full when a new message arrived - remainder would be partially read and waiting during the log process - this is probably the case now.

Also with the above double buffering - if there was interrupt and loss/contention during logging - you could time from the end of the last received message and know how long before the next would start 100 or 200 ms. Then do the logging then to avoid data loss. I wrote a serial start bit detection _isr() and some long stuff would be delayed as the transfer was about 5 ms and the calcs were about 750ms or longer on a T_3.5.

My GEO GPS ran from Teensy's 3.3V power without trouble - another on the same code was seeing some power issue and wired otherwise - that was not using the SD card in either case.
 
Suggest perhaps using charbuf to full twice then transferring to logbuf[512] - then complete buffer empty Serial read to charbuff before starting the logbuf[512] write. Or just make two [512] buffers and toggle between read and log with them.

If charbuf was all but full when a new message arrived - remainder would be partially read and waiting during the log process - this is probably the case now.
Not only "perhaps"
serious data logging should always be done with double/multiple buffering.
Rule #1: never ever allow data acquisition to be influenced by down-stream activity.
Rule #2: think about and always handle situations, where data processing (here logging) does not keep up with acquisition

IMO, defragster is right, that logging latencies (up to over 100 ms) interfered in your example with the capability of the serial port.
 
Not only "perhaps"
serious data logging should always be done with double/multiple buffering.
...

Indeed - I went back and edited in 'perhaps' for emphasis [it caught your eye] - but it was with the raised eyebrow and knowing smile modifier keys pressed as I typed.
 
Try editing serial4.c. If your install used the Windows defaults, it'll be in C:\Program Files (x86)\Arduino\hardware\teensy\avr\cores\teensy3. Look for these lines:

Code:
#ifndef SERIAL4_TX_BUFFER_SIZE
#define SERIAL4_TX_BUFFER_SIZE     40 // number of outgoing bytes to buffer
#endif
#ifndef SERIAL4_RX_BUFFER_SIZE
#define SERIAL4_RX_BUFFER_SIZE     64 // number of incoming bytes to buffer
#endif

Try increasing SERIAL4_RX_BUFFER_SIZE, maybe several hundred bytes instead of just 64. Does that help?
 
Thanks guys for all the input.

The repeat rate is 1Hz, and I turned off GGA and GSV sentences. I didn't know that you could change the size of the hardware serial buffers. Does it use DMA to capture the data or is it actually hardware?

Suggest perhaps using charbuf to full twice then transferring to logbuf[512] - then complete buffer empty Serial read to charbuff before starting the logbuf[512] write. Or just make two [512] buffers and toggle between read and log with them.

I just "randomly" picked a buffer size. I didn't know that a 512B buffer was the most efficient.


Also with the above double buffering - if there was interrupt and loss/contention during logging - you could time from the end of the last received message and know how long before the next would start 100 or 200 ms. Then do the logging then to avoid data loss.

I was experimenting with that, (logging each sentence as it came in), but I would only get one sentence. Now I know why. :D I think I may just try that.

Try editing serial4.c.

I did increase the RX buffer size to something crazy like 1024, but didn't see any difference.
 
I did increase the RX buffer size to something crazy like 1024, but didn't see any difference.

What happens if you comment out the writing to SD card, but keep the printing of everything to the serial monitor?

In other words, is the SD card latency really a factor? Since a bigger RX buffer didn't help, maybe the real problem is somewhere else? Maybe how the wiring or signals somehow?
 
what happens if you slow the GPS baud down to 4800?
are you sure your changes to SERIAL4_RX_BUFFER_SIZE were actually compiled into the sketch you were running? On T3.6 builtin SD card, my file io timing of open/write(256)/close takes about 5 ms, though one write took 110 ms. At 115200 baud, a single character takes 86 us on the serial line, if you stall for 5 ms doing file I/O, then your Rx recv buffer needs to be at least 58 characters. SD open/write/close times can vary if block allocations are required. There are several threads discussing fast logging to SD card ...
https://forum.pjrc.com/threads/43834-Real-low-latency-logging-for-Teensy-3-5-3-6-SDIO-SD
pre-allocated, pre-erased contiguous file
 
Last edited:
What makes this more interesting is that if I comment out all of the data-logging code, I no longer have this issue.
This statement from OP triggered my understanding that it is the SD write process that interferes with acquisition.
Did you try to use a freshly formatted disk?

BTW, the code
Code:
 file = SD.open("NMEA.txt", FILE_WRITE);
does it append to an existing file?
 
Like @WMXZ the OP suggested all was well when the SD write was removed. Post #5 offered a range of solutions possible - depending on time lost during the SD write completion 1,2 or 3 of them together might allow it to function: Use Serial 1/2, buffer swap and complete read to empty buffer before going to SD write, time SD write to happen when GPS not transmitting from 2nd buffer, with 512 B 2nd buffer do the write of that size each time from start to perhaps keep SD from stalling as long.

I have a GPS online with a T_3.6 and complete sketch was provided so I could attempt those changes given feedback if needed.
 
It works! I changed it so that it only logs to the SD card after 50 ms of no serial activity.

Code:
#include <SD.h>
#include <SPI.h>


File file;
const int BUFSIZE = 512;
char charbuf[BUFSIZE];
uint16_t buflevel = 0;

uint32_t ms = 0;
bool newblock = false;

const int led1 = 24;
const int led2 = 26;


void setup(){

  pinMode(led1, OUTPUT);
  digitalWrite(led1, HIGH);
  pinMode(led2, OUTPUT);
  digitalWrite(led2, HIGH);
  
  Serial.begin(115200);
  Serial4.begin(115200);  //GPS device
  while(!Serial);

  Serial.print("Initializing SD card...");

  while(!SD.begin(BUILTIN_SDCARD)) {
    Serial.println("initialization failed!");
    digitalWrite(led2, LOW);
    delay(750);
    digitalWrite(led2, HIGH);
    delay(750);
  }
  Serial.println("initialization done.");

  if(SD.exists("NMEA.txt"))
    SD.remove("NMEA.txt");  
  delay(100);
  file = SD.open("NMEA.txt", FILE_WRITE);
  
  if(file){
    Serial.print("Writing to NMEA.txt...");
    Serial.println("done.");
    file.close();
  }
  else{
    Serial.println("error opening NMEA.txt");
  }  
}



void loop(){
  
  while(Serial4.available()){             //if a new character is available

    ms = millis();                        //reset counter
    newblock = true;                      //new character block flag
    charbuf[buflevel++] = Serial4.read(); //copy character into buffer
  }

  if(millis() - ms >= 50 && newblock){

    newblock = false;                  //reset the flag
       
    file = SD.open("NMEA.txt", FILE_WRITE); //open file
  
    if(file){    
      digitalWrite(led1, LOW);         //turn on led
      file.write(charbuf, buflevel);   //write buffer data to SD
      file.close();
      Serial.write(charbuf, buflevel); //dump buffer contents to Serial monitor as well        
      digitalWrite(led1, HIGH);        //turn off led
      buflevel = 0;                    //reset buffer level to zero
    } 
    else{
      Serial.println("error opening NMEA.txt");
    }      
  }
}
 
Cool! One nice thing about the GPS strings is they arrive on a fixed interval so the time between is free for use without incoming data.

50 ms is generous on the wait - when the pause is more than 2 ms the last transfer is complete at 11,520 bytes/sec [actually under 0.09 ms]. A big SD lag could make use of that extra time.

It should be more robust and efficient with the double buffer - writing full 512 byte block.
 
I'm very surprised that the buffer size increase didn't help, but this solves the problem. The SD lib isn't supposed to disable serial interrupts, so a large buffer should have allowed capturing the incoming bytes. Very mysterious....
 
UART DMA

FWIW, here is a sketch that does UART receive ping-pong DMA with DMA interrupt on half-done and done, 512 byte chunks.
https://github.com/manitou48/teensy3/blob/master/uartdma.ino

limited testing on T3.6 Serial4 @4800 baud with sparkfun GPS. You have about 2.2 seconds to write/print the 512 byte half-buffer with the NMEA coming from sparkfun GPS, 243 chars/sec.
https://www.sparkfun.com/products/465

EDIT: Your 2 NMEA sentences amount to about 120 chars/sec, so with 512 byte DMA half-buffer, you would have about 4 seconds to write a 512-byte block to the SD
 
Last edited:
Thanks Manitou, that may come in handy. Once I make the GPS logging code a little more robust, I want to add an accelerometer so I can play around with some power saving ideas I have. Maybe even a simple gesture UI (I don't have any buttons). 2.2 seconds is way more than enough to capture some accelerometer data and process it.
 
I went ahead and added double buffers at 512B. The GPS spits out binary data which was rude to review for accuracy so I set up the T_3.1 on my desk to spit out a text string every 200 ms which is how my GPS was set. I tried to count out 100 chars to match the message length - to trap in the T_3.6 … missed it by one - so I changed to that value on Rx test. The GPS has a 2 byte start code - I put that on the end so I'd know when to be done for instrumentation.

Using Serial2 [460800 baud like GPS] on the T_3.6 receiver end, bumped it's receive buffer to 128 bytes. With the double buffer it pulls all available chars after filling "A" into "B" so none are left hanging as the write is indicated. I took out the 2 ms wait for end of GPS receive - and trusted SD to complete having a whole buffer of 128 for a 101 byte message.

Was still seeing loss so added instrumentation … currently the MAX write time is showing as 420 ms for a single 512 B block - a good average for group of 20 can be near 3 ms.

Paul - SD does not stop Serial2 FIFO!!! >>Buffer and the Detected Errors have disappeared setting up to :: SERIAL2_RX_BUFFER_SIZE 210

Until this 422.046 ms :: 5440 << # writes 422046 << MAX write time
Bumped Rx_2 buffer to 305 so it can get 3 full strings.

The SD started with a clean SDFORMATTER setup, it is a 32MB SanDisk Ultra, of course by now it has been hit by some many dozens of rude restarts. Last time I had it out I deleted the file and did a chkdsk and it was clean. To allow a clean shutdown just added:
if (Serial.available()) while (1); // pause before any restart by sending anything out USB

Pulled the card and no data errors, and 50K of well formatted data! Doing an 'Overwriteformat' to be sure it can be good for writing after so many delete/create/abort hits on the same file to see if the 422 ms goes away, first MAX values I saw were 20 to 50 ms. Will start overnight when format finishes.

Brian has a nice UBlox Parser on Github if you wanted to see the data in Teensy. Though would have to alter how it reads the data to buffer raw, or push out the structures after parsing.

If you want to see how I did the Dual Buffer code is below. Also the GPS_Dummy sketch to customize the output to your system and not have to have a GPS online ... since everyone has spare Teensys. I just connected GND and one line Rx<>Tx from the T_3.1. Lots of small changes for my system Serial2 and Baud. LED1 set to 13, and SD Write toggle is on to off as short ON flashes are easier to see/interpret which is how I knew I needed to time the SD write.

Dual 512 buffers with lots of print SPEW::
Code:
#include <SD.h>
#include <SPI.h>

File file;
const int BUFSIZE = 512;
[B]char charbuf[2][BUFSIZE];
uint16_t buflevel[2] = { 0, 0 };
uint16_t bufactive = 0;
#define SD_WAIT 255
uint16_t newblock = SD_WAIT;[/B]
uint32_t ms = 0;

const int led1 = 13;
const int led2 = 12;


void setup() {
  pinMode(led1, OUTPUT);
  digitalWrite(led1, HIGH);
  pinMode(led2, OUTPUT);
  digitalWrite(led2, HIGH);

  Serial.begin(115200);
  Serial2.begin(460800);  //GPS device
  while (!Serial);

  Serial.print("Initializing SD card... Dual Buffer size=");
  Serial.print(BUFSIZE);

  while (!SD.begin(BUILTIN_SDCARD)) {
    Serial.println(" initialization failed!");
    digitalWrite(led2, LOW);
    delay(750);
    digitalWrite(led2, HIGH);
    delay(750);
  }
  Serial.println(" ... initialization done.");

  if (SD.exists("NMEA.txt"))
    SD.remove("NMEA.txt");
  delay(100);
  file = SD.open("NMEA.txt", FILE_WRITE);

  if (file) {
    Serial.print("Writing to NMEA.txt...");
    Serial.println("done.");
    file.close();
  }
  else {
    Serial.println("error opening NMEA.txt");
  }
}

elapsedMillis mTime;
elapsedMicros wTime;
const unsigned char UBX_HEADER[] = { 0xB5, 0x62 };
void loop() {
  static int pcc = 0;
  static int wcc = 0;
  static int ecc = 0;
  static int dacc = 0;
  static int wacc = 0;
  static int tMax = 0;

  while (Serial2.available()) {           //if a new character is available
    ms = millis();                        //reset counter
    char pRead;
    static char lpRead;
    pRead = Serial2.read();
    pcc++;
[B]    charbuf[bufactive][buflevel[bufactive]++] = pRead; //copy character into buffer
[/B]    if ( pcc > 97 && pRead == UBX_HEADER[1] && lpRead == UBX_HEADER[0] ) {
      if ( 101 != pcc ) {
        Serial.print("GPS START > BAD COUNT < - Last Len==");
        Serial.print(pcc);
        Serial.print("\tms==");
        Serial.println(mTime);
        ecc++;
      }
      mTime = 0;
      pcc = 0;
    }
[B]    if ( buflevel[bufactive] >= BUFSIZE ) {
      newblock = bufactive;                      //new character block flag
      bufactive ^= 1;
      if ( 0 != buflevel[bufactive] ) {
        Serial.println("\n\nerror skipped NMEA.txt write");
        buflevel[bufactive] = 0;
      }
[/B]    }
    lpRead = pRead;
  }
[B]  if (Serial.available()) while (1);  //  pause before any restart by sending anything out USB[/B]
  //if ( (millis() - ms >= 2) && (newblock != 255) ) { // This prevents SD write mid message Rx
  if ( (newblock != 255) ) {
    wTime = 0;
    file = SD.open("NMEA.txt", FILE_WRITE); //open file
    if (file) {
      digitalWrite(led1, HIGH);        //turn off led
[B]      file.write(charbuf[newblock], buflevel[newblock]);   //write buffer data to SD[/B]
      file.close();
      //Serial.write(charbuf[newblock], buflevel[newblock]); //dump buffer contents to Serial monitor as well
      dacc += wTime;
      if ( wTime > tMax ) {
        tMax = wTime;
        Serial.println("\n   NEW HIGH MAXXXXXXXX  \\/ \\/");
      }
      if ( wTime > 50000 ) {
        Serial.print( "\n >>>  SD write end ...\t" );
        Serial.print( wTime );
        Serial.print( "\t pcc=" );
        Serial.println( pcc );
      }
      digitalWrite(led1, LOW);         //turn on led
      wcc++;
      wacc++;
      //Serial.print( wcc );
      Serial.print( wTime );
      Serial.print("  ");
      if ( !(wcc % 20) ) {
        Serial.println();
        Serial.print( ecc );
        Serial.print(" << #errs \t");
        Serial.print( dacc / wacc );
        Serial.print(" << avg write time \t");
        Serial.print( wcc );
        Serial.print(" << # writes \t");
        Serial.print( tMax );
        Serial.println(" << MAX write time ");
        dacc = wacc = 0;
      }
[B]      buflevel[newblock] = 0;                    //reset buffer level to zero[/B]
    }
    else {
      Serial.println("error opening NMEA.txt");
    }
[B]    newblock = SD_WAIT;                  //reset the flag[/B]
  }
}

Pretend Teensy is a GPS::
Code:
elapsedMillis tTxtout;

void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);
  delay(2500);
  while (!Serial && millis() < 5000 );
  Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);
  Serial2.begin(460800);  //GPS device
  while (!Serial);
  tTxtout = 0;
}

void loop() {
  if ( 200 <= tTxtout ) {
    tTxtout -= 200;
    Serial2.print( "abcdefghijklmnopqrstuvwxyz0123456789-=_=ABCEDFGHIJKLMNOPQRSTUVWXYZ0123456789-=_=MaryHadaLittleLamb!" );
    Serial2.write( 0xB5 );
    Serial2.write( 0x62 );
  }
}
 
Full Format completed - within 360 block writes I see 41 ms :: 360 << # writes 415052 << MAX write time

Then getting here I look over and see this 419.8 ms >> 380 << # writes 419858 << MAX write time

No lost strings with the larger buffer. I meant to pull full details of the card as I can only see 'san' 'ult '32' with it plugged in.

The next 123 writes passed under 50 ms . Avg write is 3.5 to 4ms. Another 220 just passed under at 4.15 ms or under before hitting an 85 and 53 ms on the next 2 writes.

This is a bare T_3.6 with female headers - no GPS - and just GND and Rx to the T_3.1 on separate port - w/T-3.6 powered by PC Front USB3 port so it should not be lack of power - just the card. I'll restart with slower baud to spread the Rx interrupts ...

This isn't conclusive - but suggestive - at 115200 the MAX wait only cross 100 ms twice with more writes than prior test:: 1580 << # writes 118060 << MAX write time

Recompiled at '921600' baud to compare - no horrid hits like before >> 700 << # writes 139924 << MAX write time

Took the message size to 200 on so Rx xfer lasts the same time as before to interfere with SD writes if it wants too - also twice as much written to SD so more chances to conflict?

Early results are no worse? Maybe moved to a fresh SD block?

Except one error message w/ length 304. This is better than was seen at half the baudrate and data volume :: 3600 << # writes 139972 << MAX write time

<EDITS>
Just after posting 423 ms: 3920 << # writes 423631 << MAX write time

Is there a valid workalike SD lib that would be good to test against?
 
A few more hours run time {Serial2.begin(921600);} indicates 18 errors {ID bytes not found at expected offset} in : 54620 << # writes 426789 << MAX write time.

Have paused logging - pulled the card to look at the strings - file is 27 MB. I should have hacked the next write buffer to flag where the trouble was so the trouble spot could be found in the file to check for corruption with a quick search ... in a way that doesn't cause corruption - or maybe serialize the strings to allow they show and log and to search when missed. It has 279,945 copies of the 'string'. Doing a quick test of SubLimeText ability to replace them all with '*' to see if anything remains as broken strings ... 'oddly' being "non responsive' for a bit ... will see if it completes the task ... taskman shows it actively working on something.

Thanks manitou I'll give that example a look.
>> class PreallocatedFile { // Use SdFatSdio or SdFatSdioEX for sd_fat_t.

First I'll drop the Serial2_Rx - and have the Teensy pretend it got a string every 200 ms and log that and see if the same long write times occur. And I should use other unique 'printable chars' for the GPS's signature bytes 0xB5 (181) and 0x62 (98) so the file editor/viewer won't be put off.
 
Yesterday I wrote the serial port emulation emulating the noted GPS messages on a single T_3.6.

The long 400+ ms write times happen with or without using the Serial port - so long delays on SD can cause missing GPS on Serial - but activity on Serial port doesn't seem to be the cause of long SD writes.

<edit>:: Left the sim running - the MAX transfer just bumped up: 425088 << MAX write us
Total of :: 36760 << # writes
with 327 of them >50ms and current average across groups of 20 is 6-9 ms rather than 3-4 ms.
So it takes a group of things made right to keep up to avoid loss.
 
Last edited:
Minor update to prior code in use:
I changed the 512 buffer to 1024 bytes wondering if that might work out better - it does not. The above numbers are below in the cryptic printed format:
13231 << avg us 67700 << # writes 1094 << # >50ms 639669 << MAX write

That file is 68MB now. Took Sublime a few minutes to load - quick scan of top and bottom suggests the file operations completed as expected, it just takes some 'variable' time.

Oh by the way the above p#22 search and replace of 279K entries of 100 bytes in Sublime did complete - it took some unknown hours. All that were left were the block terminator characters and those few strings written that were received as incomplete from buffer loss.

BTW: This was bought as a retail packed SanDisk card?
SanDisk.jpg

<edit> Found a fresh Adata 16GB card - been running some time:
14591 << avg us 49260 << # writes 577 << # >50ms 270901 << MAX write us
I don't suspect the SanDisk was used hard a lot - but the fresh Adata is not as extreme in slowness.
17071 << avg us 72020 << # writes 851 << # >50ms 270901 << MAX write us
 
Last edited:
Status
Not open for further replies.
Back
Top