Serial Flash rewrite question

Status
Not open for further replies.

linuxgeek

Well-known member
I'm not sure how to accomplish what I want. I'm logging data, and I've started by allocating one large file with SerialFlash.
I keep track in another file where the data starts in the large file, and I want to keep track of the end of the file on an on-going basis as each block is written. It works if I write the ending block out to the tracking file when stopping the device is triggered (cause I only have to do it once). But I want to keep track of the end of the file as it is being written in case the device gets turned off, so that I still know where the file ended.

The other thing that is confusing me, is that I'm initializing a buffer and writing that buffer to the tracking file initially. But after it's initialized to a value, I seem to be able to change it one more time, but that's it. Maybe you can rewrite a whole block but not a partial block? If I instead rewrote a whole bloc, that's still a problem, cause if the device gets turned off I don't want to be in a partial write for the tracking file.

Maybe I'm missing something?

I can initialize the large file to "-1", so I can tell by reading up until I encounter "-1". But I also want to allow multiple recordings within the same large file, and having the tracking file would allow me to output each separate recording after-the-fact.

I'm hoping I'm missing something simple (often the case) that someone could point me at. Thanks for any input.

Here's some code demonstrating:

Code:
#include <SerialFlash.h>
#include <SPI.h>


const int FlashChipSelect = 6;
SerialFlashFile dataFile;
int blockSize=256;
const int samplesInBlock = 64;

int32_t startByte = -1, endByte = -1, endMarkerPos = -1;
int bytesInSample = 4;
char dataFilename[12]="test.bin";
int counter=0;
int32_t intWriteBuffer[samplesInBlock];

/**
* Initialize integer buffer
*/
void initIntBuffer(int32_t * buff, int nSize, int initInt) {
  for (int i=0;i<nSize;i++) {
  buff[i]=initInt;
  }
}

void setup() {
  Serial.begin(38400);
  while (!Serial);
  Serial.print("initFlashFS filename: ");
  Serial.println(dataFilename);

  initializeSPIFlash();

  if (SerialFlash.exists(dataFilename)) {
    Serial.println("storage file exist");
  } else {
    Serial.println("storage file does NOT exist... creating");
    SerialFlash.createErasable(dataFilename, blockSize);
    initIntBuffer(intWriteBuffer,samplesInBlock,-1);    
    dataFile = SerialFlash.open(dataFilename);
    dataFile.write(intWriteBuffer,blockSize);
//can close it instead for testing
//    dataFile.close();
  }
    Serial.println("opening & leaving open");


    for (int counter=0;counter<10;counter++) {
//if it's closed above, need to open it
//      dataFile = SerialFlash.open(dataFilename);
      Serial.print("pos: ");
      Serial.println(dataFile.position());
      Serial.print("counter: ");
      Serial.println(counter);

      dataFile.seek(0);
//this works because it writes to a new spot
//      dataFile.seek(counter*bytesInSample);
      intWriteBuffer[0]=counter;
      dataFile.write(intWriteBuffer,bytesInSample);
      dataFile.close();
      delay(100);
    }
 //   dataFile.close();
    sendIntFile(dataFilename,blockSize);
    Serial.println("Done");
}

void loop() {
  delay(1000);

}

/**
* Send file full of ints with xml-like wrapper
*/
void sendIntFile(char * sendFilename, uint32_t nBytes) {
  SerialFlashFile tmpFile;
  Serial.print("sending...");
  Serial.println(sendFilename);

  Serial.print("sending...");
  Serial.println(sendFilename);
  if (SerialFlash.exists(sendFilename)) {
    tmpFile = SerialFlash.open(sendFilename);
  if (nBytes < 0) nBytes = tmpFile.size();

    Serial.print("<file=");
    Serial.print(sendFilename);
    Serial.println(">");
    while (tmpFile.position() < nBytes) {
      Serial.println(tmpFile.position());
      tmpFile.read(intWriteBuffer, blockSize);
      Serial.println(tmpFile.position());
      Serial.println(tmpFile.size());

      for (int i=0;i<samplesInBlock;i++) {
        Serial.print(intWriteBuffer[i]);
        Serial.print(",");
      }
      Serial.println();
    }
    tmpFile.close();
    Serial.println("</file>");
  }
}

/**
* Basic check if SPI Flash is connected
*/
void initializeSPIFlash() {
  Serial.println("in init spi flash");
  Serial.flush();
  delay(3000);
  if (!SerialFlash.begin(FlashChipSelect)) {
    Serial.println("Unable to access SPI Flash chip");
//    error("Unable to access SPI Flash chip");
  } else {
    Serial.println("past spi flash init");
  }
}

https://github.com/PaulStoffregen/SerialFlash
 
Last edited:
As a follow-up.

If I use this I can rewrite over it
Code:
dataFile.erase();
dataFile = SerialFlash.open(dataFilename);

This will work, as I can make a backup copy of the file before writing it, to protect from a partial write of the tracking file.

But, I'm still a little concerned on the amount of time it would take to open the file again. It could be ok, but I'm probably giving up a lot of the benefit of how fast writing would be otherwise. I'm sampling on an interrupt so I have to be a little careful of doing too much at any one time.

I guess I could move the tracker file writing to the loop. I think that's the easy way around this. Sorry for following up on my own thread. I'll try and post some example code for doing this if it works.
 
Last edited:
Tried the writing of the tracker file in loop but it conflicts with the SPI flash writes in the interrupt.
Tried turning off interrupts for writing the track file but that still locked up after awhile.

I think the solution is to initialize the whole file to "-1", and to put just the start of files in the tracker file.
For outputting the multiple recordings within the storage file, go from start of file until start of next file. And for the last one, read out until encountering "-1". Maybe this would make a decent SPI flash example? If anyone thinks it might, I'd gladly make an example sketch.
 
@linuxgeek: A good example of this would be very useful. What you found is what I expect - but haven't had the need to go through it yet. '-1' (0xff bytes) is the erased state - and once a 0 bit goes to a byte it sticks until erasure.

If I have this right - An example:: Opening the file of size x (and pretending to write/init 0xff if needed to set the length?), then writing out 'user data', reading, then doing a dataFile.erase(); to allow starting over from the 0xff state would be nice to see it work that way.
 
I'm not sure if we are thinking the same thing but I'll outline what I'm finding useful, and you can let me know if that seems good.

TrackerFile = tracker.bin (256 bytes, which can store 64 file starts)
DataFile = data.bin (very large file)

Give a menu of options:
e: Erase all files
s: start recording using interrupt
x: end recording
t: output all recordings.

The trackerFile is opened (created if needed)
The dataFile is opened (created if needed)

At the start of a recording, a new entry is added to the trackerFile (by finding 0xff), with the starting byte location within dataFile (again by starting from the last trackerFile entry and searching forward until 0xff).
The dataFile file continually gets written in the interrupt as the 256 byte buffer fills.

When the recording is stopped, interrupt is stopped, and files are closed.

Transmit files, will open trackerFile, and for each entry, it will output all the data stored in dataFile from start to next start of file. For last file, it will output from start entry until 0xff within the dataFile.
 
Last edited:
I hadn't thought of a specific use - but that looks like a good example using two files different ways.

Would a 's'-start would automatically 'x'-end an ongoing recording then restart - or just place a new Tracker entry for the 'current' sample as it runs?
 
I hadn't thought of a specific use - but that looks like a good example using two files different ways.

Would a 's'-start would automatically 'x'-end an ongoing recording then restart - or just place a new Tracker entry for the 'current' sample as it runs?

hitting 's' could just put another tracker entry, but I think I'd rather it be the equivalent of hitting 'x' to stop, and then 's' again to start. I'm afraid of spi conflicts. It shouldn't take me too long to write this, I should post it by this weekend.
 
Sounds good - like we're on the same page.
Feature Creep - might be nice to have an example of the same work on a flash drive too.
 
Sounds good - like we're on the same page.
Feature Creep - might be nice to have an example of the same work on a flash drive too.

That would be cool to transfer directly to a flash drive, but that's a USB host mode problem and I don't even know where to start for that. An sdcard transfer is feasible but mayve later. I'm almost done with it but won't get back to it for a couple of days.
 
I meant run the example directly to SD card data files instead of flash - using the same two file scheme. Should be a minor set of changes - but might allow somebody familiar with one to understand the other - and provide a decent SD example too. I'll see if I'm inspired to that that when you post - will help me understand both.
 
I think a lot of the rationale for using SPI flash is to avoid the long delay is involved with SD card while using interrupt sampling.

It would make sense to use an SPI flash buffer and use the SD card writing but I'm still worried about SPI conflicts.
I might try to make an example of that and maybe someone else could look at it and figure out how to avoid the SPI conflicts.

Does anyone know that while using interval timer for sampling, that if I use no interrupts and interrupt in the loop, will the next interrupt be called at the same interval or will it be delayed by that amount of time that I've called no interrupt?
 
Does anyone know that while using interval timer for sampling, that if I use no interrupts and interrupt in the loop, will the next interrupt be called at the same interval or will it be delayed by that amount of time that I've called no interrupt?

Sorry to respond to my own question, but I think I found the answer.

As long as you don't call nointerrupts() for so long that 2 interrupts (of the same type) pass, the one that was passed will trigger when you call interrupts(). That's reasonable, as the worst case scenario is that you call nointerrupts() a clock cycle before the interrupt would be called, and then it's delayed by however long it takes to call interrupts().

But how does one handle the problem that you could call nointerrupts() while in the middle of processing an interrupt? I assume the interrupt code keeps running until done cause it was already triggered. I think this might be where I'm getting an SPI conflict. Should I create a volatile global flag that is checked in the loop? Such as:

Code:
nointerrupts();
//copy volatile flag
//check copy of flag
//do other stuff w/copy of volatile variables, etc
interrupts();
 
Here's my sketch for writing samples (4 byte int) to spi flash from an interrupt, keeping track of files stored within a large file using a tracking file, and then allowing to output the individual files all at once.

Consecutive sample #'s are stored in interrupt, no actual sampling done, but you can put your own sampling to replace with actual sampling.

I'm sure it's not perfect, but I should post it in case I don't get around to improving it more.

First, here's the output:
Code:
i = initialize
s = start interrupt sampling
u = send tracker file
x = stop interrupt sampling
e = erase SPI Flash. Initialize after for use
t = tx data files
waiting for command...

in init spi flash
past spi flash init
storage file exist: data.bin
tracker file exist: tracker.bin
Initializing Done

sampling start
writing block. sample count: 63
writing block. sample count: 127
writing block. sample count: 191
writing block. sample count: 255
writing block. sample count: 319
writing block. sample count: 383
sampling stop

transmitting files...
<file=data.bin0>
0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,
</file>
<file=data.bin1>
0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,
64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,
128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,
</file>
<file=data.bin2>
0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,
</file>
<file=data.bin3>
0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,
64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,
128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,
192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,
256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,
320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,
</file>
Done


Code:
//author Oscar Carrillo
#include <SerialFlash.h>
#include <SPI.h>

#define FLASH_CHIP_SELECT 6

SerialFlashFile dataFile, trackerFile;
const int nBlockSize=256;
const int nBytesInSample = sizeof(int32_t);
const int nSamplesInBlock = nBlockSize/nBytesInSample;
const int32_t nFileMaxSize=12000000;
boolean bSampling = false;
int32_t nSampleCount = 0, nBufferCount = 0;
int nSampleRate=20;
IntervalTimer samplingTimer;

int32_t nStartByte = -1;

char dataFilename[12]="data.bin";
char trackerFilename[12]="tracker.bin";
int32_t intBuffer[nSamplesInBlock];

/**
* Initialize integer buffer
*/
void initIntBuffer(int32_t * buff, int nSize, int initInt) {
  for (int i=0;i<nSize;i++) {
  buff[i]=initInt;
  }
}

void setup() {
  Serial.begin(38400);
  while (!Serial);

  Serial.println("i = initialize");
  Serial.println("s = start interrupt sampling");
  Serial.println("u = send tracker file");
  Serial.println("x = stop interrupt sampling");
  Serial.println("e = erase SPI Flash. Initialize after for use");
  Serial.println("t = tx data files");
  Serial.println("waiting for command...");
  Serial.println();
}

void loop() {
  waitForCommand(1000);
}

/**
* wait for a command with a specified delay btwn checks
*/
char waitForCommand(int nMsecWait) {
  char c = ' ';

  while (c == ' ') {
    if (Serial.available() > 0) {
      c=Serial.read();
//      Serial.println(c);
      switch(c) {
      case 'i':
        initialize();
        Serial.println("Initializing Done");
        Serial.println();
        break;
      case 's':
        Serial.println("sampling start");
        if (bSampling) endSampling();
        startSampling();
        break;
      case 'u':
        Serial.println("tracker file:");
        sendIntFile(trackerFilename,nBlockSize);
//        Serial.println("datafile:");
//        sendIntFile(dataFilename,nBlockSize*nBytesInSample);
        Serial.println();
        break;
      case 'x':
        Serial.println("sampling stop\n");
        endSampling();
        break;
      case 'e':
          eraseSPIFlash();
          Serial.println("Done");
          Serial.println();
        break;
      case 't':
        Serial.println("transmitting files...");
        sendAllIntFiles();
        Serial.println("Done");
        Serial.println();
        break;
      default:
      c=' ';
    }
  }
  delay(nMsecWait);
}
  return c;
}

/**
* intialize SPI Flash
*/
void initialize() {
  initializeSPIFlash();
  initFlashFS();
}

/**
* Send file full of ints with xml-like wrapper
*/
void sendIntFile(char * sendFilename, uint32_t nBytes) {
  SerialFlashFile tmpFile;
  Serial.print("sending...");
  Serial.println(sendFilename);

  if (SerialFlash.exists(sendFilename)) {
    tmpFile = SerialFlash.open(sendFilename);
  if (nBytes < 0) nBytes = tmpFile.size();

    Serial.print("<file=");
    Serial.print(sendFilename);
    Serial.println(">");
    while (tmpFile.position() < nBytes) {
//      Serial.println(tmpFile.position());
      tmpFile.read(intBuffer, nBlockSize);
//      Serial.println(tmpFile.position());
//      Serial.println(tmpFile.size());

      for (int i=0;i<nSamplesInBlock;i++) {
        Serial.print(intBuffer[i]);
        Serial.print(",");
      }
      Serial.println();
    }
    tmpFile.close();
    Serial.println("</file>");
  }
}

/**
* Basic check if SPI Flash is connected
*/
void initializeSPIFlash() {
  Serial.println("in init spi flash");
  Serial.flush();
  if (!SerialFlash.begin(FLASH_CHIP_SELECT)) {
    Serial.println("Unable to access SPI Flash chip");
//    error("Unable to access SPI Flash chip");
  } else {
    Serial.println("past spi flash init");
  }
}

/**
* Create the tracking file and the storage file
*/
void initFlashFS() {
  if (SerialFlash.exists(dataFilename)) {
    Serial.print("storage file exist: ");
    Serial.println(dataFilename);
  } else {
    Serial.print("storage file does NOT exist... creating ");
    Serial.println(dataFilename);
    SerialFlash.createErasable(dataFilename, nFileMaxSize);
  }

  if (SerialFlash.exists(trackerFilename)) {
    Serial.print("tracker file exist: ");
    Serial.println(trackerFilename);
  } else {
    Serial.print("tracker file does NOT exist... creating: ");
    Serial.println(trackerFilename);
    SerialFlash.createErasable(trackerFilename, nBlockSize);
    //redundant because file will already by 0xff for the entire file
    initIntBuffer(intBuffer,nSamplesInBlock,-1);
    trackerFile = SerialFlash.open(trackerFilename);
    trackerFile.write(intBuffer,nBlockSize);
    trackerFile.close();
  }
}

/**
* Start the interrupt timer sampling, track the start in file, etc
*/
void startSampling() {
  nSampleCount = 0;
  nBufferCount = 0;
  nStartByte = getNextAvailStartByteForStorageFile();
//  Serial.print("startByte: ");
//  Serial.println(nStartByte);
  //store storage start in tracker file
  writeOutNewStartMarker(nStartByte);
  //move storage file to position, and leave the dataFile open, since it will be written in interrupt
  dataFile = SerialFlash.open(dataFilename);
  dataFile.seek(nStartByte);
  bSampling = true;
  samplingTimer.begin(sample, 1000000/nSampleRate);
}

/**
* Only stop sampling timer if it was started
*/
void endSampling() {
  if (bSampling) {
    samplingTimer.end();
    bSampling=false;
    dataFile.close();
  }
}

/**
* This gets called by the interrupt
* No actual sampling is done here, but a counter is written to show that it works as intended
* When buffer is full, it writes a block to the data file
*/
void sample() {
  //actual sampling would go here, this just puts a counter
  intBuffer[nBufferCount] = nSampleCount;
/*
  Serial.print("buffer: ");
  Serial.print(nBufferCount);
  Serial.print("nSampleCount: ");
  Serial.print(nSampleCount);
  Serial.println();
*/
  //if buffer is full
  if (nBufferCount >= nSamplesInBlock-1) {
    //write out a block
    Serial.print("writing block. sample count: ");
    Serial.println(nSampleCount);
    dataFile.write(intBuffer,nBlockSize);
    //reset
    nBufferCount=0;
  } else {
    nBufferCount++;
  }
  nSampleCount++;
}

/**
* Find the next valid location to place data in storage file
* Use last tracker file entry to start looking in storage file from that location
*/
int getNextAvailStartByteForStorageFile() {
  return getNextAvailStartByteInFile(dataFilename,getLastValidValueInTrackerFile());
}

/**
* reads thru file, starting at startPos until it finds -1 or end of file
* returns the last valid entry
*/
int getLastValidValueInTrackerFile() {
  int lastValidEntry = 0;
  trackerFile = SerialFlash.open(trackerFilename);
  trackerFile.seek(0);
  trackerFile.read(intBuffer, nBlockSize);
  trackerFile.close();
  for (int i=0;i<nSamplesInBlock && intBuffer[i] != -1;i++) {
    lastValidEntry = intBuffer[i];
  }
//  Serial.print("last valid entry in tracker file: ");
//  Serial.println(lastValidEntry);
  return lastValidEntry;
}

/**
* Can't access this while interrupt sampling/writing is going on
* Find the next available byte to write to a file, starting from startPos
*/
int getNextAvailStartByteInFile(char * filename, long startPos) {
  long newPos = startPos;
  SerialFlashFile tmpFile = SerialFlash.open(filename);
  tmpFile.seek(startPos);

  //just check first entry in each block for availability
//  newPos = tmpFile.position();
//  tmpFile.read(intBuffer, nBlockSize);  
  for (;tmpFile.position() < (tmpFile.size()-nBlockSize) && intBuffer[0] != -1;) {
    newPos = tmpFile.position();
//    Serial.print("tmp newPos: ");
//    Serial.println(newPos);
    tmpFile.read(intBuffer, nBlockSize);
  }
  //newPos will be at the beginning of the block that was just read
  tmpFile.close();
/*
  Serial.print("next start in ");
  Serial.print(filename);
  Serial.print(" is ");
  Serial.print(newPos);
  Serial.print(" starting from ");
  Serial.println(startPos);
*/
  return newPos;
}

/**
* Write out the start location of storage file 
*/
void writeOutNewStartMarker(int nStartByteInStorageFile) {
    //this will open/close the same file to get the next valid write position
  int elem = 0;
  trackerFile = SerialFlash.open(trackerFilename);
  trackerFile.read(intBuffer,nBlockSize);
  //search until found empty slot or end of file
  for (elem=0;elem<nSamplesInBlock && intBuffer[elem] != -1;elem++);

  //write over the relevant portion for the new marker
  intBuffer[0]=nStartByteInStorageFile;
  //write out start marker
  trackerFile.seek(elem*nBytesInSample);
  trackerFile.write(intBuffer,nBytesInSample);
  trackerFile.close();
}

/*
* I'm only doing 1 tracking entry at a time from the file, so that I don't have to use a larger buffer which uses signifcant memory
* And I can't reuse the data reading buffer, since I need to have access to both.
*/
void sendAllIntFiles() {
  int trackerBuffer[1];
  long storageStartByte=0, nextStorageStartByte=0;
  int i=0;

  //do first entry
  trackerFile = SerialFlash.open(trackerFilename);
  trackerFile.seek(0);
  trackerFile.read(trackerBuffer,nBytesInSample);
  storageStartByte=trackerBuffer[0];

  //if no entries, don't do anything, exit
  if (storageStartByte < 0) {
    trackerFile.close();
    return;
}
  trackerFile.seek(nBytesInSample);
  trackerFile.read(trackerBuffer,nBytesInSample);
  nextStorageStartByte=trackerBuffer[0];

  dataFile = SerialFlash.open(dataFilename);
  //read one at a time, to conserve memory
  for (i=2;i<nSamplesInBlock && nextStorageStartByte != -1;i++) {
//    Serial.print("tracker start: ");
//    Serial.println(storageStartByte);
//    Serial.print("tracker end: ");
//    Serial.println(nextStorageStartByte);
    //output file from start to end
    dataFile.seek(storageStartByte);
    Serial.print("<file=");
    Serial.print(dataFilename);
    Serial.print(i-2);
    Serial.println(">");
    for (;dataFile.position() < (unsigned)nextStorageStartByte;) {
//      Serial.print("outputting position: ");
//      Serial.print(dataFile.position());
//      Serial.print(" to ");
//      Serial.println(nextStorageStartByte);
      dataFile.read(intBuffer,nBlockSize);
      outputBufferOverSerial(intBuffer, nSamplesInBlock);
    }
    Serial.println("</file>");
    trackerFile.seek(i*nBytesInSample);
    trackerFile.read(trackerBuffer,nBytesInSample);
    storageStartByte=nextStorageStartByte;
    nextStorageStartByte=trackerBuffer[0];
  }

  Serial.print("<file=");
  Serial.print(dataFilename);
  Serial.print(i-2);
  Serial.println(">");
  //Serial.print("last one is: ");
  //Serial.print(storageStartByte);
  //Serial.print(":");
  //Serial.println(nextStorageStartByte);
  //datafile position will be at the start of the last file, so just read from where it is
  for (dataFile.read(intBuffer,nBlockSize);intBuffer[0] != -1;) {
    outputBufferOverSerial(intBuffer, nSamplesInBlock);
    dataFile.read(intBuffer,nBlockSize);
  }
  Serial.println("</file>");

  //now output the last one
//  for (int i=storageStartByte;intBuffer[0] != -1;i+=256) {

//  }

  trackerFile.close();
}

/**
* For testing purposes to see if we can see the ints on the serial port
*/
void outputBufferOverSerial(int32_t * outputBuffer, int count) {
  for (int i=0;i<count;i++) {
    Serial.print(outputBuffer[i]);
    Serial.print(",");
  }
  Serial.println();
}

/**
* Format the SPI Flash. clear all contents
*/
void eraseSPIFlash() {
//  SerialFlash.readID(id);
  //do this for initializing an spi flash
  initializeSPIFlash();
  SerialFlash.eraseAll();
    //Flash LED at 1Hz while formatting
  Serial.print("formatting");
  while (!SerialFlash.ready()) {
    Serial.print(".");
    delay(1000);
  }
}
 
Something odd for me with a PROP_LC on a T_3.6? What are the startup steps?
e : erase / format
i : initialize
s : start
x : stop

u : send tracker file
t : tx data files
So I did the above - in that order and::
>> This seems to work except the 't' seems to dumps the whole of flash - not stop where tracked data did? That is reading all '-1'. I've got an 8MB flash on the PROP and it isn't done dumping yet? A ton [unending?] of these:
-90811060,418775640,-51993,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
> I did a TYQT RESET and then 'i' on restart - and I see "storage file does NOT exist... creating data.bin"?
>But 'u' gives me :
sending...tracker.bin
<file=tracker.bin>
0,256,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,
</file>

>> This looks like a problem:
void sendAllIntFiles() {
int trackerBuffer[1];
// ...
trackerFile.read(trackerBuffer, nBytesInSample);

>> It gets very confused if you restart and don't do init before other commands - should this be 'implied' if init not requested? Sort of how SampleStop() is done if already sampling.

Started too late to look more at the code . . .
 
Oh! I forgot to mention that I hardcoded 12MB file as the length. You'll need to change that for sure if you only have 8MB.

Code:
const int32_t nFileMaxSize=12000000;

Yes, not the most intuitive, but I'm not sure there's an actual bug if you do things in the intended order.
You have to start with initialization, erase. Erase actually calls initializeSPIFlash() first, but doesn't call initFlashFS() since it's going to erase. After the first recording, you don't want to use 'e' anymore or else you lose your prior data.

And initialize() calls both:
Code:
void initialize() {
  initializeSPIFlash(); //sets up SPI flash communication
  initFlashFS(); //this checks if the files are there and creates if not
}

That's not a bug that you highlighted, cause trackerBuffer stores 4-byte ints, and nBytesInSample=4
trackerFile.read() reads only in bytes, that why it's like that.

e
i
s
x
u (optional)
t

s
x
u (optional)
t

If you get corrupted data, reconnect and start with 'e'. I messed myself up sometimes, so I made 'e' able to be called as the first command, rather than having to call 'i'.
 
Also, bear in mind that the size you allocate will be somewhat larger using SerialFlash. So you might want to set 7500000 to start. Or if you are running into problems you could try making it rather small first time just for troubleshooting.
 
Ah - As noted - I didn't have time to understand the code and confused:
Code:
const int nBytesInSample = sizeof(int32_t);
const int nSamplesInBlock = nBlockSize / nBytesInSample;

I had read this comment about the single entry write - but didn't sink in. If you made a larger buffer here it would be on the stack and go away after the call:
I'm only doing 1 tracking entry at a time from the file,

I modified the code to show this and respond to '?' since the command list scrolls off:
Logging to data file of nFileMaxSize=7500000
i = initialize
s = start interrupt sampling
u = send tracker file
x = stop interrupt sampling
e = erase SPI Flash. Initialize after for use
t = tx data files
? = show this command list
waiting for command...

The 't' transmit now works with nFileMaxSize properly set. I see output for three files where I started and stopped sampling. FrankB has code that reads the flash size - not sure if that could be used to detect or set the maximum file size?

Also it now does this - with added setting of bInitDone:
Code:
void startSampling() {
  nSampleCount = 0;
  nBufferCount = 0;
  [B]if ( !bInitDone )
    initialize();[/B]

I see you took out the interrupt disable in the sample() code, but you have "Serial.print" in there which is bad form. It would be better to set a flag and have code in loop show that? I made a way to do that.

Also I took out the "delay(nMsecWait);" and put in an elapsedMillis value to only check for USB input every 100ms and then return to loop ASAP. Maybe you wanted to wait - but it kept the loop() from flowing - where I added the print that a block was written.

Here is what I have working with above edits:: View attachment Flash_Log.ino

I didn't look to see if it auto stops sampling on out of room to write?
 
I'm getting pretty low on memory in my other sketches, so I wanted a way to minimize memory usage, been working with LC.
I initially read the whole tracker file into buffer, but rewrote it to just read one int at a time.

All good points for improving.

Yes, I should not have that Serial.print in the interrupt (a quick kludge so that someone could see it working). You're right I should have done it like you did. Thanks.

Obviously you can write out the raw ints over serial too, but I left it as text so that it was easy to see that it's working. I also have an extra comma when it outputs file output as comma-delimited text.

edit: Now it would be great to put an RTC datetime stamp in each file.
 
Last edited:
Yes, I see the extra trailing comma now. That should be an easy fix - if not the first printed number, then print 'comma' before the number - this works [same in sendIntFile()].
Code:
void outputBufferOverSerial(int32_t * outputBuffer, int count) {
  for (int i = 0; i < count; i++) {
    if ( i )
      Serial.print(",");
    Serial.print(outputBuffer[i]);
  }
  Serial.println();
}

Also - I noticed that doing a 't' ( or maybe a 'u' ) while sampling results in a corrupted data line?
128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,
0,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,

Bummer the LC's you use most won't usually have an RTC, but time stamp would be nice when it is handy. For now maybe arrange a placeholder of 'microseconds' at sample time? It is meaningless, until somebody with a valid RTC gets hold of it - then the sample would be more complete? Would that be a parallel file to 'tracker.bin' - with one RTC entry per file when logging started?

Might be nice to show the file name at the end of the file too and not just '/file'- like at the beginning?

I just noticed the 'u' and 't' DIE without INIT - I forgot to include bInitDone code there.
 
I see you took out the interrupt disable in the sample() code, but you have "Serial.print" in there which is bad form. It would be better to set a flag and have code in loop show that? I made a way to do that.

Does nSampleWrite need to be declared volatile, and call noInterrupts() / make copy / call interrupts() ?

I doubt there's much chance of anything to go wrong, but could it possibly report out the wrong # for nSampleWrite?

Maybe could include with a #define DEBUG, cause I don't think there's much point in an actual sketch
 
Volatile for anything changed in the isr :: void sample() { that is referenced elsewhere - and visa versa.
 
Status
Not open for further replies.
Back
Top