a question about reading very large files

[Dmax]

Hi everyone I have a a question about reading very large files, I have a simple bit of code that I modified from one of the SD card examples, it now reads a wave file rather than a text file. the problem is that the files are being read into a buffer which works fine with small files but causes the Teensy to crash once the files are bigger and some of my files are 7 or more MB. is there a way to read the files directly without the use of a buffer and if so could someone point to to a bit of code that would help or give me a good break down of what I need to do as I'm not really a programmer and I'm sort of making it up as I go along.. below is the function Im using right now...

           if     (ui.checkForButtonClicked(read_wave_file_SD))   {///
               
               myFile = SD.open("wave.wav");
  if (myFile) {
    Serial.println(myFile);
    //    int totalBytes = file.size();
    // read from the file until there's nothing else in it:

    myFile.seek(22);
    myFile.read((char*)&numChannels, 2);
    myFile.read((char*)&samplingRate, 4);
    // Read bits per sample
    myFile.seek(34);
    myFile.read((char*)&bitsPerSample, 2);
    // Read size of data in bytes
     U32 sample_length;
    myFile.seek(40);
    myFile.read((char*)&sample_length, 4);
// Allocate array to hold all the data as PCM samples
    count = sample_length / 2;
   sample_data = new PCM16[count];
//    // Read PCM data
   myFile.read((char*)sample_data, sample_length);


    
    Serial.print("number of channels = ");
    Serial.println(numChannels);
    Serial.print("sample rate = ");
    Serial.println(samplingRate);
    Serial.print("bits per sample = ");
    Serial.println(bitsPerSample);
    Serial.print("size in bytes = ");
    Serial.println(sample_length);
  
  // close the file:
  myFile.close();
} else {
  // if the file didn't open, print an error:
  Serial.println("error opening file");
}

 

[joepasquariello]

The problem is that the files are being read into a buffer which works fine with small files but causes the Teensy to crash once the files are bigger and some of my files are 7 or more MB. is there a way to read the files directly without the use of a buffer and if so could someone point to to a bit of code that would help

In the code below, you are trying to allocate an array large enough to hold ALL of the samples, no matter how many there are. If your file is 7MB, then you are trying to allocate an array of size ~7MB, then read all of the data into the array at once. Unless you have 7MB of RAM available, which you don't, the call to new() will fail, and your program will crash.

    // sample_length = bytes, count = 16-bit words
    count = sample_length / 2;                                 
    // Allocate array to hold ALL the data as PCM samples
    sample_data = new PCM16[count];                      
    // Read ALL of the PCM data
    myFile.read((char*)sample_data, sample_length);

You don't say which Teensy you are using, but let's say it is Teensy 3.2, and let's say you have space for a 4096-byte buffer. You don't say what you want to do with the data, but whatever it is, you have to process it in chunks that are within your memory (RAM) limits. You could replace the code above with something like this:

    PCM16 sample_data[2048];   // define a buffer for to hold 2048 x 16-bit values
    count = sample_length / 2;    // total number of samples in the file                                
    // process the data in chunks of 2048 samples                                 
    while (count > 0) {                                           
      if (count >= 2048)                                          
        n_samples = 2048;                                      
      else                                                             
        n_samples = count;                                     
      myFile.read((char*)sample_data, n_samples*2);  
      ...
      count = count - n_samples;                                          
   }

 

[Dmax]

Thanks for taking the time to reply joepasquariello I'm using a T-4.1 and outputting the data to a SPI DAC that in turn drives a set of Galvo's, I did think about processing the data in smaller bits but wasn't to sure if the delay would show up when displayed I also wasn't to sure how to go about it so thanks for the code I will give that a go