4.1 crashes with SD card in strange state

Fenichel said:
The test eliminates the null-pointer crashes that used to occur. Would
Code: 
if (SnapshotFile == null)
have been more correct?
Click to expand...
No. As stated by @thebigg , the FS filesystem class has an implementation of the bool operator that shows you if the SnapshotFile has a valid open file. Your test is correct as you wrote it.

As an aside, my preferred test for a null pointer is something like
C++: 
if (nullptr == mightBeNULLptr)
Using nullptr tells the compiler more about what you’re doing, and the apparently clunky / backwards check means that if you accidentally type = you get a full-blown error.

I can’t see any evidence to suggest it, but just checking you’re not accessing the SD card from inside an ISR? That almost always ends badly…
 
h4yn0nnym0u5e said:
No. As stated by @thebigg , the FS filesystem class has an implementation of the bool operator that shows you if the SnapshotFile has a valid open file. Your test is correct as you wrote it.

As an aside, my preferred test for a null pointer is something like
C++: 
if (nullptr == mightBeNULLptr)
Using nullptr tells the compiler more about what you’re doing, and the apparently clunky / backwards check means that if you accidentally type = you get a full-blown error.

I can’t see any evidence to suggest it, but just checking you’re not accessing the SD card from inside an ISR? That almost always ends badly…
Click to expand...
Thanks for the
C++: 
if (nullptr == mightBeNULLptr)
idea.

No, I'm not accessing the µSD card from inside an ISR. Good thought, though.
 
jmarsh said:
Code: 
Buffer = AF_VC0706.readPicture(BytesToRead);
Where does the memory come from that this function returns? Are you meant to release it when you're done with it?
Click to expand...

Fenichel said:
I haven't looked at the underlying library, but I have assumed that it is allocating & releasing the buffer appropriately.
Click to expand...

The memory is a fixed-size buffer in the object itself, so no heap functions are needed.
 
Fenichel said:
The idea that SD.open is failing because of memory outage from a leak somewhere else is a good thought. I will work on that.
Click to expand...
I have instrumented the top-level loop() of my code using the routines at https://github.com/mpflaga/Arduino-MemoryFree, and after limited (and continuing) testing, there is no sign of a memory leak.

I incidentally may have found the origin of my problem with SD.open. The µSD card in question has around 500 files in its root directory. While manipulating it from Windows (not via the Teensy program), the Windows File Explorer complained that it was running into a limit intrinsic to the FAT format.

I have observed SD.open(<filename>, FILE_WRITE) return a null pointer. Could this be what happens whenever the directory has too many files in it already?
  • Of course, I could and should do the experiments to prove this. I will, but at the moment I'm half out the door, and I'll be away from home until the middle of next week. I'm sorry to have created this cliff-hanger.
  • If my hypothesis is true, then this behavior of SD.open should at least be documented. Better, SD.open should then return a boolean result.
 
It does not return a pointer, it returns an object.
It is up to you to check if that object is usable or not. That's why it has a bool operator.
 
Consider if you slightly re-wrote your code thus:
C++: 
File SnapshotFile; // object exists, but obviously isn't an open file
SnapshotFile = SD.open(TheFileName, FILE_WRITE); // might now be open...

if (SnapshotFile) // ...but you have to check
{
....
}
This makes it more obvious why open() can’t possibly return a bool - as documented, it returns a File, which it has to do in order to be of any use. As a convenience, the File object provides an override to the bool operator which just calls the isOpen() method. If you wished to, you could make your test clearer by writing:
C++: 
if (SnapshotFile.isOpen()) // ...but you have to check

Although you’re asking for documentation, it rather looks as if you didn’t read what’s already available…
 
Just to further confirm, File is a C++ class, not a pointer. Internally, it has a pointer "f" to the FileImpl class that wraps the SdFat library. If the file wasn't opened, or if you have closed it, the pointer inside File will indeed be null.

File class checks if its pointer is null (the file isn't actually open) and will simply return zero without attempting any work if the file isn't open. This code is in FS.h starting at line 170:

Code: 
        // Write bytes to a file
        virtual size_t write(const uint8_t *buf, size_t size) {
                // override print version
                return (f) ? f->write((void*)buf, size) : 0;
        }
        // Write bytes to a file
        size_t write(const void *buf, size_t size) {
                return (f) ? f->write(buf, size) : 0;
        }

Hopefully it's easy to see the write() functions do indeed check whether the pointer is null and avoid dereferencing it.
 
Last edited:
I'd like to repeat my request from msg #5. Can you craft a minimal version of your program which anyone who buys the right hardware from Adafruit and connects to Teensy the same way you have can run the program to reproduce problem?

So far the most you've shown us in msg #14 "The pertinent code". But that's not a complete program I or anyone else can run on a Teensy board to reproduce the problem. We need a complete program, so there's no guesswork. We also need details like a link or exact part number for the Adafruit camera and clear description or photos showing us how to wire it the same way, so we can reproduce the same crash you're seeing.

Please, will you put a little effort into giving us such a test program to reproduce the problem?

And just to be clear, there's a chance you have indeed found a bug File / SD / SdFat. That's why I'm asking you to give us a way to reproduce the problem!
 
PaulStoffregen said:
I'd like to repeat my request from msg #5. Can you craft a minimal version of your program which anyone who buys the right hardware from Adafruit and connects to Teensy the same way you have can run the program to reproduce problem?

So far the most you've shown us in msg #14 "The pertinent code". But that's not a complete program I or anyone else can run on a Teensy board to reproduce the problem. We need a complete program, so there's no guesswork. We also need details like a link or exact part number for the Adafruit camera and clear description or photos showing us how to wire it the same way, so we can reproduce the same crash you're seeing.

Please, will you put a little effort into giving us such a test program to reproduce the problem?

And just to be clear, there's a chance you have indeed found a bug File / SD / SdFat. That's why I'm asking you to give us a way to reproduce the problem!
Click to expand...
Considering the size of the program and the infrequency of crashes, I doubted that anyone would or should try to understand the whole thing, buy a hodgepodge of sensors to duplicate my environment, try somehow to duplicate the sensors' environment, and try to find the bug. I thought (post #1) that someone might find the symptoms I described familiar, suggesting a testable hypothesis. Some good thoughts appeared (do I use SD code inside an ISR? Do I use Strings incautiously?) but (no, and no, I don't use Strings at all). As recently as post #30 I was entertaining the possibility that the problem was a memory leak somewhere in the vast body of code, somehow mounting up at irregular intervals and clogging the stack (but no, over several days of continuous operation, there is no memory leak).

The too-many-files (TMF) hypothesis that I also mentioned in post #30 is the only other explicit testable hypothesis that has been mentioned in this thread. For reasons having to do with my file-naming scheme (affected by my sensors' environment), the TMF hypothesis is attractively consistent with the irregular appearance of the crashes. It will be easy to test, and (as stated) I'll be on it as soon as I dig out from having been away. If that test shows that the TMF hypothesis is wrong, then I'll be at a loss, and I'll upload the whole thing.
 
OK, some form of the TMF hypothesis is correct; it is µSD-card-format-dependent. This code
Code: 
const int msOneSecond     = 1000;
const char RootFileName[] = "/";
const char PERIOD         = '.';

#include <SD.h>
const int chipSelect = BUILTIN_SDCARD;

bool DeletedAllFiles()
  { bool Result = false;
    File Root = SD.open(RootFileName);

    if (not Root)
      { Serial.println("Couldn't open root directory"); }
    else
      { // Root opened
        while (true)
          { // one entry
            File Entry = Root.openNextFile();
            if (not Entry)
              { break; }
            else
              { SD.remove(Entry.name()); }
          } // one entry
        Result = true;
        Root.close();
      } // Root opened

    return Result;
  } //  DeletedAllFiles()

void setup()
  { char FileName[12];

    Serial.begin(9600);
    if (CrashReport)
      { Serial.print(CrashReport); }

    while (not SD.begin(chipSelect))
      { Serial.println("Couldn't start SD package.");
        delay(msOneSecond);
      }

    if (DeletedAllFiles())
      { // uSD card empty
        for (int I = 1; I < 1000; I++)
          { // create and try to write Ith file
            sprintf(FileName, "%d.txt", I);
            CrashReport.breadcrumb(1, 10 * I + 1);
            File TheFile = SD.open(FileName, FILE_WRITE);
            CrashReport.breadcrumb(1, 10 * I + 2);
            TheFile.write(FileName, 12);
            CrashReport.breadcrumb(1, 10 * I + 3);
            TheFile.close();
            CrashReport.breadcrumb(1, 10 * I + 4);
            Serial.print(PERIOD);
            if ((I % 50) == 0)
              { Serial.printf(" %d\n", I); }
          } // create and try to write Ith file
      } // uSD card empty
    else
      { Serial.println("Couldn't clear SD card"); }
  } // setup

void loop()
  { } // loop
happily generates and writes 999 files in the root directory of a FAT32-formatted µSD card. When applied to a FAT-formatted card, its Serial Monitor output is the same, but in fact, only 508 files are written. The TheFile.write call doesn't complain, and CrashReport is never triggered, but after the first 508 files, TheFile.write does nothing.

Now, this is pathological, but it is not exactly the same behavior that I was seeing in my application. There, the app crashed because there was no File object for the write to be a method of.
 
Last edited:
Fenichel said:
File TheFile = SD.open(FileName, FILE_WRITE);
Click to expand...
After this you should check if the call succeeded with
Code: 
if (TheFile.isOpen())
or even
Code: 
if (TheFile)
before blindly assuming the IO will succeed.

TheFile.write() returns the number of bytes actually written to disk. So if it returns 0 or less than the number requested then you know there is a problem.

Is it a great API? No. But its what the Arduino guys invented years ago and Paul et al have invested heaps of effort to emulate.
 
Last edited:
thebigg said:
After this you should check if the call succeeded with
Code: 
if (TheFile.isOpen())
or even
Code: 
if (TheFile)
before blindly assuming the IO will succeed.

TheFile.write() returns the number of bytes actually written to disk. So if it returns 0 or less than the number requested then you know there is a problem.

Is it a great API? No. But its what the Arduino guys invented years ago and Paul et al have invested heaps of effort to emulate.
Click to expand...
Code: 
if (TheFile)
is essentially what I'm doing (see post #14), and it prevents crashes. The crash-prevention problem is solved, but since then this thread has been concerned with trying to find under what circumstances the result of SD.open(<filename>, FILE_WRITE) behaves like a false/0/null value. There are probably many such circumstances: improper file-name format, ill-timed physical removal of the card, and so on. None of these was present in my app, so other possibilities (e.g., interfering memory exhaustion, unrelated to the µSD-related code (disproven), TMF (likely culprit)).
 
The FAT16 filesystem has a limited number of file entries available for the root directory. FAT32 removes this limitation, or (a more sensible solution) you could place your files in a subdirectory. You should still be checking if a valid File object is returned by open() since the card may run out of space eventually.
 
jmarsh said:
The FAT16 filesystem has a limited number of file entries available for the root directory. FAT32 removes this limitation, or (a more sensible solution) you could place your files in a subdirectory. You should still be checking if a valid File object is returned by open() since the card may run out of space eventually.
Click to expand...
Yes, subdirectories are the way to go. I have to decide whether I want to organize the files by sorting them into a few subdirectories, or just use one big subdirectory whose sole strength will be that it's not the root. In any event, I consider this problem to be solved but for the scutwork.
 
Fenichel said:
The TheFile.write call doesn't complain
Click to expand...

It is supposed to tell you the number of bytes actually written. So if it wasn't able to write, it is supposed to "complain" by telling you the work it was actually able to perform. It can't do anything else to "complain" since it just returns an integer. However, your program in msg #36 isn't doing anything with the write() return value. If you were to add simple code to check the return value for zero, you would probably be able to detect when writing is no longer possible.
 
It should perhaps be noted that the test sketch Paul requested in posts #5 and again in #34 is to do with getting the SD card into a state where the first call to begin() fails, but subsequent ones (probably? possibly?) succeed. It's nothing to do with the Too Many Files issue, and proper error checking in accordance with existing filesystem documentation.
 
You must log in or register to reply here.
Back
Top