General help understanding memory management on Teensy

Status
Not open for further replies.

BlakeB

Member
Can someone point me to an explanation of how memory is managed on the Teensy device? I'm working on a decent-sized project and feel like I'm just fumbling around in the dark with this stuff, and continue to get stuck on trivial dumb (some might even say spooky) stuff that feels like I could avoid if I just actually knew what was going on.

Our project is probably about 10k lines of code at this point, so like I said decent sized but nothing crazy large. We build in VS Code on Windows, using a makefile project generated by a tool called VSTeensy. AFAIK it's all pretty straightforward. When I build/upload the sketch, the tools tell me we're using like ~45% of available memory, so again it doesn't seem like we should really have worries about this.

The "big" problem: I have a singleton object that once contained an array of 200 items, each of which is 8-16 bytes in size. When I upload the sketch, the device hangs at boot. Reducing the size of the array finally got us back up and running, and via trial and error we determined that 50 items was roughly the maximum size we could use. I later determined that if I took the array out of the class definition, and referred to the external array from within the class, it could handle 75 or so items, but at roughly 100 it would run for a little while then hang, and anything larger would just hang at boot again. Nothing is being dynamically allocated in the code, it's just a single instance of this class declared as a global variable holding a bunch of state. This one feels just like lack of knowledge on my part, like whatever memory management code there is on the system can't handle what I've done, so if I knew how to appease it or just manage the location of stuff on my own I could make this work as desired. From there, though, things got weird. Both of the following are real things that have actually happened to me in the past week:

ii. We found a global variable that is a small array of structures, like 10 items of size 20-32 bytes roughly. When we change the *name* of the variable in the code, the resultant sketch won't boot on the device. Cleaned the build from the IDE and rebuilt, no love. Cleaned the tree using 'git clean -xdf', rebuilt, still no joy. Change the name back, rebuild, all good again.

iii. Have another global boolean variable that's initialized to true. All it does is indicate whether a line of text should be shown on the display at startup, and we've built and run the code with the line displayed and without. When we change the value of the variable to false, however, the device again won't boot. Again same as before, clean build has no effect, and everybody's happy as soon as the variable is set to true.

If anyone can share any experience/knowledge/docs/whatever related to properly laying out stuff in memory and/or debugging this stuff when it happens, I'd be much obliged.
 
Assuming this is a T_4.x?
This thread has links to general T_4 memory info :: pjrc.com/threads/63695-malloc-free-for-EXTMEM-and-DTCM

That gives details on memory layout and how to place things in alternate areas and understand where they might best work.

There is a windows tool that will give an idea of true allocation after compile :: github.com/KurtE/imxrt-size
That page may link to the forum pages and there is a Frank_B verion for runtime printing as well. Look at the .h for this library to print perhaps :: github.com/FrankBoesing/T4_PowerButton

If those details can be exposed for the situation it may show where things are causing troubles or lead to info needed to help.
 
Teensy uses the GCC compiler so all of the tools used for that are available including those that tell you about memory usage. (size, readelf, etc) At root a linker script (specific to each device, mk66fx1m0.ld for example) is used to put objects from various segments (text, data, bss, and variants) into their final locations.
 
Which Teensy are you using?

an array of 200 items, each of which is 8-16 bytes in size
So, a maximum of 3200 bytes which would be no problem at all on a T3.x or T4.x and even on a Teensy 2 shouldn't cause problems.

When we change the value of the variable to false, however, the device again won't boot
That makes no sense at all. Unless, it does boot but the false value causes something to immediately blow up and it looks like it hasn't booted.

Pete
 
Which Teensy are you using?


So, a maximum of 3200 bytes which would be no problem at all on a T3.x or T4.x and even on a Teensy 2 shouldn't cause problems.


That makes no sense at all. Unless, it does boot but the false value causes something to immediately blow up and it looks like it hasn't booted.

Pete

100% concur, neither the variable rename nor the swapping of the boolean make any sense, unless there's some secondary effect as you mention. If they're there, though, I can't find 'em, or at least haven't yet :/. Device is a Teensy 3.6, FWIW. I didn't think the size of the object should cause issues, either. I'll dump the object sizes to the serial port and see if I can't get more specific about numbers. Thanks!
 
Teensy uses the GCC compiler so all of the tools used for that are available including those that tell you about memory usage. (size, readelf, etc) At root a linker script (specific to each device, mk66fx1m0.ld for example) is used to put objects from various segments (text, data, bss, and variants) into their final locations.
Cool, will look into it further. I don't have a ton of experience with GCC toolchain, most of my work over the years has been MS tools and, for the past several years, Clang on MacOS.
 
Sorry really not enough information to give too many hints as for example I am not seeing any code or even compiler information

Note: the stuff that @defragster mentioned is more specific to T4.x boards and I believe you say it is a 3.6.

When I get crashes like this some of the obvious things I look into include:
a) over running of the arrays and wiping out other memory. Some reasonably common gotchas is something like:
Code:
uint32_t my_array[10]; 
...
my_array[10] = 0;
As the array indexes are 0 based so only valid for 0-9

b) stack overruns. Example recursive function that puts lots on stack... Don't see that very often...
But do see things like someone puts a very large array into a function on the stack.

...

Steps to help debug...
Does the program startup at all and at least make it to calling setup()?
If so I make sure that I wait until Serial object is ready and start adding lots of Serial outputs to try to localize down.


I will often do things like:
Code:
void setup() {
    while (!Serial && millis() < 5000) ; // wait up to 5 seconds for Serial port to connect
    Serial.begin(115200);
    Serial.println("Setup called"); Serial.flush(); // make sure it outputs.
    init_driver1();
    Serial.println("After init_driver1"); Serial.flush();
 ...
    Serial.println("End Setup"); Serial.flush();
}
Note: You don't always need/want to do Serial.flush() as it slows your program down. But when I have crash or total hang, I often adds them as without them, several prints might be waiting around to be printed when system hangs and you try debugging from the last one... When the program may have gone a lot further.

I personally also do a lot of digitalWrite or digitalToggle of IO pins that I am not using (Of course you also need to do pinMode(pin, OUTPUT);)
As I hook up one of my logic analyzers to then monitor the sketch.

At times I may also add other debug code, like guessing how large the stack has gotten and how big the heap is and the like but again don't have any ideas from what you have specified.
 
> do Serial.flush()

I've been caught by the fact that Serial.flush() doesn't actually wait until the data is seen by the PC. For that you also need a delay.
 
We found a global variable that is a small array of structures, like 10 items of size 20-32 bytes roughly. When we change the *name* of the variable in the code, the resultant sketch won't boot on the device. Cleaned the build from the IDE and rebuilt, no love. Cleaned the tree using 'git clean -xdf', rebuilt, still no joy. Change the name back, rebuild, all good again.

Can I talk you into trying the Arduino IDE, not to actually edit your code but just to try reproducing this particular problem?

Hopefully you can copy & paste just that variable and maybe a small amount of code which actually uses it (so the compiler doesn't optimize it away) and some Serial.print() lines to see where the program hangs, to ultimately get the same problem using only the Arduino IDE. Then you could post that program here in a message, so any of us could just copy it into Arduino and load it onto a Teensy 3.6 to reproduce the problem.

We're much better at helping with problems we can reproduce. Even if using Arduino's feature-poor IDE feels like riding a tricycle, it does have the very nice advantage that we all have the exact same Arduino software, so it really is the best path to making a code sample that lets any of us easily reproduce the problem. I hope you can endure Arduino's IDE just long enough so we can help figure out what's really going wrong with this array of structures.
 
I've run into array index errors, where trying to access an undefined vector index causes a crash. Coming from the web development world myself -- is there a way to "catch" exceptions like that? I assume you'd have to just build it by making wrappers around things like the vector library or something?

Also, I'd love it if I could use CLion, as I use JetBrains IDEs for my day job. I know others have gotten it to work, but I don't want the added complexity of having to fight an IDE.
 
There are some exceptions that might get triggered and have default handlers. The default is fault_isr() which tries to print some info. You can use something else if it isn't providing the information you want.
 
Can I talk you into trying the Arduino IDE, not to actually edit your code but just to try reproducing this particular problem?

Hopefully you can copy & paste just that variable and maybe a small amount of code which actually uses it (so the compiler doesn't optimize it away) and some Serial.print() lines to see where the program hangs, to ultimately get the same problem using only the Arduino IDE. Then you could post that program here in a message, so any of us could just copy it into Arduino and load it onto a Teensy 3.6 to reproduce the problem.

We're much better at helping with problems we can reproduce. Even if using Arduino's feature-poor IDE feels like riding a tricycle, it does have the very nice advantage that we all have the exact same Arduino software, so it really is the best path to making a code sample that lets any of us easily reproduce the problem. I hope you can endure Arduino's IDE just long enough so we can help figure out what's really going wrong with this array of structures.

Thanks, I'll see what I can do with a simple app for a repro. I'm afraid it's not practical to attempt to move my entire project back to the Arduino IDE and give you a copy of the whole thing, because it's proprietary code for a prototype device, but if I really get stumped after staring at Git diffs etc I'll try to put a tiny app together that breaks the same way. Either way I'll post the resolution when I figure it out. Most likely it's PEBKAC anyway :).
 
Another possibility. If you aren't using the Arduino environment, what are you doing for startup code? The C startup can spend a fair amount of time copying data for initialized global variables and zeroing the bss. (And C++ constructors if they exist.) If that takes enough time the watchdog can fire before it is done.
 
This might be a long shot, but it sounds like you might be using enough memory that you could have a variable (a large array) crossing the SRAM_L and SRAM_U boundary, which under some access conditions can cause a hard fault. Check the memory placement in the symbol file to see if something is crossing over the 0x20000000 address. With the Arduino IDE, the memory placement can be verified in the <name>.ino.sym file located in C:\Users\<username>\AppData\Local\Temp\arduino_build_<rnd_number>. This was something I encountered on the Teensy 3.2, so it might be a different on the 3.6.

More about what I ran into and how I worked around it: https://forum.pjrc.com/threads/5832...readBytes-hang?p=233075&viewfull=1#post233075
 
...to see if something is crossing over the 0x20000000 address.... With the Arduino IDE, the memory placement can be verified in the <name>.ino.sym file located in C:\Users\<username>\AppData\Local\Temp\arduino_bui ld_<rnd_number>...

Since you are using VisualTeensy I recommend to install the new version which I just released. It uses gnu nm to generate a nicely sorted *.sym file. Should be quite easy to check if something crosses this border. The *sym file is located at {project folder}/.vsteensy/build.
 
Last edited:
Status
Not open for further replies.
Back
Top