T4 CrashReport Debugging on a Mac

TeensyDigital

Well-known member
I just spent ten days debugging a thread related crash and I am super grateful to Defragster and mjs513's help pointing me to the right tools. For future travelers also trying to troubleshoot crashes, I wanted to consolidate a few things in this thread.

The (relatively) new CrashReport capability -- as of TD1.54 -- is awesome. Instead of the board hanging, it will reboot 8 seconds after a panic and on start-up you can capture the fact that it crashed and some basic information about the crash. If you combine this with NVRAM markers that also survive a warm reboot you can get more insight into what part of your code failed. I know this is documented in many places in the forum, but below is a short sketch on how these are used together with a simple thread thrown in to show how it can span any part of your code.

Code:
// Example of using persistent NVRAM Words along with CrashReport to mark code for crash analysis on a T4
// This only uses one NVRAM word (NVRAM_UINT32[0]) but there are four available [1],[2].[3])
// This is a glorified version of serial.println("I made it this far :)");



  #include <TeensyThreads.h>
  int id_sampling;
  uint32_t *NVRAM_UINT32 ((uint32_t *)0x400D4100);
  elapsedMillis tSome;
  volatile unsigned long lClock;
  volatile byte killIt = 0; 

void setup() {
  
  SNVS_LPCR |= (1 << 24); //Enable NVRAM - documented in SDK
  
  Serial.begin(115200);

  if(CrashReport) {
    Serial.print(CrashReport);
    Serial.println(" ");
    Serial.print("NVRAM Marker Value = ");
    Serial.println( NVRAM_UINT32[0] );
    while(1);  // stop if crashed, but you don't have to
  } 

  //create the thread - nothing fancy
  id_sampling = threads.addThread(sample_thread);
  delay(2000);

  tSome = 0;
  Serial.println("PRESS A KEY TO PLAY:     '1':CRASH 1         '2':CRASH 2        'k':Kill Thread Crash    \n");
}

void loop() {
  if ( tSome >= 5000 ) {
    tSome = 0;
    Serial.println("Press a key, I'm still waiting to die... ");
    if(millis() > (lClock + 1500)) {
      Serial.print("Sorry, but the thread is dead...");
      Serial.println(lClock);
    }
  }
  
  while ( Serial.available() ) {
    char cc = Serial.read();
    if ( '1' == cc ) {
      NVRAM_UINT32[0] = 1;  // set marker value here
      int* p = nullptr;
      *p = 33;  // This will force crash and burn :-O
    }
    else if ( '2' == cc ) {
      NVRAM_UINT32[0] = 2;  // set marker marker here
      int* p = nullptr;
      *p = 33;  // This will force crash and burn :-O

    }
    else if ( 'k' == cc ) {
      killIt = 1;
    }
  }
}

void sample_thread() {

 elapsedMillis lSome;
 lSome = 0;

 while(1) {
   if (lSome > 500) {
    lClock = millis();
    lSome = 0;
   }
   if(killIt == 1) {
    killIt = 0;
    NVRAM_UINT32[0] = 3;
      int* p = nullptr;
      *p = 33;  // This will force crash and burn :-O    
    
   }
 }
}

On a Mac...

When CrashReport reports out it will display "Code was executed from address 0xC4". To associate the hex code with a line of code from your sketch you need to run addr2line against the sketch .elf file with the following convention: addr2line -e mySketch.elf 0xC4 (where 0xC4 is the line from CrashReport).

On a Mac running Big Sur and TD 1.56 the addr2line executable is in: /Applications/Teensyduino.app/Contents/Java/hardware/tools/arm/bin/arm-none-eabi-addr2line

To find the elf files on a Mac, open the Arduino preference file: Library/Arduino15/preferences.txt and find the value of "last.sketch.default.path". It should be a temp file path deep in /var/folders. Navigate to the folder in terminal and execute addr2line.

Happy troubleshooting!

Mike
 
Helpful summary note Mike.

Also note TeensyDuino 1.57 (now in Beta at #3) will be adding .breadcrumbs.

I just provided a sample to PJRC that is available here: PaulStoffregen/MyFault/blob/main/examples/BreadCrumb/BreadCrumb.ino

Here is the code for that again showing sample use of CrashReport and the new .breadcrumb()'s as well.

Code:
// MyFault - collection of examples to trigger fault exceptions

// Example for :  void breadcrumb(unsigned int num, unsigned int value) {
[B][COLOR="#FF0000"]// num is a value 1 to 6 for possible unique storage entries[/COLOR][/B]

void setup() {
  if ( !CrashReport ) {
    CrashReport.breadcrumb( 1, 0x5000000 | __LINE__ ); // Upper bits hold '5' perhaps indicating func() for ref, lower bits show line #
    CrashReport.breadcrumb( 2, millis() );
    Serial.begin(9600);
    CrashReport.breadcrumb( 3, millis() );
    while (!Serial && millis() < 4000 );
    CrashReport.breadcrumb( 4, millis() );
    Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);
    CrashReport.breadcrumb( 5, millis() );
    CrashReport.breadcrumb( 6, 0xdeadbeef );
    *(volatile uint32_t *)0x30000000 = 0; // causes Data_Access_Violation
  }
  else {
    while (!Serial && millis() < 10000 );
    Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);
    Serial.print(CrashReport);
  }
}

void loop() {
}
 
Last edited:
@defragster

Should I expect breadcrumbs to work when I have them spread out amongst a bunch of files in a larger project? I am using Platformio and have code my code spread out in a bunch of header and cpp files, but I can only get the CrashReport breadcrumbs to print that are in main.cpp

@TeensyDigital
Similarly, should I expect the addr2line program to work when code is in several different files? Do I need to compile with Teensyduino for that program to work? I am compiling the elf with platformio and see this when running addr2line with the CrashReport hex code:

Code:
❯ addr2line -e .pio/build/release/firmware.elf 0x3A160
:?
 
@defragster

Should I expect breadcrumbs to work when I have them spread out amongst a bunch of files in a larger project? I am using Platformio and have code my code spread out in a bunch of header and cpp files, but I can only get the CrashReport breadcrumbs to print that are in main.cpp

@TeensyDigital
Similarly, should I expect the addr2line program to work when code is in several different files? Do I need to compile with Teensyduino for that program to work? I am compiling the elf with platformio and see this when running addr2line with the CrashReport hex code:

Code:
❯ addr2line -e .pio/build/release/firmware.elf 0x3A160
:?

Yes, With TD 1.57 installed - if it builds then any call to CrashReport.breadcrumb() should be working as long as the passed first parameter is between 1 and 6.

Can you create/post a minimal ZIP file holding such a sketch that will build and run in the IDE?
Code:
Foo.zip of Folder: Foo
 holding files: Foo.ino, Foo.cpp, Foo.h

When the IDE is used to build the addr2line generally works to kno wthe filename and line location.
 
Yes, With TD 1.57 installed - if it builds then any call to CrashReport.breadcrumb() should be working as long as the passed first parameter is between 1 and 6.

Between 1 and 6 being the key here! MB for not looking at the breadcrumb code... I haven't had a chance to test this yet but will do so soon.
 
Between 1 and 6 being the key here! MB for not looking at the breadcrumb code... I haven't had a chance to test this yet but will do so soon.

Put Bold and Color emphasis on that line in code above. Indeed, anything but index of 1 - 6 is ignored and cannot be stored.
 
...... I am using Platformio and have code...I am compiling the elf with platformio and see this when running addr2line with the CrashReport hex code:

I use platformIO and had the same problem - you need to tell the compiler to compile with debug info - this is default on the Arduino IDE but not with platformIO.

I included the -g option in the .platformio.ini like:
Code:
[env:teensymm]
platform = teensy
board = teensymm
framework = arduino
build_flags = -D USB_MIDI_SERIAL -g

For me the right version of addr2line is in :
Code:
 /Users/paul/.platformio/packages/toolchain-gccarmnoneeabi/bin

If I point to the .elf then it decodes the line from any of the files in the project eg.
Code:
./arm-none-eabi-addr2line -a 0xDF4A -e /Users/paul/Documents/GitHub/SpaceDelay282/hex\ files/firmware.elf
0x0000df4a
/Users/paul/.platformio/packages/framework-arduinoteensy/cores/teensy4/usb_midi.c:306
 
I use platformIO and had the same problem - you need to tell the compiler to compile with debug info - this is default on the Arduino IDE but not with platformIO.

This works perfectly! This tool is going to save me a ton of debugging time. Thanks for the tip :)
 
This works perfectly! This tool is going to save me a ton of debugging time. Thanks for the tip :)

@defragster mentioned the breadcrumbs in version 1.57 - they are super useful to use along with crash report.

I sprinkle the breadcrumbs in areas that I'm interested, typically on entry to key functions e.g.
Code:
void audioboard_initialise() {
  CrashReport.breadcrumb(1, __LINE__);
...

The CrashReport will then include the line number that last updated the breadcrumbs.

To make it easy to include them and to also be able to turn them off I do this:
Code:
#define DEBUG_BCRB 1
#if DEBUG_BCRB
  #define BCRMB1 CrashReport.breadcrumb(1, __LINE__)
#else
  #define BCRMB1 (void)0
#endif

Then sprinkle them around with...

void audioboard_initialise() {
  BCRMB1;
....

There may be better way to include them / remove them, this seems to work for me. Cheers Paul
 
Hi All,

This is what I get:

CrashReport:
A problem occurred at (system time) 22:56:40
Code was executing from address 0x1C7A
CFSR: 82
(DACCVIOL) Data Access Violation
(MMARVALID) Accessed Address: 0x18 (nullptr)
Check code at 0x1C7A - very likely a bug!
Run "addr2line -e mysketch.ino.elf 0x1C7A" for filename & line number.
Temperature inside the chip was 50.09 °C
Startup CPU clock speed is 600MHz
Reboot was caused by auto reboot after fault or bad interrupt detected

Not sure what is causing this but believe it is in that loop with the extra code. I will see if I can move it out of the Interrupt function and try again.
 
Hi All,

This is what I get:

CrashReport:
A problem occurred at (system time) 22:56:40
Code was executing from address 0x1C7A
CFSR: 82
(DACCVIOL) Data Access Violation
(MMARVALID) Accessed Address: 0x18 (nullptr)
Check code at 0x1C7A - very likely a bug!
Run "addr2line -e mysketch.ino.elf 0x1C7A" for filename & line number.

Temperature inside the chip was 50.09 °C
Startup CPU clock speed is 600MHz
Reboot was caused by auto reboot after fault or bad interrupt detected

Not sure what is causing this but believe it is in that loop with the extra code. I will see if I can move it out of the Interrupt function and try again.

If the notes are followed using the ELF file - it should show the affected line of code directly.

Indication is the Interrupt is abusing a NULL pointer
 
Side note: If you are using the current released copy of Teensyduino (or anything before the current Beta versions), then the addr2line stuff works great.

For the current Teensyduino betas using the newer version of GCC, I have found that the addr2line stuff is not giving me good results.
So with these betas, what I do is to open up the generated .lst file (in the same temp directory as your elf file)
And simply search for the address. note when you do the search don't use the 0x in the search.

One of my last crashes (on MMOD) was in the function
Code:
00006fc0 <BluetoothController::handle_hci_disconnect_complete()>:
{
    6fc0:	b530      	push	{r4, r5, lr}
    6fc2:	4604      	mov	r4, r0
    6fc4:	b083      	sub	sp, #12
    DBGPrintf("    Event: HCI Disconnect complete(%d): handle: %x, reason:%x", rxbuf_[2],
    6fc6:	491e      	ldr	r1, [pc, #120]	; (7040 <BluetoothController::handle_hci_disconnect_complete()+0x80>)
              rxbuf_[3] + (rxbuf_[4] << 8), rxbuf_[5]);
    6fc8:	f894 2b64 	ldrb.w	r2, [r4, #2916]	; 0xb64
    6fcc:	f894 3b63 	ldrb.w	r3, [r4, #2915]	; 0xb63
    DBGPrintf("    Event: HCI Disconnect complete(%d): handle: %x, reason:%x", rxbuf_[2],
    6fd0:	f894 5b65 	ldrb.w	r5, [r4, #2917]	; 0xb65
    6fd4:	eb03 2302 	add.w	r3, r3, r2, lsl #8
    6fd8:	481a      	ldr	r0, [pc, #104]	; (7044 <BluetoothController::handle_hci_disconnect_complete()+0x84>)
    6fda:	f894 2b62 	ldrb.w	r2, [r4, #2914]	; 0xb62
    6fde:	9500      	str	r5, [sp, #0]
...
So for example if the address given was 0x6fc8
Searching this file for 6fc8, would bring you to the actual instruction and the comments show which function and clues for where in the function.
 
Hi All,

I couldn't get the program running using Visual Micro (Arduino) so I went around it and programmed it through MCUXpresso and now it works. There is something in Arduino and how it handles null pointers.

Now I am experiencing some very strange behavior. I ported code that works on a Nuvoton M481LGCAE device and that runs at 192 MHz and is a Cortex M4F with hardware floating point support. The reason I mention this is because it runs flawlessly on that device but doesn't work on the MIMXRT1062 as expected. I noticed in the Teensy schematic the USB goes directly into the MIMXRT1062 and not to the bootloader chip. Is the MIMXRT1062 configured or programmed in such a way that enables USB and runs a background program and then the application runs on top of that? In other words, is there something using interrupts that I should know about? Is the loader program always running? It would be helpful to know because I need the full attention of the MIMXRT1062 and I can't have anything running in the background. If something is running in the background, I will need to gut it but then at that stage might as well just design a prototype board with this device. Please let me know.

Thanks,

Eric Norton
 
Hello all! Sorry for commenting on this old thread but I am facing a very similar issue regarding debugging a crash in a Teensy sketch that uses threads (TeensyThreads library).

I have noticed that every time the crash occurs inside a thread, the crash report will report the "yield()" function from TeensyThreads.cpp as the line that crashed, even when the crash was not caused by threading or "context-switching".
For example, if I place "*(volatile uint32_t *)0x30000000 = 0;" instruction anywhere inside the thread function, the CrashReport will always report a hex address that, converted to line number using addr2line will always point to the "yield()" function. Searching the .lst file like @KurtE suggested will lead to the same line. If I place the same crashing line inside the main thread, CrashReport will point the correct line where the program crashed.

Is that the expected behavior? Is that a limitation of CrashReport when using threading? Thanks!
 
...
Is that the expected behavior? Is that a limitation of CrashReport when using threading? Thanks!

TeensyThreads is a unique process not widely used - so many things fail with thread switching in spite of the good work behind it to proved the feature.

CrashReport wasn't designed to work with it, it may take some special lookup to resolve the fault location given the method of task switching if the results are consistent as observed.

If there are crashes to debug than consider adding breadCrumbs to the threads and then localize them to find the actual last executed location.

Found this thread from a couple month back that may relate, or at least show .creadcrumbs() usage: pjrc.com/threads/72952-Specific-timing-of-thread-start-causes-hard-fault-(invalid-EXC_RETURN)-in-Teensy-core

<edit>: See also post #2 above
 
Are you using the latest Teensyduino? There were changes a while ago to make sure the correct stack is examined in the crash handler, which would affect where the fault location is gathered from.
 
If there are crashes to debug than consider adding breadCrumbs to the threads and then localize them to find the actual last executed location.

Found this thread from a couple month back that may relate, or at least show .creadcrumbs() usage: pjrc.com/threads/72952-Specific-timing-of-thread-start-causes-hard-fault-(invalid-EXC_RETURN)-in-Teensy-core

The breadcrumbs functionality seem very nice. However, I am still not sure which strategy to use to help me pinpoint unknown crashes that I have no idea where could have happened (think a 40000+ lines program that crashes intermittently, with 4 cooperative threads). Maybe if I at least knew which thread crashed, this would help. I planned to do this setting new breadcrumbs like CrashReport.breadcrumb(1, threadID) each time when there's is a context switch (threads.yield() and threads.delay() in my case). Would you recommend this approach or maybe there is something else better? Thank you!

Are you using the latest Teensyduino? There were changes a while ago to make sure the correct stack is examined in the crash handler, which would affect where the fault location is gathered from.

I am using Teensyduino 1.58. Are the mentioned changes from 1.59 beta? Thank you!
 
Interesting @jmarsh about stack tracking update. Missed that so not sure when.

Indeed starting with CrashReport.breadcrumb(1, threadID) would be a good start.

Perhaps also/next CrashReport.breadcrumb(2, __LINE__) in that thread in some fashion to track progress.

If interrupt _ISR's or 'device' code are in use, they may the problem and the thread code is where the problem gets 'caught'.
 
You would need at least Teensyduino 1.59 beta #2 (relevant change was "Fault handler use main vs process stack pointer").
 
Thanks @defragster and @jmarsh for the tips!

I will modify my code to start using the CrashReport.breadcrumb functionality to help better track the crashes.
Also, just a quick question: Is the location where CrashReport stores information to be displayed when Teensy reboots different than the NVRAM addresses on Teensy 4.1? (starting from 0x400D4100)?

As for the change in the fault handler stack pointer, I think maybe this will solve my issue! I will try to update my code to Teensyduino 1.59 beta #3. The only issue is that I use VSCode with PlatformIO, so the beta packages are not directly available. I will need manually configure them. Thanks!
 
@jmarsh Applying this patch solved the issue for me! Many thanks. Now if the crash happens inside the thread, the correct place is show in addr2line and the .lst file.
 
Back
Top