Jeff Williams
Member
Hello,
I am working on a data logger for a trackday (race track) motorcycle. The full code requires CAN bus, GPS into Serial1 etc so below is a skeleton of the same structure and though doesn't repeat the issue as often it will break and cause crashes.
Full implementation hardware is: Teensy 4.1, a NEO-M9N GPS receiver via Serial1, Adafruit Airlift ESP32 based wifi module via SPI
Test program below hardware: Teensy 4.1 + extra output on Serial4 for debugging (crash report)
In my real code I use threads and an interval timer and loop as follows:
I am outputting to Serial in my example where I would be putting strings out to the TCP server in my real implementation. Also I use a Mutex to control updating and reading of my bike stats class between the CAN handler and the rc3TimerHandler usually but did not include in my example below.
I have a 2nd Serial connection on Serial4 to view the crash reports while I debug since Serial is scrolling with data soon after the crash report when it happens.
If I remove the Mutex call and output in rc3TimerHandler for my full implementation the program is stable for 30min but as soon as I enable my rc3 output it crashes usually within seconds.
For the example code below I get the following crash report: (as you can see it crashed within 1 min sometimes, 4 min and I also have had it run for 15min so I assume some race condition?)
When I try to find what line that is using addr2line: (which isn't really helpful)
Looking in the lst file for the program it seems to be in the Teensy Threads code?
I am wondering if there is some sort of conflict between the IntervalTimer and the interrupt/timing of Teensy Threads?
Looking for any thoughts on another approach if there is some conflict between the items I am using. I have had my real implementation working with just GPS and LCD but with my new motorcycle I have CAN bus and want to add data from it to the output that goes to the phone for actual lap timing and logging. (app called Race Chrono)
Edit: Additional details once I typed initial post
I use TyCommander to monitor Serial output of the Teensy. If I click on the Serial button to stop the Serial data scrolling and then try to connect the Serial again by clicking button again the Teensy crashes within seconds without showing any Serial data. Further more if I don't try to reconnect the Serial it also crashes pretty quickly. I have yet to investigate this new finding further (it's the same crash execution address in the crash report).
Suggestions/Comments Welcome
Thanks, Jeff
I am working on a data logger for a trackday (race track) motorcycle. The full code requires CAN bus, GPS into Serial1 etc so below is a skeleton of the same structure and though doesn't repeat the issue as often it will break and cause crashes.
Full implementation hardware is: Teensy 4.1, a NEO-M9N GPS receiver via Serial1, Adafruit Airlift ESP32 based wifi module via SPI
Test program below hardware: Teensy 4.1 + extra output on Serial4 for debugging (crash report)
In my real code I use threads and an interval timer and loop as follows:
- loop will update a LCD (didn't include in the example)
- thdGPS function is a Teensy Threads thread which reads from Serial1 which is GPS, does some GPS processing and then sends the GPS data out WiFi TCP connection via Adafruit Airlift
- rc3TimerHandler function is the handler for my interval timer that runs at 40Hz. this gets data from a class which is populated by CAN bus and then puts the data out WiFi TCP connection via Adafruit Airlift
I am outputting to Serial in my example where I would be putting strings out to the TCP server in my real implementation. Also I use a Mutex to control updating and reading of my bike stats class between the CAN handler and the rc3TimerHandler usually but did not include in my example below.
I have a 2nd Serial connection on Serial4 to view the crash reports while I debug since Serial is scrolling with data soon after the crash report when it happens.
If I remove the Mutex call and output in rc3TimerHandler for my full implementation the program is stable for 30min but as soon as I enable my rc3 output it crashes usually within seconds.
For the example code below I get the following crash report: (as you can see it crashed within 1 min sometimes, 4 min and I also have had it run for 15min so I assume some race condition?)
Code:
CrashReport:
A problem occurred at (system time) 22:3:49
Code was executing from address 0x850
CFSR: 40000
(INVPC) Usage fault: invalid EXC_RETURN
Temperature inside the chip was 50.26 °C
Startup CPU clock speed is 600MHz
Reboot was caused by auto reboot after fault or bad interrupt detected
Setup Complete
Report coming ...
CrashReport:
A problem occurred at (system time) 22:7:33
<SNIP>
Setup Complete
Report coming ...
CrashReport:
A problem occurred at (system time) 22:8:25
<SNIP>
Setup Complete
Report coming ...
CrashReport:
A problem occurred at (system time) 22:12:27
When I try to find what line that is using addr2line: (which isn't really helpful)
Code:
addr2line -e C:\Users\jeff\AppData\Local\Temp\arduino_build_666607\TestCrash.ino.elf -a 0x850
0x00000850
arm-none-eabi-addr2line: Dwarf Error: Can't find .debug_ranges section.
libc_a-__call_atexit.o:?
Looking in the lst file for the program it seems to be in the Teensy Threads code?
Code:
848: f8c2 8000 str.w r8, [r2]
__enable_irq();
84c: b662 cpsie i
__asm volatile("svc %0" : : "i"(Threads::SVC_NUMBER));
84e: df21 svc 33 ; 0x21
__disable_irq();
850: b672 cpsid i
int old_state = currentActive;
852: 6813 ldr r3, [r2, #0]
currentActive = STOPPED;
854: 6014 str r4, [r2, #0]
__enable_irq();
856: b662 cpsie i
I am wondering if there is some sort of conflict between the IntervalTimer and the interrupt/timing of Teensy Threads?
Looking for any thoughts on another approach if there is some conflict between the items I am using. I have had my real implementation working with just GPS and LCD but with my new motorcycle I have CAN bus and want to add data from it to the output that goes to the phone for actual lap timing and logging. (app called Race Chrono)
Edit: Additional details once I typed initial post
I use TyCommander to monitor Serial output of the Teensy. If I click on the Serial button to stop the Serial data scrolling and then try to connect the Serial again by clicking button again the Teensy crashes within seconds without showing any Serial data. Further more if I don't try to reconnect the Serial it also crashes pretty quickly. I have yet to investigate this new finding further (it's the same crash execution address in the crash report).
Suggestions/Comments Welcome
Thanks, Jeff
Code:
#include <Arduino.h>
#include <TeensyThreads.h>
IntervalTimer rc3Timer;
Threads::Mutex mutexOutput;
#define GPS_BUFFER_SIZE 150
#define RC3_HZ 40
char gpsLine[GPS_BUFFER_SIZE];
int gpsCount = 0;
char rcLine[GPS_BUFFER_SIZE];
char tempRCLine[GPS_BUFFER_SIZE];
class StringDumper : private Print
{
public:
StringDumper(const Printable &p)
{
this->println(p);
}
operator const char *() const
{
return buf.c_str();
}
protected:
size_t write(uint8_t b) override
{
buf.append((char)b);
return 1;
}
String buf;
};
void rc3TimerHandler()
{
sprintf( rcLine, "Normalling this would be logging data: %lu\r\n", millis() );
{
Threads::Scope scope(mutexOutput);
Serial.print(rcLine);
}
}
int thdIdGPS = -1;
void thdGPS()
{
while (true)
{
sprintf( gpsLine, "Normalling this would be GPS data: %lu\r\n", millis() );
{
Threads::Scope scope(mutexOutput);
Serial.print(gpsLine);
}
memset(gpsLine, 0, sizeof(gpsLine));
gpsCount = 0;
threads.yield();
}
}
void setup() {
Serial.begin(115200);
Serial4.begin(115200);
if (CrashReport)
{
StringDumper report(CrashReport);
// doing it like this because in the real program I put the crash report out to SD and Serial4 for debug
Serial.println("Report coming ...");
Serial4.println("Report coming ...");
delay(10000);
Serial.print(report);
Serial4.print(report);
delay(10000);
}
threads.setSliceMicros(10);
Serial.println("Set default time slice");
thdIdGPS = threads.addThread(thdGPS);
Serial.println("Added GPS thread");
rc3Timer.begin(rc3TimerHandler, 1000000 / RC3_HZ);
Serial.println("Started RC3 timer");
Serial4.println("Setup Complete");
}
void loop() {
// put your main code here, to run repeatedly:
}
Last edited: