Teensy crashes at arround 60°C

[mrmof]

This is my first post on this forum.

I'm developing a commercial project using teensy 4.1 for arround 1.5 years. I'm Using teensyduino.

My project is relatively complex: It uses 7 UARTs (one receiving at least 60kbits/s at 115200 baud), reads 1 SPI sensor by interruption at 2460hz (clocked at 1Mhz), executes a PID loop at 1Khz, etc.

The code is stable and runs nicely for days if the teensy is below 60°C.

Teensy is at enclosed aluminium box with a 50mm cooler directly blowing (internal) air in the processor, so the teensy temperature is very close to the air inside the box. Also, teensy has a 10x10 aluminium heatsink on top of the processor. All heat is taken from the box by conductivity from inside to outside, and the cooler helps the air to circulate and exchange heat with the box.

I tested this with up to 5 teensy boards, and the code always crashes when a critical temperature is reached. This is done with a 1 to 2 hour run, so the temperature rises very slowly.

From the boards I tested, I saw crashes at 53°C, 60°C and 68°C. The critical temperature is consistent and the board crashes always at this temperature +- 1°C. The temperature is being measured by InternalTemperature.h lib.

I cannot post the code for commercial reasons.

Teensy is being powered by 5V in, and the power supply was tested and works above 100°C. Also, teensy is powering a 80mA load at maximum (digital isolators/transceivers) by its 3v3 out.

I tested the board with a blink program and a hot air gun, and the processor stops working at 95°C or higher, but as soon as I remove the gun, teensy restarts.

The catch is: When running my sketch, after the (much lower) critical temperature is reached, teensy crashes and become unrecoverable, until a power cycle or WDT trigger.

I implemented a WDT to reset the board and be able to comunicate again, but a reset is not desireable.

Is there a specified maximum operating temperature for teensy, considering all of its components? What is the maximum temperature of the used clock Crystal?

Is there a good way to debug this crash?

This problem works running the sketch from 150Mhz to 600Mhz at the same temperature, and it seems to not be related to the frequency.

I consider the environment teensy is put to be reasonably high in EMI, because there is some brushless motors and cryocoolers in the box. The cryocooler is the main source of heat.

I apreciate and thank all help I can get

 

[defragster]

Welcome - bummer on the heat fail.

Only time focusing on temp here was during overclocking and indeed even in open air - with heat sink - failures showed over 50°C. When at 900+ MHz the internal core voltage is raised to function and that explained the trouble when internally generated heat not well removed. The Teensy temp is from some point on the silicon - maybe not the highest point - and reported temp just before fail may lag the hot spot causing the failure. When OC'd heat sync and fan in open 'cool' room air could keep it running. And better when Teensy was fully pinned and connected to another PCB to provide more thermal mass to dissipate heat.

That may indirectly relate (in that it sounds like those OC high temp failures) only in that trapped in that box with heat (and noise?) something is going 'out of spec'. If it happens at 528 MHz (the long life rugged automotive/industrial spec speed) with lower core voltage and even 150 MHz just suggests too much heat accumulating in that enclosed box for safe/reliable/proper function.

On restart - if kept powered - the CrashReport code will give some notes - but probably just indicate the obvious:

void setup() {
  Serial.begin(9600); // With TeensyDuino 1.54 (and above) Serial.begin() may return connected
  if ( Serial && CrashReport ) { // Make sure Serial is alive and there is a CrashReport stored.
    Serial.print(CrashReport); // Once called any crash data is cleared
    // In this case USB Serial is used - but any Stream capable output will work : SD Card or other UART Serial
  }

...

 

[bicycleguy]

Need to get that Teensy near the cold side of that cryocooler

Interesting problem. FYI:
Changing CPU frequency on the fly in a vertically mounted socketed T4.1 results in a Time Constant of approx. 30 seconds. So it doesn't take long to see just Teensy effects.

24MHz CPU is about 11C above ambient

MHz     °C rise
24      11
528     21
600     23
912     31

Attached is a ILI9341_t3 screen dump. The ambient was 20C. The lower line is at 24Mhz after a few minutes. The upper shows when switching to 600Mhz a few seconds from left.

 

[JBeale]

"This problem works running the sketch from 150Mhz to 600Mhz at the same temperature, and it seems to not be related to the frequency." If it's independent of the CPU setting, I wonder if there is something to do with the crystal frequency stability vs. temperature, causing an external I/O data misread / miswrite, but the shift would not be large at only +40C over room temp. Also transmitted or conducted EMI is a possibility if any filter capacitors loose some effectiveness at higher temperatures, and/or current spikes go up when your cryocooler is working harder, due to the larger delta-T. One thing to try is run that 80 mA 3.3V load from a separate supply, not the Teensy, in case of any conducted noise there.