What causes the Teensy 3.2 to not boot up properly?

[tyzjames]

I have the Teensy 3.2 built into a product. Every now and then, we'll have a user that reports that the product is "not responding" and Teensyduino cannot detect the device as well, and the only way to resolve this would be to access the program button to load the bootloader, and then reload the firmware. This happens randomly, and probably to 1 in every 500-800 units.

I am using the watchdog timer in my firmware.

So my question is, what are potential causes the Teensy 3.2 to randomly not load the firmware correctly, such that reloading the firmware is necessary to get the device working again?

 

[defragster]

Is this a 'unit in operation' failing to function at some point while running previously? Or is this from a 'cold start'?

When in this state is it the case that removing and repowering the T_3.2 does not return it to normal function? Same question after pressing the Button then removing/repowering the T_3.2 - without reprogramming the firmware?

This happens in " probably to 1 in every 500-800 units. " - does this recur on that same unit over time, or is it generally a onetime event with that user's unit? That is does the T_3.2 generally function for some significant period after that firmware reload without same or other issue?

Does firmware reload do any 'first run' changes to EEPROM or other 'static' variables or stored reference data?

Seems the sample size must be some multiple of "500-800 units"? Impressive count!

Are these T_3.2's end user/home or Industrial environment? Used intermittently or continuously?

 

[PaulStoffregen]

Any chance a DC-DC switching power supply modules is providing the power?

Many years ago Adafruit was selling a tiny Traco brand DC-DC switcher. Its datasheet specifically said an extra capacitor at the input was only needed if the input voltage was near the upper end of the input range (I believe it supported pretty high input, like 24V). But in fact the feedback loop inside which controls the output voltage would go crazy if the input was a weak battery or long cable from a power supply, and a board like Teensy 3.0 would use almost zero current during the ramping up from 0 to about 1.5 volts, then need quite a bit of current.

At Adafruit, the answer to most power supply problems is to add a big capacitor to the output. They sell a lot of cheap power supplies which people use for LED projects, where that advice usually works. But for that little Traco one, more capacitance at the output without any at the input made everything even worse. Sometimes it would turn into about a 7 volt oscillator and never stabilize to 5 volt output. I know all this because someone had a lot of problems in the early days of Teensy 3.0, as I recall 3 of them connected to an Arduino Mega. The Mega would always boot up even with pretty terribly unstable power, but the Teensy 3.0 boards did not like it. Eventually they sent me that little Traco which I tested, and we got Adafruit to update their site with advice to add the input capacitor. We had several more conversations on this forum where that little Traco power supply caused people a lot of pain and suffering.

Is any of this relevant to your situation? Maybe, or probably not. But it's the one really weird case where boards wouldn't always boot up.

We also had a problem long ago with Teensy 3.6 not booting when the power would rise too slowly. Ultimately it was because VBAT needed to have power for the RTC init code. If the power starts up too slowly, VBAT is about half a volt "behind". This was solved by restructuring the init code to not touch the RTC until much later. I don't recall anyone ever hitting this on Teensy 3.2.

Anyway, if you have an oscilloscope, maybe set it to do single-shot capture on rising edge of the input power, and have the other channel watch the 3.3V power. Startup speed from instant to 50ms should be fine. Some reasonable overshoot is fine. You're looking for really bad problems, like if the input power doesn't rise from 0 to 5V in a smooth ramp and especially if it oscillates above and below around 1.2 to 1.7 volts.

 

[tyzjames]

Is this a 'unit in operation' failing to function at some point while running previously? Or is this from a 'cold start'?

When in this state is it the case that removing and repowering the T_3.2 does not return it to normal function? Same question after pressing the Button then removing/repowering the T_3.2 - without reprogramming the firmware?

We test all units before we ship them out. So the units would be working well until the random event where the user powers it up and the firmware does not run. Power cycling the device does not help.

This happens in " probably to 1 in every 500-800 units. " - does this recur on that same unit over time, or is it generally a onetime event with that user's unit? That is does the T_3.2 generally function for some significant period after that firmware reload without same or other issue?

Yes it will function normally for a period of time. If that issue occurs and the firmware is reloaded via Teensyduino, I don't think Ive had the same customer email again about it. Or they just don't because they know how to resolve it themelves.

Does firmware reload do any 'first run' changes to EEPROM or other 'static' variables or stored reference data?

Not sure if I understood this correctly. On boot up, the firmware does load data from the internal and external EEPROM. Is there a scenario where this might affect the program?

Seems the sample size must be some multiple of "500-800 units"? Impressive count!

Are these T_3.2's end user/home or Industrial environment? Used intermittently or continuously?

Thanks..! It is typically used in a home environment. It won't stay powered up 24/7 but when used, it will be powered from an external power supply. Usually 9v and then the voltage is stepped down to 3v3/5v using a voltage regulator.

Any chance a DC-DC switching power supply modules is providing the power?

Yes, the units are powered by an external power supply. I did come across some power supply models where the power ramps up slowly, or not fast enough for the T3.2, and this causes the T3.2 to not be able to boot up. But in this instance, a power cycle will resolve the issue, no need to load the firmware again. Anyway, I've resolved this issue by using the watchdog timer, where if the T3.2 fails to boot due to the power supply ramp up issue, the watchdog timer can will detect this and do a reboot a few seconds later without issues. So it's a different issue from this.


I realise that this issue occurs too randomly to properly diagnose what the cause might be. One thing I noticed between the T3.2 and T4.1 is that if the unit hangs due to bad code, stack overflow or whatever, the T4.1 can still work after a power cycle whereas the T3.2 requires the firmware to be reloaded again. Is this a known issue, and could this be related to the issue I am facing?