Program accidentally erased?

J

John Knoll

Member
Yesterday I experienced something odd. Two years ago I built a motor control system using a Teensy 4.1. It's inside a closed cabinet, but communicates with a laptop computer over a serial port to receive commands and send acknowledgments via a serial terminal. It works great, and I've built several of these for my own use.

I have a colleague at work who is using one of my systems and he called me yesterday for help, saying it wasn't working for some reason. I went to have a look at it and somehow the program had been erased. It was running the default blink sketch. Not a big deal, I just reflashed it and he was fine, but I'm trying to understand what happened. It was in a closed cabinet, so he couldn't have reset it via the button, and he swears he didn't reprogram it via Teensyduino. He doesn't have Teensyduino installed on his machine. He is using Arduino as his serial terminal, so I imagine there may have been something he did that accidentally reset it, but I've never seen that before.

Any idea how someone can mistakenly erase a Teensy 4.1 without touching the button or using Teensyduino?

Not urgent, just curious. Thanks!
 
Your issue sounds very similar to this thread.

There isn't a known way to accidentally go back to the default blink program without either a long Program button push with power applied or via a software download that I am aware of.

Is the Teensy that had the issue an older one (from 2 years ago) or one that you bought more recently (within the last year)?
 
It's in San Francisco. It's in a climate controlled building that doesn't deviate from ~70 degrees F by more than 10 degrees really ever.

The odd thing is it's been working fine for years already and just yesterday it's suddenly in default blink. Pretty certain nobody was holding the button down for the magic 13-17 seconds as the Teensy in an enclosure. This left me thinking that it was related to it being connected via USB to a host computer, and somehow being reset that way. My colleague swears he didn't accidentally try to upload a sketch, and I kind of believe him since he doesn't have Teensyduino installed on his computer. He does have Arduino, as that's what he's using as his serial terminal.

At the moment, my two leading theories are:
1: There IS some way in Arduino to accidentally trigger this kind of reset to a connected Teensy and my colleague did this inadvertently.
2: There is some super odd rare circumstance where a power transient, a cosmic ray, or something else glitches the Teensy to cause a reset.

I read the other thread, thanks! Super weird.

As I said before, this was not a huge problem for me. I just reflashed the Teensy and everything was fine, but I'm baffled how it happened.
 
The enclosure in question:
8_axis_RP4_system_small.jpg
 
KenHahn said:
Your issue sounds very similar to this thread.

There isn't a known way to accidentally go back to the default blink program without either a long Program button push with power applied or via a software download that I am aware of.

Is the Teensy that had the issue an older one (from 2 years ago) or one that you bought more recently (within the last year)?
Click to expand...
This is the same teensy that's been in the box actually since 2020. It has been updated with newer versions of the software several times, most recently in February, just to add additional features. All told I have 8 different generations of systems like this, and this is the first time I've seen this happen
 
As said again and again, the only way (for us users) to put Teensy into factory state (copying factory blink vode to lower flashnlocation) is to hold program button (program pin) for 13 s at ground level. this can be done by pushing the program button, which can be excluded for the cases we are discussing here, but ANYthing that connects PROG pin with adjacent GND will do it. Even humid dust maybe sufficient to reduce the voltage level in PROG pin.
As the PROG pin is hardly used, best is to tie it up to 3.3V, say with 10k resistor, and keep the board clean.
 
Another possibility is some kind of EMI, radiated or conducted, wich could cause random behaviour.
 
WMXZ said:
As the PROG pin is hardly used, best is to tie it up to 3.3V, say with 10k resistor, and keep the board clean.
Click to expand...
If you're REALLY sure that it will NEVER be used, I guess you could short it to 3.3V too. Good luck EMI'ing *that*! Or a wet or dirty board.

But it would also mean that the button now shorts the supply, so you really do need to be sure that it will NEVER be pressed with power on.

For reference, a 10k pull-up would cause about 0.3mA of additional current when the button is pressed, and a 1k about 3mA. Or, it would take that much induced or leaked current to cause a spurious trigger...for 13 to 17 seconds, no more, no less...*if* that is indeed the problem.

Given the specific window of time, it seems unlikely, but technically possible. Maybe as a conductive path slowly builds, one noise event with digital thresholds just happens to land inside the magic window?
 
AaronD said:
Given the specific window of time, it seems unlikely, but technically possible. Maybe as a conductive path slowly builds, one noise event with digital thresholds just happens to land inside the magic window?
Click to expand...

For each event that happened to hit the time window I'd expect lots of events, mostly shorter, that didn't.
A shorter press of the button would put the unit into upload mode and require a power cycle to recover back to running the normal program. If that was happening to something that was normally left on and running I'd expect the user to have noticed the need for constant reboots.
 
AndyA said:
If that was happening to something that was normally left on and running I'd expect the user to have noticed the need for constant reboots.
Click to expand...
Does the system get rebooted anyway, so the user might not necessarily notice? Or possibly accepted the "fact" that "systems need rebooting", because of shoddy code in commercial consumer systems for so long, and so didn't think to report *that* as a problem?
 
AaronD said:
Does the system get rebooted anyway, so the user might not necessarily notice? Or possibly accepted the "fact" that "systems need rebooting", because of shoddy code in commercial consumer systems for so long, and so didn't think to report *that* as a problem?
Click to expand...
This system is turned on to perform a task, and then turned off afterwards. Usually it would be on for about 10 hours at a time. While powered up it was pretty reliable.
 
John Knoll said:
This system is turned on to perform a task, and then turned off afterwards. Usually it would be on for about 10 hours at a time. While powered up it was pretty reliable.
Click to expand...
How much of that 10 hours is dead time at the end? So it could have stopped then, unnoticed, and then powered off?

The picture looks to me like theatrical gear, and that stuff often does get turned on and tested way early, left on for the show, and left on afterwards too, while the crew is focused on other things and finally gets around to taking it down. So that 10 hours of being on may include one show from hours 4 to 6 or something like that, and the rest is testing and dead time.

Of course, with what we know so far, it could just as easily stop earlier, and that would definitely be noticed, but again, technically possible. Maybe its local environment changes during operation, to make it more likely to fail after several hours, and probably *after* the show? Then it's just shut down, without noticing that anything's wrong.
 
There are moments of dead time, but not huge amounts. While it's powered up, it spends probably half of its time running motors. This is a device that controls the speed and position of a number of stepper motors, and those stepper motors control the position of a camera and a subject, most often a miniature. I use this thing to photograph models for tv shows and movies. If you're curious, this is a little more detail about a newer 16 axis version:
 
So it's pretty much running for the entire 10 hours. No short critical time and then nobody cares. There goes that theory.

As far as anyone can tell, it was perfect for all of the previous shoot, and then immediately on the next startup, it's unresponsive and blinking?
 
I've been using Teensys since 2017 for all kinds of personal and professional projects, and never saw the program get erased like this. They have all worked very reliable and flawlessly until the unexplained reset the other day. Last summer I did a 5 week shoot where the system in the video was running ~10 hours a day, 5 days a week, and never skipped a beat.
 
John Knoll said:
It's in San Francisco. It's in a climate controlled building that doesn't deviate from ~70 degrees F by more than 10 degrees really ever.

The odd thing is it's been working fine for years already and just yesterday it's suddenly in default blink. Pretty certain nobody was holding the button down for the magic 13-17 seconds as the Teensy in an enclosure. This left me thinking that it was related to it being connected via USB to a host computer, and somehow being reset that way. My colleague swears he didn't accidentally try to upload a sketch, and I kind of believe him since he doesn't have Teensyduino installed on his computer. He does have Arduino, as that's what he's using as his serial terminal.

At the moment, my two leading theories are:
1: There IS some way in Arduino to accidentally trigger this kind of reset to a connected Teensy and my colleague did this inadvertently.
2: There is some super odd rare circumstance where a power transient, a cosmic ray, or something else glitches the Teensy to cause a reset.

I read the other thread, thanks! Super weird.

As I said before, this was not a huge problem for me. I just reflashed the Teensy and everything was fine, but I'm baffled how it happened.
Click to expand...
So far as far as I can tell, assuming we have the same issue, it has to be #2, some sort of black swan electrical event must be tripping the reset function. I wonder if the reset line has a pullup or pulldown on it... might be time for me to get out a scope and investigate
 
Been following these threads with interest, and a thing just occurred to me…

The Teensy bootloader chip is just another processor, and presumably spends the vast majority of its life running a very dull state machine looking for activity on the Program pin and comms from the Teensy. The only exits to Interesting Stuff are to the EraseAndProgramFromUSB() function, or to the EraseAndCopyBlink() one.

There is also a non-volatile flag somewhere which says “this is a new Teensy”, which causes the Blink program to be loaded from the bootloader to the top 4k of a completely blank Flash, and copied to the execution area. This is how people can make DIY Teensy-like hardware.

It sounds like the systems which reset to Blink did so at power up, so maybe what’s happened is that a minor glitch at power up has caused the bootloader to decide it’s a new Teensy, or (somewhat less likely IMHO) branch wildly directly to EraseAndCopyBlink(). This theory has the merit of not needing any weird science of how to make it look as if the Program button has been pressed by Dark Forces for exactly the wrong length of time.

I know Paul works extremely diligently to make the bootloader as bulletproof as he can, but just maybe this is a single point of failure he’s overlooked. It’s probably quite fixable by doing a bit of cross- and double-checking, but it’s not a field update apart from swapping in a Teensy with a new bootloader.
 
Hmmm. On closer reading of the Hardware Initialisation section for the bootloader, it appears the “new Teensy” process still requires two physical steps - a short press on Program to load the safe copy of Blink, then the long press to copy it down for the first time. So the “wild branch at power up” for the bootloader becomes the more plausible theory…
 
Exactly my thoughts, too.
Power glitching is a common technique to make the MCU do weird things.
It could also explain the other case with Teensy arriving blank: the restore did not happen somehow magically during the shipping, but the 1st power up could have glitched the GD32 to reflash the firmware. The red led turned on (if it did at all) while it happened might have been mistaken for a normal power led, esp. if the user is not familiar with the Teensy's operation.

@John Knoll
Do you know if the sudden blink restore happened while the system was already working, or, perhaps on power up?
 
You must log in or register to reply here.
Back
Top