Teensy 3.6 looses programming

D

dundakitty

Well-known member
Under what circumstances would a Teensy 3.6 loose its programming?
Several times I've had a Teensy 3.6 appear to die but really it just needed re-programming. I have to hold down the reset button for 15 seconds and then the Teensy 3.6 comes back to life.
The teensy 3.6 is in an MP3 player application, as documented here: https://forum.pjrc.com/threads/55551-posted-RCA-118-quot-tombstone-quot-MP3-player
I play the "radio" for a half-hour almost every night then turn it off. After several months the teensy 3.6 appears dead and has to be reprogrammed. Power-cycling doesn't have any affect. When I attach the USB cable the computer doesn't see a teensy present. Only after holding down the reset button does the computer see the Teensy and it can be reprogrammed.
After reprogramming the teensy 3.6 works fine for several months, then one day it will appear dead.

Does anyone have an idea where to look?
 
I would say it's unlikely the T3.6 is "losing" its program, as opposed to being corrupted by something your software is doing. I checked the original thread and don't see any links to code. Are you using EEPROM or saving any data in flash? Those come to mind as things that could eventually corrupt your program flash. As a first step, it might be worthwhile to see if you can update your project from TD 1.54 to the latest 1.59b2. There have been two compiler updates, big changes to SD, etc. Maybe the problem will resolve by doing that, and if not, at least you're looking for something with the latest tools, which will make it easier for others to try to help.
 
The code is only reading from EEPROM, it's not saving any data there. I do use Serial to read commands and update the EEPROM, but I'd have to have the USB connected for that and it is only rarely connected. Even then the code only writes to EEPROM if the data from Serial is properly formatted. I've compiling with TD 1.58, I'm waiting for TD 1.59 to come out of beta.
My current thought is a power-supply transient during power-up. I'm using a 12v 3-amp power supply for the audio board, then a 5v buck switching regulator for the teensy. It's possible I have too large of a filter capacitor after the switching regulator. I'll have to tear the "radio" down to get to the board to check.
 
I just had a 2nd T3.6 lose it's programming. After a power "flicker" the T3.6 appeared dead. The PC didn't even see the T3.6 over the USB connection. After holding the reset button down for 15 seconds the PC could see the T3.6 and I could reprogram it.
The power supply for this T3.6 is a simple 5v wall-brick. The 5v/USB trace is cut on the T3.6 and I only connect the PC when needed.
 
dundakitty said:
I just had a 2nd T3.6 lose it's programming. After a power "flicker" the T3.6 appeared dead. The PC didn't even see the T3.6 over the USB connection. After holding the reset button down for 15 seconds the PC could see the T3.6 and I could reprogram it.
The power supply for this T3.6 is a simple 5v wall-brick. The 5v/USB trace is cut on the T3.6 and I only connect the PC when needed.
Click to expand...

This exact thing happens to me and the Teensy 4 when I compiled my code and it produced code that (I think) was too big for the teensy. When not uploading and just building, the compiler says space free for local variables is a negative value.

If I tried to upload that it compiles just fine, works for a few seconds and then crashes and loses connection. Unplugging the USB and plugging back in does nothing, you have to press the button and then recompile with code that works.

My guess is something goes out of bounds in memory somewhere.
 
frankzappa said:
This exact thing happens to me and the Teensy 4 when I compiled my code and it produced code that (I think) was too big for the teensy. When not uploading and just building, the compiler says space free for local variables is a negative value.

If I tried to upload that it compiles just fine, works for a few seconds and then crashes and loses connection. Unplugging the USB and plugging back in does nothing, you have to press the button and then recompile with code that works.

My guess is something goes out of bounds in memory somewhere.
Click to expand...

When 'free for local' is <=0 this code github.com/PaulStoffregen/teensy_size/blob/main/teensy_size.c#L99

Should cause a build error to be displayed. At this point the build is complete and the HEX remains and could be uploaded - but would not function, and the build should break stopping the upload attempt? Though Teensy.exe loader may have the path and could upload?

But this would be different than claiming valid code was there once for some time and then seemingly goes away.

Cases where 'working' code goes missing has been reported but not observed AFAIK. If the Teensy.exe is active and has a path to a HEX file (and the ELF file to confirm the target that would stop pushing code for the wrong MCU) it can be triggered to erase and reprogram a connected device. Unless the code overtly writes to the program flash area the code on a healthy Teensy should persist indefinitely (i.e. some years). If the code relies on some EEPROM or other elements to run 'properly' then those program choices would generally be suspect for failure to run.
 
defragster said:
Should cause a build error to be displayed.
Click to expand...

Indeed this is supposed to cause an error and not allow upload.

Maybe some non-Arduino software like PlatformIO is in play here?

Or perhaps we have a previously unknown bug where known bad code is being allowed?
 
PaulStoffregen said:
Indeed this is supposed to cause an error and not allow upload.

Maybe some non-Arduino software like PlatformIO is in play here?

Or perhaps we have a previously unknown bug where known bad code is being allowed?
Click to expand...

This was noted:
Code: 
When [U]not uploading and just building[/U], the compiler says space free for local variables is a [B][U]negative value[/U][/B].

If it shows NEG that should trigger ERROR and stop, but the note says build not upload. So Teensy.exe may have HEX Path from that or prior build? A button press would then proceed?

So, some user intervention occurred to move to upload reported bad HEX? An alternate IDE might mishandle the build error - but note says it wasn't expected to upload. But any prior working HEX was replaced.

Without a way to repro that seems to be behind the 'T_3.6 loss of programming' as well - unless there is a code path to unexpectedly halt execution (p#6) - or flash overwrite?
 
If this is PlatformIO, maybe an issue should be opened? They really should be checking the return status from teensy_size and consider the build a failure if it returns any error.
 
I just had it happen again on a Teensy 3.6, power supplied through a 5v wall-brick. The device was working fine until the AC power went out. When the power was re-applied the Teensy 3.6 appeared dead. I connected the Teensy 3.6's USB to a PC and it wasn't recognized until I held the program button down for 15 seconds. After that I could reprogram the Teensy 3.6 with my program and it started working again.
I'm not close to any memory limit, I have plenty of flash and ram left. The device has run for months without an issue. There just seems to be a small chance the programming goes missing when powered up.
 
I have four Teensy 3.2 systems in daily use, with another twenty or so out in the wild. I also have two Teensy 4.1 systems running. I've only had a problem with the Teensy 3.6 systems. A Teensy 4.1 and a Teensy 3.2 are plugged into the same power strip as the recently failed Teensy 3.6, all using separate 5v wall-bricks.
 
Please check which version of Teensyduino created the .HEX file you're programming onto Teensy 3.6. If it's older than version 1.49, you should update for this improvement in 2019.

github.com

Improve startup behavior on Teensy 3.6 with external slowly rising power · PaulStoffregen/cores@1b1f953

Teensy Core Libraries for Arduino. Contribute to PaulStoffregen/cores development by creating an account on GitHub.
github.com github.com

If using Arduino IDE 1.8.x, click Help > About (or Arduino > About on MacOS) to check the version numbers.
 
Every time a Teensy 3.6 has lost its programming I've rebuilt the image from source, using the current Teensyduino. I've had a loss of programming using Teensyduino version 1.58. When I had to reprogram it a few days ago I used Teensyduino 1.60. In my case if after a power cycle the Teensy 3.6 doesn't respond, it stays that way until the program button is held down for 15 seconds. It doesn't matter if I swap out the power supply or rapidly power cycle the Teensy, it stays unresponsive.
 
This really feels like it's most likely a software problem. Does this happen with more than one Teensy 3.6?
 
When in the 'No Normal State" :: With Teensy Loader open with HELP/Verbose Window open

Holding Button while Plugging in the Teensy, and releasing after a second or two, should show something as it overrides normal sketch startup, and should take it to waiting to upload.

Doing that with no sketch or Loader AUTO disabled would have it stop and wait with RED LED for an upload.

If that works, then try unplugging and plug in again normally??

If that doesn't work, then something the "15sec Restore" fixes was 'in need of correction' for some reason, given that safely reloads
 
In the early days of Teensy 3.0 and Teensy 3.1, we regularly heard complaints that Teensy wasn't retaining the loaded program. The actual observed behavior was everything would work fine after uploading new programs, but then Teensy would appear to be blank after power cycling.

After many of these reports, most heavy on the conclusion of problems with the memory and light on details of the actual observed behavior and rarely giving the specific program used, the cause finally became clear. They were using MEMS motion sensors, which were relatively new and popular at the time (the era of the original Nintendo Wii) and a library developed on Arduino Uno. The bootloader on Uno runs at startup and waits about 1 second for USB serial communication to initiate reprogramming. Teensy's bootloader does NOT run at startup. On Teensy 3.x the chip boots up very fast, where the delay until code starts running normally is only the time for the 16 MHz crystal to start oscillating.

Most MEMS motion sensors take about 100 to 150ms to boot up. During that time they're unresponsive to I2C communication. The libraries only tested with Arduino Uno had never encountered that initial sensor startup time, because they ran after Uno's bootloader had waited, which gave the sensor plenty of time to start up. Those libraries had no checking for non-response from the sensor. Even worse, when the sensor was later up and running, they wouldn't manage to recover and use it because they were written with very simple code that assumed initialization was successful.

People who reported Teensy lost its program were using programs that used those motion sensors as their only or primary mode of input. If the sensor wasn't responding, their program had no other way to inidicate it was actually running. We rarely saw any code, but in many cases the programs may have entered infinite loops or otherwise crashed. When we asked people to add LED blinking or other status indication into their programs at startup, they would see Teensy did indeed still have their program in flash memory.

Ultimate the MEMS motion sensor problem was "solved" by adding a 300ms delay in Teensy's startup code!

Having now said all this, I will add it is theoretically possible for code running Teensy to alter or overwrite its firmware. But that takes pretty special programming. It's not the sort of thing which tends to happen by mistake. Also theoretically possible is faultly flash memory. But that too is quite unlikely, especially if the memory is able to work successfully after reprogrammed. Sadly, if it is actually a hardware problem, there's little which can be done now, since Teensy 3.6 was discontinued years ago.

But given the long history of startup problems being mistaken for memory loss, I hope you can understand when I ask for emphasis on the actual observed behavior.

As a first troubleshooting step, please add code which does some sort of distinctive startup behavior like LED blinking. Best to use startup_middle_hook() so it runs before USB is up or C++ constructors. If you're really ambitious, you could compute a checksum of the flash memory and communite it in some way. But starting with a simple distinctive LED blink before anything use runs is an easy first step which can really help confirm whether Teensy starts up and actually runs your program.
 
Last edited:
The first project that "lost" programming is an MP3 player in a 1920's "Tombstone" radio case. It has two LEDs as a tuning indicator, a bluetooth module connected to Serial2, some buttons, and two analog inputs for volume and tuning. The USB port is only connected for programming and debugging. The MP3 player worked for months/years, being power cycled several times a week, until one day the MP3 player wouldn't play. The tuning indicators were both dark, and adjusting the tuning did nothing. The volume control did nothing. Bluetooth would connect, but wouldn't respond to the UART commands. Power-cycling didn't help. I connected the USB to a laptop and the laptop did not see the Teensy. I tried quick presses of the Program button and still the laptop wouldn't see the Teensy. After a 15-second press of the Program button the laptop was able to see the Teensy and I was able to reprogram it from source using the latest (at the time) Teensyduino.

The second project that "lost" programming is a clock driving an LCD display using SPI. It has a GPS module connected to Serial1, an analog light sensor, and a capacitive touch screen. The USB port is only connected for programming and debugging. The clock worked for years, surviving several power cycles. Recently I unplugged the clock for a week while I was on vacation, and when I plugged it back in the display stayed dark. It didn't respond to the touch screen. I've left the clock unplugged for a week (or more) before and haven't had a problem. I opened the case and the "SCK0" LED (pin 13) was dark. Normally it flashes as data is sent to the LCD screen. Additional power-cycles didn't help. I connected the USB to a laptop and the laptop did not see the Teensy. I tried quick presses of the Program button and still the laptop wouldn't see the Teensy. After a 15-second press of the Program button the laptop was able to see the Teensy and I was able to reprogram it from source using the latest Teensyduino.

I'll try Defragster's suggestion the next time it happens. I'll add an LED to the other projects the next time they fail. I have a spare Teensy 3.6, I'll build another MP3 player with an LED and see if I can get it to fail too.

Both the MP3 player and the clock have required the 15-second Program button push and reprogramming twice now.
 
I built an MP3 player out of a spare Teensy 3.6, the audio library, and Frank Boesing's codec library, using parts I had laying around. I use a FRAM chip to store filename hashes for "shuffle" play. I added an LED that's first lit in startup_middle_hook and then blinks at a 0.5 Hz rate under normal operation and 2 Hz rate when a file is paused. I stuffed it into an old audio amplifier I built in the 70's. Luckily I had a DC-DC converter that could handle the 32v supply.

MP3_player-1.jpg
MP3_player-2.jpg
MP3_player-3.jpg
MP3_player-4.jpg
 
You must log in or register to reply here.
Back
Top