Teensy firmware being corrupting during shipping?

S

SaturnGoddex

Member
This has been a real head scratcher for me. I sell a little kit that includes a teensy 4.1 with special firmware that I flash before mailing them out. I've had multiple reports from customers on the other end that they're in the blinking led reset state when they get there. We go through troubleshooting, I have them flash the firmware and everything works perfectly. At first I thought that maybe I was just accidentally mixing in unflashed boards with the kits so I started marking them when I program them, and I can confirm with 100% certainty that I had a board in my hands with properly flashed firmware and about two weeks later it showed up in Australia blank.

I am really curious if anyone has had similar experiences. Someone suggested to me that maybe it's getting x-rayed while in transit and that's corrupting it, and while that sounds farfetched to me I can't come up with a better answer.
 
If the old firmware was wiped/corrupted they wouldn't automatically reset to the blink program. Only holding the button for ~15 secs would do that, because it causes it to be copied from the top to the base of the flash memory.
 
Is something in the shipping process holding down, or intermittently holding down, the program button?
 
The restore process is triggered by holding the pushbutton for at least 13 seconds but no longer than 17 seconds. The design tries to minimize the odds of accidentally triggering by human pressing which is rarely more than a few seconds, and by accidentally pressed for lengthy time like another object rests on top of Teensy and makes contact with the pushbutton (probably for much longer than 17 seconds).

But if the button is pressed for 13-17 seconds, indeed the restore process completely erases the flash memory (including the portion used for EEPROM emulation) and writes a known-good LED blink program.

Of course this is only when Teensy has power. When inside a plastic bag without power, pressing the pushbutton can't have any effect. Also the flash memory requires power to erase or write its memory. Internally, all flash memory uses Fowler-Nordheim Tunneling to change the trapped electrons on floating mosfet transistor gate. This transfer of electrical charge can not happen without power.
 
Is your "little kit" shipped with a sufficiently sized battery (and connected) to keep Teensy powered during shipping? This would have to be the case for @shawn's scenario to play out. If Teensy is not powered, then pressing the program/restore button will not erase/re-program the default LED Blink program.

I'd hazard a guess that your customer is pressing the button (with Teensy powered ON) and doesn't realize what the result is from doing that. Maybe the customer thinks it's a Reset button (which it's not).
 
Does this product use a LOCKED Teensy? If locked the Button cannot trigger a 15s Restore, only program with a valid encrypted Hex
 
Or its simply been X-rayed in transit? Cargo X-raying apparently uses very high X-ray energues to penetrate the steel shipping containers and this can be enough to disrupt electronics.
 
jmarsh said:
An x-ray would not replace the firmware with the blink program.
Click to expand...
I was thinking the same. It can only erase, not put something else there.

However, what happens if the program is indeed corrupt? Does it execute garbage code? Is there a checksum or other verification? If it does have an integrity check, what happens if that fails? Perhaps automatically load a known-good program? Or does it sit and do nothing, appearing broken when all it needs is a re-flash?

As an immediate workaround, I suppose you could intentionally ship them blank, since they're going to arrive blank anyway, and just make it part of the process for the customer to "update firmware" before use?
I've seen several commercial products that do that. They come new from the factory with only a bootloader that does nothing else, and the customer must "update" it before it does anything useful.
 
AaronD said:
However, what happens if the program is indeed corrupt? Does it execute garbage code? Is there a checksum or other verification?
Click to expand...

The answer depends on whether using secure mode, which is only possible on lockable Teensy, and only used if you run the programs which burn your key into the hardware and permanently lock into secure mode.

Secure mode uses a secure sha256 hash (basically a 256 bit checksum) and some other tests to verify your program is intact and was encrypted with the correct key.

Standard mode only checks metadata in the headers, just a few dozen bytes.

These tests are performed by code in the NXP ROM inside the chip. If the tests fail, the ROM will not jump to your program. Usually if the ROM continues running, it will turn on the USB port and appear as a USB HID device with a name like "NXP Blank" or "SE Blank".

In the case of standard mode, if the flash has become corrupted but the header bytes are still valid info, the ROM will jump to your program. What happens depends on which parts of your program are corrupted. Attempting to execute corrupted code will very likely result in a memory fault or other fault condition which is caught by the ARM processor. The fault handlers are similar to interrupts. The default fault handler waits 8 seconds and then stores info for CrashReport and tries to soft reboot. But if that fault handling code is corrupted, it may generate another fault. The ARM processor goes into a "lock up" mode where it basically just stops, if another fault occurs while running the fault handler code. In theory, randomly corrupted code could to anything, but in practice the faults which detect invalid instruction opcodes or attempts to access memory addresses (32 bits for 4GB space) which aren't implemented are the very likely.
 
AaronD said:
intentionally ship them blank
Click to expand...
defragster said:
That would allow them to arrive untested?
Click to expand...
Each individual unit, yes. But if the design hasn't changed and the hardware QA is good, then the chance of shipping a dud is still low.

Of course, nothing prevents a complete test before shipping, then reset so that just the wrong type of corruption can't wreck things, and then having the customer put the same correct code back into it again. As far as they're concerned, there's really no difference: it comes blank with a guarantee, and they have to "update" it before it works.
 
Teensy isn't locked and since customer firmware updates are a feature (it's an open source project) that isn't an option. The kit doesn't have a battery which is what confuses me. I assumed with corruption that it would fail dead rather than the blink program but I was curious if maybe the teensy had some sort of automatic recovery mechanism. I'd rather not ship them blank but I will if I must, the current policy has just been letting customers know it's an issue I'm working on and offering a discount if they need to reprogram them. I'll double check with some of the customers whether they stumbled into the reset sequence, it doesn't match what they told me but the general public is always a varriable
 
I appreciate the expertise. This has been weird. It's unfortunate I can't dump the firmware on the other end and see what's going on
 
SaturnGoddex said:
It's unfortunate I can't dump the firmware on the other end and see what's going on
Click to expand...
Can you offer a free-return warranty? Pack it back up as received, put a new label on that you pay for, and drop it off. See what shows up.

How expensive would that be for you? Is it worth a couple, to find out what they actually got?
 
AaronD said:
Can you offer a free-return warranty? Pack it back up as received, put a new label on that you pay for, and drop it off. See what shows up.

How expensive would that be for you? Is it worth a couple, to find out what they actually got?
Click to expand...
I'd be happy to try but I'm not sure it's possible to actually read the flash off a teensy?
 
You must log in or register to reply here.
Back
Top