Frightening rare failure... Any wisdom?

[Davidelvig]

I use a standard Teensy 4.1 NE on my own board, connecting D+ and D- with pogo pins so that I can present USB C to the outside.
I've shipped over 50 of these without getting this issue.

I left the device (a digital trumpet) plugged in to my Mac overnight (I often do) and the app running on the Teensy was hung in the morning. That's not that unusual... My code has bug remnants...

But it would not restart:
- power off then power on
- boot attempt via PlatformIO and with Arduino (USB C)
- hold the Teensy reset button during power on did not affect this

Upon opening my device, I see the repeating sequence of 4 red LED blinks near the bootloader chip on the Teensy.
I was able to flash the Teensy through the USB Micro port, and it's operating normally now, including flashing through USB C

Questions:
- what might cause the Teensy to get in this state (4-flash LED)?
- What might be different about the USB micro connection and the pogo-pin D+ D- connection and the USB C connection.

In USB C mode, GND is hard wired and VUSB goes through an OnOff circuit on my board that delivers slightly less that VUSB to VIN on the Teensy when "On".When USB Micro connected, of course, it delivers full VUSB to the Teensy.

Attached are 2 pics of the pogo pin connection.

Any ideas for how I can sleep at night with these units in the field (and how to design better for the next 50?)

Also, this may be a board from PJRC or one from SparkFun. I don't know how to tell, and I've mixed my stock.

 

[Davidelvig]

Oh my goodness... as I continued testing, I saw the state again...4-flash and otherwise unresponsive.

When I unplugged a screen module (in the 1/10th inch header above) the flashing went away and the board booted normally. Plugged the screen back in and it continued to work, and has worked since. I'm trying wiggling the display, and no repeat of failure yet.

So, I'm doubting it was the USB C and pogo pins.

I'm expecting something with a faulty connection of one of the 8 display pins which are:

1: Teensy 33: Backlight_Display - 220 own series resistor
2: Teensy 36: CS_Display - 10K ohm pull-up
3: Teensy 37: DC_Display
4: Teensy 38: Reset_Display
5: Teensy MOSI
6: Teensy SCK
7: Teensy 3v3
8: Teensy GND

Any of these pins seem especially tempting for a short to GND or 3v3 causing this 4-blink Teensy state?

And the question about what is this 4-flash mode? I get this from ChatGPT:

The red LED blinking four times near the USB port on your Teensy 4.1 typically indicates that the bootloader chip is trying to communicate but no user code is loaded or there was a problem with the uploaded code.

Thanks


[edit - DC and Reset pin nums corrected]

 

[jmarsh]

Does the USB C port have pulldown resistors?

 

[Davidelvig]

yes, on C1 and C2. 4.7K each

 

[Davidelvig]

Latest:

Problem restarted on its own while sitting on the desk, plugged in.
4-blink red LED near USB persists.
I can press and hold reset for 15 seconds (13-17) and the red LED glows for 45 seconds, then returns to 4-blink pattern
This on USB C or Micro-USB power with no display plugged in.

Any more ideas?

 

[KenHahn]

This link to the PJRC website shows the official explanation of the blink codes if you scroll down near the bottom of the page. https://www.pjrc.com/store/ic_mkl02_t4.html

If you have 50 units in the field and no customer complaints, you are probably dealing with an issue specific to your current setup rather than a design issue.

I would suspect the USB cable getting worn out if you plug it in and out a lot in testing as I have seen somewhat similar random red LED blinking issues when a USB cable has passed its useful life, perhaps due to high contact resistance. The fact you see it with both USB ports and with different USB cables would seem to rule that out.

Do you have another system you can try to reproduce the issue with? Perhaps the one you are working with just happens to have a marginal crystal or solder joint, especially if it is an engineering unit that may have been abused a bit.

 

[Davidelvig]

Thanks for that link @KenHahn. So 4 blinks has something to do with:
- the 24MHz crystal on the board not oscillating (I can see this and think I could test this)... or
- a particular connection between the boot loader chip and the main IMXRT (I would not know how to test this)

These are all new components in this failed unit (the Teensy, my carrier board). And like you say, different USB cables. Both of which work to flash other Teensys.

This is likely above my pay grade, but I expect an oscilloscope on the crystal pins would tell me about the first possibility. And if it's that, bad Teensy, I guess. I've had hundreds of issues to work through over the last decade, and I don't think it's ever been the Teensy itself.

I can live with a single bad board, but I'd sleep better if I knew why.

Thanks for your help!

 

[KenHahn]

That page with the blink codes is mainly for people who are designing their own Teensy boards and integrating the bootloader chip into their own PCB. The explanations of the blink codes is oriented more toward troubleshooting PCB design defects, like forgetting to run a trace or not attaching a crystal, though it may give you a clue.

As for the crystal, it can be hard to probe as an oscilloscope probe capacitance is usually enough to throw it out of whack or stop it from oscillating altogether. Using x10 probe has less negative effect than x1 probe. The parts are also very small and hard to probe without shorting things out and inducing issues like red blink codes. This I know from experience.

The only other thing I can think of is that though your Teensy looks nice and clean, it might be worth giving it another good cleaning with some IPA to make sure there isn't some flux residue or other contaminate hiding somewhere that is causing your issue. The nature of crystals make them sensitive to contaminates. A spray bottle with something like 99% IPA can be useful for flushing stuff off the board.

 

[Davidelvig]

@KenHahn , I inspected those areas (24MHz crystal and boo loader) and saw no visible defects. Then I did a scrubbing with a paper towel with some 99% IPA. I even doused it a bit and scrubbed again.

It didn't come back alive right away. And now I think that was because it was still perhaps a little wet with the alcohol (through my friend ChatGPT says it's a very poor conductor).

Two hours later now I plugged it in and did not get the evil four blinks. And I was able to flash the board through my usual mechanisms.

I'll be thinking through what I should do with my processes and perhaps whether I should be giving my newer teen a bath before shipping them to customers. I hope this was a one off. I'll be testing this board for the foreseeable future after different scenarios of leaving it on and leaving it off.

Thanks again, @KenHahn and @jmarsh for your comments!

 

[PaulStoffregen]

Washing with isopropyl alcohol probably fixed it.

4 blinks almost always means a problem with the 24 MHz crystal, especially on a properly manufactured board that went though testing.

Even on custom PCBs, it's the same JTAG pins for communication except for 1 signal which selects what part of the chip will respond to JTAG. Part of the chip works with JTAG even without a crystal. The other part only talks if the crystal is working. If you get 4 blinks rather than 2 blinks, that means JTAG was working when talking to the circuitry that doesn't need the crystal. The problem can theoretically be with the signal that selects which part of the chip talks JTAG. But that's pretty reliable and gets well tested on every manufactured Teensy. The crystal also gets well tested. But crystal oscillators are high impedance circuits that are quite sensitive to physical contamination.... the sort that cleans up easily with isopropyl alcohol.

 

[Davidelvig]

Thanks @Paul
The problem is now intermittent on this one board, and may be related to time-on. Blinking yesterday after being plugged in a while. Unplugged overnight. Working correctly this morning on first plugging in. I'm going to leave it for a few hours to see what happens.

Temperature maybe? Nothing feels hot when it's failing.

SerialNumber: 69295218-3F2041D7

... is the serial number of the chip using
teensyUID64(uint8_t *uid64)
from TeensyID.cpp
(Copyright (c) 2017 Stefan Staub)

[Edit] 90 minutes later after being left plugged in. 4 blinks.

[Edit] 2 hours after that and having been unplugged. Plugged in to see 4 flashes.
Now it's in the fridge for a while. I'll see if cooling causes any positive changes.

After that, perhaps it's time for progressive destructive changes to the Teensy's environment. Desolder from the daughter board if I can, and see if the problem persists.