Repeated Teensy 4.1 Failures - Processor appears fried

[robws]

We have experienced ongoing failures of Teensy 4.1 modules in our system over the last 18 months. These failures appear random and arbitrary except:
1) Never fails during normal operation. In other words once up and running fine during that power cycle.
2) The symptom of the failure is that the 3.3V rail on the Teensy is shorted to ground. We have removed the 3.3V regulator the the rail remains shorted.
3) It is our strong impression that these failures only happen when a USB cable is connected/disconnected, Note we often, but not always connect USB cables with the USB 5V power disconnected.
4) We have placed diodes from the 3.3V line to the 5V rail as per some of the posts here to ensure that we are not backfeeding the 5V rail through the 3.3V. This did not help.

We have never had this type of problem with the 3.2 Teensy. This is becoming a critical problem for us as we need USB access to the our system via the Teensy but at this point we risk system failure every time we plug in the USB cable.

Any thoughts or what we can check on the Teensy 4.1 to further try and diagnose this issue?

 

[BillFM]

Digital pins on T3.2 are 5V tolerant.
Digital pins on T4.x are NOT 5V tolerant.

In general, connecting T4.x GPIO to any 5V peripheral is a hazard waiting to happen. And end result is (generally) a blown Teensy processor with a shorted 3.3V rail to ground. The solution is to use voltage level translators such that Teensy GPIO never touches 5V.

And this is worrisome...
"4) We have placed diodes from the 3.3V line to the 5V rail as per some of the posts here to ensure that we are not backfeeding the 5V rail through the 3.3V. This did not help."
I hope that's not what you wired. I suspect you meant to say that you diode OR'd Vusb with Vin and cut the jumper between pads connecting Vusb to Vin on bottom of Teensy pcb. Please confirm!

Some solder flux is conductive. Simple cleaning can remove potential shorts.
You might also inspect the soldering for solder shorts between pins or stray tiny balls of solder floating around the pcb. Post a photo of circuitry/wiring. We can sometimes spot things.

We'd need to know what devices you've connected to T4.1 and how they are powered in order to help further.

 

[robws]

Hi Bill, thanks for the thoughtful reply.

- Our 4.1 design is a full 3.3V compliant design, with 5V translators where appropriate.
- On the diode issue, there was a post in these forums somewhere around backfeed from the 3.3V internal supply into an already turned off 5V rail causing unexpected current flows and failures. I agree this is a bit weird and we will probably remove as it did not do anything to prevent our latest failure.
- Regarding the Vusb and Vin and the jumper on the Teeny's, I personally believe this is one of the weakest design points on the whole teensy. Essentially one cannot do a diode or with Vusb and Vin without literally running jumper wires onto the teensy. We have our external Vusb or'd into the Teensy, but there is no way, without hacking the teensy to add the second Diode or on Vusb. Most importantly in real-world designs you want to be able to connect the Teensy from our external USB connection (for software updates, removal of logs, etc.), AND power it from our host system under normal operation, where you also want to be able to read logs, interact with the command line etc. We continually have to ensure no USB power from any USB connection when the boards are in situ, meaning we have USB hubs with power switches, or cut USB cables, etc. A small fix on the 5V Vin/Vusb connection with an active switcher would resolve all of these problems at minimal expense.

While it is definitely possible that we are getting noise on one of the signal lines going into the teensy (we operate in a noise environment), this seems unlikely as in normal system operation we have not had any failures, all of our failures are related in some way to situations where the USB is also being actively used, and the failure occurs either at system power up or power down (in other words once the system is running on USB it keeps running, and doesn't fail in mid operation). We have tried on multiple occasions to recreate failures but have been unable to replicate on command.

As I type this I wonder if we can ohm out each of the IO lines on one of these failed processors and identify the IO which caused the short. I will test. Note yesterday our tech removed the processor from one of the failed teensy boards and the 3.3V short followed the processor so we are definitely frying the processor on the board.

 

[jmarsh]

I personally believe this is one of the weakest design points on the whole teensy. Essentially one cannot do a diode or with Vusb and Vin without literally running jumper wires onto the teensy.

All you have to do is cut the join between the pads and solder the diode on...

 

[PaulStoffregen]

I'm not aware of any way to get further info from a blown chip.

As for figuring out what's going wrong, I'm afraid the best we can do is guessing, and without seeing specifics of how things are really connected it's going to be mostly blind guessing. Any chance you could show us photos or a clear diagram of how you're really connecting things?

Do you have access to an oscilloscope? Might be worthwhile to observe the startup and shutdown behavior of the power supply that's connected to Teensy's VIN pin. Some switching power supplies can have pretty terrible overshoot at startup. In the early days of Teensy 3.x there was a particularly bad little Traco brand switcher that Adafruit was selling. It did horrible things if powered from a weak battery without a capacitor located at its input (not output... adding extra output capacitors without any at its input made the problem *much* worse). Viewing the behavior with an oscilloscope was the only way to tell, since a regular voltmeter just shows 5V and looks like everything is fine.

 

[Angelo]

I read : we operate in a noise environment
Are signals comming from outside filtered, protected, limited,... ? or are they directly connected to teensy pins ???
How is your electronics connected to earth ground ??? And the PC used to connect to the Teensy ??

 

[robws]

Attached please find the power subsystem, as well as the main processor block. The overall board is fed by a 24V power supply (high quality) which is then broken down into 12, 5 and 3.3V. I will check the 5V startup, but in the past its pretty clean. And again on normal (non USB cable connected) startup/operation the system operates without any issues or failures.

 

[BillFM]

Thanks for posting more details.
I understand now what you meant by the 3.3V <-- 5V diode. Thanks for explaining.

As for Vbus/Vin diode OR'ing, most people cut the track between 2 large pads on bottom of Teensy and solder a diode across those 2 pads. No flying wires required. Using your method for updates, provided you can guarantee nobody ever plugs in a USB cable supplying Vbus (when Teensy is powered from Vin), the diode issue is resolved.

From your observation of Teensy possibly dying when USB is plugged/unplugged from a PC/Mac/whatever, I wonder if you have a grounding issue? There could be a potential difference (voltage) between the 2 grounds. When the USB connection is made, that voltage difference has to go somewhere.

For 5V peripheral devices, do you have voltage translators for inputs and outputs?
Or, only translators on Teensy outputs driving 5V device inputs?

 

[defragster]

Does DVM show anything from unconnected USB GND/Shield measuring to Teensy connector GND/hood?

<crosspost. - same concern p#8
Since the USB 5V is protected - but the GND will be 'active'
Had a Teensy here with external power and a 32x32 LED matrix board that didn't have enough GND and on plugging in the Teensy it found an easier LARGE GND path through Teensy to the PC with a USB power dongle. And that was with a normal 5V USB supply to the main PCB not an external supply 24V. This was only noticed when the Teensy was 'hard to reliably program' - fix was connecting another GND wire from 32x32 matrix to the PCB.

 

[robws]

I read : we operate in a noise environment
Are signals comming from outside filtered, protected, limited,... ? or are they directly connected to teensy pins ???
How is your electronics connected to earth ground ??? And the PC used to connect to the Teensy ??-

- The 24V base power supply is AC connected, with earth ground
- We have had these failures when the board is not connected to any external ports (we control motors, collect signals, etc) but they

Thanks for posting more details.
I understand now what you meant by the 3.3V <-- 5V diode. Thanks for explaining.

As for Vbus/Vin diode OR'ing, most people cut the track between 2 large pads on bottom of Teensy and solder a diode across those 2 pads. No flying wires required. Using your method for updates, provided you can guarantee nobody ever plugs in a USB cable supplying Vbus (when Teensy is powered from Vin), the diode issue is resolved.

From your observation of Teensy possibly dying when USB is plugged/unplugged from a PC/Mac/whatever, I wonder if you have a grounding issue? There could be a potential difference (voltage) between the 2 grounds. When the USB connection is made, that voltage difference has to go somewhere.

For 5V peripheral devices, do you have voltage translators for inputs and outputs?
Or, only translators on Teensy outputs driving 5V device inputs?

translators on both inputs and outputs.

 

[robws]

Does DVM show anything from unconnected USB GND/Shield measuring to Teensy connector GND/hood?

<crosspost. - same concern p#8
Since the USB 5V is protected - but the GND will be 'active'
Had a Teensy here with external power and a 32x32 LED matrix board that didn't have enough GND and on plugging in the Teensy it found an easier LARGE GND path through Teensy to the PC with a USB power dongle. And that was with a normal 5V USB supply to the main PCB not an external supply 24V. This was only noticed when the Teensy was 'hard to reliably program' - fix was connecting another GND wire from 32x32 matrix to the PCB.

This is what I am leaning towards, that we are getting essentially a ground loop through the processor. We have checked several failed teensies and the two we checked had IO pins which also were failed (eg 4 ohms VS the 38-40K ohms we see on "good" pins). Note this is for the teensy completely removed from our circuit, so only whats on the Teensy itself at that point.

 

[BillFM]

There's a signal called Legacy_Tension with resistor voltage divider (huge ratio).
Do the resistors have a voltage rating high enough to meet Legacy_Tension voltage? Resistors break down and become conductive as you exceed this rating.

BTW, use extreme caution if you measure ground potentials. It can be a shock hazard!!! Don't be touching DVM probes with your fingers (or any other conductive parts).

 

[sicco]

…We have checked several failed teensies and the two we checked had IO pins which also were failed (eg 4 ohms VS the 38-40K ohms we see on "good" pins). Note this is for the teensy completely removed from our circuit, so only whats on the Teensy itself at that point…

That should be a good hint. So which io pins exactly? And what does a voltmeter show when you measure from those pins to GND in a not yet failed unit?
What type and voltage are the zener diodes Z7 and Z11?
How exactly do you guarantee that the volts on all Teensy io pins cannot exceed 3.3V? Just one resistor as in R56, R57, R58, R83 will likely not be enough.