Another Burned out Teensy Thread......

[nanokaztz]

Hello All,

I've recently started a new job and have taken over a in progress project which uses Teensy 4.0 and 4.1.Boards were designed and built before I started here. We have a modular system where various boards and be stacked together. The boards and connected by simple 4 pin (long) header with power (12V) and I2C lines. The top most board is then supplied with power and then each board has its own 5V linear reg as well as 3.3 from the Teensy. So far so good.


However we have run into a problem with assembly it seems. Some times after assembling a stack (including a stack that was taken apart to add a case) and applying power all teensys in the stack are instantly dead. 3.3v line shows ~0.15v and obviously sinks a good amount of current. This seems to imply that somehow the teensy are getting more then ~3.9V on a gpio. That it happens to all teensys (and to stacks that have worked before) it would need to be the SDA or SCL line as those are the only connection between the boards.
When this first occurred I thought the pins were perhaps misaligned resulting in 12V getting on the SDA line. However in retrospect that should only effect the boards below the first, not to mention its something we check for now. I thought maybe residual voltage on the 12V line but that doesn't seem possible after checking how long the 12V line takes to decay. Sadly I never had the mind to measure the connector pins right when a stack first died.

Edit: I took a look at the startup of the 5V reg and it is infact spiking to 6 volts. It's over 5.5 for about 30us. Maybe at time spikes higher./longer It is just an 7805 and so has no softstart. May also be unrelated.....
I'm at a bit of a loss at this point as to how the teensy might be getting an overvoltage. Is there another failure mode I'm just to blind to see...
I've thought about adding some overvoltage protection to the top board but I'd like to be sure that this is actually the problem.

I've attached the layout and schematic of the upper board, as well as a photo showing a stack. Normally there are of course standoffs between the boards to avoid contact.

 

[BillFM]

Hello.
According to schematic, if connector JP1 was wired incorrectly, or installed backwards/mis-aligned, +12V could be applied to SDA/SCL and, +12V would back drive the 3.3V rail thru R12. Assuming these connections are all headers (and not ribbon cable), not likely an applicable scenario.

If the pcb footprint was wrong for Q1 and Q2, or a solder short from collector-to-base, +12V could also get applied to Teensy pins (28, 29). If either of these transistors shorts out, collector-to-base, that would also apply +12V to Teensy. (assuming F1 and/or F2, are connected when things went wrong)

An assumption is that you know not to apply external 5V to Teensy while simultaneously plugged into USB power unless you've cut the Teensy pcb track (to isolate VUSB from VIN).

There are numerous 3-pin and 4-pin connectors that include +12V, all heading back to Teensy. Just another place to check the wiring.

Have you taken Teensy off the stack and tested it with nothing else connected except a USB cable?

 

[nanokaztz]

Thanks for the input!

Hello.
According to schematic, if connector JP1 was wired incorrectly, or installed backwards/mis-aligned, +12V could be applied to SDA/SCL and, +12V would back drive the 3.3V rail thru R12. Assuming these connections are all headers (and not ribbon cable), not likely an applicable scenario.

Yes the connections between boards is all through very long unstablized pin headers....which were not always soldered straight(As I said this is project I've taken over). We have certainly had cases of the pins not getting in the headers and such. Thats why it was my first thought.

If the pcb footprint was wrong for Q1 and Q2, or a solder short from collector-to-base, +12V could also get applied to Teensy pins (28, 29). If either of these transistors shorts out, collector-to-base, that would also apply +12V to Teensy. (assuming F1 and/or F2, are connected when things went wrong)

F1 and F2 are not being used at the moment. Solder shorts I haven't seen and many of the boards have run for weeks before being disassembled; so while loose solder is possible unlikely I would say.

An assumption is that you know not to apply external 5V to Teensy while simultaneously plugged into USB power unless you've cut the Teensy pcb track (to isolate VUSB from VIN).

Yes I am aware of this. The boards got fried with only the 12V input connected.

There are numerous 3-pin and 4-pin connectors that include +12V, all heading back to Teensy. Just another place to check the wiring.

Like the F1/F2 connector none of those ports are being used at the moment....of course doesn't mean something cant happen but with the number of times it was happened I would be skeptical.

Have you taken Teensy off the stack and tested it with nothing else connected except a USB cable?

No, mostly becuase I have been too lazy to desolder the ethernet connection preventing me from removing one.


I have however tried to find out some more info. On applying power the PMIC on signal is sitting at 1.9V (I believe this should be 1.1 but I am not sure if that is merely the minimum). Removing the regulator however showed no change. Applying 3.3V then directly (after the PMIC line goes high) gets me pretty much a hard short with 200ma/0.02V. I have also wittnessed the magic smoke coming from the main IMX Chip so I am pretty sure they are all fried as expected.

I may try to add some BAT54s to try and prevent this but have to make sure that plays well with the I2C lines

 

[BillFM]

Sounds like you've already visited the Power UP Sequence description on Bootloader web page.
https://www.pjrc.com/store/ic_mkl02_t4.html

Maybe on the board sending smoke signals, remove IMX chip. Apply 12V source and look for a spike on 3.3V regulator output (resulting from 7805 output over-shoot at startup). Also check the spec on 3.3V regulator input level. This chip changed a few times during shortages, so make sure you identify the part first. See Teensy 4.1 web page, Design Changes (near bottom of web page) for 2 sentences describing regulator changes.
https://www.pjrc.com/store/teensy41.html

Is the case (enclosure) made of metal or plastic?
Any chance the case mounting method shorted something? Nylon screws vs metal screws? Metal washers too big?
Maybe components/tracks touched each other between pcb's in the stack? Looks pretty scary where upper pcb sits above rf module pcb.
Any heatsinks? Not all thermal tape is non-conductive and neither is all thermal grease.

My $0.02 worth.

edit: I may be mistaking an rf module when in fact it might be a Teensy 4.0

 

[sicco]

Many Teensy’s and they all share their 3V3 pins? That’s asking for trouble. Especially when there’s also a 3V battery on VBAT.
The 5 to 3V3 LDO’s have a MOSFET at their output that shorts to GND when LDO is receiving ‘OFF’ input. They have this FET to discharge any caps on the 3V3 rail, so that off is really off with caps discharged.
If one Teensy goes into sleep mode, but others not yet, then expect the two Teensy LDOs ending up in fighting mode.

 

[nanokaztz]

Sounds like you've already visited the Power UP Sequence description on Bootloader web page.
https://www.pjrc.com/store/ic_mkl02_t4.html

Maybe on the board sending smoke signals, remove IMX chip. Apply 12V source and look for a spike on 3.3V regulator output (resulting from 7805 output over-shoot at startup). Also check the spec on 3.3V regulator input level. This chip changed a few times during shortages, so make sure you identify the part first. See Teensy 4.1 web page, Design Changes (near bottom of web page) for 2 sentences describing regulator changes.
https://www.pjrc.com/store/teensy41.html

Yeah we actually have a mix of boards with different regulators. One has a limit of 5.5 and one of 6V. The overshoot issue thought I am bit skeptical of as the boards have always died right after re/assembly. Once they run once they don't have any issues. I'm just not sure how in an unpowered state (and the 12V and 5V come down in ms) something could go wrong here.

I haven't tried without the IMX but certainly in working boards even with that overshoot there was no spike on the 3V3 line

Is the case (enclosure) made of metal or plastic?
Any chance the case mounting method shorted something? Nylon screws vs metal screws? Metal washers too big?
Maybe components/tracks touched each other between pcb's in the stack? Looks pretty scary where upper pcb sits above rf module pcb.
Any heatsinks? Not all thermal tape is non-conductive and neither is all thermal grease.

Enclosure is just printed PLA. Standoff is pretty close to the ground pin for stacking but not really much else. No heatsinks or washers or such. Once assembled there seems to be enough clearance (and yes those are teensys 4.0s). But yes....its tight.

I've noticed that that connection between the SHD pin from the EM4095 is direct connected to the teensy...and it has a 5V 50k pull up interally. Doesn't seem to be a problem and wouldn't have anything to do with assembly but just something to note....

Many Teensy’s and they all share their 3V3 pins? That’s asking for trouble. Especially when there’s also a 3V battery on VBAT.
The 5 to 3V3 LDO’s have a MOSFET at their output that shorts to GND when LDO is receiving ‘OFF’ input. They have this FET to discharge any caps on the 3V3 rail, so that off is really off with caps discharged.
If one Teensy goes into sleep mode, but others not yet, then expect the two Teensy LDOs ending up in fighting mode.

The Teensy's do not share the 3.3V line. Each is supplied with 5V from the onboard 7805. They are only connected by GND and the i2c lines.

 

[BillFM]

EM4095 is a 5V part with at least 2 digital 5V outputs (DEMOD_OUT, RDY/CLK) going to a uP as shown in datasheet.
Are these signals going into a Teensy? Ouch, Teensy 4.0, 4.1, is not 5V tolerant.

Or, are the EM4095 outputs logic-level translated before hitting Teensy? That would be a good thing.

 

[nanokaztz]

EM4095 is a 5V part with at least 2 digital 5V outputs (DEMOD_OUT, RDY/CLK) going to a uP as shown in datasheet.
Are these signals going into a Teensy? Ouch, Teensy 4.0, 4.1, is not 5V tolerant.

Or, are the EM4095 outputs logic-level translated before hitting Teensy? That would be a good thing.

Yes they are being "translated"....but only by a voltage divider (I've attached the schematic below) It's 3.3k/1.8k so even at 6V should be **just** okay for the teensy inputs. The EM4095 even has a seperate 5V low noise reg which at least in my tests has a nice soft start and no spikes.

Edit: While these are all interesting points that maybe could be improved....I can't see how they would be affect by assembly.

 

[sicco]

Those 3k3/1k8 do not lower the 5V voltage if you wire them as in the schematic…

 

[BillFM]

Thanks for sending schematic.
To start with, the voltage dividers are implemented backwards. So, 5V is being sent directly into Teensy.
R1 should be on the other side of R2.
R3 should be on other side of R4.

edit: sicco types faster than me.

 

[nanokaztz]

Jesus am I blind. Yup no real translation just current limiting in that setup. Check the boards and they are in fact built that way. The Teensy pins are getting a nice 4V right on the pin right now. Surprised then this issue hadn't turned up before. Guess time to cut some traces.
Now what that would have to do with assembly I have no idea....

 

[BillFM]

If you decide to re-spin pcb, might want to take care of SHD input with a level shifter to decouple it from Teensy.

Yeah, no idea how stack assembly caused the issue. It was a train wreck waiting to happen.

 

[nanokaztz]

Ya ...here are a few things I would have changed anyway if/when we make a new version or need more.

Have to see if in the next few weeks if we get any spontaneous failures