Electric Fast Transient issue when USB connected

Status
Not open for further replies.
K

kyle_IRIS

Member
Hey everyone.

We're trying to have a product pass FCC certification. It's a 2-axis flight simulator controller, run by a Teensy3.1. Future units are powered by a Teensy3.2

The Teensy is being powered by an ATX switching supply's 5V standby line into its Vin pin, and the ground is connected to the supply's ground lines. There is a 1uF and 0.1uF ceramic cap between these connections. The ATX power supply is electrically connected to the chassis.

The USB power trace has been cut. The Teensy USB port is connected to a short usb connector that mounts to the back of the case, where it is grounded to the chassis and provides a user USB connection.

When the USB is connected to a computer and a fast electric transient is applied to the ATX's 110V line, the unit loses USB communication to the PC, the code apparently stops working and the watchdog does not restart the firmware. Cycling the USB connection has no effect; the unit is not recognized by the PC. Cycling the power connection results in the unit resuming normal function with no apparent damage.

When the transient is applied with the USB disconnected, normal operation resumes uninterrupted and there is no sign of data corruption.

I'm guessing there is an issue with some kind of latch up? However, the lack of a watchdog reset is making this impossible to pass class B specifications, and we'd rather not concede class C, if possible.

Does anyone have any experience or general suggestions on how we could make our design more robust?

If useful, schematics or layout snapshots can be provided.

Thanks much for your time,
Kyle
 
Xenoamor said:
You should definitely definitely have a watchdog timer
Click to expand...

Yeah, agreed. We do have a watchdog timer that works under every other circumstance we've come across. I mean, it doesn't really get used much anymore since most of the crashes have been removed, but if we put a while(1); in the code, the watchdog shuts it down after a couple seconds. After the reset if you look at the reset type register, it will confirm it was a COP Watchdog reset, so we're confident the watchdog is ok.

It seems as though the spike of current flow from the USB lines into the PC are latching something up and preventing the watchdog from resetting things.

Our immediate reaction is to put common mode chokes on the D+ and D- lines, in case the current being drawn out of the chip on those lines is latching the K20...does that sound like a viable solution? It would require some kind of in-line USB board, unfortunately.
 
Sorry I probably should have read your OP better.

Make a tiny usb connector board and try some ESD protection to find out if that's what's causing it. The MCU is rated for +/-500V ESD and a +/-100mA latch-up current.
You should be able to probe around and see if any pins are latched up

It maybe something to do with that polymeric fuse on the VUSB line but I rather doubt it

You could try using just the blink program and see if it occurs with that. Just to rule out any software possibilities

It does sound like a latchup though. Monitor the D+/- pins before, during and after a surge. I'm kind of assuming you have the equipment there to make the transient. Is this the case?
 
Last edited:
Xenoamor said:
I'm kind of assuming you have the equipment there to make the transient. Is this the case?
Click to expand...

Not currently, the unit under test is at a facility separated from us by a 2 hour ferry ride.

We were going to try using a barbecue lighter spark to generate the event though.

I like the idea of using the blink program though to rule out the code. I'll see if we can get some sort of event generator and post the results. I believe the test they're using is a 1000 V spike, so that might explain the latchup...fortunately it doesn't seem to be frying anything.

How might one determine or confirm a latched up circuit?
 
kyle_IRIS said:
How might one determine or confirm a latched up circuit?
Click to expand...
You could measure the current the teensy is drawing before and after. If this increases by 50+mA you most likely have a latchup.

A latchup is shorting the positive and negative rails so check to see if this is happening. I advise you read up on how latchups manifest and why they are stuck in that state. Wikipedia has a good article on this
 
Excellent feedback, thank you.

I read the wikipedia article, as well as some other literature. Turns out CMOS latchup is a very well documented issue... Looks like if the transient is latching us up, the only cure is a power cycle. Fortunately we can still meet FCC - we'd just require a line or two in the manual.

Tomorrow we'll try and reproduce the problem here with the lighter and put an ammeter in line with the teensy's power to confirm or rule out the latchup. I'll do so with blink running, as well as our firmware. We've modified the USB code (including the interrupt) so it's entirely possible this is a firmware issue. I've always been under the assumption that a hang in an interrupt would eventually trigger the watchdog, but I admittedly haven't studied the watchdog's circuitry.

Thanks again. More to come...
 
kyle_IRIS said:
I've always been under the assumption that a hang in an interrupt would eventually trigger the watchdog, but I admittedly haven't studied the watchdog's circuitry
Click to expand...
It should. It's as close to an automatic power cycle you can get without actually power cycling.
When you do test it check the lines between the MK20DX256VLH7 and the MKL02Z32VFG4. It's possible for some reason the Reset line controlled by the bootloader chip is latching up and preventing the chip from operating.
I've just checked and the MKL02Z32 has the same ESD protection in chip.

It's hard to say how the watchdog is wired inside the MK20 but I do know it has a separate clock for better separation. Are you using this clock by setting WDOG_STCTRLH |= CLKSRC?

PaulStoffregen said:
Let me know what you find? Maybe I'll try to set the same up here....
Click to expand...
I think we'd all be interested to hear what you find. Keep us in the loop Kyle!
 
Last edited:
Xenoamor said:
It should. It's as close to an automatic power cycle you can get without actually power cycling.
When you do test it check the lines between the MK20DX256VLH7 and the MKL02Z32VFG4. It's possible for some reason the Reset line controlled by the bootloader chip is latching up and preventing the chip from operating.
Click to expand...
I will check this out. The crash only happens with the USB connected to a PC and since it looks like PTA6 of the MKL02Z32VFG4 is connected to pin 4 of the USB, this chip could be latching up. Im not exactly sure what pin 4 of the micro-AB is connected to, but if it makes a connection to the PC and all levels on the Teensy come up by 1kV, this could very well be part of our problem.

Xenoamor said:
It's hard to say how the watchdog is wired inside the MK20 but I do know it has a separate clock for better separation. Are you using this clock by setting WDOG_STCTRLH |= CLKSRC?
Click to expand...

This is what I have for the watchdog, but I admittedly copied some example code and haven't read that part of the K20's manual...

Code: 
#ifdef  WATCHDOG_RESET_TIMER  
extern "C" {
  void startup_early_hook() {
		// The next 2 lines sets the time-out value. This is the value that the watchdog timer compare itself to.
    WDOG_TOVALL = 1000; 
    WDOG_TOVALH = 0;
    WDOG_STCTRLH = (WDOG_STCTRLH_ALLOWUPDATE | WDOG_STCTRLH_WDOGEN | WDOG_STCTRLH_WAITEN | WDOG_STCTRLH_STOPEN); // Enable WDG
    //WDOG_PRESC = 0; // prescaler 
  }
}
#endif

And to feed the dog (we dont kick dogs where Im from)
Code: 
// Do not call this function more frequencly than 1 ms intervals
int feedDog() {  

	ACCESS_TIMER(3000000);
	noInterrupts();
	WDOG_REFRESH = 0xA602;
	WDOG_REFRESH = 0xB480;
	interrupts();
  
	return 1;
}

Code: 
// Use in periodically called function calls to prevent high frequency access
#define ACCESS_TIMER(__ACCESS_PERIOD__)  {static uint32_t update_time = 0;\
   if ((uint32_t)(micros() - update_time) < __ACCESS_PERIOD__) return 0;\
   update_time = micros(); }
 
I've just noticed CLKSRC defaults to one and it should be zero.
So calling:
Code: 
WDOG_STCTRLH = (WDOG_STCTRLH_ALLOWUPDATE | WDOG_STCTRLH_WDOGEN | WDOG_STCTRLH_WAITEN | WDOG_STCTRLH_STOPEN | WDOG_STCTRLH_CLKSRC);
Will actually set this to zero which is correct

Everything appears to be in order. Although bit 8 of that register is reserved and defaults to 1 so changing it to 0 is undefined beheaviour. I imagine it would be read only though if it altered anything
 
Separately: If the problem happens when there are transients on the ATX power supply input, wouldn't a better option be to attempt to protect that input?
A TVS or MOV between live and zero on the mains input could conceivably take care of that just fine!

Another thing you can do is add capacitance and inductance to the output of the ATX power supply for more filtering. Maybe even a TVS there, too?

Finally, there are specific chips for ESD protecting USB ports that you could wire up on the USB side. They are basically 4-way TVS diodes with 6V threshold in a single package.
 
jwatte said:
Separately: If the problem happens when there are transients on the ATX power supply input, wouldn't a better option be to attempt to protect that input?
A TVS or MOV between live and zero on the mains input could conceivably take care of that just fine!
Click to expand...
Great idea, unfortunately we can't be taking the power supplies apart and reworking them. Im not sure why whatever protection they've included isnt prventing this energy from reaching the Teensy's USB lines, but it seems to be nevertheless.

jwatte said:
Another thing you can do is add capacitance and inductance to the output of the ATX power supply for more filtering. Maybe even a TVS there, too?
Click to expand...

Yes, in hindsight this would have been a good idea. Every chip has its own pair of ac decoupling but there's no TVS on the 5 or 12 volt rails and I think you're right in saying there should be...maybe even ferrite beads between the ATX power header and the board's bulk decoupling?

If there's a standard setup for treating an external DC level that comes onto your board, or just rules of thumb, I'm all eager ears. All we've put on our board this time was ceramic and some bulk electrolytic decoupling.

jwatte said:
Finally, there are specific chips for ESD protecting USB ports that you could wire up on the USB side. They are basically 4-way TVS diodes with 6V threshold in a single package.
Click to expand...

This is a good idea as well, but the way I see it, this would have to be a Teensy development since without pogo pins, we cant grab the USB lines and condition them.


From what I can tell, all of the levels of the product are coming up together which is why TVS's and extra capacitance wouldn't work.

More to come...
 
So we've got a barbecue piezoelectric spark generator. It's working quite well - we can generate a spark about 7mm in length which is fun to watch if nothing else. By some crude calculations, this amounts to somewhere between 10 and 30 kV (depending on humidity which is high here).

This is a higher voltage than they're using in the lab.

By shocking various points I can achieve mixed results. Sometimes shocking the same point multiple times produces different results.

The majority of the time, shocking the chassis at any point produces no result (unless you're touching the chassis in which case you get shocked...)
The most common error is that when the unit is connected via USB with an Arduino serial window (running the USB Serial, not the emulated Serial), the serial connection goes dead and comes back right away. As soon as the connection returns, it works fine.
Sometimes, the unit immediately restarts. A query to the Teensy's reset type register indicates an external pin reset.
More rarely, the unit restarts a second or two after the shock. Reset type is a Watchdog (COP) reset.
Only once was I able to get the unit to apparently latch up where a power cycle was required to cause a restart. I have not been able to reproduce this result, but will continue trying.

Our theory is that although the lab is using a lower voltage, the pulse is likely a longer duration than we can produce with the piezoelectric lighter. This longer duration jump in all voltage levels causes more current to move through the USB lines and ultimately latch either the K20 or its bootloader. I will confirm the pulse duration when we next talk to the lab.

In the mean time, I will wire up an ammeter to the Teensy's power lines and try and get the thing to crash hard so we can confirm a latchup state. Once I can reproduce this, I will attempt to determine the latched line, although I dont have much that can accurately probe the USB lines.

Thanks again for all the feedback, gentleman. It's great to have this community based around Paul and Robin's fine work to turn to when things get confusing.
 
Last edited:
Depending on how desperate you are:

1) You could use a USB extension cable between case and Teensy and add the extra protection there.

2) You could tap into the D+, D-, USBV, and USBG signals on the back of the Teensy board, and add the extra protection there.

3) You could solder TVS-es between the poles of whatever entry you use for 5V and 12V rails.

4) You could add an additional power entry module, and use a pigtail to forward to the ATX, and put the external mains TVS (and a fuse?) into that module.

5) You could design a small intermediary board for adding filtering between ATX and the rest of your device.

6) You could wrap ferrite rings around whatever cables you do have access to (USB, power, etc.)

Does the lab document its test procedure anywhere? If so, you should probably do what you can to reproduce that more precisely...
 
The barbecue lighter trick sounds very similar to 29441-Teensy-LC-resets-due-to-EMI, just holding the piezo element close to the PCB causes a reset. I could also just reset the Teensy (with blink, no external circuitry) by touching the program pin with a piece of wire of 20cm. The Teensy randomly stopped working when placed inside my car, I believe I fixed this with the suggestion from Constantin to add a pullup and a capacitor to the program line.
 
Saw this and thought back to my Piezo post. But T_3.1 was more resistant than the LC - and I only did noise not direct strikes as it had a ground shield shroud. I recalled a change to the program delay/debounce under test and never tried added parts. My bare Teensy's hang off of USB cables and get touched a lot with no issues - except one touch display with a floppy header connection to a hand wired board I stopped using that would take some effort to reconnect or program - now wondering if that was getting taken to a similarly odd state - except it took more than a simple power cycle to recover.

@iwanders - I had an ESP8266 unit that would reset touching a wire's free end - or jump baud rate (and keep running) when connected to a Teensy that started with the connected reset pin in default (Hi-Z) state when changed to OUTPUT and setting HIGH. It had a pullup onboard - but adding a #104 cap fixed that on three copies of the ESP board.
 
Hey everyone,

I just thought Id follow up on this. We ended up stipulating that a power cycle is required following an electric fast transient.

Attached is an image of the transient that was applied.

Because this was sufficient to pass FCC, and because our team is swamped with other work, we didn't investigate much further. Our intention for further products is to clamp the USB lines using TVSs, and possibly put inline ferrite beads to limit the current transients on these lines. When we do this, I'll post our results here.

Thanks everyone for your insights.
Electric Fast Transient.jpg
U is in units of kV

This was applied to the AC power input of an ATX power supply (P/N:9PA250CZ54) which powered the Teensy via the 5VSB power rail (standby power).
 
Last edited:
kyle_IRIS said:
Hey everyone,

I just thought Id follow up on this. We ended up stipulating that a power cycle is required following an electric fast transient.

Attached is an image of the transient that was applied.

Because this was sufficient to pass FCC, and because our team is swamped with other work, we didn't investigate much further. Our intention for further products is to clamp the USB lines using TVSs, and possibly put inline ferrite beads to limit the current transients on these lines. When we do this, I'll post our results here.

Thanks everyone for your insights.
View attachment 6563
U is in units of kV

This was applied to the AC power input of an ATX power supply (P/N:9PA250CZ54) which powered the Teensy via the 5VSB power rail (standby power).
Click to expand...

Considering most quality power supplies have some form of HV protection that seems more an issue with the supply then the Teensy. Also good UPS and protected power strips should be able to prevent that also, kind of why they exist.
 
I've just witnessed a similar latch-up while I was running some tests using a USB connection between a laptop and a (in-circuit) Teensy 3.1.

I'm using the Teensy board to operate & control a sine wave inverter. I was reading some inverter parameters using the USB serial connection then, after disconnecting the serial terminal, I have removed the USB cable from my laptop.

In that instant, the inverter's Mosfets went on fire (actually, on smoke). Being a device under test, the inverter had no overcurrent/shot-through protection (I was testing the voltage feedback algorithm with no load connected).

What could possibly happen?? Teensy has an isolated power supply (flyback topology), my laptop is using an isolated power supply, too.. they only shared the ground wire.

I don'd know if there was a glitch or a complete latch-up (as the smoke were my first priority) but either way I hardly want to avoid that in the future.

How to make a reliable/safe USB connection with a Teensy board?
 
If you need the strongest isolation, use one of the serial ports, and use a USB serial adapter, and use opto isolators or similar to decouple them.

However, why did the FETs catch on fire? Do you not use a FET driver that ensures safe operation, like a 2101 or similar?
 
jwatte said:
If you need the strongest isolation, use one of the serial ports, and use a USB serial adapter, and use opto isolators or similar to decouple them.
Click to expand...

Many thanks for your suggestion, I guess that's supposed to be the best option (although the existing USB port would have been more convenient). Btw, there's no isolated (passive/active) USB cables available?

Maybe a custom USB extension cable, using some ferite beads/TVS diodes (like you've sugested few posts above) could be enough?

jwatte said:
However, why did the FETs catch on fire? Do you not use a FET driver that ensures safe operation, like a 2101 or similar?
Click to expand...

Of course I'm using a Mosfet driver, but any glitch in PWM signal generation could lead to both switches of the same h-bridge side (upper and lower Mosfets) being turned on at the same time (shoot-through condition).
 
someteen said:
Btw, there's no isolated (passive/active) USB cables available?
Click to expand...

I don't know of any. USB actually uses pull-up resistors of different values to select the various bus version numbers / speeds, so that would be super hard to build.

Of course I'm using a Mosfet driver, but any glitch in PWM signal generation could lead to both switches of the same h-bridge side (upper and lower Mosfets) being turned on at the same time (shoot-through condition).
Click to expand...

I would recommend a driver with built-in shoot-through protection. For example:
http://www.infineon.com/dgdl/ir2111.pdf?fileId=5546d462533600a4015355c810e51682
http://www.infineon.com/dgdl/ir2184.pdf?fileId=5546d462533600a4015355c955e616d4
IR2101 was a mis-statement: it's not the right tool for this particular job!
After burning a bunch of FETs, I finally started using them :)
(And in fact, even though they're often called "half bridge drivers" they can also be used as synchronous rectification drivers.)
 
Well, dead time insertion alone is not fail-proof enough (especially a fixed value ones).

Adaptive dead time is one step further but the current sense is the ultimate protection (if one switch fail short, a dead time is useless).

Like I've just said, it was a device under test and I was trying to optimize some operation parameters (dead time included, though not at that time).

I usually don't rely on MCU dead time generation neither; I'm using a separate (logic gates) circuit to generate the complementary signal for the half-bridge and for dead time insertion. This time, I have by-passed that circuit to manually adjust/test those parameters.

Sure thing, Teensy knows when to strike! ;)
 
Status
Not open for further replies.
Back
Top