Latching issue preventing programming custom Teensy 3.2 with MKL02 until power cycle

Status
Not open for further replies.
I have a custom PCB based on Teensy 3.2 with a MKL02 (purchased within the last 6mo). This PCB is normally powered by 24V --buck--> 12V --buck--> 5V --Diode--> USB/Debug --LDO--> 3.3V. So the 3.3V section of the board can also be power & programmed with just the USB connection.

I also have a Raspberry Pi with GPIOs connected to Program and Reset (and Serial1), powered via the 5V power stage.

On boot up, I can reliably program and use the device and all features of the board. I can use Teensy loader (on Windows) to perform a firmware update without pressing a button, as long as the virtual serial port is given time to "close" (see https://forum.pjrc.com/threads/67834-PlatformIO-not-entering-programming-mode-reliably). This seems to be related to sending ~1kbps of utf8 (which I don't understand. It's not *that* much data).

I had some growing pains (this is my second custom Teensy 3.2 PCB) around connecting the Pi to Reset and Program because most GPIOs on the Pi have default pull-ups or downs, which partially interfered with the pull ups on Reset and Program (especially when the Pi is unpowered). But those problems I'm pretty sure I've got covered by setting those pull-ups in config.txt. Future revisions will use an N-Channel mosfet to ensure the single direction.

So, onto my actual issue.

After running and exercising the system for some time, at some point, reprogramming fails and does not recover until a full power cycle is performed.

  • Does not seem to be time based - left the device on overnight and it is still programmable in the morning.
  • Teensy loader shows: "Waiting for Teensy device... (hint: press the reset button)"
  • Pressing the "Program" button does nothing
  • COM port is still listed in available ports on Windows
  • I can open the COM port but no data is sent
  • Resetting the MK20 starts the program normally
  • Unplugging the Pi from the board (disconnect external IO) does not fix anything
  • Removing both USB and 24V un-bricks the device.
  • I haven't been able to watch what happens to the Reset line with a scope when Program is pressed while bricked

Probing around, I thought I found the culprit, PTA3 of MKL02 was sitting at 0.2V. My design leaves this pin floating but I expect the MKL02 to pull it up. Or maybe it only pulls it up during programming? In any case, this made me think maybe the debug enable was kicking in on the MKL02 and it was refusing to do anything because of that. However, after a power cycle, I still see 0.2V and programming works. Did the "Future debug enable" ever get enabled? How thoroughly tested is the debug enable? Does it maybe only turn on after a high to low transition on PTA3? This would explain why on power up, everything works fine. But after running the device for a while, maybe it briefly exceeds the threshold of PTA3, goes back low, and latches on?

Unfortunately, I still can't reliably get this behavior to start. So for now, I'm just getting tests ready so that when the bricking does happen, I can analyze. Any thoughts on tests to try would be greatly appreciated.

I've made two of these boards, but one got fried after an unfortunate encounter with 110VAC on the ground plane. I think the other board exhibited some of these issues but that was a while ago and, while I think I've seen these issues before, I'd attributed them to other problems that have since been fixed so I can't say for certain. I'll have more boards to try with this week, if all goes well.

I believe I've checked everything in the "Troubleshooting" of the MKL02 page. I've not seen an issue that leads to reliably broken programming (coming from an otherwise reliable and working system) like I'm seeing.

I've been trying to regularly try programming to see if I can't find which actions of using the machine trip this soft bricking issue, but no luck yet.

Thanks for the look!
 
Pressing the Program button should recover from almost any type of crash on the MK20 chip. The one case I know about, and this probably isn't relevant here but I'll mention it anyway, was early in the Teensy 3.0 days where someone used Traco DCDC converters which turned out to have terrible startup performance. The MK20 chip can get into a locked up start if the 3.3V power is highly unstable. Turns out those little Traco converters would go into an oscillation mode unless a capacitor was added at their input (not output), and wild oscillation on the 3.3V power line would put the chip into some strange lockup state where it never recovered until power was cycled off and back on again.

Part of the issue, at least in that case, is the MK20 chip isn't anything like a resistor when it comes to current consumption. It draws almost zero current until the voltage reaches some threshold, as I recall somewhere around 1.5 volts. If you have a power supply with poor startup or low stability, that sudden non-linear load can play havoc, causing the voltage to not rise monotonically from zero to 3.3V. A large dip to a lower voltage and then rising back up, rather than a smooth ramp up, can put the chip into a lockup mode.

If this sounds like a possibility, I'd suggest using an oscilloscope to watch how your 3.3V power line performs during power up. You might also wish to look at whether the chip really is fully powering down, or if power is leaking through the pins if signals are driving it while power is off.

Also double check the capacitor you're using with that LDO regulator. LDOs are notorious for appearing to work but having unstable oscillation under some conditions if the capacitor isn't right or isn't located very close to the LDO. Read the datasheet carefully. Some LDO regulators require the capacitor's ESR to be within a range, where too high is a problem, but also too low can cause it to go unstable. Most newer LDOs work with ceramic capacitors, but not all. But one thing is common to all LDO regulators - their control loop stability depends heavily on the characteristics of the capacitor connected at their output.


And to answer this specific question:

PTA3 of MKL02 was sitting at 0.2V. .... Did the "Future debug enable" ever get enabled?

No, this feature was never implemented. That pin is ignored.
 
Thank you for the info Paul, in particular about the debug enable.

I'm thinking it is not the LDO because I can reset the MK20 and my normal program runs fine.

For reference, I'm using an LP3961 with 2.2uF on the input and output for my 3.3V rail. I'll look into the 3.3V stability just in case. Curiously, I've never seen problems on power up. Only after running for some time.
 
Status
Not open for further replies.
Back
Top