Teensy 3.2 boot problem with 3.3V power (200 board statistic)

Status
Not open for further replies.
Hi everybody,
I’m using 150 Teensy 3.2 to do motion control for an interactive installation.
We have problem at boot. Randomly some board do not boot with the sketch.
Sometimes for the boards that do not boot pressing the reset button make them work, sometimes not.
To have them boot we have to power them multiple time.


All the board are powered with 3.3V provided by Murata OKI-78SR-3.3 DC/DC converter and connected in parallel to 3 main power lines with linear regulator that provide 24VDC on each line. When we power the system the booting behaviour is not consistent.

Some numbers:
We have 6 board out of 200 ordered that systematically don’t boot the program when we power the system.
To have them running we have to disconnect and reconnect them multiple times.

We have around 15 board out of 200 ordered that don’t boot the program around 1 time out of 10
To have them running we have to disconnect and reconnect them one time.

We have around 10 boards that don’t boot the program around 1 time out of 50
To have them running we have to disconnect and reconnect them one time.

The other boards are behaving correctly and boot all of the time.

All the teensy are mounted on a custom PCB and the PCB’s are machine assembled.

We checked the 3.3V power line with scopes and the rise time is the same when the board boot properly and when the board do not boot properly.
We have a 300ms delay as suggested in the forum as first line of setup function.
I would exclude a software problems since lot's of board boot every time and the software is the same.
We have also tried to load the blink example on the board that don't boot consistently and the behaviour is the same.

What is going on?

Best,
G.
 
Years ago a similar startup problem was reported with Traco power supplies. In the end, it turned out those little DC-DC modules were not stable unless they had an extra input capacitor. The Traco datasheet said the extra input cap was only necessary with certain input voltages, but it turns out the supply would go unstable without it, even in the ranges where the datasheet implied it wasn't needed.

Here's that old thread.

https://forum.pjrc.com/threads/23647-Powering-Teensy-3/page2

Also, you should be aware the button on Teensy is "Program", not "Reset". It puts Teensy into programming mode.
 
We have more that 100 boards that boot every time without problems and all have same DC converter, all the PCB are machine assembled.
Here I attach you two images of the 3.3V rising front, you can see that the powerup ramp is the same when the teensy boot or not.
Sorry for the "reset" button!

This is the front when the board boot
IMG_5954.jpg

This is the front when the board do not boot
IMG_5953.jpg
 
Well then, I'm out of guesses, based on only the info written so far.

As a next step, I'd suggest adding a distinctive LED blink at the beginning of your code. Lots of startup problems can appear as if your program never booted up. Plenty of other people have wasted a tremendous amount of time believing their program wasn't starting, only the later discover it was indeed running but doing something that crashed before they could observe any signs of life.

Also pay careful attention to whether any C++ classes or Arduino libs you're using run stuff in their constructors. Those execute before setup(), so any problems in constructors can appear as if your program never ran.
 
Interesting.
Whats the output load on the Murata 78SR and what does your input voltage look like at start up?
We use the 78SR's at work and I also use them at home. They are very handy and fairly clean even without external caps. We have been using them as drop in replacements for 7805's on one of our VFD drives.
 
As a next step, I'd suggest adding a distinctive LED blink at the beginning of your code. Lots of startup problems can appear as if your program never booted up. Plenty of other people have wasted a tremendous amount of time believing their program wasn't starting, only the later discover it was indeed running but doing something that crashed before they could observe any signs of life.

Also pay careful attention to whether any C++ classes or Arduino libs you're using run stuff in their constructors. Those execute before setup(), so any problems in constructors can appear as if your program never ran.

All the teensy (200 units) have the same program and we have the problem just on specific hardware modules.
I'm starting thinking this is a Teensy problem because I really can't find anything else.
 
Interesting.
Whats the output load on the Murata 78SR and what does your input voltage look like at start up?
We use the 78SR's at work and I also use them at home. They are very handy and fairly clean even without external caps. We have been using them as drop in replacements for 7805's on one of our VFD drives.

Hi, the images attached in my previous post there are two startup ramp of the Teensy input voltage. As you can see they are the same but in one case it boot and in the other not.
 
Do your 6 worst boards boot when you disconnect all I/O? Do they boot if you replace the switching regulator with a linear regulator?. Do they boot if they are just running a very simple program?
 
Since it might be a borderline symptom, happening only on a part of your Teensy boards, I wonder if a delayed boot with a 2.2uF from the MK20's reset pin to ground might solve it... If the issue persists, you can definitively exclude the power supply issue. But if this solves the problems, it would mean that the ramping up of the supply voltage is near the Teensy's tolerance band.
 
We have observed a similar issue with a MC34063 DC-DC converter set for a 5V output on the PCB, fed into a 3.3V linear regulator. Fortunately our boards are powered individually, so usually takes just unplugging and replugging to bring them back to life. I too, could not find an issue with rise times or a discrepancy between units that frequently suffered the issue and those which booted every time.

We later identified an alternate mains AC/DC converter to replace the one we were using which remedied the issue for 80% of cases.

This probably isn't particularly helpful information for you, other than to say we had what sounds like the same issue; however we never really solved it, just worked around it.
 
We have observed a similar issue with a MC34063 DC-DC converter set for a 5V output on the PCB, fed into a 3.3V linear regulator. Fortunately our boards are powered individually, so usually takes just unplugging and replugging to bring them back to life. I too, could not find an issue with rise times or a discrepancy between units that frequently suffered the issue and those which booted every time.

We later identified an alternate mains AC/DC converter to replace the one we were using which remedied the issue for 80% of cases.

This probably isn't particularly helpful information for you, other than to say we had what sounds like the same issue; however we never really solved it, just worked around it.

This is exaclty our situation. I tried to do other tests but I didn't find any difference in voltage rise at power up between boards that boot and boards that don't.
 
Since we would like to use the teensy again in other projects is there a way to avoid the booting problem when using external regulated power? The problem is related to the external power converter I think.
 
Problem is that the Teensy is (compared to older AVR or PIC processors) so quick starting up and initializing internal build blocks that you can run sometimes into these issues if you have the power more or less slowly ramping up. In my modest experience, an additional capacitor from the reset pin to ground, delaying the boot by 0.1 to 1s is the cheapest but efficient solution. The more professional way is using a voltage regulator with an open collector "power good" output which keeps the reset pin low until the output voltage has stabilized. IIRC, all the ATX power supplies in common PCs work this way.
 
Problem is that the Teensy is (compared to older AVR or PIC processors) so quick starting up and initializing internal build blocks that you can run sometimes into these issues if you have the power more or less slowly ramping up. In my modest experience, an additional capacitor from the reset pin to ground, delaying the boot by 0.1 to 1s is the cheapest but efficient solution. The more professional way is using a voltage regulator with an open collector "power good" output which keeps the reset pin low until the output voltage has stabilized. IIRC, all the ATX power supplies in common PCs work this way.

This seems the best solution.
There is just the problem of the reset pin position that on teensy 3.2 is not exposed on a pad or a ring.
How do you cable that pin under the board?
I'm thinking about soldering the teensy board on my PCB as it was an SMD component.
 
There is a small round pad marked "RST" on the Teensy 3.2's bottom side which allows to solder a capacitor towards the nearby USB GND pad. As I'm somewhat 1980ies when it comes to circuit design, my Teensys all get pins and are plugged in corresponding sockets on the PCB. That leaves enough space for the capacitor.
 
As I mainly only do things here for my own fun and on a onesy or twosy basis, the others here talking about the hardware are much more likely to be on the right path...

But if it were me, I would also try to do things to verify that it is not something with the software. For example if all of the chips are programmed with just something like Blink, do you still have the same problems?

I know Paul mentioned about putting in a distinct blink as the first thing that runs... Which in #9 you implied you have done something like this. But without seeing the code it is sometimes hard to say, what is going on.

At one time I had issues with a program that sometimes booted up and other times did not. Turned out there was an issue with a set of constructors which tried to touch the hardware, but depended on other things like the Teensy init code to have been completed... Sometimes worked sometimes not... I fixed this, by NOT having the constructors touch the hardware, but instead having an Init function that was called during setup... But again just throwing darts...
 
There is a small round pad marked "RST" on the Teensy 3.2's bottom side which allows to solder a capacitor towards the nearby USB GND pad. As I'm somewhat 1980ies when it comes to circuit design, my Teensys all get pins and are plugged in corresponding sockets on the PCB. That leaves enough space for the capacitor.

How you program the board after you solder a capacitor between rst and gnd?
When i try to program the board the capacitor hold up the pin causing this behaviour:

Screenshot 2018-04-18 11.49.34.png
 
Ah, good question. Up to now, I programmed first, then soldered the capacitor and assembled everything. I never tried to re-programm a board with capacitor.
 
I think the best option is:
- Use 5V DC converter
- Put a PMOS between the 5V converter output and the VIN of the teensy
- Activate the PMOS when the 5V line is above 4V

In this case the rise time of the 3.3V line should be ok.
 
I think the best option is:
- Use 5V DC converter
- Put a PMOS between the 5V converter output and the VIN of the teensy
- Activate the PMOS when the 5V line is above 4V

In this case the rise time of the 3.3V line should be ok.

This should give the same result as using a power supply with an additional “power good” signal output.
 
Status
Not open for further replies.
Back
Top