Teensy 3.2 not booting

Status
Not open for further replies.

stg

Member
Hello,

I recently purchased 2pcs Teensy 3.2 from Mouser, 474-DEV-13736 reportedly from SparkFun.

When connecting one of them for the first time I noticed the lack of the typical "device connected sound" present on Windows machines, but thought little of it.
Tried downloading code and it didn't work until I pressed the PROG button, which was also unusual.

I've worked with hundreds of Teensy 3.2 and never had a problem (with a fresh one), but I've also never bought a Teensy that didn't come from PRJC.

What happens is that there is about a 50/50 chance the processor boots up or not.
When it does not, the firmware does not launch and the computer does not recognize anything being connected, as it still would if there had been an infinite loop in setup for example.
Pressing the PROG button works fine and as soon as the new firmware is written it launches without issue.
The problem never re-occurs until the Teensy is powered down and up again and the behavior is the same when powered from the VIN rail (3V3 not tested).

This behavior happens regardless of what program is loaded (e.g. "blink" behaves exactly the same way).
"Normalclocking" the processor at 72MHz has no effect.

Quite stupidly, I've already assembled this into a product and replacing it would be quite a bit of effort, so I was hoping that someone could offer some insight into what may be going and and possibly suggest a way to work around the issue.

Best,
stg
 
With a good cable - with data lines not charge only - a new T_3.2 will blink but not present as USB as shipped. Button push takes to bootloader for programming - that will sound the Windows Arrival and it sounds as if it will upload code.

Code uploaded then must have USB Type Serial of some sort to present as on USB to Windows.

Some other nots and sample blink to test and makes notes against if it doesn't work: Can-t-communicate-with-Teensy-3-2-through-Teensyduino
 
Dear defragster,

Every new Teensy 3.2 I have on hand does present itself on USB when connected, just not as a CDC serial port.
They show up with a VID and PID under ports and can be programmed without pressing PROG.

There is no problem with the sketch, it does not matter what sketch is loaded and setup() is never reached.
The USB cable is not an issue - the problem remains when powered from VIN.

I've had similar things happen with other bare Kinetis processors, most notably the MK20DX128 which I have used a lot - but never with a factory assembled Teensy or a 'DX256.
 
Yes, not as CDC Serial. I see a new one showing as this { MSFT USB Device Viewer } so it isn't a blank chip:
Device Descriptor:
bcdUSB: 0x0200
bDeviceClass: 0x00
bDeviceSubClass: 0x00
bDeviceProtocol: 0x00
bMaxPacketSize0: 0x40 (64)
idVendor: 0x16C0
idProduct: 0x0486
bcdDevice: 0x0100
iManufacturer: 0x01
0x0409: "Teensyduino"
iProduct: 0x02
0x0409: "Teensyduino RawHID"
iSerialNumber: 0x03
0x0409: "124538"
bNumConfigurations: 0x01

ConnectionStatus: DeviceConnected
Current Config Value: 0x01
Device Bus Speed: Full
Device Address: 0x12
Open Pipes: 4

And indeed the newer TeensyDuino with TeensyPorts listing on the 'Tools / Ports' menu shows it as an active device with that RawHID type from new.

Given it can be programmed at all - 'Pressing the PROG button works fine ...' - It isn't blank - at least not the programming MCU.

Indeed a mystery with ''already assembled this into a product '' and without feedback from a known sketch and IDE/TDuino Version
 
I've been digging around in the core, using PTC5 (LED) to figure out exactly where things go wrong.
The processor does indeed boot up, but at the following line (compiling with -O0 of course)...

mk20dx128.c 752: if (!(RTC_CR & RTC_CR_OSCE)) {

...the processor hard-faults (ending up in fault_isr).

The location is perfectly consistent, but only happens on cool boot and even then - not always.

This, of course, makes no sense - the only way I know this could happen is if the clock gate for the RTC peripheral was not enabled, which is done just a few instructions earlier.
For good measure, I verified that it does in fact get set.

Seems to be an unlucky case of bad silicon that managed to slip through the usual testing.

In any case, I noticed Paul had been nice enough to leave in some code to dump the hard-fault over uart0 and decided to try that out to see exactly what the the CPU had to say about it.
Interestingly enough - since fault_isr() links at an earlier location than ResetHandler(), this pushed ResetHandler forward a few hundred bytes in the flash...
...and, voila - no more hard-faults!

Not too happy with this "solution", but if the processor shows no other signs of problems I'm gonna ship and hope for the best ;)
 
Dear defragster,

I'm not sure what more I could have said regarding known sketches?
NO sketch, including blink (which is very well known, AFAIK), will run - about 50% of the time when properly powered up via either VIN or USB.
IDE version (1.8.8 for the record) seemed beside the point, as no other Teensy have ever behaved this way - which clearly indicates a single unit problem.
The PCB it was assembled onto is very simple: the Teensy, an LDO, an RS-485 tranceiver and a few SPOX jacks - not much worth mentioning.
However, it has already been potted for harsh environments - hence the difficulty in replacing the Teensy.

I appreciate your feedback, however.

Luckily I got around the problem with a bit of a hack, and since this is far from safety critical I can live with shipping out a possibly slightly wonky processor as long as it keeps behaving from now on.

Best,
stg
 
Does it only affect that one unit - or all T_3.2's you've tried?

That is really odd and unexpected. There was an adjustment for T_3.6 power up in some recent TD version - not sure what version you have - and didn't watch it go in to see if it was #ifdef k66 or other.

I've been on the T4 beta thread - I made a debug lib that brings out faults for T_3.x's with sketch #include - then broke it trying to share it on T4 before USB worked like it does now - and left it at that given the T4 wasn't faulting for anyone.

Also confusing that with 3 posts on this thread you are now up to #5 - always tough to guess the 'audience' :)
 
We cross posted … see my note about post count … hard to get a baseline on who is typing … also what user sees that isn't apparent ...

… also as noted there was a recent TD 1.45 or so update to boot time setup though not sure it should have affected T_3.2 cores …

Seems to be T_3.6 specific change ...
Version 1.46
Support for Arduino 1.8.9
Improve startup behavior on Teensy 3.6 with external slowly rising power
 
no other Teensy have ever behaved this way - which clearly indicates a single unit problem.
The PCB it was assembled onto is very simple: the Teensy, an LDO, an RS-485 tranceiver and a few SPOX jacks - not much worth mentioning.
However, it has already been potted for harsh environments - hence the difficulty in replacing the Teensy.

So you don't have physical access to it?
Sounds like a soldering defect. Sometimes this can be assessed by keeping the board pushed down with the thumb: if it works better (boots more often), there it is.
 
This is very likely an *extremely* slow power supply start up issue.

The processor does indeed boot up, but at the following line (compiling with -O0 of course)...

mk20dx128.c 752: if (!(RTC_CR & RTC_CR_OSCE)) {

...the processor hard-faults (ending up in fault_isr).

The location is perfectly consistent, but only happens on cool boot and even then - not always.

First, check which version of Teensyduino you're using. Click Help > About in Arduino, or Arduino > About if using Macintosh.

The startup code was changed in 1.46 to allow more time for lower power rise. So if you have 1.45, upgrade and maybe everything will be magically fixed.

As far as I know, this problem has only been experienced by people using Teensy 3.6, which is able to boot up much faster starting with a lower power supply voltage. But it's theoretically possible to happen with Teensy 3.2 also, if you have a very abnormally slow voltage rise when your power turns on. To understand the problem, Teensy starts running as the voltage rises past about 1.7V. Since VBAT is powered through a schottky diode, it is approx 0.4V "behind" the main 3.3V power as the voltage rises. If your power supply voltage rises up *very* slowly, it's possible the RTC init could be attempted before VBAT has enough voltage for the RTC to operate, causing a fault condition.

Again, Teensyduino 1.46 restructured the startup code to do the RTC init much later. You didn't mention which version of the software you're using, which I guess is understandable since you're convinced this is a hardware malfunction. But my guess is you've probably got an older version with the RTC init fairly early. Check it and update to 1.46, which I'm guessing will solve this issue for you.

You might also use an oscilloscope to view how the 5V and 3.3V power start up. If your hardware has a relatively weak power supply and you've done something like adding a huge capacitor to try to compensate for the lack of power (a commonly suggested but sometimes shortsighted hack), that may be making matters worse in unexpected ways. When viewing the 3.3V rise, you want to make sure the rise is "monotonic". If you see anything like the voltage rising slowly, then suddenly taking a downward slope (likely around 1.5 to 1.8V) before it continues rising the rest of the way, that's a sure sign you have a terrible power supply problem that needs to be fixed. Even if you get Teensy to reliably boot up under those conditions, you should fixed the power supply startup so the rise is a steady ramp from 0 to 3.3V.
 
Looking at the latest code, the RTC oscillator is being started early on Teensy 3.2.

This code is at line 748:

Code:
#if defined(KINETISK) && !defined(__MK66FX1M0__)
        // If the RTC oscillator isn't enabled, get it started early.
        // But don't do this early on Teensy 3.6 - RTC_CR depends on 3.3V+VBAT
        // which may be ~0.4V "behind" 3.3V if the power ramps up slowly.
        if (!(RTC_CR & RTC_CR_OSCE)) {
                RTC_SR = 0;
                RTC_CR = RTC_CR_SC16P | RTC_CR_SC4P | RTC_CR_OSCE;
        }
#endif

and this is at line 1108:

Code:
#if defined(__MK66FX1M0__)
        // If the RTC oscillator isn't enabled, get it started.  For Teensy 3.6
        // we don't do this early.  See comment above about slow rising power.
        if (!(RTC_CR & RTC_CR_OSCE)) {
                RTC_SR = 0;
                RTC_CR = RTC_CR_SC16P | RTC_CR_SC4P | RTC_CR_OSCE;
        }
#endif

So if you have this version, maybe try altering those #if checks so the late start is also done with Teensy 3.2.

Also, I'd like to repeat the suggestion to measure the actual 5V and 3.3V power up waveforms. They might be ok and simply much slower than usual, but this might also be a sign of something much worse going on with your power supply startup. Even if a startup code edit "fixes" this problem, check those power up waveforms!
 
Post #6 showed this: - about 50% of the time when properly powered up via either VIN or USB.

So unless something is hurting USB power - maybe check current draw too
 
@XFer: No, not really, I mean I can access the USB via a short chassis mount extension cable and I can press the PROG button and punch probes through the silicon with enough accuracy to get at the breakout pins, but not really any SMT pins reliably.

@Paul: You're right that I am using an older version of Teensyduino, namely 1.42.

Checking the power rails was one of the first thing I did. 3.3V is a clean ~1.6ms ramp. See https://imgur.com/a/w1cbvEC for details.
I wouldn't say this is "extremely" slow, but I suppose that depends on your definition - it's certainly slower than the ~0.1ms when powering a solitary Teensy 3.2 from a 2meter USB cable.

The reason I'm convinced it is a hardware issue is that I've designed PCBs for about 40 prototypes and products using Teensy 3.2 and gone through well over 200 Teensy 3.2's over the last 6 years and this problem is specific to this one single unit. I've always developed on the same computer and powered them from USB, or a lab supply, depending on application. Cables have changed on occasion and the IDE has been updated only a select few times, when absolutely necessary. I've had one boot up issue before, but it was determined that the device may have earlier been accidentally powered up with 5V on the 3V3 rail which very likely damaged the processor.

This particular hardware setup is extremely simple, and current consumption is only 39mA (Teensy+idle LTC2862A) for the whole board.
I am using a 2A 5V switched regulator, backed by a 3A lab power supply at 12V - over 40cm/~16" 1.5mm²/~AWG15 wires, identical to many other boards I've constructed.
Capacitance beyond what's on the Teensy is a 100nF decoupling cap for the tranciever, 30µF bulk caps in the regulator design and, of course, some pico-range stray capacitance.

But, you are right - a later version of Teensyduino solves the problem. However, so does manually moving the RTC start-up code to another location, later or *earlier*, actually.
Note that the fault always occurs when *reading* RTC_CR, not when setting RTC_CR_OSCE.
I've also determined it is possible to get a fault by reading *other* registers (but not all) than RTC_CR, only when the read is located within a particular range in flash.
At least, it certainly *seems* more related to location than time - as moving ResetHandler() in flash should have no effect on time of execution, but does resolve the hard-fault.

I suppose I could modify the link file to lock ResetHandler() to a specific location and get that fault dump, if it's of interest. But as it stands now I feel like I've spent more time than I'd like on this and the processor has been behaving well for the last 50 or so power cycles - ever since relocating ResetHandler() by enabling fault dumps. I don't mind keeping the core like this for future development, as it may be useful on occasion, with the only penalty being a few hundred bytes of extra flash use.

I still lean towards this being a silicon defect, but if you want that dump or any other further data, ask and I'll try to get it for you - otherwise, enough of me taking up peoples time.

Have a wonderful day,
Davey
 
Can you separate it from the hardware?

Sorry, no - this is why I started the thread in the first place; it is already assembled into a product and potted for harsh environments.
Had it been easy, I would have instead replaced it as soon as I realized there was an issue.

I'm shipping later this week, however if the product comes back due to further problems - I will keep in mind that you want it.
I'd have to sent it to you including pcb and potting though. USB and select pins are accessible, anything else and you'd have to dig it out silicone.

Best,
Davey
 
Status
Not open for further replies.
Back
Top