Teensy 3.6 program loader and I/O glitches

Status
Not open for further replies.

bicycleguy

Well-known member
This problem started similar to posts about the program loader not responding such as https://forum.pjrc.com/threads/40313-Power-restart-on-Teensy-3-6 and https://forum.pjrc.com/threads/37531-T_3-6-acting-oddly-Program-Button-doing-a-reset-no-bootloader-reprogram but has gotten worse. After much hassle with the program I was trying to debug I wrote this program to narrow the issue.

It loops around saving the I/O values and if any changed since the last loop prints out all the pins monitored:
Code:
#define MYVERSION_STUFF "Teensy 3.6 I/O Test 12/3/2016"

void setup() {
  Serial.begin(9600);
  while( !Serial )
    ;
  Serial.println(MYVERSION_STUFF);
  Serial.println("#\tTime\tPins high");
  for(int i=0;i<40;i++){
    pinMode(i, INPUT_PULLUP);
  }
}

void loop() {
  static int lastvalue[40];
  static int count;
  boolean prnt=false;
  int temp;
  
  for(int i=0;i<40;i++){
    temp=digitalRead(i);
    if(temp!=lastvalue[i]){
      lastvalue[i]=temp;
      prnt=true;
    }
  }
  if(prnt){
    Serial.print(count++);Serial.print("\t");
    Serial.print(millis());Serial.print("\t");
    for(int i=0;i<40;i++){
      lastvalue[i]==0 ? Serial.print("_"):Serial.print(i%10);
    }
    Serial.println();
  }
}

All the I/O pins are connected to ground like this:

P1030447.jpeg

The results I get when holding by the white proto board ends and mildly torquing back and forth look like this:
Code:
Teensy 3.6 I/O Test 12/3/2016
#	Time	Pins high(modulo 10)
0	3801	___________________________7____________
1	3833	________________________________________
2	3889	___________________________7____________
3	4043	___________________________7___________9
4	4043	___________________________7____________
5	4050	___________________________7___________9
6	4123	___________________________7____________
7	4141	___________________________7___________9
8	4281	__________________________67___________9
9	4283	___________________________7___________9
10	4284	__________________________67___________9
11	4284	___________________________7___________9
12	4298	__________________________67___________9
13	4298	___________________________7___________9
14	4300	__________________________67___________9
15	4316	___________________________7___________9
16	4336	___________________________7____________
17	4343	___________________________7___________9
18	4344	__________________________67___________9
19	4344	___________________________7___________9
20	4366	___________________________7____________
21	4372	________________________________________
22	4375	___________________________7____________
Pins 26, 27 and 39 occasionally go high and sometimes stay high. I have verified with an oscilloscope that the top pads on the pcb on these pins never glitch so the problem is in
the board or the chip connections, not the proto board connection. Also twisting in the direction of continuity allows the boot loader to run correctly, otherwise I get a reset when pressing either the Arduino upload button or the switch on the board.

No twisting is required if started from cold and waiting.

Looking at the I/O locations on the chip the affected pins are:
IO.png
The red ones are for sure, the yellow, RESET_b, I''m speculating on since I don't understand the interaction of the MKL chip with the pin.

Hope this technique will help anyone else having problems.
 
That test and setup looks valid as I read and see it - and perhaps relevant as well.

The Proto pic is low res but seems GND is in the loop on both sides and across only to Digital pins 0-40. As the MCU is oriented - which corner is that with RESET_b L_12?

The only point I'm not sure how to read and see you measured is the second part of this about "top pads ... never glitch"?:
Pins 26, 27 and 39 occasionally go high and sometimes stay high. I have verified with an oscilloscope that the top pads on the pcb on these pins never glitch ...
> Reading that again now I see it as saying that the oscilloscope on the through pins/pads shows that they never leave GND level even though internally the MCU is reading them as HIGH?

Question: in your testing does the Teensy ever RESET?

In line with the linked problem I saw - what does the oscilloscope on the Reset and Program pins show:
>> during this test (probably no change?)
>> when the Program Button is pushed?

Forum Posts indicated the RESET pin can be repurposed. That post was some weeks back - and I'm not recalling the way and extent - but I'm wondering if the sketch could be updated to monitor that pin for change while pressing the program button? Probably not, as it appears the Program button is only 'not seen' because the MCU is getting a reset which is immediately taking the bootloader MCU back offline. In any case if the K66 does a reset or goes to Program mode the sketch will stop reporting. I suppose an oscilloscope wired to the correct pin of the Bootloader MCU could detect this transition.

If the DAC0 pin had the low 12 bits of millis() written to it a scope could watch that - perhaps the low 12 bits of microseconds() to DAC1 as well. That would show the health of those pins (L-3 & L-4 ?) and also indicate when the MCU does a reset and restart when the Button is pushed - though USB shows that.
 
As the MCU is oriented - which corner is that with RESET_b L_12?
I think that would be the corner nearest board pin 24
> Reading that again now I see it as saying that the oscilloscope on the through pins/pads shows that they never leave GND level even though internally the MCU is reading them as HIGH?
correct
Question: in your testing does the Teensy ever RESET?
no, it requires the addition of either the boards reset switch or the Arduino download button.
In line with the linked problem I saw - what does the oscilloscope on the Reset and Program pins show:
>> during this test (probably no change?)
>> when the Program Button is pushed?
I didn't look at those pins, they would require additional soldering in this setup
Forum Posts indicated the RESET pin can be repurposed. That post was some weeks back - and I'm not recalling the way and extent - but I'm wondering if the sketch could be updated to monitor that pin for change while pressing the program button? Probably not, as it appears the Program button is only 'not seen' because the MCU is getting a reset which is immediately taking the bootloader MCU back offline. In any case if the K66 does a reset or goes to Program mode the sketch will stop reporting. I suppose an oscilloscope wired to the correct pin of the Bootloader MCU could detect this transition.

If the DAC0 pin had the low 12 bits of millis() written to it a scope could watch that - perhaps the low 12 bits of microseconds() to DAC1 as well. That would show the health of those pins (L-3 & L-4 ?) and also indicate when the MCU does a reset and restart when the Button is pushed - though USB shows that.
Sorry, my time spent on this board is over. I had hoped this might be some other issue but its looking more like bad BGA solder joints. My experience with intermittent BGA pins, especially with large arrays is that any failure is a sure sign of more to come. Whatever caused these formerly connected solder joints to become intermittent likely affected adjacent pins and its only a matter of time. BGA pins on large packages are highly stressed and even made perfect will fail. Thats why they sell a leaded version of this part. Also why you don't see free warranties for more than a year on your iPhone. Hopefully I can get PJRC to replace this early failure and even more hopefully this isn't an omen of things to come because I love the Teensy.
 
Paul isn't seeing a lot of these reported (though perhaps there was at least one T_3.5 and two T_3.6's)- I'm wondering if a panel just hit the oven wrong . . . time will tell. Your unit was soldered to pins - my unit was not - so user solder heat not a factor in mine at least.

Did you only get a single T-3.6 to work with? Would be nice if you had another on hand the same setup worked it. I got one Beta KS unit in T_3.5 and T_3.6 that I soldered and plugged more than a few times with no problems these past months - though with more rigid female header strip. Then from my KS set of 6 only one showed as ODD new out of wrapper. I'm playing with at least one bare unsoldered board regularly at this desk - and maybe another downstairs that seems fine in handling and just bare wires in the pin holes. I've hit Button on this one more than a bit.
 
As a typical engineer I can't leave well enough alone. I noticed that the I/O glitches are very sensitive to temperature changes. It would be useful to run a program on the chip that gets it as hot as possible while running the test. You wouldn't have to even touch the setup to get results. I don't want to use loading the I/O for the test.

What could I run during the loops that would heat up the chip the most?
 
Just looking back at this thread I kind of left it in limbo. While attempting to update the program above to include software self heating and temperature reporting the problem became so bad that I can't seem to program the part anymore. RIP long skinny Teensy.
 
Ok, I couldn't leave it alone. Put the Teensy in the freezer and lo and behold it sprang to life at some temperatures while warming up. So I added some code to read the internal temp and some floating point loops to heat it up. Then with the program running hit the corner with a hot air gun at 410C with the smallest nozzle while watching the output. At about 125C all the I/O connected. At about 135C the terminal com quit. Kept heating for another 20 seconds or so. The insulators and my proto board melted slightly, the pwb measled slightly.

After cooling down a little, re-plugged in and success! The I/O appear good and the Teensy can be programmed without pushing on the top. Temperature cycled it 0 to 80F a few times without problems. Flexing the proto board has no effect. Unfortunately in my excitement I closed the terminal window before saving the output. It was kind of neat. You could see the temp rising and various pins becoming intermittent and then solid, then comm loss.

Here is some output from one of the freezer tests. After 15 minutes of running I pulled a ground wire to make sure it was running and check the temp. Then plugged back in.
Code:
Teensy 3.6 I/O Test 6/6/2017

Start temperature= 5.1°F
#	Time ms	Temp°F	I/O grounded? _ =good, pin%10 =bad
1	923272	81.8	_________________________________3456789
2	926441	82.0	________________________________________

The reliability of this fix is questionable but it was a fun test and now I have a 3.6 to play with again. FYI the speed test incorporates https://forum.pjrc.com/threads/25755-Teensy-3-1-overclock-to-168MHz?p=48695&viewfull=1#post48695 and the temperature library is https://forum.pjrc.com/threads/43297-New-Internal-Temperature-Library?p=139979&viewfull=1#post139979 and the max temp running the float test continually (without printing) is about 30°F above ambient.
 
I have been having problems with my T3.5 and T3.6 for a while now, I have to bend the pcb to get it to program and keep it somewhat bent for it to keep running. These Teensy's do not have any pins soldered to them yet. I wonder if there is some way to see what batch these come from. I don't know if paul cares or not but there is definitely something up with the ones I have which is similar to what other people reported. In their current state they are almost unusable. I'm going to solder pins on one of them and see if that will help.
 
Myself and only a couple others have posted on this since the KS release. So one bad panel could explain it - or maybe an end/corner on a couple panels.

Duff: You have multiple showing issues? I only had one odd in my 6 pack - it worked except the button acted more like reset if it did anything.
RE "somewhat bent " - bent to flat or otherwise? One poster put pencils under the ends to remove a bow to see it work. Not sure if I ever typed this - but on ones like yours I was wondering if the POGO pin press assured it was FLAT and passed factory tests?
 
duff,
Although I managed to get mine to work I'm sure the fix is only temporary. As a former electronic packaging engineer in the aerospace industry with much leadless package design, analysis and testing experience I can guarantee problems with this design, even if perfectly soldered. In the typical test we used to run -55 to +125C with about 45 minute cycle time, this kind of pga package wouldn't last 300 cycles. Thats why we didn't allow pga packages above a certain size. Of course in a less drastic thermal environment you will do better. However, mechanically with the pwb so flimsy and the package-bga-pwb interface so stiff, a huge stress concentration occurs at the corner solder joints when you inadvertently bend the board. I would solder a female header on to stiffen up the board and then be very carefull.
Another thing to point out is that my test only looked at 40 of the 144 pins. What about the others?
 
I don't know about BGA failure rates or anything like that, all my T3.5/6's are from either the Kickstarter campaign or shortly after that. One thing I haven't tried is to power them from my power supply and see if they work without bending?
 
duff,
Although I managed to get mine to work I'm sure the fix is only temporary....

Thought I'd update this. So far the resoldered board is still working! It hasn't seen many on/off cycles, maybe 200 max.

Its becoming a SDR radio next so it should see some use.
 
Status
Not open for further replies.
Back
Top