New Teensy MicroMods defective

Ouch...

So far, I have not run into that. For example, I put a new MMOD in the breakout board that Paul made.
View attachment 29099

And I used the test program that @defragster and myself hacked on:
Code:
#ifdef ESP_PLATFORM
#define digitalWriteFast digitalWrite
#define digitalReadFast digitalRead
#endif

void setup()
{
  Serial.begin(115200);
  while (!Serial && millis() < 4000 );
  Serial.println("Compile Time:: " __FILE__ " " __DATE__ " " __TIME__);
  Serial.printf("Num Digital Pins: %d\n", NUM_DIGITAL_PINS);

  testForShorts();
  
}

uint32_t cnt = 0;
void loop() {
  cnt++;
    allPinTest( cnt );
}

uint32_t pinLast[NUM_DIGITAL_PINS];
void allPinTest( uint32_t cnt ) {
  uint32_t ii, SET;
  Serial.print("PULLDOWN Start Vals:\n  ");
  SET = 0;
  Serial.print("PULLDOWN :: TEST to 3.3V\n  ");
  for ( ii = 0; ii < NUM_DIGITAL_PINS; ii++) {
    pinMode( ii, INPUT_PULLDOWN );
    delayMicroseconds( 5 );
    pinLast[ii] = digitalReadFast( ii );
    if (pinLast[ii]) {
      Serial.print("\nd#=");
      Serial.print( ii );
      Serial.print( " val=" );
    }
    Serial.print( pinLast[ii] );
    Serial.print(',');
  }
  Serial.println();
  Serial.println();
  while ( 1 ) {
    uint32_t jj, dd = 0, cc = 0, ee=4;
    cc = 0;
    for ( ii = 0; ii < NUM_DIGITAL_PINS; ii++) {
      jj = digitalReadFast( ii );
      if ( jj != pinLast[ii] ) {
        dd = 1;
        cc++;
        pinLast[ii] = jj;
        Serial.print("d#=");
        Serial.print( ii );
        if ( pinLast[ii] ) Serial.print( "\t" );
        Serial.print( " val=" );
        Serial.print( pinLast[ii] );
        Serial.print(',');
      }
      if ( cc > 1 && ee ) {
        Serial.println(">>> MULTI CHANGE !!");
        ee--;
      }
      if ( Serial.available() ) {
        while ( Serial.available() ) Serial.read();
        if ( 0 == SET ) {
          SET = 1;
          Serial.print("PULLUP :: TEST TO GND\n  ");
        }
        else {
          SET = 0;
          Serial.print("PULLDOWN :: TEST to 3.3V\n  ");
        }
        for ( ii = 0; ii < NUM_DIGITAL_PINS; ii++) {
          if ( 0 == SET )
            pinMode( ii, INPUT_PULLDOWN );
          else
            pinMode( ii, INPUT_PULLUP );
          delayMicroseconds( 20 );
          pinLast[ii] = digitalReadFast( ii );
          if (SET != pinLast[ii]) {
            Serial.print("d#=");
            Serial.print( ii );
            Serial.print( " val=" );
            Serial.println( pinLast[ii] );
          }
        }
      }
    }
    if ( dd ) {
      dd = 0;
      Serial.println();
      delay( 50 );
    }
  }
}

void testForShorts() {
  uint32_t ii;
  Serial.print("Quick Test for Shorts to adjacent pin");
  Serial.println("First pull pins down and see if the next one follows");
  for ( ii = 0; ii < NUM_DIGITAL_PINS-1; ii++) {
    pinMode( ii+1, INPUT_PULLDOWN );
    pinMode( ii, OUTPUT);
    digitalWrite(ii, HIGH);
    delayMicroseconds( 5 );
    if (digitalRead(ii+1)) {
      Serial.printf("%d:%d ", ii, ii+1);
    }
  }
  Serial.println("\n Now try Pull up and see if setting low follow");
  for ( ii = 0; ii < NUM_DIGITAL_PINS-1; ii++) {
    pinMode( ii+1, INPUT_PULLUP );
    pinMode( ii, OUTPUT);
    digitalWrite(ii, LOW);
    delayMicroseconds( 5 );
    if (!digitalRead(ii+1)) {
      Serial.printf("%d:%d ", ii, ii+1);
    }
  }
  Serial.println();  
}

And I rang out all of the IO pins on it and it, all were found.

Note the green board is my own breakout, that I had partially assembled. Earlier I rang out all of the pins on that one.

Nice info, thanks KurtE. I'm gonna try that test program!
 
It would be interesting to x-ray the bad boards to see if it is a BGA issue or socket issue.

I use the MM in a couple of projects. I haven't tested every pin, but I did find they can be finnicky with the M.2 connector.

To me it feels like there is too much wiggle room in the mating and the M2.5 screw. The 22mm length may be the issue. Or the width at the connection end. Being careful when installing is usually sufficient.

I looked at what I could to see if they did something wrong and possibly improve on my carrier board design. The PCB hole size Spark Fun uses for the standoff is correct to the manufacturer's spec, but I still find it a bit tool loose for my taste. Same with the TE connector, the Eagle footprint design checks out.

I tried looking into the M.2 spec, but it's 2400 euros I didn't want to spend. https://www.normadoc.com/english/pci-express-m-2-specification-revision-3-0-version-1-2.html

One can usually find 22mm wide M.2 components that are as short as 30mm. The spec seems to allow for narrow (1216) and short boards (2226), but they are solder down only, not socket.

Most M.2 have a center standoff or 2 edge standoff . The single offset standoff on the Sparkfun I've never seen elsewhere. Potential for board twist? Maybe, but likely minimal.

Do I trust it? I've had some prototypes in intermittent use, and they are holding up. I never remove the MM once it's been seated and tested.

I'd feel much better if there was a 2230 version (why deviate from a working standard), with centered standoff (or dual standoff) that was code lockable.
 
We have several test boards, some assembled in house, some at JLC as well as the Sparkfun ATP - no matter which board it’s in - get the same issues.
On the ATP every now and then it wont establish USB communication and I’ll get a 2/4/7 blink on the boot chip. Reseating the MM will fix this in most cases.

I’m happy to hear that most of you are getting good results on the test, but I still believe there is mot enough QA being done on SparkFuns end before shipping out the units.

@Paul is there anyone there you can connect us to? We want to make a 50-100 unit order and need to be sure we get 100% working units.
 
I'm in the same boat with the MicroMods. I ordered 4 test boards and an ATP back in October and have been using them since. I completed a MicroMod version of a product I've been developing on the Teensy 4.0 for some time. Over time each of the four MicroMods developed persistent boot code errors (2/7 mostly) and eventually all four got to the point that they would not boot, connect, load code or self-reload the blink sketch.

With no operating boards, I set out to reflow one. For each board I first reflowed the bootloader chip with CQ4LF-0.5 flux at 260c air temperature, cooled and tested. I then reflowed the flash chip, cooled and tested. I then reflowed the other components, cooled and tested. Lastly I reflowed the CPU, cooled and tested. After reflow of the CPU, the three boards I've tried so far booted code. All three now boot properly, run code, communicate through USB and self load the blink program on my boards and the ATP. Prior to reflow, all three processors gave NXP ROM in open mode errors on USB connect (the fourth won't connect).

Code:
07:59:50.074 (loader): nxp_write: success
07:59:50.089 (loader): HAB open mode, bootcfg=80018
07:59:50.092 (loader): Opps, NXP ROM in open mode, but we do not yet have code for this case :(
07:59:50.095 (loader): start ignoring usb:0/140000/0/3/2
07:59:50.098 (loader): end operation, total time = 0.144 seconds

My use case depends on the availability of USB, so the MMod allowing main-board mounted USB is a huge selling point for me. I've had prototypes of the Teensy version of the device running in cars for more than a year with zero issues, but 4/4 failed MMODS on the bench are leaving me...concerned.
 
@Paul is there anyone there you can connect us to?

Unfortunately, no. The addresses I have are (probably) meant to be private.

I sent an email around the time I wrote msg #4. So far, no reply. But it's only been 2 full business days. Remind me later this week and I'll ping Sparkfun again if they haven't replied.
 
I'm in the same boat with the MicroMods. I ordered 4 test boards and an ATP back in October and have been using them since. I completed a MicroMod version of a product I've been developing on the Teensy 4.0 for some time. Over time each of the four MicroMods developed persistent boot code errors (2/7 mostly) and eventually all four got to the point that they would not boot, connect, load code or self-reload the blink sketch.

With no operating boards, I set out to reflow one. For each board I first reflowed the bootloader chip with CQ4LF-0.5 flux at 260c air temperature, cooled and tested. I then reflowed the flash chip, cooled and tested. I then reflowed the other components, cooled and tested. Lastly I reflowed the CPU, cooled and tested. After reflow of the CPU, the three boards I've tried so far booted code. All three now boot properly, run code, communicate through USB and self load the blink program on my boards and the ATP. Prior to reflow, all three processors gave NXP ROM in open mode errors on USB connect (the fourth won't connect).

Code:
07:59:50.074 (loader): nxp_write: success
07:59:50.089 (loader): HAB open mode, bootcfg=80018
07:59:50.092 (loader): Opps, NXP ROM in open mode, but we do not yet have code for this case :(
07:59:50.095 (loader): start ignoring usb:0/140000/0/3/2
07:59:50.098 (loader): end operation, total time = 0.144 seconds

My use case depends on the availability of USB, so the MMod allowing main-board mounted USB is a huge selling point for me. I've had prototypes of the Teensy version of the device running in cars for more than a year with zero issues, but 4/4 failed MMODS on the bench are leaving me...concerned.

Well, I'm happy to hear that we're not alone, but sad to hear that this issue is not isolated to us.
Would you be able to run the the test sketch I posted on each of those working boards to see if you have full pin connectivity after reflowing them?
Just reminding that you need to ground all the GPIOs to run this test.

Unfortunately, no. The addresses I have are (probably) meant to be private.

I sent an email around the time I wrote msg #4. So far, no reply. But it's only been 2 full business days. Remind me later this week and I'll ping Sparkfun again if they haven't replied.
Fully understand the privacy. I've also dropped them an email and waiting for their response. I'll bounce the thread end of the week to check in with you.
 
Well, I'm happy to hear that we're not alone, but sad to hear that this issue is not isolated to us.
Would you be able to run the the test sketch I posted on each of those working boards to see if you have full pin connectivity after reflowing them?
Just reminding that you need to ground all the GPIOs to run this test.

Sure thing. I'll run them through and see what I get.
 
Quick followup. I heard from Sparkfun about a week ago. Just now catching up to everything from last couple week while we've worked to catch up to Teensy 4.0 backlog.

They are reviewing and updating their test procedures to make sure the BGA chip isn't touched while testing the boards. They're going to retest every MicroMod Teensy they have in stock. Sounds like they're taking this seriously.


@Paul is there anyone there you can connect us to? We want to make a 50-100 unit order and need to be sure we get 100% working units.

I'll pass this on to Sparkfun. Normally we never share your contact info, but in this case I'm assuming you want them to contact you directly.
 
Wow. I'm very glad I came across this. I've been going crazy trying to recode, design, and re-order prototype PCBs. There was just always something that wasn't working properly. This explains it.

Both of my boards have some defective pins (one more than the other).


Board #1
Code:
Pins set to input
Pin 0 state: 0 
Pin 1 state: 0 
[COLOR="#FF0000"]Pin 2 state: 1 [/COLOR]
Pin 3 state: 0 
Pin 4 state: 0 
Pin 5 state: 0 
Pin 6 state: 0 
Pin 7 state: 0 
Pin 8 state: 0 
Pin 9 state: 0 
Pin 10 state: 0 
Pin 11 state: 0 
Pin 12 state: 0 
Pin 13 state: 0 
Pin 14 state: 0 
[COLOR="#FF0000"]Pin 15 state: 1 [/COLOR]
Pin 16 state: 0 
Pin 17 state: 0 
Pin 18 state: 0 
Pin 19 state: 0 
Pin 20 state: 0 
Pin 21 state: 0 
Pin 22 state: 1 
[COLOR="#FF0000"]Pin 23 state: 1 [/COLOR]
Pin 24 state: 0 
[COLOR="#FF0000"]Pin 25 state: 1[/COLOR] 
Pin 26 state: 0 
Pin 27 state: 0 
Pin 28 state: 1 
[COLOR="#FF0000"]Pin 29 state: 1 [/COLOR]
Pin 30 state: 0 
[COLOR="#FF0000"]Pin 31 state: 1 [/COLOR]
Pin 32 state: 0 
Pin 33 state: 0 
Pin 34 state: 0 
[COLOR="#FF0000"]Pin 35 state: 1 [/COLOR]
Pin 36 state: 0 
Pin 37 state: 0 
Pin 38 state: 0 
Pin 39 state: 0 
Pin 40 state: 0 
Pin 41 state: 0 
[COLOR="#FF0000"]Pin 42 state: 1 [/COLOR]
Pin 43 state: 0 
Pin 44 state: 0 
Pin 45 state: 0 
Test Completed


Board #2
Code:
Pins set to input
Pin 0 state: 0 
Pin 1 state: 0 
Pin 2 state: 0 
Pin 3 state: 0 
Pin 4 state: 0 
Pin 5 state: 0 
Pin 6 state: 0 
Pin 7 state: 0 
Pin 8 state: 0 
Pin 9 state: 0 
Pin 10 state: 0 
Pin 11 state: 0 
Pin 12 state: 0 
Pin 13 state: 0 
Pin 14 state: 0 
Pin 15 state: 0 
Pin 16 state: 0 
Pin 17 state: 0 
Pin 18 state: 0 
Pin 19 state: 0 
Pin 20 state: 0 
Pin 21 state: 0 
Pin 22 state: 1 
Pin 23 state: 0 
Pin 24 state: 0 
[COLOR="#FF0000"]Pin 25 state: 1 [/COLOR]
Pin 26 state: 0 
Pin 27 state: 0 
Pin 28 state: 1 
Pin 29 state: 0 
Pin 30 state: 0 
Pin 31 state: 0 
Pin 32 state: 0 
Pin 33 state: 0 
Pin 34 state: 0 
Pin 35 state: 0 
Pin 36 state: 0 
Pin 37 state: 0 
Pin 38 state: 0 
Pin 39 state: 0 
Pin 40 state: 0 
Pin 41 state: 0 
Pin 42 state: 0 
Pin 43 state: 0 
Pin 44 state: 0 
Pin 45 state: 0 
Test Completed


Really hoping that this actually gets resolved. 2/2 that I ordered had issues. One from Sparkfun, and one from Digikey.
 
Just tested a batch of MM bought from Digi-Key in Feb, of the 15, 6 either don't boot or have pins not connected.
Thanks for the scripts and detective work.
Any advice on getting them replaced - direct to Sparkfun or through Digital-key?

Cheers, Paul
 
@Rezo - apols, a bit off topic but you mention above that you got some boards assembled at JLC, I use JLC for boards but would like to try their smd assembly.
Can I ask - were you able to find the M2 connector in JLC's part lib or did you supply it yourself? Cheers, Paul
 
@Rezo - apols, a bit off topic but you mention above that you got some boards assembled at JLC, I use JLC for boards but would like to try their smd assembly.
Can I ask - were you able to find the M2 connector in JLC's part lib or did you supply it yourself? Cheers, Paul
I am not sure if it helps. I had a few boards (5) assembled in Jan/Feb time frame with PCBWay.
Screenshot.jpg

I only had them do the SMD stuff. Note with most of the components they ordered from places like Digikey.

When it came to the MMOD connectors: I gave them what I thought were some part numbers as well as the Sparkfun package of I think both 5 of the main connectors as well as 5 of the stand offs.

I don't remember which way they went.
 
Sorry to resurrect this. I have a TeensyMM that had problems, I fixed it by blasting the top with a hot air gun which fixed it for a few months. It's failed again and blinks nine times over and over, and no USB recognition.Is the blinking a particular code? Thanks.
 
9 Blinks = ARM JTAG DAP Init Error
The ARM JTAG DAP was detected (4 blinks) but could not be initialized. This error is rather unlikely!


Oh dear! Is this a problem with communication to the IMXRT IC or bootloader chip?

The TeensyMM is a BGA chip on a comparatively small, thin PCB. Are the failures mechanical issues caused by the bending of the PCB by the connector and screw? Mine failed after being left on for a couple of hours after months of working fine. Sparkfun aren't testing properly, obviously. I'll try heating the actual BGA, my other efforts before, were on the bootloader chip mainly.

This is a real shame, I have another synth project/product that uses these because they're easier for end users, but the failure rate is pathetic.
 
I believe the small pcb footprint and it’s thin form makes it too flexible if not handled with care, and this can cause the (probably poor) solder points on the BGA to come loose.
Try pushing on the corner of the NXP chip thats closer to the bootloader and then power it up and see if it makes a difference.
We’ve started to work on developing our product with the NXP and parts integrated into our main pcb, hopefully avoiding any of these issues in the future.
 
Board flex is a known problem with BGA parts, maybe SparkFun could consider a thicker board design? I am unfamiliar with the physical specs on the MicroMod form factor.

My work project had several failures of a 144-ball BGA mounted K60 part (similar to the K66 in the Teensy 3.6) in a very harsh physical environment: several hundred G parallel to the board, but inducing significant movement normal to the board. We solved it by adding 2 mount points close to the BGA and potting between the board and the enclosure. Before this fix boards died within months of installation. After the fix, there have been no BGA failures in about 3 1/2 years.

We conclude that preventing flex was the solution.
 
One minute with my cheap SMD hot air gun at 320C with some flux has revived it to fight another day (producing cool synth sounds hopefully...) I concentrated it on the MCU and masked the rest of the board with foil. The pin test program in this thread also shows all pins working.

Yes it's board flex isn't it. I expect the M.2 specifications state the exact PCB thickness to mate with the connector, so a thicker board probably isn't an option. Not having read these specs, I wonder what it says about chip size and type? The IMXRT IC as a BGA package (is there any other?) is too large for this size of M.2 board - again my non-expert opinion. M.2 comes in a range of lengths I think.
 
Aren't the chips on most M.2 SSDs BGA package?

Could very well be board flex. Just wondering how normal M.2 SSDs manage this?
 
Longer boards? Larger Radius of bend?
EDIT: Also reduced bending force due to longer boards.
May or may not be significant but I see that there is a TWIST applied to the board by the use of an offset screw mounting.
 
Last edited:
The M.2 connector definitely does push the board to bow upwards, causing potential flexing especially when the board is screwed to the boss. I can see how this would stress the outer BGA pads on the RT1062.

I have a Kioxia/Toshiba M.2 NVMe 128G drive in 2230 form factor (used in my Mikrotik RBM33G router board), and it has a much, much stiffer PCB.

There might be enough room on the sides to add structural PCB supports (non-plated PCB, edge-wise, perpendicular to the MicroMod), like side support ribs. They need a very precise slot for the MM to fit into and to stop the flexing. There are a few components on the edges of MM that need clearance milled to the ribs, making it quite delicate.. Or perhaps CNC-milled side support ribs from say ⌀5mm nylon rod, milled with a 1mm end mill to 2-3mm deep, and just pushed onto the sides of MM?

Another option is to have a couple of ⌀2.5-3.0mm holes just outside the MM footprint, nylon studs, and a counter-spring bar (or heatsink) to counteract the flexing/bowing. The studs could even have a small ledge for the bottom of MM to ensure the bar/heatsink cannot flex the MM the other way.

Interesting, but also an annoying, problem.
 
Aren't the chips on most M.2 SSDs BGA package?

Could very well be board flex. Just wondering how normal M.2 SSDs manage this?

Structural label? Different PCB material? More layers? Different solder alloy? Tighter reflow tolerance? End to end soldered chips most of the board width - an SSD Board feels more stout than the EPS MM I picked up.

But it is indeed the same thickness and connector width, and all chips are BGA.
 
Just tested a batch of MM bought from Digi-Key in Feb, of the 15, 6 either don't boot or have pins not connected.
Thanks for the scripts and detective work.
Any advice on getting them replaced - direct to Sparkfun or through Digital-key?

Cheers, Paul

Replying to my own post here, Digikey got in touch today - their product specialist has told me to scrap the six that are defective, they don't need them returned and have offered to ship 6 replacements at no charge or credit onto credit card.

I'm going to get them replaced and see if they are any different/better.

I hope Sparkfun sticks with it and finds a fix the quality issues, the MM format is really attractive for me.
 
Back
Top