New Teensy MicroMods defective

[Dogbone06]

Replying to my own post here, Digikey got in touch today - their product specialist has told me to scrap the six that are defective, they don't need them returned and have offered to ship 6 replacements at no charge or credit onto credit card.

I'm going to get them replaced and see if they are any different/better.

I hope Sparkfun sticks with it and finds a fix the quality issues, the MM format is really attractive for me.

I know for a fact that Sparkfun now has a much better quality control. I recently ordered 250 Micromods and only 8 were defective, I test all Micromods in a custom PCB that checks all the pins. Those 8 were sent back so they can check them and ofc refund. My order before that was 50pcs and on that order only 1 was bad.

John at Sparkfun support has been an absolute legend with handling this. I’ve worked with them on this issue by providing feedback and they’ve actually implemented another step in their QC for the Micromods.

 

[Mcu32]

I know for a fact that Sparkfun now has a much better quality control. I recently ordered 250 Micromods and only 8 were defective, I test all Micromods in a custom PCB that checks all the pins. Those 8 were sent back so they can check them and ofc refund. My order before that was 50pcs and on that order only 1 was bad.

John at Sparkfun support has been an absolute legend with handling this. I’ve worked with them on this issue by providing feedback and they’ve actually implemented another step in their QC for the Micromods.

If they had *working* quality control, not a single unit would be defective.

 

[Dogbone06]

If they had *working* quality control, not a single unit would be defective.

Sure but this is a Sparkfun Dev product, bare that in mind. Atleast their listening to us, the customers. I know of many companies that doesn’t care at all. And like I mentioned, their support has been great in this case.

 

[UHF]

Micromods passing QC can still fail in service. I have two. One failed on my own PCB and then gradually failed on the ATP board - presumably small differences in the PCB bending. The other failed recently after being turned on and off daily for a year - again the bending of the PCB with perhaps thermal cycling affecting it. Is the problem actually just soldering? It would interesting to see if the same thing happens to these: https://www.sparkfun.com/products/18030

 

[Nominal Animal]

I just have to say, this shows how good a product PJRC puts out. I've been a user since 2.0++ came out, and I've always been happy with every single Teensy I've bought, and had zero quality problems with them.

While I'm no EE, it seems obvious to me that the problem with MicroMods is twofold: the major one is the spring force induced flexing and thermal cycling, and the minor one is/was a bug in their production/QC (applying pressure to the RT1062 chip while testing, which masked some problems). For longevity, counteracting the flexing by some kind of supports seems necessary, until they can switch to a more rigid PCB material.

In the long term, more rigid PCB material for MicroMods is absolutely required. To me, the material used in MicroMods seems especially flimsy and transparent in comparison; perhaps it indicates lower glass fiber and higher epoxy content, explaining why it feels so much less rigid than the Kioxia M.2 2230 NVMe drive I have? (Closest comparable one I have: the same connector, just ~8mm longer, all-BGA too.)

 

[Rezo]

The form factor of the Micromod is more preferable to some of us, as Dongbone and myself use it in a product. But the reliability is an issue, especially in harder environments.

On the other, the T4 and T4.1 I have here at home have been working with zero issues since they were both released by PJRC. They fall on the floor, they get chucked in boxes, they get sat on by cats. I even have one running in a car for the past 3-4 years daily and it's still alive and kicking.

I just with the MM was as reliable..

 

[UHF]

Other than the issues, I think perhaps it's worth noting that these - Teensys, Micromods, Arduinos etc. are intended as development boards and some of us, myself included, are exploiting them to use in products. It is very convenient to use a Teensy or get an end user to to use one on a PCB, without having to deal with getting the actual MCU and support components soldered on by a PCB house or yourself. Plus the impedance, high-speed traces, grounding etc has been taken care of. That's a lot of, in some cases specialist work and monetary outlay getting it all working. There was a problem with manufacturing a previous Teensy that required x-raying I believe? Thank god those problems were solved for us! I hope the MicroMod problems are solved too, because it's fantastic for my project.

I was bemoaning JLC scratching my PCB front panels a couple of years ago and then I remembered that I was paying comparatively nothing for what is intended to be a fast prototyping service. We're being spoiled!

 

[houtson]

Replying to my own post here, Digikey got in touch today - their product specialist has told me to scrap the six that are defective, they don't need them returned and have offered to ship 6 replacements at no charge or credit onto credit card.

I'm going to get them replaced and see if they are any different/better.

I hope Sparkfun sticks with it and finds a fix the quality issues, the MM format is really attractive for me.

Of the 6 that were replaced with new boards 2 of them have failed pins. I got them through Digikey so they may not be from the latest batches.

 

[Rezo]

Of the 6 that were replaced with new boards 2 of them have failed pins. I got them through Digikey so they may not be from the latest batches.

Regardless of being from the latest batch or not, that’s still a shame and quite annoying I must say.

We’re paying a premium price for these Micromods, just like the Teensy 4.0/4.1, and therefore I would expect SparkFun to deliver the same build quality as Paul does.
As mentioned before, my cats love (sitting on) the Teensies, and out of roughly 10-12 units (3.2/4.0/4.1) I have purchased over the past few years from PJRC, none of them have issue. They all work flawlessly!

 

[houtson]

Regardless of being from the latest batch or not, that’s still a shame and quite annoying I must say.

We’re paying a premium price for these Micromods, just like the Teensy 4.0/4.1, and therefore I would expect SparkFun to deliver the same build quality as Paul does.
As mentioned before, my cats love (sitting on) the Teensies, and out of roughly 10-12 units (3.2/4.0/4.1) I have purchased over the past few years from PJRC, none of them have issue. They all work flawlessly!

I've had similar experience with the PJRC teensy over the years - rock solid and wouldn't even cross my mind to test them before use, no need to.

I really hope the Sparkfun guys work through the MMod issues, such a convenient format if it can be made to be reliable.

 

[Ausplex]

It would be interesting to see inside Sparkfun's manufacturing facilities. Placing 100 pin BGA packages requires a more diligent approach to quality and testing than regular surface mount manufacturing. For example, production batches should regularly have samples x-ray inspected to ensure the reflow process within the grid array is effective. It is not enough to just test the board and see if it is working as contacts may appear to work but may not be fully wetted or reflowed. I have placed 1000's of BGA packages on my pick and place line but no longer do it as I don't have those production test facilities to provide the level of confidence needed.

There seems to be quite a few users that are using significant quantities of Teensy MicroMod's. It would be great to see PJRC release a Teensy 4.1 derivative that is in a high density format that lends itself to SMT production (ie. the M.2 connector). I'm sure there will be great demand for it.

 

[BriComp]

The MicroMod is NOT a PJRC product so it is NOT incumbent upon PJRC to pass any comment upon it.

 

[Sanworks]

I'm lucky I found this thread (+ similar ones on the Sparkfun forums) only 3 days into product development with Teensy Micromod. I love the concept of Micromod, but the failure rate in the field is too high for us to work around. If board flex is the issue, I'm not convinced that Sparkfun's changes to the QC process will result in anything close to PJRC-level reliability.

Please add my voice to the calling - this is a great opportunity for PJRC to spin off a socket-pluggable or reflowable version of T4. The Teensy user community, software environment and reputation for reliability are leagues above what Portenta developers are faced with. I think it would be a real hit!

 

[Dogbone06]

I'm lucky I found this thread (+ similar ones on the Sparkfun forums) only 3 days into product development with Teensy Micromod. I love the concept of Micromod, but the failure rate in the field is too high for us to work around. If board flex is the issue, I'm not convinced that Sparkfun's changes to the QC process will result in anything close to PJRC-level reliability.

Please add my voice to the calling - this is a great opportunity for PJRC to spin off a socket-pluggable or reflowable version of T4. The Teensy user community, software environment and reputation for reliability are leagues above what Portenta developers are faced with. I think it would be a real hit!

Sparkfun has been very good in the matter. I have kept close contact with them. The failiure rate is now for me almost none, I order 100-200 units each time and only a few have been bad. Which I’ve sent back for a refund.

So don’t hesitate to talk to them, they’re great!

 

[PaulStoffregen]

... what Portenta developers are faced with.

Do you mean Arduino Portenta? So far I've not heard of Portenta hardware issues, but I'm not really connected to that community. Are people having problems with those boards?

 

[Sanworks]

My concern isn't failure rate at the time of purchase - of course not pressing on the chips during testing will unmask the worst solder joints. The issue is that several users in this thread and others have reported spontaneous failure of seemingly working pins after less than 1 year of operation. Beyond QC, I'd guess that they probably need to address the issue with changes to the Micromod board material and mounting scheme to get PJRC's long-term reliability. I'll stay tuned to find out whether their changes to QC have improved the outcome in the field.

reaul, Yes. I haven't heard of Portentas failing, but various other Arduino products have been problematic over the years. It was more a comment about your brand's reputation for excellent reliability and developer resources - both major selling points for commercial scale-up.

 

[isaacjacobson]

Just to reiterate some of the things that others have already said:
• Initially 50% of my MicroMod units were defective (upon arrival or at a later time).
• Since additional QC measures have been put in place, failure rate has improved drastically to about 10%.
• Sparkfun (and John specifically) have been incredible to work with and have gone above and beyond to resolve issues! Fantastic customer service.


That said, I noticed that a copy protection version of Teensy MicroMod was recently released.

Has anyone noticed a better/worse failure rate with the copy protection version?

 

[Ausplex]

The concern from a long term use perspective, is if there is a high failure rate on delivery, then it is possible that the ones that do work are actually marginal and could fail in the field. After all, we have learned that they are supposedly getting a 100% pin test before leaving the factory. The only way to provide a long term quality product with a BGA package of this density is to have regular X-ray inspection QA on the production line. I wonder if Sparkfun have the equipment and experience to do that.

 

[defragster]

...
Has anyone noticed a better/worse failure rate with the copy protection version?

From that link: "The SparkFun MicroMod Teensy Processor with Copy Protection utilizes the same hardware as the "Lockable" Teensy from PJRC. "

So, this is just shipping with altered bootloader that allows complete secure encryption. Only diff is the bootloader chip, and how it initializes the 1062.

Likely to be newer production with current QA being used/passed. No reason it would be any worse.

Having seen the lockable T_4.x's since they were beta - they work and run the same with the added ability to require secure code when locked, that is not an option on regular production boards - where the newer boards CAN be set to run encrypted code - but not be mandatory. That is when encrypted the code stored on the external flash is ... just that. With mandatory locking set the board can not be reprogrammed except with the pre-defined stored key having been used to encrypt any upload.

 

[PaulStoffregen]

With mandatory locking set the board can not be reprogrammed except with the pre-defined stored key.

The following may seem like nit picking over semantics, but it really is an important feature of the security model. The key (eg, key.pem file) is not used for reprogramming. Anyone without access to the key who is given the encrypted .EHEX file can use Teensy Loader to program it onto these boards.

This is the main value you get from the copy protection (aka "lockable") feature, for people who wish to create commercial products. You can send locked hardware into the world and if an update is needed, you can safely send the .EHEX file to anyone to allow them to perform the update. The key is not used by Teensy Loader. The system is designed to allow untrusted people to securely update, because the key isn't used for loading the code.

The key is only used in 2 ways, to initially lock the hardware, and to create the .EHEX file from the .HEX file.

 

[defragster]

The following may seem like nit picking over semantics, but it really is an important feature of the security model. The key (eg, key.pem file) is not used for reprogramming. Anyone without access to the key who is given the encrypted .EHEX file can use Teensy Loader to program it onto these boards.

This is the main value you get from the copy protection (aka "lockable") feature, for people who wish to create commercial products. You can send locked hardware into the world and if an update is needed, you can safely send the .EHEX file to anyone to allow them to perform the update. The key is not used by Teensy Loader. The system is designed to allow untrusted people to securely update, because the key isn't used for loading the code.

The key is only used in 2 ways, to initially lock the hardware, and to create the .EHEX file from the .HEX file.

Quite right - finished the sentence in p#69:

With mandatory locking set the board can not be reprogrammed except with the pre-defined stored key [B]having been used to encrypt any upload[/B].

 

[Nominal Animal]

Would the following help clarify the locking scheme to those wondering?

Locking uses a very clever bit of mathematics called public key cryptography. The core of this is that you have a pair of keys, so that what one can encrypt, the other can decrypt, and vice versa. You cannot, however, derive or guess any better the other key, even when you have the other one of the pair. (This is why it is called "public key": you keep one secret, and publish/give out the other, so that anyone can encrypt data and send it to you without anyone else being able to decrypt it (because only you have the secret key); and whatever you encrypt, can be decrypted (and authenticated) by anyone with the public key.)

Aside from "encryption" and "decryption", the same key pair can be used for authentication ("MAC" for Message Authentication Code). Basically, one half of the key is used to calculate a cryptographic checksum of sorts of the message, and appended to the message (with the message itself either plainly visible or encrypted). Anyone with the other half of the key can check if the checksum is correct, but they cannot generate a checksum themselves.

For lockable Teensies, the key is used to create the .EHEX file from the .HEX file. The .EHEX file is encrypted, but also contains such a MAC.
You have your key of the key pair in a .PEM file, but the other key of the pair is secreted inside the Teensy (in a way that it cannot be extracted by any normal means), so it will only accept .EHEX files that have been encrypted by its paired key.

Thus, it is the .HEX to .EHEX step, that makes the firmware image trusted and encrypted. You cannot modify it without both keys; although of course only the key in the .PEM file is needed to encrypt a different .HEX to a .EHEX file that the Teensy would accept.

When you first lock your Teensy, both keys are created from nothing using random numbers (so it isn't like PJRC or SparkFun has any kind of a "master key"), one saved as a .PEM file, and the other "burned" into Teensy. After being set, it cannot be changed.

---

In any case, for the lockable MicroMods, I do believe the only hardware change between lockable Teensies and non-lockable Teensies is the MKL02 bootloader chip (which is basically the same chip but with slightly different proprietary PJRC code in it).

So, isaacjacobson's question in #67 is basically whether SparkFun has done additional changes (beyond the earlier QC changes that dropped the failure rate drastically) in their Teensy MicroMod production, because only that would affect the failure rate compared to non-lockable Teensy MicroMods.

I would guess not, but obviously, until we have reports from people having used a bunch of lockable Teensy MicroMods, we don't know.

So, anyone using lockable Teensy MicroMods: How's the reliability now?

 

[Dogbone06]

Just to reiterate some of the things that others have already said:
• Initially 50% of my MicroMod units were defective (upon arrival or at a later time).
• Since additional QC measures have been put in place, failure rate has improved drastically to about 10%.
• Sparkfun (and John specifically) have been incredible to work with and have gone above and beyond to resolve issues! Fantastic customer service.


That said, I noticed that a copy protection version of Teensy MicroMod was recently released.

Has anyone noticed a better/worse failure rate with the copy protection version?

I just wanted to give my two cents regarding John. He's been great, he has been a key person in making this all better. I always order 100+ micromods at the time, these days for me, only about 5% are faulty at the most.

 

[defragster]

Would the following help clarify the locking scheme to those wondering?
...
So, anyone using lockable Teensy MicroMods: How's the reliability now?

PJRC has posted here: pjrc.com/store/teensy41.html#codesecurity
and here pjrc.com/teensy/td_code_security.html

Those are complete description of the details ... within any constraints imposed by the associated NDA ... on those pages.

A lockable T_MM was sent here in Beta and it is just a normal T_MM that ships with altered bootloader/MCU fuse configuration to allow and support the Lockability as indicated in the PJRC notes.

Didn't mean to open a can of worms ... just that having used Teensy 4.0, 4.1 and MM with locking enabled ... other than a 'lock' inking on the PCB - they seem otherwise unchanged as noted:

Lockable Teensy
Secure mode can only be activated on Lockable Teensy. [B]While Standard and Lockable Teensy are identical hardware[/B], the permanent fuse configuation differs.

 

[UHF]

...only about 5% are faulty at the most.

I would regard that still, as a disaster. No one has complained about T4.1s failing like this. I presume you're producing a product for customers, are you concerned about or have had failures in the field? I'm working on a synth using TMMs because I like the ease with which customers can buy and slot it in without soldering. But both my micromods failed, one a few months after the other. Both have been resurrected by reflowing (blasting with heat actually) the IC.