Teensy 4.0 code security?

Greetings! We've shipped dozens of units of an industrial product containing the T3.6 as its brains. I'd be very keen to migrate to the 4.1 once I can make our code secure. Any updates on when this might become available?
cheers
Doug

So have I. If there is a donation page to buy hardware or anything I'd contribute in to make it a working feature.
 
Hi all, after long time without a sign from me I will give you an update of what I have so far:

I used the time do the development and repeated testing of a secondary bootloader and a releated host tool, which are now working stable and reliable even some security features are not yet implemented. After having a look at the bootloader code provided by NXP and trying to understand how it needs to be adjusted to get it work on UART for several days I found it too complex to go this way further. As I already had begun development of a flash bootloader for another processor I decided to work on that and imlplement one from scratch. Meanwhile I had several difficulties with the one or another issue, so it took actually more time than I had expected.

Anyway, its now working and it's time to get the focus back on the security topics.

During the time I also wrote a perl script that helps to generate the configuration file for NXP's Code Signing Tool (CST).

So I will share the code as soon as I have an idea how. That's because I used for the development a different environment than the arduino. My development environment is based on a bunch of Makefiles and has several dependecies on external tools. But the most dogy is, that it also contains other projects. So I do not yet have a good idea how to cut out the relevant part in a way that gives you the possibility to rebuild and use it. So I think I will at least share the plain source code and Makefiles where possible. Maybe someone here who's probably more experienced with arduino comes up with a good idea how to get it into the arduino environment or how to get a build environment prepared for the community.
 
Hi all, after long time without a sign from me I will give you an update of what I have so far:

...
Anyway, its now working and it's time to get the focus back on the security topics.

...

Great you are back to look into the security topic!

BTW for anyone who hasn't seen it on this third thread Paul posted this - the currently 'not SECURE capable' but bootloader no locking fuses is available in 'test' quantities:

After it gets some confirming feedback Paul noted the order limit will be removed.

Also noted the code on that chip with unlocked fuses will ideally be updatable (by PJRC) for use when the security issue is resolved ... that is on another thread.
 
Great you are back to look into the security topic!

In fact I've never been away. But it took me some of my spare time to get prepared for further investigations. BTW nice to see what Paul did in the meantime.

Also noted the code on that chip with unlocked fuses will ideally be updatable (by PJRC) for use when the security issue is resolved

As I understand the, update process, the bootloader is injected into the iMXRT via JTAG by the MKL02 chip. So as long as you do not disable or protect JTAG access this way will still work. Only, once security is enabled, your flashed application image will not be allowed to execute from power-up (by HAB) if not properly signed. Anyway, if one wishes to have a really secure device one will also disable JTAG access, which then excludes further updates in the known way.
 
Hello everybody,

I made an attempt to set one of the unlocked teensy boards to closed configuration after having burnt the SRK hash fuses and successful HAB authentication in open configuration. But rather undexpectedly the board didn't answer anymore after reset. After long second thought what mistake I might have made and what I should do different, I decided to do the same steps at my MIMXRT1060-EVK with the difference that on this board I'm able to get into the running software via JTAG Debugger. So the result was quite similar right after programming the signed bootloader image via debugger. Anyway, I was still able to debug the code an step/jump arround and inspect memories and registers. First idea was to check the watchdogs and indeed they where enabled. Later it turned out, that this is the proper behaviour and when the image gets authenticated successfully, they are turned off by the HAB before the image starts to execute.
Somhow I managed to send a command via UART into my bootloader to evaluate and dump the HAB status and this is what I got:
Code:
Hello FBL
FBL>habStatus

HAB Configuration: 0xcc (closed) HAB State: 0x66 (non-secure)

----- HAB Event 1 -----
Event Data:
  0xDB 0x00 0x2C 0x43
  0x33 0x30 0xE1 0x1E
  0x00 0x00 0x00 0x00
  0x80 0x00 0xBB 0x00
  0x80 0x00 0x00 0x02
  0x00 0x00 0x00 0x20
  0x00 0x00 0x00 0x00
  0x00 0x00 0x00 0x00
  0x00 0x00 0x00 0x08
  0x00 0x00 0x00 0x00
  0x00 0x00 0x00 0x00
  Tag: 0xDB -> EVT
  Len: 0x002C -> 44 Bytes
  Par: 0x43 -> HAB Ver: 4.3
    Evt Status:  0x33 -> HAB_FAILED
    Evt Reason:  0x30 -> ENGINE_FAIL
    Evt Context: 0xE1 -> ENTRY
    Evt Engine:  0x1E -> SNVS

----- HAB Event 2 -----
Event Data:
  0xDB 0x00 0x2C 0x43
  0x33 0x30 0xE1 0x1E
  0x00 0x00 0x00 0x00
  0x80 0x00 0xBB 0x00
  0x80 0x00 0x00 0x02
  0x00 0x00 0x00 0x20
  0x00 0x00 0x00 0x00
  0x00 0x00 0x00 0x00
  0x00 0x00 0x00 0x08
  0x00 0x00 0x00 0x00
  0x00 0x00 0x00 0x00
  Tag: 0xDB -> EVT
  Len: 0x002C -> 44 Bytes
  Par: 0x43 -> HAB Ver: 4.3
    Evt Status:  0x33 -> HAB_FAILED
    Evt Reason:  0x30 -> ENGINE_FAIL
    Evt Context: 0xE1 -> ENTRY
    Evt Engine:  0x1E -> SNVS

No more HAB Events found!

According to High Assurance Boot Version 4 Application Programming Interface Reference Manual, Page 57, the 3rd to last line of the hex dump of the events show the following SNVS registers:

If an entry, exit or test operation fails, an audit event is logged with status field HAB_FAILURE, reason field HAB_ENG_FAIL, engine field HAB_ENG_SNVS and data field containing the following registers (in order):
o HP Security Violation Control
o HP Status
o HP Security Violation Status
o LP Control
o LP Master Key Control
o LP Security Violation Control
o LP Status
o LP Secure Real Time Counter MSB
o LP Secure Real Time Counter LSB

Inspecting the very same registers via JTAG debugger shows like the hex dump, that the HP Security Violation Status Register (HPSVSR) has Security Violation 1 set, which according to Security Reference Manual for the i.MX RT1050 Processor, Rev. 1, 04/2018, Page 50 is linked to SJC JTAG active. Obviously this security violation cannot be disabled, just configured as fatal or non-fatal violation.
Later I did a power-on reset on the EVK and I was very surprised to see the proper prompt of the bootloader image on the terminal. So what happened? By powering off the board, the JTAG connection was killed and the HAB obviously successfully authenticated the bootloader image. Experimenting with the running image confirmed the suspicion that a connected JTAG might be the reason for the failed authentiation. Here is the HAB status after power-on reset (1) and after re-attaching the JTAG debugger on the EVK board:

Code:
Hello FBL
FBL>boot
Enter FBL
FBL>habCheck
ROM-Boot ok.
FBL>habStatus

HAB Configuration: 0xcc (closed) HAB State: 0x99 (trusted)
No HAB Events Found!

Code:
FBL>habStatus

HAB Configuration: 0xcc (closed) HAB State: 0xcc (fail_soft)
No HAB Events Found!

As you can see, after attaching the JTAG debugger, the HAB state changes from "trusted" to "fail_soft" and this is a reproducable issue.

So my assumption for the teensy now is, that due to the JTAG access by the MKL02 chip the SNVS asserts a security violation already while the ROM bootloader is running and so the HAB fails due to security checks as seen in the very first terminal output.

What can be seen on the teensy is, that the Halfkay injecttion via JTAG is still working. So when I shortly press the push button, I'm able to program another image, but obviously this doesn't run because of the reboot following the reprogramming. Also due to the failed HAB authentication the teensy switches to serial download mode, which might be a last resort to un-brick this board. Until now I was not able to get any useful information out of the "bricked" teensys by the sdphost tool as it repeatedly answers with status "HAB enabled" and no reasonable value returned.

So I'm writing this here the in order to possibly get some confirmation or feedback for my assumptions befor I may brick another unlocked teensy in the attempt to disable JTAG access before setting it to closed configuration. Especially I'd like to get confirmed, that the MKL02 chip is connecting to the IMXRT on every reset, not just after pressing the push button, which would further proof the assumption that the reason for the failed authentication is the JTAG connection.

A question here is, if there is a way to temporarily prevent the MKL02 to JTAG the IMXRT after reset. A further question is whether the MKL02 will still apply the necessary signals to let the IMXRT run, if the JTAG connection doesn't due to disablement. A 3rd question is, whether there is a way to tell the Halfkay to directly jump to an address instead of resetting the chip, when it has programmed an image.

Does anybody know how to prevent HAB checking the SNVS status registers or disable SNVS during boot? The latter is implied in a note in the HABv4 API RM:
Note: If a failure occurs when the SNVS is not enabled then the audit event reason field is HAB_WARNING rather than HAB_FAILURE.
 
Probably just Paul can give substantated answers to those 3 questions...

A question here is, if there is a way to temporarily prevent the MKL02 to JTAG the IMXRT after reset.
A further question is whether the MKL02 will still apply the necessary signals to let the IMXRT run, if the JTAG connection doesn't work due to disablement.
A 3rd question is, whether there is a way to tell the Halfkay to directly jump to an address instead of resetting the chip, when it has programmed an image.

So please Paul, if your time allows, please let me know whether the Teensy 4.0 will boot and run also if JTAG between MKL02 and IMXRT is disabled. I don't want to brick a fresh non-locked Teensy for finding out that this way might not work, if probably you can give me the answer in advance.
 
Hello everybody,

here is the latest news about HAB...

As I didn't get a response yet, I started experimenting with a fresh board. So tried to hardly pull up/down single wires of the JTAG interface by which I was able to prevent the MKL02 form accessing the IMXRT, but the SNVS security violation remained.
Next I burnt JTAG_SMODE to 0b01 (Secure JTAG) which normally should block JTAG access to the CM7 DP and only allow access to SJC via callenge-response authentication. Result was, that the MKL02 was still able to reprogram the IMXRT and SNVS still reported SV1 (JTAG active).
Next I burnt JTAG_SMODE t0 0x11 (No Debug) which should even block JTAG access through SJC cia challenge-response authentication. Anyway, the result was the same as before: MKL02 was able to reprogram the IMXRT and SNVS still reported SV1 (JTAG active).
In a for now final step I also burnt SJC_DISABLE fuse to 1 which should disable all JTAG access to the processor including boundary scan and indeed the MKL02 is no longer able to reprogram the IMXRT. Unfrotunately the SNVS still reports SV1.

So I do not try to set the IMXRT on this board to secure configuration until I find a way to convince the SNVS to stop complaining about JTAG access. I also started a new thread in the NXP comunity and hopefully get some useful information there.

In conclusion for the time being I'm not able to make the teensy secure, obviously due to the JTAG access by the MKL02.
 
Ok, I found a reasonable exlanation for the MKL02 still beeing able to access the IMXRT via JTAG when JTAG_SMODe is burnt to 0b01 or 0b11: AN12419 in chapter 2.3 says that additionally the KTE fuse hes to be burnt to have JTAG secure mode work, which I did not. Anyway tos doesn't change anything on the fact that even with fully blocked JTAG the SNVS indicates security violation 1 (JTAG active).
 
Ok, I took the board, where all the JTAG is blocked now by fuses and as SNVS still reports SV1 I tried to physically block JTAG access. I soldered very thin wires to those pads of the MKL02 which are most probable stimulated for JTAG access (P13 - JTAG_MOD, P8 - JTAG_TCK, P7 - JTAG_TDI, P5 - JTAG_TMS) and tied them hard to GND even with the risk to damage the MKL02 by possible over current.

teensy-jtag-tied-to-gnd.jpg

Anyway, by doing so after power-up the SV1 (JTAG active) was no longer reported by SNVS. Instesd now it reported SW_SV (Software Security Violation) which is probably by the ROM bootloader due to the fact that security configuration is still open, but that's a guess. I'm not really sure. I also tested whether releasing single pins from being tied to GND will have the same result, but this wasn't the case. So it seams that SNVS is tracking signals on those pins and if one looks like JTAG it will report SV1.

Ok, so far so good. Now having a first sign I did the same aproach with that board previously having secured. Unfortunately it still is not willing to start, so it's not quite sure whether there's someting wrong with the image, SNVS still reports SV1 maybe because of JTAG_nTRST which remains because it's directly connected to DCDC_PSWITCH or SNVS reports something different e.g. SW_SV.
 
As even with the modification previously described the IMXRT didn't boot successfully and because the reason is not obvious, I decided to make an attempt to completely dedolder the MKL02 in order to gain access to the IMXRT's JTAG interface. As I do not own special desolder station for chips, it was not very easy and I succeded only by destroying the MKL02. So here's what left over of the boot chip:

teeny-mkl02-left-over.jpg

Unfortunately also 3 pads fell victim to my attempt of which one was the JTAG_nTRST/DCDC_PSWITCH so it looked as if I screwed up this board completely. For good luck I found the 100k resistor on the bottom of the board, which connects this wire to GND and I soldered the related wire there and indeed it works. Here''s how the board now lokks:

teensy-mkl02-removed.jpg

So in the end I'm now able to access the IMXRT via JTAG, with the result, that still SV1 is reported by SNVS. But searching the HAB log shows also the ENGINE_FAILED entry due to SNVS and this even if I start the board with no debugger attached. Compared to the MIMXRT1060-EVK this would be expected only, if the debugger is already attached during early startup in ROM boot. I will further investigate what's the differnce between Teensy and the EVK an will keep you informed.
 
Right few minutes ago I tried to check the HAB entry again but watching the memory with the debugger (target was halted) showed randomly changing RAM contents which was a bit misterious. After disconnectiing the debugger from both, the target and my PC and restarting the debug host application I firstly was not able to reconnect to the target. So I billed and reconnected target power to find the reason for the connection failure during this I also tried several combinations of the JTAG_nTRST/DCDC_PSWITCH and JTAG_MOD wire obviously without success. Next I tried to find a way to check whether the IMXRT does really start and checked my USB devices for the serial boot USB HID to appear, but it did not, firstly. At this point I had the JTAG_nTRST/DCDC_PSWITCH wire connected to GND. As I connected it to VCC (3V3) suddenly the desired USB HID appeared. Ok, this was not really unexpected, as the IMXRT normally doesn't start befor the DCDC_PSWITCH goes HI certaint time after power stabilized. Anyway in yesterday's attempts I didn't bother. Afterward I removed the USB cable to proove that the USB HID diappers again, but it didn't! Why? Obviously the IMXRT got some parasitic voltage by a USB-to-UART adapter connected by 3 wires to UART4 RX/TX and GND. Ok, some point to keep in mind! Now again I tried to re-connect the debugger and after a few unsuccessful attepts I checked the error messages which told me that the debugger was trying to connect via SWD. BTW had the same issue yesterday, but didn't remember. Ok, problem identified, problem solved. No I was able to re-attach to the target. Once halted I checked again the HAB log entries but today found something different! The entry was:
Code:
0xDB 0x00 0x08 0x43
0x33 0x22 0x0A 0x00
Looking up the codes in the HAB API this can be translated into:
Code:
Tag: 0xDB -> EVT
Len: 0x0008 -> 8 Bytes
Par: 0x43 -> HAB Ver: 4.3
  Evt Status:  0x33 -> HAB_FAILED
  Evt Reason:  0x22 -> INVALID_ADDR
  Evt Context: 0x0A -> AUTH
  Evt Engine:  0x00 ->ANY

Ok, this is no longer the SNVS security violation but a real authentication failure!
So next I inspected again the image vector table (IVT) of the image in the nor flash at address 0x60001000 which had a CSF address of 0x00000000. Which is definately wrong and is a reasonable explanation for the seen HAB log entry. In order to fix this I reprogrammed the flash by the debugger, so that now the CSF address field got a proper address.

Finally testing to run HAB the commandline of my bootloader came up which indicates successful boot now.
 
Trying to reproduce the HAB log entry of yesterday I found out that it not had to do with the UART, but with how I configured the debug connection settings and how I used the debugger. Yesterday I configured the connection settings to use JTAG_nTRST for a reconnect and as after plugging in the USB into the board the IMXRT did obviously not run because of the missing rising edge on DCDC_PSWITCH, it was first started when I attached the debugger, hence SNVS detected JTAG active.
Today I disabled usage of JTAG_nTRST and thus had troubles to re-connect the debugger, obviously because the IMXRT was not yet running. Giving the wire a manual rising edge and afterwards re-connecting the debugger worked well and because now the processor started before the debugger was attached SNVS couldn't report SV1 during HAB. Anyway, after attaching the debugger SNVS still reports SV1 and SSM state changes form trusted to soft_fail as seen also on the EVK.
 
KEep going Dresden-Fx, you doing well!

Thank you for your support and encouragement. That helps a lot to increase my motivation.

Actually, behind the scenes, I still keep going, albeit at half throttle. I'm in contact with Paul in order to discuss and tune technical solutions in accordance with his master plan. Unfortunately Paul has more than just this single topic to spend time for, even though, I can tell you, he put a lot of effort also into this topic. But surely it's still a complex matter.
 
Ok, it's finally time for an update! I've been working on code security support for a last few months. It's been a long journey and the end is finally in sight. Soon we'll begin a first beta test for Teensyduino 1.55 with this feature.

Today I ported all the code to Windows. It also works on Linux x86-64, and probably the other Linux architectures (no others have been tested yet). No work has happened on MacOS yet, but I will eventually get it ported to Macs too. It has some issues on Windows 7 which I haven't been able to resolve. I plan to start the beta test with only Linux and Windows 10 supported.

Here's a screenshot, so you can get a visual idea of how this will work.

capture.png

Teensyduino 1.55 will add a "Teensy 4 Security" entry to the Tools menu. Clicking it brings up this little window with the 4 buttons.

After you've created a key (which the button still doesn't do... a script running lots of openssl commands is still needed) then every compile creates a .ehex file from the .hex file. You don't do anything special to encrypt your code, as it happens automatically with every compile if you've got a key.pem file in the {Documents}/Arduino (or whatever location the Arduino IDE is configured to look for your sketchbook - see File > Preferences).

Those 3 buttons generate code which opens in a new Arduino window. Hopefully the descriptions are pretty self explanatory? To configure your Teensy hardware to use the key, you just upload those programs.


The one catch is the final LockSecureMode sketch doesn't work on standard Teensy. The fuses on standard Teensy are configured in such a way that you can't alter the boot configuration, which is meant to keep Teensy from possibly becoming bricked. Once we're finished the beta testing, PJRC will start selling lockable Teensy models, which are identical to standard Teensy except the fuse configuration will allow locking security.

After security is locked, you can still upload .ehex files, but not normal .hex. So a locked Teensy remains usable, as long as you have the correct key.pem file to create .ehex files which match the key you wrote.

The stand-alone Teensy Loader will also be able to securely program a .ehex file (if it's made with the correct key). So if you put a locked Teensy inside a product and sell it, you can later published a .ehex file and give it to anyone to be able to securely upgrade the firmware.
 
This is the biggest news ever for Teensies!

To be clear, the single .eHEX will be able to update multiple teensies, correct?
 
Paul:

Thanks for the great work that you have put into this . . . I am excited to give this a try. One question does come to mind: Will pressing the "Generate Key" button cause a warning if there is already a key in the destination folder (to avoid accidentally overwriting a previous key) ??

Thanks again !!

Mark J Culross
KD5RXT
 
I'm planning to make it rename the file(s) for your previously created key(s), so nothing ever gets overwritten. But if you click it many times, you'll end up with many key files! Only the one named "key.pem" actually gets used.

In future versions I might support other (better?) ways to manage keys. Optional password protection is high on the list of missing features. But keeping it simple for now.
 
I'm planning to make it rename the file(s) for your previously created key(s), so nothing ever gets overwritten. But if you click it many times, you'll end up with many key files! Only the one named "key.pem" actually gets used.

In future versions I might support other (better?) ways to manage keys. Optional password protection is high on the list of missing features. But keeping it simple for now.

Writing to pem.key and a copy to pem.YYMMDDHHMMSS.key at the same time would allow user tracking/storage of the source and assure any pem.key overwrite wouldn't cause loss, and wouldn't take any UI change for now, or rename complexity.
 
Could the Loader obtain the the Teensy Serial number and log to file together with date so, together with @defragster suggestion, one can get a log of which teensy got which key?
 
Making a running log file might be handy? On key create - maybe time date stamp and toss the Serial# there, but serial # won't help when the same key covers 2 or 200 boards. Maybe the public key used for ref to bootloader feedback, if that can always be read? Not sure if the Serial# will always be 'viewable', if not running USB code - can the bootloader get that from a locked unresponsive mcu?
 
Could the Loader obtain the the Teensy Serial number and log to file together with date

This could be done, but probably won't be added anytime soon. My short term goal is to keep key management and other non-essential details as simple as possible. And my goal beyond that short term goal is to focus on several other important development tasks, like NativeEthernet testing & debugging, better MTP integration, non-blocking Wire library, lots of audio features, future Teensy hardware, and so on.

Teensy Loader is meant to look & feel simple. But some of that simplicity is quite difficult to implement. The main issue is with dialog boxes. They come in 2 types, either modal or non-modal. Modal dialog boxes take over the entire GUI and all event processing. Those are relatively simple. I believe File > Open is implemented as a modal dialog. In the early days of Teensy all the dialog boxes were modal, which left a lot to be desired. Today most of them are non-modal, which takes a *lot* more work to implement, but it pays off because Teensy Loader remains responsive to USB changes and to remote commands from Arduino. For example, if you have Tools > Boards set to the wrong board and try to upload, you'll get the dialog about the code not matching the hardware. If you then set the correct board and recompile, you'll see the dialog automatically goes away when the problem is resolved. Likewise if you plug in different hardware.

So while a lot of nifty features could be implemented if I pour more time into Teensy Loader, I'm a bit reluctant to add more, especially if it involves even 1 extra dialog box. It's really easy for a feature that seemed simple to turn into a much longer than expected project in a program which tries to remain as responsive at possible to GUI events, localhost network connections, USB hardware changes, and filesystem changes - which is built cross platform for Linux, MacOS and Windows.
 
Back
Top