Anyone else seeing MacBook Air hard crash when using a Teensy 4?

Second very hard crash today while trying to upload. Something is very wrong with the Mac’s Thunderbolt/USB drivers.
 
I’m seeing this intermittently too. I’ve tried to isolate the problem by trying various combinations of hardware/software sequences when uploading new code to the Teensy, but without much success. Keeping an eye on this thread!
 
Might help to have some additional information.
Things like:
What version of Teesnyduino are you running?
What version of Mac are you running?
What version of Mac OS are you running?
Any particular type of Teensy4 USB type on it? i.e. are you doing USB Type: Serial or something else?

My Mac is now about 10 years old, which I only use for secondary stuff...
 
Recently I've done a little bit more coding Teensy wise than before and I've had a handful of very hard crashes in the sense that the system just locks up/freezes and nothing can be done besides holding the power button and turning back on. In this case no crash report is even generated nor does a popup show up once it turns back on indicating that it was restarted because of an issue. In my case I've pretty much concluded that it's happening when I have Wireshark listening on my USB to Ethernet adapter connected to my Teensy 4.1 I've been working on and I go to reflash the code. Reason I believe my issue is exclusively related to Wireshark is because for one the Teensy is directly connected to this adapter and if the Ethernet PHY isn't started on the Teensy then the adapter does not show as active in Wireshark so it's not able to be in a listening state when that happens during reflashing. Most of the time it does complete a reflash while Wireshark is still listening with no issues though so really it's anyones guess there, but I haven't had a crash yet when not using Wireshark.

Teensyduino: 1.55
Mac: MacBook Pro (16-inch, 2019)
macOS: Monterey Beta
Teensy 4.1: connected directly through USB-C to USB-C from custom carrier board
USB Type: Serial with Serial Monitor open and receiving text
 
I've noticed that I've never seen hard crashes when using the PlatformIO toolchain. I see at least several a day, and sometimes several per hour, if I use the Teensyduino IDE v1.55 regularly. We're talking full computer freeze and then that screen-of-death several minutes later.

My setup: MacBook Pro (13” 2020), macOS Big Sur 11.6.1, Teensy 4.1 connected via a CalDigit TS3 Plus, Teensyduino IDE v1.55.
 
I’ve been seeing this for several months now too, using the latest platformio (currently 5.2.3) and a Teensy 4.0 (I’ve tried several different brand new Teensy 4.0 boards). If I run the platformio build command to upload new code to the Teensy while serialmon is running, 50% of the time I get the full grey screen hard crash on my Mac. The other times it works fine. I’ve been running serialmon from the command line as Paul suggested, and the same thing happens.

I’m uploading some pretty complicated code and I can’t rule out user error, so I’ll try to test it out with a simple sketch soon. I’m also using a USB-C hub with four active connections: HDMI to an external monitor, ethernet, USB-A to an audio interface and USB-A to the Teensy. I’ll try to eliminate some of those too and see if I can isolate the problem any further.

The crash log looks very similar to others posted on this thread:
Code:
panic(cpu 2 caller 0xffffff8012bc5396): Kernel trap at 0xffffff8013056f73, type 14=page fault, registers:
CR0: 0x0000000080010033, CR2: 0x0000000000000018, CR3: 0x000000045cccc01c, CR4: 0x00000000003626e0
RAX: 0x0000000000000000, RBX: 0xffffff86abca9200, RCX: 0x0000000000000000, RDX: 0x0000000003000000
RSP: 0xffffffa0e8603e30, RBP: 0xffffffa0e8603e40, RSI: 0xffffff9371d30600, RDI: 0x0000000000000000
R8:  0xffffff8013a78818, R9:  0xffffffa8ba0c7700, R10: 0xffffff9373751c00, R11: 0x0000000f93737510
R12: 0xffffff86b8ea8e40, R13: 0xffffff86abca9200, R14: 0xffffff9371d30600, R15: 0xffffff86b13b2120
RFL: 0x0000000000010207, RIP: 0xffffff8013056f73, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0x0000000000000018, Error code: 0x0000000000000000, Fault CPU: 0x2, PL: 0, VF: 0

Backtrace (CPU 2), Frame : Return Address
0xffffffa0e8603850 : 0xffffff8012a8d25d 
0xffffffa0e86038a0 : 0xffffff8012bd49d3 
0xffffffa0e86038e0 : 0xffffff8012bc4fca 
0xffffffa0e8603930 : 0xffffff8012a31a2f 
0xffffffa0e8603950 : 0xffffff8012a8ca7d 
0xffffffa0e8603a70 : 0xffffff8012a8cd73 
0xffffffa0e8603ae0 : 0xffffff801329d8fa 
0xffffffa0e8603b50 : 0xffffff8012bc5396 
0xffffffa0e8603cd0 : 0xffffff8012bc507d 
0xffffffa0e8603d20 : 0xffffff8012a31a2f 
0xffffffa0e8603d40 : 0xffffff8013056f73 
0xffffffa0e8603e40 : 0xffffff8012ffe108 
0xffffffa0e8603eb0 : 0xffffff8012fede16 
0xffffffa0e8603f40 : 0xffffff801313fd2e 
0xffffffa0e8603fa0 : 0xffffff8012a321f6 

Process name corresponding to current thread: teensy_serialmon

Mac OS version:
20G224

Kernel version:
Darwin Kernel Version 20.6.0: Tue Oct 12 18:33:42 PDT 2021; root:xnu-7195.141.8~1/RELEASE_X86_64
Kernel UUID: ABC69550-60C2-34FE-B307-C24A8C39309C
KernelCache slide: 0x0000000012800000
KernelCache base:  0xffffff8012a00000
Kernel slide:      0x0000000012810000
Kernel text base:  0xffffff8012a10000
__HIB  text base: 0xffffff8012900000
System model name: MacBookPro14,1 (Mac-B4831CEBD52A0C4C)
System shutdown begun: NO
Panic diags file available: YES (0x0)
Hibernation exit count: 0

System uptime in nanoseconds: 453418793398
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x0000006991ddbcc9
  Sleep   : 0x0000000000000000 0x0000000000000000 0x0000000000000000
  Wake    : 0x0000000000000000 0x0000000d67ab307a 0x0000000000000000
last started kext at 35558044589: >!A!BMultitouch   99 (addr 0xffffff8013e24000, size 24576)
loaded kexts:
...
 
A quick update: I’ve eliminated a few possible causes, but haven’t narrowed it down to a reproducible set of conditions yet.

The USB-C hub was not the problem. The crash still happens when the Teensy is the only USB device connected to the MacBook Pro (using a single Apple USB-C to USB-A dongle).

The crash does not happen with a simple test sketch nor with the MidiSynthKeyboard example sketch (which compiles to a similar size as my program and also uses the Audio library and sends characters to Serial constantly. So there’s something about my program that is triggering the issue. I’ll try to narrow it down further, but not today...
 
Can you reproduce the error if you use the Arduino IDE? Or does it only happen when PlatformIO is in the mix?

I have a few Macs here, but all are older hardware. My Macbook Air is a 2015 model with the latest MacOS Monterey. I also have a 2013 "trashcan" Mac Pro running Mojave. Also have an older Mac Mini. So far I've never managed to reproduce the problem on them.

I really want to investigate this, but first I need a way to reproduce the crash.
 
This is probably a long shot, but here's a copy of teensy_serialmon with debug symbols compiled in. If you can reproduce the crash with this copy running, maybe the MacOS crash report might give more info?
 

Attachments

  • teensy_serialmon.zip
    17.1 KB · Views: 46
Some extra info: I tend to see these crashes with the Arduino IDE, and when I open/close/reopen the serial monitor, with some combination of plugging/unplugging/programming the Teensy. I've never seen it when I use PlatformIO.
 
I reproduced the crash with this version of serialmon, but the crash report looks very similar:
Code:
panic(cpu 3 caller 0xffffff80033c5396): Kernel trap at 0xffffff8003856f73, type 14=page fault, registers:
CR0: 0x0000000080010033, CR2: 0x0000000000000018, CR3: 0x00000002672bf15b, CR4: 0x00000000003626e0
RAX: 0x0000000000000000, RBX: 0xffffff86b83da400, RCX: 0x0000000000000000, RDX: 0x0000000003000000
RSP: 0xffffffa0b5d43e30, RBP: 0xffffffa0b5d43e40, RSI: 0xffffff935b1ca200, RDI: 0x0000000000000000
R8:  0xffffff8004278818, R9:  0xffffffc2c3e88960, R10: 0xffffff9365677c00, R11: 0x0000000f93656770
R12: 0xffffff86a8241000, R13: 0xffffff86b83da400, R14: 0xffffff935b1ca200, R15: 0xffffff86b58fd5e0
RFL: 0x0000000000010207, RIP: 0xffffff8003856f73, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0x0000000000000018, Error code: 0x0000000000000000, Fault CPU: 0x3, PL: 0, VF: 0

Backtrace (CPU 3), Frame : Return Address
0xffffffa0b5d43850 : 0xffffff800328d25d 
0xffffffa0b5d438a0 : 0xffffff80033d49d3 
0xffffffa0b5d438e0 : 0xffffff80033c4fca 
0xffffffa0b5d43930 : 0xffffff8003231a2f 
0xffffffa0b5d43950 : 0xffffff800328ca7d 
0xffffffa0b5d43a70 : 0xffffff800328cd73 
0xffffffa0b5d43ae0 : 0xffffff8003a9d8fa 
0xffffffa0b5d43b50 : 0xffffff80033c5396 
0xffffffa0b5d43cd0 : 0xffffff80033c507d 
0xffffffa0b5d43d20 : 0xffffff8003231a2f 
0xffffffa0b5d43d40 : 0xffffff8003856f73 
0xffffffa0b5d43e40 : 0xffffff80037fe108 
0xffffffa0b5d43eb0 : 0xffffff80037ede16 
0xffffffa0b5d43f40 : 0xffffff800393fd2e 
0xffffffa0b5d43fa0 : 0xffffff80032321f6 

Process name corresponding to current thread: teensy_serialmon

Mac OS version:
20G224

Kernel version:
Darwin Kernel Version 20.6.0: Tue Oct 12 18:33:42 PDT 2021; root:xnu-7195.141.8~1/RELEASE_X86_64
Kernel UUID: ABC69550-60C2-34FE-B307-C24A8C39309C
KernelCache slide: 0x0000000003000000
KernelCache base:  0xffffff8003200000
Kernel slide:      0x0000000003010000
Kernel text base:  0xffffff8003210000
__HIB  text base: 0xffffff8003100000
System model name: MacBookPro14,1 (Mac-B4831CEBD52A0C4C)
System shutdown begun: NO
Panic diags file available: YES (0x0)
Hibernation exit count: 0

System uptime in nanoseconds: 15196246045601
Last Sleep:           absolute           base_tsc          base_nano
  Uptime  : 0x00000dd226d8f800
  Sleep   : 0x00000cca1696be6e 0x00002d5be9ba19bc 0x00000cbe323c267e
  Wake    : 0x00000cca2156db6a 0x000033fc3c5a3dce 0x00000cca1f8e743e
last started kext at 55302693347: |SCSITaskUserClient   436.140.1 (addr 0xffffff8005e0a000, size 20480)
loaded kexts:
...
 
Ok, I’ve eliminated almost every cause I can think of now... I can reliably reproduce the bug and here’s all the info that might be of interest:

  • MacBook Pro (13-inch, 2017, Two Thunderbolt 3 ports)
  • 2.3 GHz Dual-Core Intel Core i5 processor
  • Brand new Teensy 4.0 connected to one of the Thunderbolt 3 ports via an official Apple Thunderbolt 3 to USB-A connector and a USB-A to Micro USB cable
  • Teensyduino 1.56
  • Tools > USB Type: Serial
  • 50% chance of a hard crash uploading any sketch while serialmon is running in a terminal window
  • Not yet seen a crash when Serial Monitor is running (started from inside Teensyduino)

This is the command I’m using to run serialmon in a terminal window:
Code:
/Applications/Teensyduino.app/Contents/Java/hardware/tools/teensy_serialmon usb:14120000

Here’s the sketch I’m uploading (I’ve tried all sorts – it doesn’t seem to make a difference):
Code:
#include <Arduino.h>
void setup() {}
void loop() {}

The same thing happens when I use platformio to upload or the Deviot plugin for Sublime Text to have the serial monitor output appear in my text editor, but neither of those are necessary to reproduce the bug.
 
Good news, I am able to reproduce the problem on my Macbook Air.

Just spent the last couple hours repeatedly crashing! Finally found using "fcntl(fileno(fp), F_FULLFSYNC);" forces MacOS to actually write buffered data to disk (before the whole system goes down). I'm slowly getting closer. Seems the crash might be happening in or somehow related to CFFileDescriptorInvalidate().

This absolutely is a bug in MacOS Monterey (and other recent versions). A user level function like CFFileDescriptorInvalidate() is not supposed to be able to crash the entire machine!

I'm going to take a break and then read up on MacOS Core Foundation APIs again. Pretty sure a workaround can be found, but I want to take some time to really understand this before just putting an ugly kludge in the code to keep MacOS from crashing.
 
I can also confirm that the IDE will crash every now and then when trying to upload a sketch on a late 2016 Macbook Pro i5
I’ve noticed it happens if I unplug a Teensy and then plug back in while the serial monitor has been open, and then try upload a sketch. Sometimes it will throw some Java exceptions and just freeze.
 
Continuing to work on this. I can prevent the crash by removing CFFileDescriptorInvalidate() and some other cleanup code. But then it crashes MacOS at other times. Very mysterious....
 
Sorry about the delay. I haven't forgotten this problem. In fact, while recently working on serial monitor support for upcoming Arduino 2.0, I completely rewrote the Mac serial monitor code to avoid CFFileDescriptor stuff.

If anyone is still watching this thread, please run this and let me know if it's stable on your Mac? It seems to work well on my 2015 MacBook Air which definitely does crash with the test from msg #37. But I could really use some confirmation with other Macs. If this is looking good, I'll start work to back-port it for use with Arduino 1.8.19.

To run this test, first download arduino-cli. Get the nightly build from this page (scroll past "Latest release")

https://arduino.github.io/arduino-cli/0.20/installation/

When you extract, it's just a single command line executable and a license text file.

To start, run "./arduino-cli config init". It will create "arduino-cli.yaml". Edit that file to add "https://www.pjrc.com/teensy/td_156/package_teensy_index.json" to the additional_urls, like this:

sc.jpg

Then run "./arduino-cli core install teensy:avr". It may take a minute to download and extract stuff.

Once this is installed, you should be able to run "./arduino-cli monitor --port usb:14200000". Of course, change the number to whatever location is used on your Mac. You should see it in the bottom status bar of Arduino 1.8.19. You can also run "./arduino-cli board list" ask it to detect what boards are connected.

When the arduino-cli monitor command is running in Terminal, it should act as the Arduino Serial Monitor. Then try uploading, plugging the USB, etc. Run Arduino 1.8.19 while this is going in Terminal and try uploading new programs. It should automatically reconnect after each upload.

The big question: is this stable on your Mac? Should I start work to replace the old teensy_serialmon with this new code?
 
I've ported the new serial monitor code back for use with Arduino 1.8.19. This copy runs fine on my Macbook Air, both with the serial monitor and the stand-alone test on msg #37.

Please let me know how it works on your Mac?
 

Attachments

  • teensy_serialmon_2.zip
    19.2 KB · Views: 38
FWIW, I'm pretty sure this a bug in MacOS related to CFFileDescriptor. I rewrote the serial monitor stuff to not use CFFileDescriptor.

In searching, found this Google Groups conversation: https://groups.google.com/a/chromium.org/g/net-dev/c/nTOu2ie74Bg?pli=1

Relevant info copied here (in case Google Groups disappears....)

wukai:
I see that there are a few crashes
...
I get some information(API MISUSE: Over-release of an object) from crash_info , and I found that the release com.apple.CFFileDescriptor queue by analyzing the App coredump.
...
In my app, only Cronet uses CFFileDescriptor. So I suspect it's Cronet problem.

yanshua:
I got the same crash in my app, and i can confirm, this crash is triggered by CFFileDescriptor.
Looking forward to the answer.

wukai:
I fix this crash, but very trick~

Hao Chen:
How did you fix the crash? These crashes are still annoying us.

Chidera Olibie
Hi, could you please file a bug via https://bugs.chromium.org/p/chromium/issues/list with the word "Cronet" in the summary line. Add the stack trace and other relevant information to enable us to debug this properly.

yanshua:
I fix this crash ,but very trick too. Get the dispatch_queue which over released and forced change its ref count to INT_MAX.
No need to worry about memory leak,because it could be destroyed when app exists.

This last comment really resonates with the problems I was seeing. At one point, I was able to make the problem temporarily disappear by removing a call to CFFileDescriptorInvalidate(). But then it would just crash later, like when the teensy_serialmon process would exit. I experimented with adding CFRetain() on various references and a lot of other changes, but nothing ever seemed to make the problem completely disappear.

I'm not planning to investigate the crash further, as this rewrite probably avoids the problem completely, but wanted to at least get what little I know about the CFFileDescriptor crashes into a forum message before I move on to other works and forget all about this, for (hopefully never needed) future CFFileDescriptor reference.
 
Please let me know how it works on your Mac?

I replaced the teensy_serialmon script in /Applications/Teensyduino.app/Contents/Java/hardware/tools/ with the version you posted and so far have seen no crashes! I’ve tried keeping the Serial Monitor open in Teensyduino while building and uploading sketches to the Teensy, running the serialmon script standalone, and using the Deviot serial monitor window in Sublime Text. ~30 uploads so far with no crashes, and I was seeing a crash every 2-3 uploads before.

So it looks very promising! I’ll keep using it in my day-to-day development and let you know if it crashes.
 
Thanks for testing & confirming! :)

Without your report on msg #37, I would not have been able to work around this problem (basically, redesign to avoid all use of CFFileDescriptor). Really do appreciate your effort to make it more reproducible!

I sent a feedback report to Apple, with the steps to reproduce the crash and even a demo video showing the crash happen. The report has been assigned number FB9891995. Hopefully they will eventually fix this in MacOS.

But long before then, I'm going to release Teensyduino 1.57 with this redesigned teensy_serialmon. Will probably start betas in a couple weeks.
 
Yesterday I sent a detailed bug report to Apple, including a video demo showing the steps to reproduce the crash. Got a (likely form letter) reply today that they are investigating.
 
Back
Top