Anyone else seeing MacBook Air hard crash when using a Teensy 4?

[shawn]

This is a report without much technical information. I figured I'd post anyway as a placeholder and as a place for others to note if they've experienced the same thing.

Basically, when I frequently upload code to a Teensy 4, eventually, my MacBook Air grey screens (the system experiences a hard crash). This has happened often enough and just with this device that I thought it was worth a post. I haven't seen crashing at a specific point, just that when I do a bunch of upload/test/monitor cycles, this happens eventually somewhere during one of these cycles. I don't know if it's during upload or during executing/monitoring.

I'd like to emphasize that I'm not saying that the Teensy 4 or its stack is doing anything wrong, just that my Mac is experiencing hard crashes and am wondering if others have seen this too.

My system:
MacBook Air (model 6,2 from mid-2014), 8GB RAM, OSX 10.14.16 with all the latest updates.

Some possible explanations:

• Bad USB ports on my laptop.
• Something about OSX's handling of USB in certain cases is buggy.
• Teensy 4's USB stack does something wrong or out-of-spec, or even some timing is a little off or has a race condition.
• Teensy 4 is using some feature that OSX can't handle well.
• Apple's USB and USB-related drivers do something wrong.
• It's something specific to my machine setup.

I haven't seen crashing like this since Bluetooth was starting to become normalized in OSX, more than 10 years ago. I'm sorry I'm not able to pin down this elusive crash. The only log I have is the window that pops up when the machine restarts that sends the report to Apple.

[blackketter]

I'm not on that specific hardware, but I've been on a MacBook Pro on that version of the OS and have not seen any system crashes.

I'll throw out a few things to try toward narrowing it down:

1. I wonder if you have the same problem connecting the teensy through a USB hub or a different cable.
2. What happens if you just repeatedly unplug and plug the teensy and just have it boot over and over again without uploading?
3. Can you reproduce the problem by repeatedly uploading a simple blink sketch?
4. Does the system reboot spontaneously with a sketch running and doing a simple print to the virtual serial portD

My daughter has a MBA of similar specs, I'll see if she'll let me try to crash it.

[PaulStoffregen]

I've never seen it happen with my Macbook Air, which is early 2015, model "MacBookAir,7,2".

[shawn]

I'm going to continue playing and see if I can't narrow down when this happens. I'm willing to accept this may be specific to my setup. I'm also willing to bet this doesn't happen on newer computers. Heck, maybe I have an intermittent internal short somewhere? Maybe it's a coincidence that I haven't seen it with other USB devices and that some unrelated problem just started? Still unknown, but it sounds like this isn't at all common.

At the very least, if people do see this, and if they experience the same thing, they can come to this thread.

[shawn]

In reply to @blackketter:
I've tried different cables and both with and without a powered USB hub (Amazon Basics 7-port USB 3.0 hub); all configurations so far have seen crashes. I'll be trying the other things you mentioned, too.

[PaulStoffregen]

I'm also willing to bet this doesn't happen on newer computers.

I have an older Macbook Air, 11 inch mid-2012, model "MacBookAir5,1". It has MacOS 10.12.2. The software is Arduino 1.8.9 and Teensyduino 1.47.

I uploaded the LED blink example a couple dozen times using that old laptop. No crashes. I did the first few by pressing the button on Teensy 4.0, and the rest by clicking Upload in Arduino.

[shawn]

I've been poking around the kernel panic logs, and the log says it crashes in a `teensy_serialmon` thread. In fact, every single one of the kernel panics that have happened recently says that it's in a thread from this program. It also says that the last loaded kext is `com.apple.driver.usb.cdc.ncm`. I wonder what USB madness is crashing the kernel? Some bug in Apple's implementation maybe? Do I have bad RAM? I ran the startup diagnostics tool by holding 'D' at startup but it says there's no issues. (Thinking aloud.)

Also in the crash report, below (snipped so it's not too long), it says only `com.apple` kexts are loaded.

Here's the relevant portion of the log, if it's useful to anyone:

Wed Dec  4 19:34:18 2019

*** Panic Report ***
panic(cpu 1 caller 0xffffff8011cdc9bd): Kernel trap at 0xffffff80120fad63, type 14=page fault, registers:
CR0: 0x0000000080010033, CR2: 0x0000000000000018, CR3: 0x000000025a84f113, CR4: 0x00000000001626e0
RAX: 0x0000000000000000, RBX: 0xffffff80430d8ab0, RCX: 0x0000000000000000, RDX: 0x0000000003000000
RSP: 0xffffff9143753de0, RBP: 0xffffff9143753e00, RSI: 0xffffff8037cebe00, RDI: 0x0000000000000000
R8:  0x00000000448b9b80, R9:  0xffffff80124a8ad0, R10: 0xffffff803c4cd648, R11: 0x0000000000000001
R12: 0xffffff804962fcc8, R13: 0xffffff802c7fc9b0, R14: 0xffffff8037cebe00, R15: 0xffffff8037cebee0
RFL: 0x0000000000010207, RIP: 0xffffff80120fad63, CS:  0x0000000000000008, SS:  0x0000000000000010
Fault CR2: 0x0000000000000018, Error code: 0x0000000000000000, Fault CPU: 0x1, PL: 0, VF: 0

Backtrace (CPU 1), Frame : Return Address
0xffffff91437538b0 : 0xffffff8011baf57d 
0xffffff9143753900 : 0xffffff8011ceb065 
0xffffff9143753940 : 0xffffff8011cdc79a 
0xffffff91437539b0 : 0xffffff8011b5c9d0 
0xffffff91437539d0 : 0xffffff8011baef97 
0xffffff9143753af0 : 0xffffff8011baede3 
0xffffff9143753b60 : 0xffffff8011cdc9bd 
0xffffff9143753cd0 : 0xffffff8011b5c9d0 
0xffffff9143753cf0 : 0xffffff80120fad63 
0xffffff9143753e00 : 0xffffff80120abaae 
0xffffff9143753e90 : 0xffffff80120973df 
0xffffff9143753f00 : 0xffffff801209c034 
0xffffff9143753f40 : 0xffffff80121b9bfd 
0xffffff9143753fa0 : 0xffffff8011b5d196 

BSD process name corresponding to current thread: teensy_serialmon

Mac OS version:
18G1012

Kernel version:
Darwin Kernel Version 18.7.0: Sat Oct 12 00:02:19 PDT 2019; root:xnu-4903.278.12~1/RELEASE_X86_64
Kernel UUID: DFB5D0E2-3B41-3647-A48B-D704AFCC06B4
Kernel slide:     0x0000000011800000
Kernel text base: 0xffffff8011a00000
__HIB  text base: 0xffffff8011900000
System model name: MacBookAir6,2 (Mac-7DF21CB3ED6977E5)

System uptime in nanoseconds: 18523322240797
last loaded kext at 12687175825037: com.apple.driver.usb.cdc.ncm	5.0.0 (addr 0xffffff7f9330b000, size 69632)
loaded kexts:
[...snipped...]

[defragster]

What sketch is running - does it do this with Simple Blink - or is this a full speed write to Serial sketch for SerMon message viewing?

What version of the IDE and TeensyDuino are in use?

[PaulStoffregen]

You might try running teensy_serialmon in Terminal, without Arduino. To do this, first make a note of the USB number shown in the lower right corner of the Arduino window.

Then open Terminal and navigate to the folder within Arduino that has teensy_serialmon. For example, if Arduino is in your Downloads folder, you'd use:

cd Downloads/Arduino.app/Contents/Java/hardware/tools

Then just run it with the USB number. For example:

./teensy_serialmon usb:14200000

This will cause it to print to the Terminal window. You can completely quit Arduino and all other programs.

You could also try using "cat" with the serial port device name (which you can see in the Tools > Ports menu). For example:

cat /dev/cu.usbmodem65570901

Both of these should print everything Teensy is sending to Terminal, without any use of the Arduino software. With "cat" nothing from PJRC runs on your Mac at all.

Maybe try a fresh reboot and run the "cat" approach. Does that crash your Mac?

[shawn]

Thank you both for attempting to help me.

@defragster - I'm running tests using a development version of my TeensyDMX program, but I have seen some panics a couple of times when testing something simple (10's of lines) using the Arduino IDE (as opposed to PlatformIO). I see the crash a few times a day during upload-monitor cycles. I haven't pinpointed the moment yet, sadly. I believe it's just after an upload is finished and serial monitoring starts. I'm using the latest VSCode (1.40.2), the latest PlatformIO (Home 3.0.1 and Core 4.1.0), and the latest Teensy FW download that goes with it (4.6.0 with the 1.148.0 attached Arduino framework and toolchain; I'm assuming this aligns with the 1.48 Teensyduino). I'm going to guesstimate and say that the crash occurs about once every 2-3 hours, and every about 20-30 test cycles.

@PaulStoffregen - I will play with these suggestions and see if there's more crashing. Thank you for these tips.

[defragster]

That is more than a simple blink ... so added details might be helpful. Also interesting if the same process using the standard IDE install with known TeensyDuino can reproduce - it may be T4 or PlatformIO interaction with the system?

Once every couple hours and couple of dozen upload cycles is not easy to reproduce. I'm on Windows and spent some dozen hours with many dozen uploads to a T4 in the last 11 months and not ever seen this. Time is split between Typical IDE interface - and a Windows batch file scheme that lets my editor of choice run a Commnd Line to trigger the IDE build without actually using the OPEN IDE interface. This CmdLine process uses TyCommander for the uploading.

To extend on what @Paul wrote - perhaps save a copy of the .HEX file as built [ PlatformIO or idealy Arduino IDE ], and close PlatformIO/IDE. Then using TeensyLoader and the CmdLine run of teensy_seralmon to a load Hex file and upload and reset manually and see if that can inspire the system fault. Alternatively install TyCommander that allows Upload and provides GUI SerMon to use with that HEX file.

Might help to see setup() code - if there is a wait like : while ( !Serial && millis()<2000 ); To make sure the computer is online before sending Serial.print(?) data that may or may not make it as expected.

[shawn]

Yep, this is a hard one because it's very difficult to reproduce. I'll take your suggestions and do the best I can to figure out what the HEX file was, which step in the procedure was happening, etc., and then re-try with that HEX using the command-line `teensy_serialmon`.

Here's my `setup()`. Not asking you to debug it, it's just informational. Note that all the `NullSketch` overrides from `Sketch` are no-ops (except `setup()`)

// The current sketch
std::unique_ptr<Sketch> currSketch{std::make_unique<NullSketch>()};
int currSketchType = -1;

void setup() {
  // Initialize the serial port
  Serial.begin(115200);
  while (!Serial && millis() < 4000) {
    // Wait for initialization to complete or a time limit
  }
  Serial.println("Starting main program.");

  // Set up any pins
  pinMode(kLEDPin, OUTPUT);
  digitalWriteFast(kLEDPin, HIGH);  // Start with the LED on
  pinMode(kTXPin, OUTPUT);
  digitalWriteFast(kTXPin, LOW);

  // Initialize with the Null sketch
  changeSketch('n');

  Serial.println("Hello, DMX World!");
}

void changeSketch(int sketchType) {
  if (sketchType == currSketchType) {
    return;
  }

  // Check for a valid sketch type
  switch (sketchType) {
    case 'c':
    case 'f':
    case 'n':
      break;
    default:
      return;
  }

  // Destroy any current sketch
  currSketch->tearDown();

  // Create a new sketch
  switch (sketchType) {
    case 'c':
      currSketch.reset(new Chaser{});
      break;
    case 'f':
      currSketch.reset(new Flasher{});
      break;
    case 'n':
      currSketch.reset(new NullSketch{});
      break;
  }
  currSketchType = sketchType;

  Serial.printf("Changing sketch to: %s\n", currSketch->name().c_str());
  currSketch->setup();
  Serial.printf("Free RAM: %d\n", freeRAM());
}

void NullSketch::setup() {
  digitalWriteFast(kLEDPin, HIGH);
}

int freeRAM() {
#if defined(__IMXRT1052__) || defined(__IMXRT1062__)
  extern unsigned long _heap_end;
  intptr_t end = reinterpret_cast<intptr_t>(&_heap_end);
#else
  int v;
  intptr_t end = reinterpret_cast<intptr_t>(&v);
#endif
  return end - reinterpret_cast<intptr_t>(_sbrk(0));
}

To me, `setup()` is pretty innocuous, except for maybe the `_sbrk(0)` call, but looking at the implementation doesn't raise any flags for me when `0` is passed as the argument.

I'm aware that this is a difficult one, and that there isn't much anyone can do here to solve the issue, short of giving me debugging-approach suggestions, and I'm ok with that. I'm sure the root cause will eventually rear its head.

I'd like to summarize some of the suggestions given so far in this thread:

1. See which HEX file was present at the time of the crash. Or any of them, for that matter, and use the file with command line tools.
2. Command line `teensy_serialmon`.
3. Using `cat` direct from the USB device file.
4. Command line tools instead of IDE-based tools.

[blackketter]

Another data point: I use platformio on macOS with T4 and have never had a system crash.

Also, can you reproduce the problem _without_ reprogramming the teensy?

[shawn]

@blackketter this is a good question. You mean like cycling the power to the board or something and then loading up a serial monitor of some sort?

[PaulStoffregen]

Have you tried with 1.49-beta1? Either the normal way or this build for Catalina? (which should work on Mojave)

The USB device code has some very substantial improvements in 1.49-beta1. The raw USB speed is approximately double compared to 1.48.

The new signatures and hardened runtime (which is used if you have Mojave 10.14.5 or 10.14.6) probably won't make any difference. But it is built against a new Apple SDK, so maybe worth a try comparing 1.49-beta1 all-in-one download to the 1.49-beta1 installer-based Teensyduino.

[shawn]

I should add: I'm seeing this with Teensy 3's too, as of late. 3.2 and 3.5, so that probably narrows it down to my machine and nothing anything the Teensy is doing.

[shawn]

Have you tried with 1.49-beta1? Either the normal way or this build for Catalina? (which should work on Mojave)

The USB device code has some very substantial improvements in 1.49-beta1. The raw USB speed is approximately double compared to 1.48.

The new signatures and hardened runtime (which is used if you have Mojave 10.14.5 or 10.14.6) probably won't make any difference. But it is built against a new Apple SDK, so maybe worth a try comparing 1.49-beta1 all-in-one download to the 1.49-beta1 installer-based Teensyduino.

I will try this too. I'm likely to copy files over my PlatformIO install too. Let's see if this approach with PlatformIO is viable...

I haven't switched to Catalina yet because of the 64-bit-only support. I still use LogicWorks, which unfortunately doesn't have a 64-bit version.

[blackketter]

@blackketter this is a good question. You mean like cycling the power to the board or something and then loading up a serial monitor of some sort?

Yes, by cycling the power or having the teensy reboot itself.

[PaulStoffregen]

I haven't switched to Catalina yet because of the 64-bit-only support.

Don't update to Catalina. But DO run the 1.49-beta1 for Catalina on your Mac running Mojave. Also run the normal 1.49-beta1, where you install it into a copy of Arduino 1.8.10.

Those 2 are pretty much identical, except the beta for Catalina is built using a newer Apple SDK and uses the hardened runtime. My guess is they will both crash the same way. But if they are different, that could be a clue....

[shawn]

Thank you all for the suggestions. I will report back if I know more. Hopefully this crashes again in a timely fashion. My plan is to try one thing at a time until crash or "timeout" of a few days. This could be a while...

[PaulStoffregen]

One other variable to consider is your Mac's power source. Apple's CDC driver uses a *lot* of CPU time (for reasons unknown, Windows & Linux drivers don't do this) when receiving data at high speed. Sustained high speed receiving may be causing your Mac to draw quite a bit more power than normal.

If the laptop's battery is low (or old & not holding much charge anymore) and your power adaptor is old or going bad or the AC line voltage is low, your Mac might simply be crashing due to lack of adequate power.