Teensy 4.0 entering HalfKay autonomously during normal operation — no USB connected

[jenlewin]

We are running Teensy 4.0 boards in a large deployed sculpture installation (100+ boards). Periodically, individual boards enter what appears to be the HalfKay bootloader on their own during normal running operation. When this happens the board stops running its firmware, the onboard LED goes dark, and the small red bootloader LED comes on. The board sits in this state indefinitely until physically reflashed via USB by manually pressing the program button (only).
In a sculpture installation with 100+ inaccessible nodes, this is a serious problem.

Hardware context:

• Custom PCB with Teensy 4.0 socketed via side headers - this PCB ran 500+ Teensy 3.2 fixtures over the last 6 years and is very well tested for the 3.2 - (but this is the first use with the 4.0)
• XBee 802.15.4 radio on the same board
• PROG_N on the Teensy is connected via an N-FET (NX7002AKVL) to XBee DIO0 for remote programming capability - when connected over Xbee, I an have a 'master' that can remotely pull the Prog_n high or low. This works great for the 3.2 and is battle tested.
• PROG_N has no external pull-up — relying on companion chip internal pull-up only
• Boards are powered from 5-24V DC supplies, no USB host connected during normal operation

What we have ruled out through testing:

• Firmware crashes / HardFault — confirmed via test sketch that the fault handler does NOT enter HalfKay, board restarts normally
• CPU_RESTART() — confirmed does not enter HalfKay
• CPU_REBOOT() / _reboot_Teensyduino_() — confirmed enters HalfKay, but verified this is never called during normal operation
• XBee DIO0 — the condition occurs when no XBee communication is active and no DIO lines are not being driven. The XBee is completely idle when the condition occurs.

Environment:

• Boards are susceptible during AC power fluctuations and high static environments
• Only individual boards are affected at a time, not all simultaneously (we are losing 3-5% ever 48 hours on average. By losing, they enter this mode, and again, can only be brought back by physically pushing the program button.)
• The condition always occurs during normal running operation, not on power-up, and not correlated with any XBee activity
• We have run projects of similar scale successfully with Teensy 3.2 — this appears specific to the T4.0

Our current suspicion:PROG_N has no external pull-up resistor and may be floating or sensitive to electrical noise — power fluctuations, EMI from switching LED strips, or static discharge. We are wondering if the T4.0 companion chip's PROG_N input is more sensitive than the T3.2, and whether the lack of an external pull-up is leaving us vulnerable.

Questions for the community:

1. Is PROG_N on the T4.0 particularly sensitive or prone to noise triggering? What is the recommended external pull-up value? Has anyone seen this before?
2. Can electrical noise, power fluctuations, or static on PROG_N cause spurious HalfKay entry on T4.0?
3. Is there any sensitivity to +3V3 rail noise or drooping that could cause a false HalfKay trigger?
4. Is there any software mechanism to detect or prevent unintended HalfKay entry, or to recover from it without physical USB access?

Any insight from the community or PJRC would be greatly appreciated.