I had something similar happen to me. My board designed for Teensy 3.5 worked fine. Then switched to Teensy 4.1 and surprisingly the linear regulator got quite hot without a heatsink. DC-DC converter fixed that.
Seems way easier to just cut open a USB cable and add a through-hole diode to the red wire with the line end pointing towards the Teensy. This cable could be used for multiple Teensies. And as mentioned previously, a diode for Vin would protect...