I'm having the same 'problem' with my Teensy 4.0, though I don't think it's a true problem, just a feature. The note on the back of the Teensy 4.0 'Welcome Card' regarding the jumper trace between the VUSB and the VIN pin says, "Cut to separate VIN from VUSB, if using battery or external power." I think this would be more properly worded, "...battery or external power for VIN."
I too am powering my board with 3.3V directly (using both 3.3V pins actually). When I do that alone I cannot program the device. And even after the device is programmed, if I want to use the Serial(0) port for diagnostic data, it is unreliable - working intermittently.
To program the device or use the Serial(0) port I have to either 1) leave the jumper trace intact, or 2) connect VIN to 5V. For my application, since I won't always be using Serial(0) for diagnostics, and I won't be programming that often, I'm considering reconnecting the VIN/VUSB jumper with a blob of solder. This works, but I'm not sure if there are considerations for the long term.
The long-term question for me is about the TLV75733P regulator. What happens if the 3.3V I provide isn't exactly the 3.3V the regulator outputs when I have the USB cable connected? For those times will the regulator be 'unhappy'? TI has some seemingly applicable App Notes about reverse current and the like, but they're over my head.
Is there any reason it would be better to re-cut the jumper trace and use the VIN pin instead? Or, what about removing the 3.3V regulator completely?
Thanks!