When are you seeing your problem? When the sketch is first downloaded and run? Or when the T4 has been powered off and then powered back on again?
If the former, that's how it's supposed to work; the Teensy Loader...
I pretty much duplicated your setup; Raspbian instead of Ubuntu, but otherwise identical, down to the version of Teensyduino and the library used. It's been running now for over 10 minutes with nary a hiccup or...
It sounds like that person may be using an older version of the Teensy loader that didn't know about the 4.1. If Help -> About doesn't say "Teensy Loader 1.53" (or newer), it's time to upgrade.