I've just been doing a project in arduino IDE where I was listening to a serial communication between 2 devices and dumping it to SD card with a Teensy 3.5, and have been tearing my hair for a while as to why the communication would be disturbed any time I had the Teensy connected AND the serial port initialized. Spent some time looking into the code of the teensy3 core lib, and found that the pullup was enabled for the RX line. Changing that solved my problem.

Not sure why it's enabled, but if there's a good reason I'd suggest adding a parameter to the HardwareSerial constructor to choose whether that pullup should be enabled or not.