I believe I also got this by email. Yes, please sent it back and I'll replace it. I'll followup to your email with details.
Certainly Teensy is now supposed to work as you described. When you plug it in (but without pressing the button or running any software), the LED should continue blinking. The chip should not get hot.
I can tell you we do fully test every board. A bed-of-nails test checks for connectivity to each pin. As a final test, the LED blink program is loaded onto the board by USB, which is how we verify the USB port works.
It's entirely possible the board could have been damaged in shipping or as we packaged it. It's also possible something went wrong on your end. Usually I ask more questions, as a minimum precaution so there is something wrong with your setup, the replacement won't suffer the same fate. It sounds like you simply plugged the USB cable in to the board and your computer, right?
Whatever you do, when the replacement arrives, check it carefully by plugging in to your computer using a good cable. The LED should blink and continue blinking until you push the button or run the Arduino software and attempt to upload new code. The chip should not get hot.