Also check which version of Teensyduino you have. Click Help > About in Arduino.
A problem with startup on 3.6 with slow rising power supplies was recently fixed. If you have an older version of the software and one of those slow-starting power supplies, you might be experiencing that problem.