Agreed. The internal sensor is usually not very accurate when it comes to measuring ambient conditions. But it can still be very useful for certain tasks, such as calibrating the ADC.
I did an experiment with a MCP3241 16-bit ADC to see how temperature affected it and the MUX that was supplying it. That in turn convinced me of the need to run calibration runs for the ADC and to use the output to smooth out / eliminate these offset and gain errors. This is what the analog read does inside the Teensy, IIRC, as part of every setup.
I now have a DHT22 also designed into the Teensy shield I am developing. I am then planning something altogether more ambitious, i.e. measure drift as a function of temperature and compensate for that. A 3.3V-based GPS sensor provides the signal to compare to... the result could be a super-accurate thermally-compensated clock like the DS3231 at a bargain price. But even 5PPM is a pretty amazing spec - if I understand it correctly, per year you'd be losing / gaining 157 seconds, worst case. That's 13 seconds per month... good enough for most work, even if all you do is compensate for drift at the beginning.