A slightly different ground is the USB power ground -- this also has varying currents flowing in it and will have deviations from 0 V.
For instance, you might expect that an I/O pin driven to logic '0' will generate precisely 0 V -- it won't -- the internal driver connects this to VSS, and if with a good enough 'scope you could see the noise on this w.r.t AGND.
The Teensy chip will work well when these grounds are not equal; as long as the difference doesn't get more than a few 100 mV.
On the Teensy schematic, you will see that VREF is decoupled directly to VSSA (== AGND) with 0.1 uF. AGND and VSS are connected with a ferrite bead (to filter VSS noise from AGND).
So--in general--analog signals and things to do with ADC inputs should always be referenced to AGND. If the current they consume is small and doesn't change quickly (i.e. doesn't have a sharp clock), it might be OK to connect the power supply ground of these blocks to AGND also; if not, they should be powered from VSS/GND.