You have a potentiometer going between the voltge rail, which is about 3.3volts, and ground. A portion of that voltage goes into the analog to digital converter. The 3.3 volt rail is not a stable voltage source; there is noise on it and the actual voltage varies a bit depending on the current draw of the teensy and any other chips powered from it. Thus, that shows up as noise on your analog readings.
A voltage reference chip provides a steady, low noise voltage that is not much affected by noise and instability on the power rail. Thus, you get less noise on your readings.
By frequency I mean how fast the changes can occur. Some sensors can produce values thatchange at the rate of kiloherts or even megahertz. Since you mention potentiometers, those can't change values very fast (you can't sweep from one end tothe other more than a few times a second). So you can filter out any frequencies above, say, 100Hz which are due tointerference, noise, etc. That can be a simple capacitor and resistor to make an RC filter. In addition to that analog filtering you can do averaging etc which is a type of digital filtering.
Teensy 3.1 is capable of reading the ADC very fast. Normally it is doing multiple reads and averaging behind the scenes. The limit is not so much how many times it can read one input, but how quickly it can switch to a different input and get a reliable, stable reading from it.