I'm hesitant to say it's the audio library hanging, but on my Teensy 3.0, I have a couple problems with this simple program (below). I've got an Adafruit MAX4466 board wired up to pin 9, running off 3.3v. If I read raw analog values, I see more or less what I'd expect. But with the peak meter in the loop - if the mic ever clips (thump it) then the peak never goes below "10" (0 - 30) after that point. Trying to rule out that its not something actually holding a charge on the pin, I stuck an analog read in there, and it SOMETIMES prints the analogread value but other times prints it and then hangs the processor. It's not trying to print a float - it's a %d and cast to an int. And once in a while it actually lets another value escape to the serial console. If I completely remove the audio library and just print the raw value the pin goes to 1023 on clipping and then immediately falls back to around 510 (the 1/3 scale offset does not occur when reading the pin.)

Running Arduino 1.6.9 and Teensyduino v1.29-beta3

#include <Audio.h>
#include <Wire.h>

// GUItool: begin automatically generated code
AudioInputAnalog         adc1(A9);           //xy=164,95
AudioAnalyzePeak         peak1;          //xy=317,123
AudioConnection          patchCord1(adc1, peak1);
// GUItool: end automatically generated code

void setup() {

// for best effect make your terminal/monitor a minimum of 31 chars wide and as high as you can.

elapsedMillis fps;

void loop() {
  if (fps > 24) {
    if (peak1.available()) {
      fps = 0;
      int monoPeak = peak1.read() * 30.0;
      Serial.printf("%d\r\n", monoPeak);

      if (monoPeak >9) 
        Serial.printf("Raw: %d\r\n", (int)analogRead(A9));