For just playing sounds for a gun, you might want to look at the ISD1820 chips. These provide 8-20 seconds of a single sound. and are often used in the musical greeting cards. You can either get the raw chip, which you would wire into a proto board or pcb, or you can get one with pins, etc. that you could hook to your teensy. I bought the following, but I haven't done more than test it playing a simple sound using a simple unpowered speaker I got at Radio Shack:
http://www.ebay.com/itm/ISD1820-Voice-Board-Voice-Module-Sound-Recording-Module-/221180636859?pt=LH_DefaultDomain_0&hash=item337f64aebb
If you want to play multiple sounds, you can get WTV020-SD-16P chips that have a micro SD card to hold the sounds. This chip is the basis of many of the Arduino sound shields. A common complaint is these units are real picky about the size and quality of the micro SD cards.
http://www.ebay.com/itm/WTV020-SD-16P-U-Disk-Audio-Player-SD-Card-Voice-Module-MP3-Voice-Module-/221152563949?pt=LH_DefaultDomain_0&hash=item337db852ed
Over on the Arduino forum, I first found about the WTV020-SD-16P from this post which includes a basic library. The original author used it to create a light saber with sounds. I haven't looked into details yet, but I suspect the programming is either i2c or spi. I don't think it used asm, so you might be able to use it on the teensy 3.0 unchanged.
http://arduino.cc/forum/index.php/topic,117009.0.html
For basic sounds without speaker, you could use a passive beeper or buzzer, assuming Paul has ported the Tone library to the Teensy (beeper makes a single click and turning it on/off rapidly can produce a tone, while buzzer emits a continuous sound once triggered until it is canceled). The ones I bought are 5v that I used with the Arduino, and I don't think I've tried it on the Teeny yet, but you should be able to find 3.3v versions.
http://yourduino.com/sunshop2/index.php?l=product_detail&p=211
On my steampunk camera setup (see my icon picture), I have a telegraph key to fire that camera, and I added telegraph sounds using the beeper on the Arduino. Like you, I like the form factor of the Teensy, and I have been planning to move it to the Teensy. I want to upgrade to using recordings of more complex sounds, but so far, I haven't been focused enough to get to the sound portion yet.
I have found 2.5mm and 3.5mm stereo connectors for mounting on panels at my local electronics store (even Radio Shack has them if you search through the cabinets at the back), but these aren't as convenient to mount on proto boards. I made a little wood frame to hold it, but after a search I found a Shogyo component that I just ordered that provides a 2.5mm connector for PCB mount (SJ-0252-3RT)t:
http://www.shogyo.com/pdf/sj-0252-3rt.pdf
If you wanted the more standard 3.5mm connector, you can get breakouts like:
http://www.ebay.com/itm/300655692210?ssPageName=STRK:MESINDXX:IT&_trksid=p3984.m1436.l2649
If you want the ability to remove/connect devices, besides the 2.5mm/3.5mm phono connector, you can get RJ-11 (US phone cable) for 6 wires or RJ-45 (ethernet) for 8 wires connectors. In addition, I've gotten a bunch of wafer headers from dipmicro, that are a little tedious to make, but it allows me to make a wire connection of a particular size:
http://www.dipmicro.com/store/SET-HDR