You can indeed use the main USB port as MIDI, but only in USB device mode. Details here:
https://www.pjrc.com/teensy/td_midi.html
If Teensy will talk MIDI with a computer or something else with USB host MIDI, then this works great.
If you need Teensy to work in USB host mode, where you would plug MIDI keyboards or other instruments into Teensy, then you'll need the host port on Teensy 3.6.
If all this host vs device stuff is confusing, the first part of chapter 4 in the USB 2.0 spec explains it pretty well. Here's a direct link to the PDF.
https://www.pjrc.com/teensy/beta/usb20.pdf
This file is huge, but chapter 4 is short and easy to read to get just the essential concepts.