What does "at other times the download halts and reports errors that aren't there" really mean? Is there an error message? Remember, we can't see your screen. Nobody can know what you're seeing if you do not give a screenshot.
The only description is " I notice when this happens that when I go into Tools that the Port appears grayed out". I'm trying to understand if the error you see is the port menu, or if you're observing a correlation between this error and something happening in that menu? If it is merely a correlation, it's not clear if you observe the port menu this way before or after the error. I also can't tell what settings you're using. So many unknowns....
Screenshots of the tools menu normally and in this condition would help quite a lot. So would a screenshot or video clip of the whatever error happens. If the error only appears momentarily, perhaps use your phone or a camcorder to record it? Often the easiest way to share a video clip is uploading to an unlisted video on youtube.
To really get to the bottom of any problem, at the very least you'll need to collect logs of what really happen. There are two of these.
1: Teensy Loader has a Verbose Info window. It's hidden in the Help menu. When you open the Verbose Info, it has a "Log" menu which lets you save all the info to a file.
2: The Arduino-side utilities log their activity to a file named "teensy_reboot_log.txt". On Windows and Mac it's likely to be in temporary folder which the operating system hides. You'll probably need to use search to find it. Knowing the exact name of the file "teensy_reboot_log.txt" hopefully can help.
Please try rebooting your computer, so you're starting fresh. Then recreate the error. Take screenshots or record a video, so you can show us exactly what it really happening. Collect those 2 log files.
We can talk at length about how the USB stuff works or is supposed to work, but it's all just blind guessing if we can't see what really happened on your screen, and without those 2 log files, there won't be enough info. I know it requires more work, but I hope you can post a reply with all this info. I really would like to understand what's going wrong and find a way to solve it. If there's something Teensy isn't doing correctly, I definitely want to investigate and fix that for everyone.