First to directly answer your "bare metal" question, several people have gone down this path and shared their work. Here's one of the best. Maybe others will recall links to more of them, or perhaps you'll find them with a little searching...
http://kevincuzner.com/2014/12/12/teensy-3-1-bare-metal-writing-a-usb-driver/
If you don't already have it, you really must get Joseph's Yiu's book.
https://www.amazon.com/dp/0124080820
Freescale's reference manual documents all the peripherals, but it has only a few pages that summarize how they configured the ARM core. There are tons of critically important features about the ARM code which you really must understand if you're going to achieve best performance. This book is really the only approachable source for that info. ARM's reference manuals document but don't explain how things are meant to work. Joseph Yiu does. No matter what approach you take, the info in this book is essential for getting the most out of the ARM processor.
Now I can tell from your question you're determined to achieve best possible performance. That's good. You also seem convinced discarding all the Teensy core library work would be a best way to accomplish your goals. That's not so good. With a lot of work, you'll manage to trim some flash memory and RAM usage (which amounts to just a few percent of the total memory), but you'll gain nothing performance-wise. If you decide to support USB or serial communication, you'll be hard pressed to achieve the levels of performance already present in Teensyduino's core library, which makes effective use of advanced features like USB DMA and serial FIFOs.
Of course, if you build this as an Arduino library & sketch, as GRBL does, you get full access to the hardware registers. Teensyduino gives you a good base to build upon, and it'll be much easier to share your work with others. I highly recommend using it to avoid all the mundane stuff, so you can focus on making your code run fast. It really won't be holding you back. If you're used to 8 bit AVR or PIC, probably the best thing you can do is read Joseph's Yiu's book and spend some time experimenting with the powerful hardware features.
For example, you're probably conditioned to avoid floating point math, if you're used to lesser processors. On Teensy 3.5, the FPU is basically the same speed as 32 bit integer math, as long as you're careful to avoid avoid 64 bit double (the FPU only does 32 bit floats). As soon as you go to special effort like extra if-else code to use "fast" integers, you're usually going to end up slower than if you'd just done it the easy way with 32 bit float. It really is that fast.
Interrupts are another feature that sometimes has a poor performance reputation from the AVR world. Especially using more than 1 interrupt on AVR is problematic, where serial communication interrupts mess up low-latency interrupt response. On ARM you get prioritized nesting with hardware optimizations for context save/restore. You'd be doing yourself a tremendous disservice to apply that "only 1 interrupt" strategy from the 8 bit world, when the ARM NVIC can allow your high priority interrupt to always get low latency service, even with lower priority communication interrupts are running. You'll discover Teensyduino already has sensible default interrupt priorities (unlike Arduino Due where they never supported this feature), and the highest 2 priority levels are unused by any existing libraries so you can have them fully available.
The DMA channels are also an incredibly powerful feature you can leverage in some cases, especially moving data on/off chip a fast sustained speeds, using basically zero CPU overhead other than setting up the DMA transfers.
Of course, if you want to start with a blank slate and do everything yourself from scratch, you certainly can. That first link will show you the way. But odds are you'll spend a year or more getting up to the already very good performance offered by Teensy's core library. It's your choice, but I hope you'll leverage what already exists and focus your efforts where they will matter most. Either way, get Joseph's Yiu's book!
)
