There are 2 ways of viewing this question.
#1 - Many people do use something other than the Arduino IDE. The most common are Visual Micro, Eclipse with Jantje's plugin, and running a makefile from the command line. For this last option, the Teensyduino installer puts a sample makefile in hardware/teensy/avr/cores/teensy3. Usually the same core library and libraries you'd use with Arduino get used with these environments.
I personally use Arduino's "use external editor" feature, from File > Preferences, and they I edit code with my favorite text editor.
#2 - Often we get people who believe they'll get higher performance if they write everything from scratch. This question comes up over and over on this forum. Usually it's a pretty misguided idea, partly because it already is a bare metal system, partly because people familiar with very old architectures like PIC16 vastly underestimate the complexity & learning curve of these modern parts and their peripherals, and partly because so much of the existing code has a tremendous amount of optimization work that very few people could manage to even match with less than a man-year of intense work.
However, not everything is fast. For example, digitalWrite versus digitalWriteFast, or analogRead versus the ADC library, or SD versus SdFat. Often we get people asking these sorts of questions, and after much discussion it's finally revealed they're doing something like bitbashing waveforms which could have been accomplished using timers (maybe with DMA). Taking the best approach and fully leveraging powerful peripherals usually results in far better overall performance than hand optimizing code (much of which is already pretty well optimized in the libraries).
But if you really want to go full from-scratch bare metal, the makefile and linker script are there for your hacking pleasure. Maybe at least start with the exist code and replace it piecewise. Some parts like the USB have a particularly steep learning curve!
If you think long term though, one pretty compelling reason to use the existing code through its published API is easy compatibility with future chips. In the not too distant future 400 to 600 MHz microcontrollers will become the norm. If you want to use those, you'll have a lot of work to redo if you've tied all your code very tightly to today's 100 to 180 MHz chips.