You need either the "Windows Installer, for Windows XP and up" or "Windows ZIP file for non admin install". Both of those work well.
https://www.arduino.cc/en/Main/Software
The "Windows app Requires Win 8.1 or 10" version does not work at all for Teensy.
Before you do anything else, make sure you've backed up all important files. That's just ordinary generic computing advice. If you don't have a system of regular backups, do yourself a huge favor and make sure you copy everything important onto a flash memory stick or portable hard drive.
Now, about Arduino, having only 1 copy on your PC is simplest. If you want simple, go with the installer.
Arduino is designed to allow more than 1 copy. That's what the non-admin zip file is all about. Beyond that, Arduino has a special "portable" mode where a copy of Arduino becomes completely self contained, not using any other files or config on your computer.
https://www.arduino.cc/en/Guide/PortableIDE
Teensyduino does support the portable mode. If you want to have a copy that you use for Teensy and not have it affect the other stuff on your PC in any way, extract the ZIP file and set it up for portable mode *before* you run the Teensyduino installer to add Teensy's stuff into that copy. Then it will all live completely isolated in that folder, not connected to any of the other Arduino stuff on your PC.
The downside to multiple copies is complexity. Many people who do this end up later getting confused. Usually the problems come in when they click on a file which causes Windows to open Arduino automatically. But which copy gets opened? The look alike. You can click Help > About to get the version, and if Teensyduino is present you'll see its info. But even that takes awareness and some diligence to remember you have multiple copies that look alike, but support different boards.
