I'm going to guess you're using Windows or Linux, since you said "PC". The following works on both, but not on Macintosh.
First, you can just put more than 1 copy on your PC. In using Windows, download the ZIP version. The important (and difficult part for some people) is remembering where you put each copy. If you run Arduino by clicking on files, your system will pick 1 copy which may not be the one you want. Best to explicitly run the copy you want, then use File > Sketchbook or File > Open to open your files.
Normally all copies of Arduino share settings. If you set Tools > Board in 1 copy, then quit it and run another, it will pick up the change you made in that other copy.
Arduino supports a "portable" mode where all settings are self-contained within that 1 copy. Teensyduino supports this mode. To use it, first follow these instructions (create the "portable" folder) to set a copy of Arduino into portable mode.
https://www.arduino.cc/en/Guide/PortableIDE
Then run the Teensyduino installer. Don't just blindly click "Next". When you get to the step to select the location, be sure to select the correct copy of Arduino. If that copy has the "portable" folder, Teensyduino's installer will detect it and install appropriately for portable mode. Then you can use that copy without affecting any of the others.