This is a pretty big question.
You can also ask this about how Arduino DUE and Arduino M0... Work as well as they are not Atmega8 processors as well.
Several parts to this:
a) First off, with the Arduino if you build for one of the Atmega boards, it only will work on one of the Atmega boards. i.e. the generated hex file will only run on the board that it was built for.
b) When you install different architectures, they download a complete build platform for type of chip, including compiler, include files... There are files included as part of this download (boards.txt, platform.txt...) which tells the Arduino ide which commands to use to do things like build, or upload...
c) A lot of source code does not depend on the underlying hardware specifics and can be simply be recompiled for the different processors. Are there as some gotchas to watch out for. Things like if your program uses data types like: int, or float or... The size of these variables may be different on different boards. Example int on Atmega8 boards are 16 bit, whereas on the ARM boards they are 32 bits... So if you need/want specific size you should use data types like int16_t or uint32_t or ... which are the same size on the different boards.
d) In lots of libraries and programs, if you need/want to do hardware specific code, you can put them under #if compiling so you can do specific things for specific boards. You will see some of that in many different libraries.
e) Paul (and others here ) have worked hard to make AVR specific code work on the Teensy boards. This includes things like emulating several of the AVR specific registers. For example there is code that if you had SPI code that was developed to use the AVR specific registers, the emulation code will translate those register accesses to use the standard SPI code...
f) Likewise: a lot of work has gone into trying to get all of the major (and lot of minor) libraries to work on Teensy. That is why for example if you download Teensyduino and installed it, you will see a lot of optional libraries to be installed.
Hope that helps