Arduino boards are generally larger and it seems would generally draw less power - and not have to step the 5V down to 3.3V for use creating another heater on the 'Teensy' PCB.
So somewhat warmer is to be expected with more energy dissipated on a smaller board, where unless attached to a larger breakout board to spread the heat - it remains centralized.
Last check a Teensy 4.0 running normally takes about 100 mA at 5V - about one third of the 5V goes to heat from the regulator - the rest heats the processor and parts taking power.
At 600 MHz - versus 16 MHz - the resulting internal processor temp is likely 40 to 50 degrees C and about 100 degrees F is typical processor temp +/- a few depending on ambient air temp with a bare PCB with nothing else using power.
The T_4.1 has a bit more surface area from a larger PCB - but is similar.
Without quantitative numbers or knowing what if anything is connected ( even soldered pins help cooling ) it is hard to understand 'quite hot', the notes and numbers above should help for general reference and expectations.